Francesco Gérard

Paweł Jean-Philippe Michael

Yu-Han Raphaël

Travailler

Francis

Gérard Jean-Philippe Gauthier

Jules Linus

Claire Arnaud

Stéphane

Merci Adeline

Alexandra

Alice Alexis

Ariane Antonio

Cyprien, Eddy, Francesco, Gloria, Iqraa, Lucas, Ludovic Miguel Camila

Nicklas Nathan

Thibault Ugo Paul

Yazid Ma Thèse N'

Clément, Corentin, Eloïse, Guillaume, Linus Margaux Bénédicte

Keywords: theory of deep learning, neural networks, statistical learning, non-convex optimization

Deep learning has emerged as a transformative paradigm in the past decade, with major impact in various fields of artificial intelligence. However, the properties of this family of machine learning methods are not yet fully understood. In this PhD thesis, we present contributions, mostly theoretical in nature, to the field of deep learning. We study various families of neural networks (shallow neural networks, residual networks, recurrent networks, Transformer) and various types of mathematical problems, most notably in the fields of statistics (generalization bounds) and optimization (convergence of the gradient flow). A first setting that is of particular interest for us is the large-depth limit of residual networks. It has been remarked that this large-depth limit may correspond to a neural ordinary differential equations. Under appropriate conditions, we show that it is indeed the case, although other limits such as stochastic differential equations can also hold. We investigate optimization and statistical properties of neural networks in this setting. In the second part of the thesis, we move on to prove results on finite-depth neural networks. We prove convergence of gradient flow for shallow neural networks with a moderate number of neurons in a simple setting. Finally, we investigate properties of the more recent Transformer architecture from a more practical point of view.

Résumé

L'apprentissage profond a largement transformé le paysage de l'apprentissage automatique au cours de la dernière décennie, avec un impact majeur dans divers domaines de l'intelligence artificielle. Cependant, les propriétés des méthodes d'apprentissage profond ne sont pas encore entièrement comprises. Dans cette thèse de doctorat, nous présentons des contributions, principalement d'ordre théorique, dans ce domaine. Nous étudions différentes familles de réseaux neuronaux (réseaux neuronaux à une couche cachée, réseaux résiduels, réseaux récurrents, Transformer) et différents types de problèmes mathématiques, notamment en statistique (bornes de généralisation) et en optimisation (convergence du flot de gradient). Dans un premier temps, nous nous intéressons à la limite en grande profondeur des réseaux résiduels. Il a été remarqué dans la littérature que cette limite en grande profondeur pourrait correspondre à une équation différentielle ordinaire neuronale. Sous des conditions appropriées, nous montrons que c'est effectivement le cas, bien que d'autres objets limites peuvent aussi apparaître, en particulier une équation différentielle stochastique. Nous étudions les propriétés d'optimisation et statistiques des réseaux neuronaux dans ce cadre. Dans la deuxième partie de la thèse, nous nous intéressons à des réseaux neuronaux de profondeur finie. Nous prouvons la convergence du flot de gradient pour des réseaux à une couche cachée avec un nombre modéré de neurones dans un cadre simple. Enfin, nous étudions les propriétés de l'architecture plus récente du Transformer avec une approche plus pratique.

Mots-clefs : théorie de l'apprentissage profond, réseaux de neurones, apprentissage statistique, optimisation non-convexe

Contributions and thesis outline

The thesis is organized in two parts, preceded by an introduction and followed by a conclusion. Each part is separated in several chapters, which each correspond to a standalone contribution. As a consequence, the chapters are independent and self-contained. The notation may vary from chapter to chapter, although we try to keep some identical conventions throughout the thesis. Each chapter has led to or should lead to a publication, as detailed below.

Part I: From discrete to continuous architectures, neural networks in the large-depth regime This part is dedicated to the mathematical analysis of the large-depth limit of residual networks, both from a statistical and optimization point of view. [START_REF] Marion | Scaling ResNets in the large-depth regime[END_REF] Scaling residual networks in the large-depth regime, P.M., Adeline Fermanian (Sorbonne Université at the time, now Califrais), Gérard Biau (Sorbonne Université), and Jean-Philippe Vert (Google Research at the time, now Owkin). Submitted. [START_REF] Marion | Implicit regularization of deep residual networks towards neural odes[END_REF] Implicit regularization of deep residual networks towards neural ODEs, P.M., Yu-Han Wu (Sorbonne Université), Michael E. Sander (ENS Paris), and Gérard Biau. Submitted. [START_REF] Marion | Implicit regularization of deep residual networks towards neural odes[END_REF] Generalization bounds for neural ordinary differential equations and deep residual networks. Published at NeurIPS 2023. [START_REF] Fermanian | Embedding and learning with signatures[END_REF] Framing RNN as a kernel method: A neural ODE approach, Adeline Fermanian, P.M., Jean-Philippe Vert, and Gérard Biau. Published at NeurIPS 2021 (oral presentation).

Part II: Contributions to finite-depth neural networks

This part gathers two contributions related to modern topics in deep learning, this time for finite-depth neural networks. The presented contributions are the following: [Marion and Berthier, 2023] Leveraging the two-timescale regime to demonstrate convergence of neural networks, P.M. and Raphaël Berthier (EPFL). Published at NeurIPS 2023. [START_REF] Marion | Structured context and high-coverage grammar for conversational question answering over knowledge graphs[END_REF] Structured context and high-coverage grammar for conversational question answering over knowledge graphs, P.M., Paweł K. Nowak (Google Research) and Francesco Piccinno (Google Research). Published at EMNLP 2021. This contribution differs from the others in the thesis since it has a more applied flavor. It follows a work carried out during an internship at Google Research in 2020.

Introduction

This introduction presents the general context of the manuscript and an overview of our contributions. We start by an introduction to the mathematics of deep learning and to some deep learning architectures in Section 1.1. In the following sections, we delve more into the details of the models we consider in this thesis, to introduce the specific context to each model and present our contributions. Section 1.2 presents the first part of this manuscript (Chapters 2 to 5), while Section 1.3 discusses the second part of the manuscript (Chapter 6 and 7). Section 1.4 gives a detailed summary of our work in French.

Mathematics of deep learning

1.1.1 Why study the mathematics of deep learning?

Deep learning is a subset of machine learning regrouping a large variety of algorithms that have three major characteristics. First, they involve an iterative data-driven optimization procedure, whose goal is to tune a parameterized function called a neural network. Second, the neural network can take different forms but is always a non-convex function of its parameters, which involves the composition of successive elementary parameterized operations. Third, the number of parameters of this function is large, and thus they also require a massive amount of data and compute to be tuned. The algorithm returns the trained neural network, which can then be used to perform a given task (classification, regression, generation, etc.).

Deep learning methods have allowed major breakthroughs in various fields in the past decade, such as computer vision or natural language processing, by taking advantage of an explosion in available compute resources and data, as well as algorithmic improvements. More recently, deep learning is behind the current successes in generative artificial intelligence (epitomized by chatGPT), and the renewed expectancy of major societal impacts of artificial intelligence.

The great empirical successes and even greater promises of deep learning methods call for a mathematical theory of neural networks. More precisely, the need for strong mathematical grounding of deep learning is justified for at least three major reasons: efficiency, effectiveness, explainability.

First, the call for more efficient learning approaches has been growing, in order to make it less resource-intensive and accessible to a wider community. This concern is particularly important to limit the environmental impact of machine learning.

Second, a deeper understanding of the fundamental underpinnings for the empirical success of deep learning may lead to proposing more effective methods, by unlocking some of the issues in the field. For instance, large language models such as chatGPT have a known tendency to produce false statements. Designing algorithms with theoretical guarantees on the validity of their outputs is therefore a crucial endeavor.

Third, explainability is necessary to foster adoption in critical applications (health, banking, cyber-security, transports, etc.) and will probably also be increasingly demanded in the Internet industry.

It is essential to emphasize at this point that deep learning is an extremely active area of research encompassing efforts at the intersection of many fields in mathematics (approximation theory, probability, statistics, optimization) and computer science (algorithmic, software engineering, hardware design), not to mention all the fields of applications. Therefore, our ambition in the following is not to give an exhaustive presentation of deep learning, but rather to introduce the models and ideas which are useful to understand the main contributions of this manuscript. We refer to the textbooks by [START_REF] Goodfellow | Deep Learning[END_REF] and [START_REF] Fleuret | The Little Book of Deep Learning[END_REF] for general introductions to deep learning, and to [START_REF] Anthony | Neural network learning: Theoretical foundations[END_REF] for an introduction to the statistical theory of deep learning. Section 1.1.2 presents the general framework of deep learning theory, in the simplest case of shallow neural networks. We then briefly present in Section 1.1.3 the main deep learning models this thesis will be concerned with. In Section 1.1.4, we describe where our contributions stand in the landscape of deep learning theory and models.

What are the main mathematical challenges of deep learning?

In this section, we give an overview of some main questions related to deep learning theory. To fix ideas, we take as our running example a simple setting, namely shallow neural network trained with stochastic gradient descent on a non-parametric regression task. Although this algorithm has been replaced in practice by empirically stronger methods, it remains a central object for mathematical study, in particular to understand its optimization and statistical properties. Furthermore, note that most of the exposition in this section applies to many other settings in deep learning and beyond in machine learning in general.

Shallow neural networks.

Let us begin by describing our class of neural networks. Shallow neural networks are parameterized functions

f v,W,b : x → v • σ(W x + b),
(1.1) where x ∈ R d is the input, the outputs belongs to R, while v ∈ R q , W ∈ R q×d and b ∈ R q are the parameters of the neural network. v and W are referred to as weights and b as the bias. Finally, σ : R → R is a function applied component-wise, called an activation function.

Common activation functions include the ReLU function x → max(x, 0), the logistic function x → (1 + exp(-x)) -1 , and the hyperbolic tangent function x → (e xe -x )/(e x + e -x ). The shallow neural network can equivalently be written

f v,W,b (x) = q k=1 v k σ(W k • x + b k ), (1.2)
where the triple (v k , W k , b k ) is referred to as a neuron. In the following of this section, for simplicity, we omit the bias b, the exposition easily extends to the case where it is present.

Non-parametric regression. Take X a random variable in R d , f * : R d → R a regression function, and Y = f * (X) + ε, where ε is some additive noise independent of X. The regression problem is the following: given an i.i.d. sample (x i , y i ) 1 i n with the same distribution as (X, Y ), find v and W such that f v,W ≈ f * . More formally, can we find v and W such that

E(|f v,W (X) -f * (X)| 2
) is small? Note that this is equivalent to finding (v, W ) such that

R(v, W ) := E(|f v,W (X) -Y | 2 ) ≈ E(|f * (X) -Y | 2 ) = min f measurable E(|f (X) -Y | 2 ),
or in other words, to find f v,W achieving the optimal Bayes risk for predicting Y given X.

The stochastic gradient descent (SGD) algorithm aims at solving this problem by tuning (or training) the parameters v and W by an iterative optimization procedure. The algorithm picks at each step a data point (x i , y i ) and updates the parameters in the direction opposite to their gradient with respect to the squared loss evaluated at (F v,W (x i ), y i ). The pseudocode is presented in Algorithm 1.

We now present a first line of theoretical analysis of this algorithm, which is an instantiation of the classical statistical learning framework.

The classical waltz (uniform law of large numbers)

The analysis of Algorithm 1, and beyond of many machine learning algorithms, has historically been decomposed in three subproblems, each corresponding to a field of applied mathematics. More precisely, given classes of parameters V and W, this analysis requires the three following conditions to hold simultaneously (see, e.g., Bach, 2023, Sections 4.2 and 5.1, for similar decompositions).

• Approximation: there exist weights v * ∈ V and W * ∈ W such that f v * ,W * is close to f * (according to some norm), or in other words, such that

R(v * , W * ) ≈ min f measurable E(|f (X) -Y | 2 ).
Algorithm 1 Training of shallow neural networks with stochastic gradient descent on a regression task Input: Sample (x i , y i ) 1 i n , initial weights v 0 and W 0 , learning rate γ, number of steps P Output: Trained weights v final and W final 1: Let r(x, y, v, W ) = |f v,W (x) -y| 2 .

2: for k = 1, 2, . . . , P do 3:

Choose uniformly at random some i ∈ {1, . . . n}.

4:

v k+1 ← v kγ ∂r ∂v (x i , y i , v k , W k ).

5:

W k+1 ← W kγ ∂r ∂W (x i , y i , v k , W k ). 6: end for 7: Return (v final , W final ) = (v P , W P ).

• Statistics: for any (v, W ) ∈ (V, W), the difference between the empirical risk

Rn (v, W ) := 1 n n i=1 |f v,W (x i ) -y i | 2 (1.3)
and the theoretical risk

R(v, W ) = E(|f v,W (X) -Y | 2 )
is small, typically decaying to zero as n grows, uniformly over all (v, W ) ∈ (V, W).

• Optimization: Algorithm 1 converges to v final and W final such that the empirical risk

Rn (v final , W final ) = 1 n n i=1 |f v final ,W final (x i ) -y i | 2
is small, typically not far away from the global minimum of the empirical risk over all possible parameters (v, W ) ∈ (V, W).

If these three conditions hold at the same time, then simple algebra shows that it is possible to bound the difference between the theoretical risk of the trained network R(v final , W final ) and the Bayes risk min f measurable E(|f (X) -Y | 2 ). Indeed, denoting (v n , Ŵn ) a minimizer of the empirical risk Rn over (V, W), we have

R(v final , W final ) - min f measurable E(|f (X) -Y | 2 ) |R(v final , W final ) -Rn (v final , W final )| + Rn (v final , W final ) -Rn (v n , Ŵn ) + | Rn (v n , Ŵn ) -R(v * , W * )| + R(v * , W * ) - min f measurable E(|f (X) -Y | 2 ).
The four terms can be controlled under the conditions presented above. The connection is straightforward for all of them, except perhaps for the third term, which can be controlled by the statistical error since

| Rn (v n , Ŵn ) -R(v * , W * )| = inf (v,W )∈(V,W) Rn (v, W ) - inf (v,W )∈(V,W) R(v, W ) sup (v,W )∈(V,W) | Rn (v, W ) -R(v, W )|.
Let us now examine whether it is reasonable to hope that the three conditions above be satisfied.

Approximation. The first approximation results for shallow neural networks were proven in a landmark paper by [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. This paper shows that shallow neural networks are universal approximators of continuous functions when the width q of the hidden layer can grow arbitrarily large, as a consequence of the Stone-Weierstrass theorem. We refer to [START_REF] Devore | Neural network approximation[END_REF] for a review of more recent and sophisticated results including rates of approximation on various function spaces.

Statistics. Bounding the difference between the empirical and the theoretical risk requires a uniform law of large numbers, in the sense that it amounts to bounding the difference between an expectation and an empirical mean uniformly over a function class. This condition is typically proven by bounding the capacity of the function class, here a class of neural networks. Statistical learning provides us with a toolbox to do so. For instance, using covering numbers argument, Anthony and Bartlett (1999, Theorem 19.2) show that, under the assumption that f v,W takes its values in [0, 1] for all (v, W ) ∈ (V, W), we have

P ∃(v, W ) ∈ (V, W), | Rn (v, W ) -R(v, W )| ε O 1 ε log( 1 ε ) D exp(-ε 2 n) , (1.4)
where the probability P holds over the sample (x i , y i ) 1 i n , and D is the so-called pseudodimension of the neural network, which can in turn be bounded by O(dq log(dq)) (Anthony and Bartlett, 1999, Theorems 8.8 and 14.1). In particular, if the sample size n is sufficiently large with respect to the pseudo-dimension of the neural network, then the right-hand side of (1.4) is small, and therefore the uniform bound on the difference between the empirical and the theoretical risks holds with high probability. Note that this implies in particular that the number of neurons q should be less than the sample size n. Nevertheless, it is also possible to construct bounds that do not depend explicitly on q but instead on the magnitude of the parameters measured according to some norm. Let us present a result in this spirit: denote W 1,∞ the maximum of the 1 -norm of the rows of W . Then, under the assumptions that f v,W takes its values in [0, 1] for all (v, W ) ∈ (V, W), that σ is Lipschitz-continuous and bounded, and that X is bounded almost surely, we have (Anthony and Bartlett, 1999, Corollary 14.16 and Theorem 17.1)

P ∃(v, W ) ∈ (V, W), | Rn (v, W ) -R(v, W )| ε O exp M 6 ε 4 log d -ε 2 n , (1.5)
where M is such that

∀(v, W ) ∈ (V, W), v 1 + W 1,∞ M.
We see that in this case, the measure of statistical complexity of the neural network is not the number of parameters anymore, but rather a norm of the parameters. The key insight behind the proof is that a network with bounded weights can be approximated by one with few weights. We refer to Bartlett et al. (2017[START_REF] Bartlett | Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks[END_REF] for more recent results on the statistical complexity of neural networks.

Optimization. The problematic time in our waltz is optimization: since the shallow neural network (1.1) is non-convex in W , the objective function Rn of the optimization algorithm is also non-convex. This breaks the key convexity assumption ensuring that gradient-based optimization procedures (such as stochastic gradient descent) converge to a global minimum. For this reason, there is no guarantee that SGD converges to a global minimum, and there are even negative results describing settings where SGD for shallow neural networks converges to local minima with high probability [START_REF] Brutzkus | SGD learns over-parameterized networks that provably generalize on linearly separable data[END_REF]. Nevertheless, in practice, it is observed that trained neural networks do converge to very low training errors. For instance, it is shown in [START_REF] Zhang | Understanding deep learning (still) requires rethinking generalization[END_REF] that it is easy to train neural networks to interpolation (i.e., zero training error), even with random labels. A framework to explain this phenomenon has emerged in the past years, which we present next.

The modern tango (large number of neurons, implicit regularization)

The classical decomposition into the trio approximation-statistics-optimization, which we sketched above, is not entirely satisfactory for several reasons. First, it provides no guarantee that the optimization algorithm converges close to a global minimizer of the empirical risk. Second, the statistical guarantees are typically given over a bounded subset of the space of parameters, and it is not a priori clear that the output of the optimization algorithm belongs to this subset. Finally, it has been remarked that uniform capacity control provably leads to vacuous bonds in some settings (Nagarajan and Kolter, 2019).

These reasons have prompted a more complex line of analysis (see [START_REF] Belkin | Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation[END_REF], for a general presentation), which analyzes jointly the optimization and statistical aspects instead of relying on a uniform deviation bound over a class of parameters. This analysis holds when the number of neurons is large enough (typically larger than the sample size). The reasoning in this new line of analysis is two-fold: first, prove that, when the network is wide enough, the optimization landscape becomes more favorable, which allows proving global convergence of the empirical risk to zero despite the lack of convexity. Second, even more agreeable, the optimization algorithm does not converge towards just any minimizer of the empirical risk, but towards a minimizer that possesses structural properties. More precisely, among all minimizers of the empirical risk, the one found by SGD minimizes some measure of complexity of the parameters. In other words, the optimization algorithm implicitly solves the problem

min v,W ∈(V,W) Rn(v,W )=0 c(v, W ),
where c is an appropriate measure of complexity, which is typically a norm (see below) or a matrix rank (Razin and Cohen, 2020). Furthermore, neural networks such that c(v, W ) is low enjoy favorable generalization properties, hence the name of implicit regularization given to this phenomenon.

Let us now delve a bit more into the details of the approach, first regarding the proof of global convergence, then the implicit regularization.

Global convergence of SGD. Regarding the first facet of the reasoning, global convergence of gradient-based algorithms for (shallow) neural networks can be explained in the case where the width q of the network is large enough with respect to the number of data n. Recent works in this direction can roughly be clustered in two main categories.

On the one hand, the neural tangent kernel (NTK) regime (Jacot et al., 2018;[START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF]Zou et al., 2020a) corresponds to small movements of the parameters of the neural network. In this case, the neural network can be linearized around its initial point, and thus behaves like a linear regression. In spirit, provided that there are more parameters than data points, the linear regression is then overparameterized, which means that there exists a solution to the linear regression problem with a null empirical risk.

On the other hand, the mean-field regime (Chizat and Bach, 2018;[START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF][START_REF] Rotskoff | Trainability and accuracy of neural networks: An interacting particle system approach[END_REF][START_REF] Sirignano | Mean field analysis of neural networks: a central limit theorem[END_REF] describes the dynamics in the regime q 1 through a partial differential equation (PDE) on the density of neurons, which takes the form of a Wasserstein gradient flow. It can then be shown that, under the assumption that this PDE converges, then the limit distribution must be optimal. This regime is sometimes referred to as 'rich' (see, e.g. [START_REF] Woodworth | Kernel and rich regimes in overparametrized models[END_REF] since it describes non-linear feature learning, contrarily to the NTK regime which is akin to a kernel method. The transition from the kernel to the rich regime depends on the scale of the initialization (Chizat et al., 2019;[START_REF] Woodworth | Kernel and rich regimes in overparametrized models[END_REF].

Implicit regularization. A number of recent works study implicit regularization for neural networks. Let us present one specific example, which fits in the framework of regression with shallow neural networks presented above, in order to give the flavor of these results. Boursier et al. (2022) study the case of a ReLU activation function, a large enough number of neurons q and orthogonal inputs, which in particular implies that d n. They show that, if the weights are initialized infinitesimally close to zero (corresponding to the rich regime described above), then the gradient flow converges to a global minimizer of the empirical risk, achieving a zero empirical risk. Furthermore, this minimizer is a solution of the problem min v,W ∈R q ×R q×d Rn(v,W )=0

v 2 + W 2 ,
where • denotes the Euclidean norm. This result can be rephrased in terms of the prediction function f v,W : it shows that the prediction function found by gradient flow minimizes the so-called variation norm among all interpolators of the training data that write as an infinite-width neural network. Furthermore, it is known that prediction functions with low variation norm generalize well (Bach, 2017, Proposition 7). Note that many other implicit regularization results have been shown, we refer to Vardi (2023) for a review.

Remark 1.1. Since the analysis presented in this section holds when the number of neurons is large, it is often referred to as the 'overparameterized' regime, by opposition to an 'underparameterized' regime which would correspond to the analysis of the previous section. However, this formulation is somewhat improper since it was already known that uniform deviation bounds can hold independently of the number of parameters, as shown for instance by the bound (1.4). It seems more precise to distinguish both analyses by stating that the uniform deviation bound is replaced by a characterization of the implicit regularization implied by the optimization algorithm.

From shallow neural networks to Transformer

Having sketched the landscape of deep learning theory, we now present in this section some important neural network architectures by increasing level of complexity. Let us stress that we give a simple and unified presentation of the models, at the cost of generality. For instance, we remove biases and normalizations (batch normalization or layer normalization). More general models will be considered in the manuscript, with slight variations depending on the chapters.

Deep neural networks. The shallow neural network model introduced in Section 1.1.2 can be generalized to deep neural networks, which consists in composing linear mappings and non-linear component-wise activations, according to the relations

h 0 = Ax,
(1.6) (1.8) where L denotes the depth of the network, A ∈ R q×d , W 1 , . . . , W L ∈ R q×q , and B ∈ R d ×q are weight matrices, and σ is still an activation function. The quantity h k is referred to as a hidden layer. The weight matrices can either be full matrices, in which case the neural network is called fully-connected, or sparse with weight-sharing, corresponding to convolutional neural networks.

h k+1 = σ(W k+1 h k ), 0 k L -1, (1.7) F (x) = Bh L ,
In practice, convolutions play a critical role in the performance of neural networks for visions tasks, in particular because they encode symmetries of the network with respect to translation in the image. Nevertheless, we do not present them in more details here, since the results in this manuscript are given for fully-connected neural networks. We refer the interested reader to Goodfellow et al. (2016, Section 9) for a detailed presentation of convolutional neural networks.

Residual neural networks. In practice, vanilla deep neural networks (1.6)-(1.8) are difficult to train when L is larger than a few units, due to instabilities in the training procedure, as demonstrated in particular in a landmark paper by He et al. (2016a). These authors propose important improvements, allowing to reach depth of order to several hundreds or even thousands, thereby achieving (at the time) state-of-the-art results in image recognition. The crucial modification is the presence of skip connections, meaning that the definition of the hidden layer (1.7) is replaced by

h k+1 = h k + σ(W k+1 h k ), 0 k L -1.
(1.9)

Comparing (1.7) and (1.9), we see that, in the second case, the mapping h → W k+1 σ(h) does not parameterize directly the new hidden layer h k+1 as in (1.7), but rather the difference between two successive hidden layers h k+1h k , hence the name residual neural network given to this model. This intuition is key to the developments on the properties of this model made in the first part of this manuscript.

Recurrent neural networks. Note that, for now, we have placed ourselves in the simple case where the input and the output are vectors, belonging respectively to R d and R d . Another key context is where the input data consists in a collection of vectors x ∈ R L×d . This setting encompasses time series (in which case L corresponds to the number of time steps) or textual data (where L corresponds to the length of the text). We now present a first model that is adapted to this context, recurrent neural networks [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. They are defined by a sequence of hidden states h 1 , . . . , h L ∈ R q . These hidden states obey the following recurrence, for an input x = (x 1 , . . . , x L ):

h 0 = 0, h k+1 = σ(W h k + U x k+1 ), 0 k L -1 F (x) = Bh L , (1.10)
where σ is an activation function, and W ∈ R q×q , U ∈ R q×d , and B ∈ R d ×q are weight matrices.

Note that here, contrarily to the models presented above, the weight matrices do not depend on the layer index k. Naturally, it is also possible to consider a residual version of recurrent neural networks (see, e.g., [START_REF] Yue | Residual recurrent neural networks for learning sequential representations[END_REF]. The iteration then writes

h k+1 = h k + σ(W h k + U x k+1 ), 0 k L -1.
(1.11)

Transformer. Recurrent neural networks have several drawbacks that make them unsuitable in some use cases (Kolen and Kremer, 2001); they are slow to train and suffer from instabilities during training. They also have difficulties picking up long-range dependencies in the data. An important mechanism to mitigate these problems is the so-called attention [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF], which is in particular present in Transformer (Vaswani et al., 2017). The Transformer architecture forms the basis of modern natural language processing and in particular of large language models. There are different variants of this architecture, we focus here on the Transformer encoder (a.k.a. BERT-like Transformer) to give a flavor of the main ideas. Just like recurrent neural networks, the Transformer encoder handles a collection of vectors x = (x 1 , . . . , x L ) ∈ R L×d . However, instead of performing sequential computations involving successively each x i for i ∈ {1, . . . , L}, it manipulates the whole sequence at once. More precisely, the Transformer encoder consists in a series of blocks, where each block takes as input a matrix x ∈ R L×d and returns a matrix B(x) ∈ R L×d , so several blocks can be composed together to form the encoder. Each block consists in two layers, an attention layer and a feedforward layer, which are defined as follows:

A(x) = x + H h=1 σ(xQ h K h x )xV h O h B(x) = A(x) + ReLU(A(x)W 1 )W 2
The parameters of the attention layer are Q h , K h , V h , O h that are all R d×r matrices, for h ∈ {1, . . . , H}, and the parameters of the feedforward layer are W 1 ∈ R d×m and W 2 ∈ R m×d . The dimensions r and m are hyperparameters of the model. The function σ : R L×L → R L×L performs a row-wise softmax operation, that is, for M ∈ R L×L , we have

σ(M ) ij = exp(M ij ) L k=1 exp(M ik )
.

Finally, ReLU : x → max(x, 0) is the activation function applied element-wise. We emphasize that the architecture presented here is a simplification of the actual Transformer models, which eludes important concepts (in particular the contrast between encoder-only, decoder-only, and encoder-decoder architectures). We refer to [START_REF] Phuong | Formal algorithms for transformers[END_REF] for a more in-depth presentation of Transformer-related algorithms.

1.1.4 What to expect in this manuscript?

Now that we have presented a brief panorama of deep learning theory and of neural networks models, we are in a position to examine where our contributions stand in this landscape. The situation is summarized in Table 1.1, where the chapters are sorted by architecture, and Table 1.2, where the chapters are sorted according to the nature of the results.

Architecture Type of results

Residual networks

Other properties (Chap. The analysis of shallow models presented in Section 1.1.2 does not fully adapt to deep networks, and furthermore does not provide a consensual explanation for the role of depth, despite its empirically-proven importance (see, e.g., [START_REF] Wang | Deepnet: Scaling transformers to 1,000 layers[END_REF]. This remark, along with the empirical success of residual networks (He et al., 2016a), motivated a line of research devoted to understanding the properties of residual networks in the limit where the depth tends to infinity.

Let us consider in this section a more general formulation of residual networks than (1.9), which writes

h 0 = Ax, h k+1 = h k + 1 L β V k+1 σ(W k+1 h k ), 0 k L -1, F (x) = Bh L , (1.12)
where we added a scaling factor 1/L β with β > 0, additional weight matrices V k ∈ R q×q for k = 1, . . . , L, and we recall that, as in (1.9), A ∈ R q×d , B ∈ R d ×q , W k ∈ R q×q for k = 1, . . . , L, and σ is an activation function. The presence of a scaling factor differs from the residual network models presented above. This matter is thoroughly discussed in Chapter 2; in a nutshell, the scaling factor is necessary for the model to be well-posed in the absence of other normalization techniques such as batch normalization. As for the additional weight matrices V k , they make the formulation of the model actually closer to the original residual networks of He et al. (2016a), and they are required for some results below.

Looking at the discrete recursion (1.12), a natural idea is to substitute it with a continuous counterpart dH ds = V σ(W H), s ∈ [0, 1].

(1.13)

In other words, the discrete layer index k is transformed into a continuous layer index s and the discrete updates are replaced by an ordinary differential equation. This idea has been popularized by the seminal paper of Chen et al. (2018a), which coined the terminology of neural ordinary differential equations (neural ODEs). Subsequently, numerous works have relied on the intuition that the neural ODE (1.13) is the limit of the residual network (1.12) in the large-depth limit L → ∞ (see, e.g., [START_REF] Haber | Stable architectures for deep neural networks[END_REF]E et al., 2019;[START_REF] Dong | Towards adaptive residual network training: A neural-ODE perspective[END_REF]Massaroli et al., 2020;[START_REF] Kidger | On Neural Ordinary Differential Equations[END_REF], although the picture is more complex as we will see in Section 1.2.1.

Remark 1.2. Equation (1.13) is ambiguous in that it does not specify whether V and W depend on s. The answer depends on the context, which is why we leave it ambiguous for now and make more precise statements in the following. More precisely, neural ODEs were first introduced by Chen et al. (2018a) with constant (i.e., depth-independent) parameters. However, since the parameters of (1.12) depend on the layer index, it seems clear, and will be made formal in the following, that the continuous version of (1.12) has depth-dependent parameters V (s) and W (s).

Finally, we also study in Chapter 5 a continuous version of recurrent neural networks, in which case the parameters of the limit ODE are depth-independent (see Section 1.2.4).

The continuous-depth limit (1.13) appeals both from an algorithmic and a theoretical point of view. Let us delve more into the details of both aspects.

Algorithmic advantages of the large-depth limit. The continuous-depth viewpoint on deep learning has received a lot of attention from an algorithmic and practical point of view. We review a few relevant works here, although we emphasize that this manuscript focuses mostly on theoretical analysis. For this reason, the reader interested in algorithmic developments is encouraged to investigate the PhD thesis of [START_REF] Kidger | On Neural Ordinary Differential Equations[END_REF], which contains a comprehensive overview of models and algorithms related to neural ODEs. This being said, an important motivation for continuous-depth models is that they offer memory efficient training by using the adjoint method to retrieve gradients, which removes the need to store the value of hidden layers (Chen et al., 2018a). This property was later extended to finite-depth residual networks (Sander et al., 2022b). Another line of work has used the continuous viewpoint to design new architectures that discretize more efficiently continuous-depth differential equations, and thereby benefit from favorable properties of their continuous-depth equivalent, such as stability [START_REF] Haber | Stable architectures for deep neural networks[END_REF][START_REF] Chang | AntisymmetricRNN: A dynamical system view on recurrent neural networks[END_REF][START_REF] Benning | Deep learning as optimal control problems: Models and numerical methods[END_REF]. Novel optimization algorithms coming from optimal control have also been proposed by casting the optimization problem over continuous-depth networks as an optimal control problem [START_REF] Li | Maximum principle based algorithms for deep learning[END_REF]. The continuous viewpoint was also used to design efficient generative models (Chen et al., 2018a;[START_REF] Grathwohl | Scalable reversible generative models with free-form continuous dynamics[END_REF][START_REF] Kidger | Signatory: Differentiable computations of the signature and logsignature transforms, on both CPU and GPU[END_REF], which sample noise then continuously transforms it into the target distribution. In particular, the recently acclaimed diffusion models can be seen as instances of neural ODEs [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF]. Finally, in a time series context, continuous-depth networks have been praised for natively handling irregularly sampled data (Rubanova et al., 2019;Kidger et al., 2020), contrarily to standard models.

Theoretical advantages of the large-depth limit. From a theoretical point of view, the large-depth limit is conceptually simple and provides a very natural interpretation of depth as the time flow of a differential equation. Furthermore, it allows to leverage the well-established tools of differential equations in a deep learning context. This prompted interest into deriving mathematical properties of deep learning models by leveraging the continuous viewpoint, as proposed by E (2017). For instance, E et al. (2019) leverages the optimal control theory to obtain statistical guarantees on infinite-depth residual networks, by formulating a Pontryagin's maximum principle version of the learning problem. [START_REF] Cuchiero | Deep neural networks, generic universal interpolation, and controlled odes[END_REF] also uses tools from optimal control (Lie brackets and controllability) to show that infinite-depth residual networks are able to interpolate arbitrary training sets with a limited number of parameters.

However, the precise derivation of the connection between (1.12) and (1.13), and the exploration of the consequences of the continuous viewpoint on residual networks remain not yet fully investigated. Our goal in this first part of the thesis is to provide mathematical statements on these two challenges. We will answer the following four key questions: first, under what conditions and in which sense does the neural ODE limit hold for deep neural networks, both at initialization and after training? Second, are there other possible deep limits than neural ODEs? Third, can global convergence of the training algorithm be proven in this setting? Fourth, what does this ODE-like regime imply in terms of the generalization abilities of deep neural networks?

Before delving into the core of our contributions, let us note that the setting and precise assumptions vary slightly between the different chapters. We give here a unified and simplified presentation, and leave rigorous statements, related work, as well as generalizations of the results sketched below, to the core of the manuscript. In particular, rather than providing a literature review on the theory of large-depth residual networks at this point, we refer the reader to the related work sections of Chapters 2 to 5 for an in-depth survey of the literature relevant to the topics addressed in each chapter.

Scaling of residual networks at initialization [Chapter 2 of the manuscript]

We begin by investigating in Chapter 2 whether the scaled residual network (1.12) converges at initialization towards a differential equation in the large-depth limit, depending on the initialization scheme and on the scaling factor β. Several possible initialization schemes are investigated, the two main ones being i.i.d. initialization and smooth initialization. I.i.d. initialization corresponds to the standard practice of initializing every weight as an i.i.d. random variable, e.g., uniform or Gaussian. Smooth initialization is less common and amounts to taking the V k and W k as discretizations of some smooth (possibly random) functions V : [0, 1] → R q×q and W : [0, 1] → R q×q , that is, V k = V(k/L) and W k = W(k/L) for k ∈ {1, . . . , L}. This includes in particular the case where the weights are initialized weight-tied across the depth, i.e., where V and W are constant functions. Another more complex case is where the entries of V and W are independent Gaussian processes with expectation zero and squared exponential covariance.

Our main contribution in this chapter is to show that the large-depth properties of the network at initialization depend on the joint choice of β and of the initialization scheme, as reported in Table 1.3. Both in the i.i.d. case and in the smooth case, a specific value of β corresponds to a differential equation limit that separates two antithetical dynamics, explosion and identity. Furthermore, the differential equation limit is a stochastic differential equation (SDE) in the i.i.d. case and an ODE in the smooth case. 1.3: Properties of the residual network at initialization as a function of the scaling factor and of the initialization. Explosion means that the output of the network tends to infinity when the depth L goes to infinity. Identity corresponds to the fact that h L ≈ h 0 when L goes to infinity.

Scaling factor

0 < β < 1/2 β = 1/2 1/2 < β < 1 β = 1 β > 1 I.i.d
To make things more precise, the first row of Table 1.3 can be formalized by the following result, which is a simplified version of Corollary 2.4 in Chapter 2.

Theorem 1.3. Consider the residual network (1.12) with i.i.d. weights.

(i) If β > 1 /2, then h L -h 0 h 0 P ----→ L→∞ 0. (ii) If β < 1 /2, then h L -h 0 h 0 P ----→ L→∞ ∞.
(iii) If β = 1 /2, then, for any δ ∈ (0, 1), with probability at least 1δ,

exp 3 8 - 22 qδ -1 < h L -h 0 2 h 0 2 < exp 1 + 10 qδ + 1.
The proof of this result relies crucially on the martingale structure of ( h k ) 0 k L , as well as on state-of-the-art concentration inequalities for random matrices with sub-Gaussian entries.

In summary, among the possibilities we examine, the only case where the ODE limit (1.13) holds is when initializing with smooth weights and taking a scaling factor β = 1 (corresponding to the framed cell in Table 1.3). This is the case which we consider in the following chapters.

Furthermore, we also perform some preliminary experiments showing that this ODE-like weight structure is preserved after training, as shown in Figure 1.1. This is the topic of the next chapter.

Implicit regularization of deep residual networks towards neural ODEs [Chapter 3 of the manuscript]

In Chapter 3, we also consider model (1.12), this time focusing on the case of a smooth initialization and a 1/L scaling factor. According to the previous chapter, we know that, at initialization, the neural network converges towards a neural ODE. We are interested in showing that the trained neural network still exhibits an ODE-like weight structure, or in other words, to shed light on Figure 1.1a. To this aim, we assume that the neural network is trained with a gradient flow that writes as the following ODEs (1.14) where Rn denotes the empirical risk (as defined in (1.3)). Gradient flow can be seen a simplifying limit of SGD (see Algorithm 1) when the learning rate γ tends to zero. It is a standard tool in the analysis of optimization algorithms. We refer to Bach (2020) for a user-friendly introduction to gradient flows in a machine learning context. The scaling factor L in (1. [START_REF] Frei | Algorithm-dependent generalization bounds for overparameterized deep residual networks[END_REF] is the counterpart of the scaling factor 1 /L in (1.12), and is necessary to avoid gradient vanishing as L goes to infinity. Note that the time variable t of the ODE differs from the continuous-depth variable s in the neural ODE (1.13). Our first main contribution in this chapter is to prove that the neural ODE limit holds after training, as shown in the following theorem (Theorem 3.4 of Chapter 3), stated informally here.

∂V k ∂t (t) = -L ∂ Rn ∂V k (t), ∂W k ∂t (t) = -L ∂ Rn ∂W k (t), t 0,
Theorem 1.4. Consider the neural network (1.12) with β = 1. Assume that it is initialized smoothly in the sense of Section 1.2.1, and trained with the gradient flow (1.14) for t ∈ [0, T ].

Then there exist Lipschitz-continuous functions V, W : [0, 1] × [0, T ] such that, uniformly over s ∈ [0, 1] and t ∈ [0, T ], V sL (t) → V(s, t) and W sL (t) → W(s, t) when the depth L goes to infinity.

Furthermore, for any k ∈ {0, . . . L}, denote h k (t) the value of the layer k at training time t. Then, for any s ∈ [0, 1], the layer h sL (t) converges when L goes to infinity to the value at time s of the solution of the neural ODE

H(0) = x dH ds (s) = V(s, t)σ(W(s, t)H(s)), s ∈ [0, 1]. This convergence is uniform over s ∈ [0, 1], t ∈ [0, T ] and x ∈ K for any compact K ⊂ R d .
This large-depth convergence holds for any finite training time t ∈ [0, T ]. However, the convergence of the optimization algorithm when T goes to infinity is not guaranteed without further assumptions, due to the non-convexity of the optimization problem. We can obtain such a convergence by proving a Polyak-Łojasiewicz (PL) condition, which is a key modern tool in analyzing the properties of optimization algorithms for deep neural networks [START_REF] Liu | Loss landscapes and optimization in over-parameterized non-linear systems and neural networks[END_REF]. Importantly, the PL condition implies the convergence of gradient flow to a global minimum. As our second main contribution in this chapter, we prove that such a condition holds when the width q of the hidden layers is larger than some constant times the number of data n. As a consequence, we obtain the convergence with high probability in large depth and large training time, namely the existence of Lipschitz-continuous functions V ∞ and W ∞ such that the trained neural network converges when both L and T go to infinity to the ODE

dH ds (s) = V ∞ (s)σ(W ∞ (s)H(s)), s ∈ [0, 1]. (1.15)
The convergence holds in a sense similar to the one of Theorem 1.4. Furthermore, the limit ODE achieves zero training loss. This analysis is a first step towards understanding the implicit regularization (see Section 1.1.2.2) of gradient flow for deep residual networks, that is, characterizing the properties of the trained network among all minimizers of the empirical risk.

Generalization bounds for neural ODEs and residual networks [Chapter 4 of the manuscript]

Now that we know that trained deep residual networks converge in the large-depth limit towards neural ODEs of the form (1.15), we are interested in understanding the statistical properties of this class of models. This is our aim in Chapter 4. We examine the simpler case where the inner weights W ∞ are equal to the identity matrix, or in other words, we only consider outer weights V ∞ . Even with this simplification, this remains a delicate problem since the output of the model is a non-linear function of the parameters, and the latter belong to an infinite-dimensional space. Nevertheless, we are able to bound their statistical complexity by leveraging results on the covering number of bounded Lipschitz-continuous functions. More precisely, for a matrix-valued function

V , denote V 1,1,∞ = max s∈[0,1] i,j |V ij (s)|.
Then we consider the set of neural ODEs

H(0) = x dH ds (s) = V (s)σ(H(s))ds, s ∈ [0, 1] F V (x) = H(1), (1.16)
where V belongs to the set of functions

V = {V : [0, 1] → R q×q , V 1,1,∞ R V and V ij is K V -Lipschitz-continuous for i, j ∈ {1, . . . , q}}. (1.17)
It is a simplification of the large-depth limit model (1.15) obtained in the previous chapter, in the sense that we removed the initial and final projections matrices A and B, as well as the inner weights W (s). We place ourselves in a standard supervised learning setup with n i.i.d. data samples, which we assume to be almost surely bounded. We consider some general Lipschitz-continuous loss function. The empirical risk is denoted by Rn , the theoretical risk by R, and the empirical risk minimizer over V by Vn . The following result can then be proven as a straightforward corollary of Theorem 4.4 of Chapter 4.

Theorem 1.5. Consider the class of neural ODEs (1.16), where V belongs to (1.17). Then, there exists a constant B > 0 depending on the parameters of the problem such that for n large enough, for any δ > 0, with probability at least 1δ,

R( Vn ) Rn ( Vn ) + B(q + 1) log(R V mn) n + B q 2 √ K V n 1/4 + B √ n log 1 δ .
This result belongs to the family of generalization bounds, which aim at bounding the difference between the theoretical risk and the empirical risk of Vn . It is a consequence of a uniform law of large numbers as presented in Section 1.1.2.1. Three terms appear in our upper bound of R( Vn )-Rn ( Vn ). The first and the third ones are classical (see, e.g. Bach, 2023, Sections 4.4 and 4.5). On the contrary, the second term is more surprising with its convergence rate in O(n -1/4 ). This slower convergence rate is due to the fact that the space of parameters V is infinite-dimensional. However, we retrieve a classical O(n -1/2 ) convergence rate in the case where K V = 0. This corresponds to constant functions V , i.e., depth-independent weights, which belong to a finite-dimensional space.

As a second main contribution in this chapter, we prove similar generalization bounds for finite-depth residual networks of the form (1.12). Although these bounds hold for finite-depth networks, their derivation is strongly inspired by the infinite-depth analysis.

Recurrent neural networks as kernel methods [Chapter 5 of the manuscript]

In Chapter 5, we consider a slightly different model, namely residual recurrent networks. Similarly to the previous chapter, we introduce a scaling factor 1/L and thus consider the rescaled version of the residual recurrent network (1.11), which writes as follows

h k+1 = h k + 1 L σ(U h k + V x k+1 ). (1.18)
We further assume that each input series (x k ) 1 k L corresponds to the discretization of a continuous time process X : [0, 1] → R d . Our goal is to show that the rescaled residual recurrent network is actually a kernel method, yielding as a byproduct generalization and stability bounds. Since we recognize in (1.18) an Euler discretization of an ODE, we first show (in a precise sense) that the network approaches at distance O(1/L) the continuous-time network

dH ds (s) = σ(U H(s) + V X(s)), H(0) = 0, (1.19)
where the output of the network is a linear projection of the value of the solution of the ODE at time t = 1, that is, y = BH(1). Then, our main theorem in this chapter states the existence of a Hilbert space H and of mappings S and ς such that, under some regularity assumptions, .20) In this equation, S(X) is the signature of the time series X, which is a feature map for time series [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF], while ς(B, U, V ) is an expression involving powers of B, U , V and higher-order derivatives of σ. The key insight of the proof is to use a specific variant of the Taylor expansion that applies for ODEs involving a time-dependent parameter X, which are known as controlled differential equations [START_REF] Lyons | Differential Equations Driven by Rough Paths[END_REF]. This result allows to reinterpret the action of the recurrent network as a scalar product in an (infinite-dimensional) Hilbert space, thereby framing the recurrent network as a kernel method. Hence we can use the usual kernel machinery to derive generalization bounds and stability bounds. Let us illustrate the second idea, whose derivation is easier: for two time series X and X , we can bound the difference between the corresponding outputs y and y by using the Cauchy-Schwartz inequality, as follows

y = S(X), ς(B, U, V ) H . ( 1
|y -y | = | S(X), ς(B, U, V ) H -S(X ), ς(B, U, V ) H | ≤ S(X) -S(X ) H ς(B, U, V ) H .
(1.21) Since the mapping S is continuous, S(X) -S(X ) H is small if X and X are sufficiently close. Hence this result shows that the Hilbert norm of the weights mapping ς(B, U, V ) H controls the stability of the network. This suggests using this quantity as a regularizer when training the network.

Contributions to finite-depth neural networks [Part II of the manuscript]

This second part of the manuscript gathers various contributions related to modern topics in deep learning, this time for finite-depth neural networks, contrarily to the first part of the manuscript which focused on large-depth limits. We first present results on shallow neural networks, then on Transformer.

Convergence of shallow neural networks in the two-timescale regime [Chapter 6 of the manuscript]

In Chapter 6, we study the training dynamics of shallow neural networks of the form (1.2), in a two-timescale regime in which the step sizes for the inner layer are much smaller than those for the outer layer. We consider a simple univariate setting where the neural network writes

f v,b (x) = q k=1 v k σ(x -b k ), for an input x ∈ R and parameters v k , b k ∈ R.
Comparing with the general formulation of shallow neural networks (1.2), we see that we fix the multiplicative inner weights w k to 1. Similarly to Chapter 3 (see equation (1.14)), we assume that the neural network is trained by gradient flow, this time directly on the theoretical risk

R(v, b) = E(|f v,b (X) -f * (X)| 2 ),
where X follows a uniform law on [0, 1]. Minimizing directly the theoretical risk instead of the empirical risk allows us to set aside statistical issues to focus on optimization, which remains a delicate matter since the risk is non-convex in b. The gradient flow equations write

∂v ∂t (t) = - ∂R ∂v (t), ∂b ∂t (t) = -ε ∂R ∂b (t), t 0, (1.22)
where ε parameterizes the ratio between the step sizes for the inner layer and for the outer layer. By taking ε 1, the neural network can be thought of as a fitted linear regression with slowly evolving features σ(•b k ), k = 1, . . . , q. This reduction enables us to precisely describe the movement of the inner layer parameters b k . In this two-timescale regime, we prove convergence of the gradient flow to a global optimum in the case where f * is a piecewise constant function.

More precisely, we show the following theorem (Theorem 6.2 of Chapter 6), stated in a simplified manner here.

Theorem 1.6. Let ξ, δ > 0, and f * a piecewise constant function with N pieces whose sizes are all lower-bounded by ∆v. Assume that the neural network has q neurons with q 6 ∆v 4 + log N + log 1 δ .

(1.23)

Assume that, at initialization, the biases b 1 , . . . , b q are i.i.d. uniformly distributed on [0, 1] and the weights v are equal to zero. Then there exist an activation function σ, ε > 0, and T > 0, such that, with probability at least 1δ, the solution to the gradient flow (1.22) is defined at least until T , and

R(v(T ), b(T )) ξ .
Note that the lower bound on the number of neurons (1.23) does not depend on the target precision ξ, but only on the function f * and on the probability of failure δ. This distinguishes our result from the neural tangent kernel or mean-field regimes, mentioned in Section 1.1.2, which both require the width of the network to grow large in order to achieve an arbitrary precision.

Experimental illustration is provided, showing that the stochastic gradient descent behaves according to our description of the gradient flow and thus converges to a global optimum in the two-timescale regime (see Figure 1.2), but can fail outside this regime (see Figure 1.3). ). The target function is in blue and the neural network is in orange. The biases b 1 , . . . , b q of the neurons are indicated with vertical dotted lines (the dots are only present for black and white visibility). In a first short phase, only the weights v 1 , . . . , v q of the neurons evolve to match as well as possible the target function (second plot). Then, in a longer phase, the neuron closest to each target discontinuity moves towards it (third plot). Recovery is achieved.

Structured context and high-coverage grammar for conversational question answering over knowledge graphs [Chapter 7 of the manuscript]

We present in Chapter 7 results on the Transformer architecture, turning our attention to more algorithmic matters related to natural language processing. Since this topic may be less familiar to the reader than the rest of the thesis, we present the context and our contribution in a somewhat more detailed fashion than the other chapters. Our goal is to design a Transformer-like neural network for the task of conversational question answering over knowledge graphs. Let us first describe the task, before presenting our approach, contributions and results.

Figure 1.4: A small subgraph of Wikidata

To begin with, a knowledge graph (KG) is a particular type of graphs that encodes real-world factual information. It is a directed graph consisting of vertices, called entities, which represent concepts, and labeled edges, which represent relations between concepts. For instance, Figure 1.4 presents a small subgraph of Wikidata, which is the largest publicly available KG and which we work with in this chapter. Question answering over KGs refers to the task of building a language model that can answer factual questions by querying the graph. Furthermore, we want our system to handle conversations, that is, not only be able to answer a one-off question but also follow-up questions on the same topic. This task is referred to as conversational question answering over KGs. An example of such a conversation is presented in Table 1.4. It is taken from ConvQuestions [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], which, together with CSQA [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF], is one of the main publicly available datasets for this task. Note that conversational question answering over KGs is of direct interest for developing personal assistants that can reliably answer factual-based questions. One of the difficulties associated with this task is the scale of the graph: for instance, there are over a hundred million nodes and a billion edges in Wikidata, and private KGs can be orders of magnitude larger.

Q

Who played the joker in [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF]. The goal of the system is to find the right answer at each turn of the conversation.

We address this task by building a so-called semantic parsing model. A semantic parser is a language model that takes as input a sentence phrased in natural language (typically English) and transforms it into a query formulated in some programming language, which can then be executed, or evaluated, to return an answer to the question. From a high-level perspective, a semantic parser can be seen as translating English into a machine-executable language. Let us introduce some vocabulary that is heavily used in Chapter 7: the query is called a logical form, and the programming language in which it is expressed is called a grammar.

Our semantic parser works in several steps depicted in Figure 1.5. Given a question, entities in the graph that are likely to be relevant are identified using a technique called named entity linking. Then local exploration of the graph around these entities is performed to extract a context of tractable size. The context is organized in a tree structure, where each node of the tree contains a vector of information. Technically, this tree takes the form of a JSON file. In a third step, a Transformer-like neural network takes this context tree as input and returns the logical form.

Our work features two main methodological contributions: first, we build a grammar that can model a larger scope of questions over knowledge graphs than previous propositions. Second, we introduce a variant of the Transformer architecture that can operate on tree-structured data, contrarily to the standard version, which can only process a list of vectors. One of the difficulties we face is that we do not have at our disposal readily-available supervised data to train our Transformer model, since there exists no dataset mapping English questions to logical forms in our grammar. To circumvent this issue, we first have to craft such a dataset, before training our model. This setting is referred to as weak supervision. The dataset is created in a compute-intensive manner: for every (question, answer) pair in CSQA and ConvQuestions, we explore the space of all possible logical forms to find one whose execution gives the right answer, and which has reasonable chances to be a correct formalization of the question. Heuristic metrics are used to assess the accordance between the question and the logical form. The space of logical forms can be seen as a tree, which we explore with breadth-first search in order to favor the simplest possible logical forms.

The capabilities of our system are evaluated on the two datasets mentioned above. Our approach improves over the state-of-the-art on both datasets in terms of answer accuracy.

Résumé détaillé en français

Cette thèse présente des contributions à la théorie des réseaux de neurones, séparées en deux parties. La première partie s'intéresse à l'analyse mathématique des réseaux résiduels dans la limite en grande profondeur, et comporte quatre chapitres. La deuxième partie présente des contributions reliées cette fois aux réseaux de profondeur finie, et regroupe deux chapitres. Nous introduisons ici les chapitres dans l'ordre du manuscrit. La présentation est moins détaillée que celle qui précède en anglais, aussi nous recommandons au locuteur anglophone de se référer à la version anglaise.

Échelle des réseaux résiduels à l'initialisation [Chapitre 2 du manuscrit]

Nous commençons dans le Chapitre 2 par nous intéresser aux propriétés des réseaux de neurones résiduels qui s'écrivent

h 0 = Ax, h k+1 = h k + 1 L β V k+1 σ(W k+1 h k ), 0 k L -1, F (x) = Bh L , où la donnée est x ∈ R d , la matrice A appartient à R q×d , les états cachés h k sont dans R q , les matrices V k+1 , W k+1 appartiennent à R q×q , et B ∈ R d ×q . L'objectif
V : [0, 1] → R q×q et W : [0, 1] → R q×q , soit V k = V(k/L) and W k = W(k/L) pour k ∈ {1, . . . , L}.
Notre contribution principale dans ce chapitre est de montrer que la limite en grande profondeur du réseau à l'initialisation dépend conjointement de β et du schéma d'initialisation, comme présenté dans le Tableau 1.5.

Facteur d'échelle

0 < β < 1/2 β = 1/2 1/2 < β < 1 β = 1 β > 1 Initialisation i.i.d.
H(0) = Ax, dH ds (s) = V(s)σ(W(s)H(s)), s ∈ [0, 1], F (x) = BH(1).
(1.24)

C'est le cas auquel on s'intéresse dans les deux chapitres qui suivent.

∂V k ∂t (t) = -L ∂ Rn ∂V k (t), ∂W k ∂t (t) = -L ∂ Rn ∂W k (t), t 0,
où Rn désigne un risque empirique. Notons en particulier que la variable temporelle t de l'EDO qui décrit l'évolution des poids n'est pas la même que la variable s de l'EDO neuronale (1.24) qui décrit la limite en large profondeur. Notre première contribution dans ce chapitre est de prouver que la convergence (lorsque L tend vers l'infini) du réseau résiduel vers une EDO neuronale est également valide après entraînement. Cette convergence est valide pour tout temps d'entraînement fini t ∈ [0, T ].

Néanmoins, la convergence de l'algorithme d'optimisation lorsque T tend vers l'infini n'est pas garantie sans hypothèse supplémentaire, du fait de la non-convexité du problème d'optimisation.

Nous prouvons cette convergence grâce à une condition de type Polyak-Łojasiewicz (PL), un outil majeur dans l'analyse des algorithmes d'optimisation pour les réseaux de neurones [START_REF] Liu | Loss landscapes and optimization in over-parameterized non-linear systems and neural networks[END_REF]. La condition PL implique la convergence du flot de gradient vers un minimum global. Notre seconde contribution dans ce chapitre est de prouver que cette condition est vérifiée lorsque la largeur q des couches cachées est plus grande qu'une constante fois la taille de l'échantillon n. Nous obtenons par conséquent la convergence en grande profondeur et en grand temps d'entraînement, c'est-à-dire l'existence de fonctions Lipschitz V ∞ et W ∞ telles que le réseau de neurones entraîné converge lorsque L et T tendent vers l'infini vers l'EDO Maintenant que l'on sait que certains réseaux de neurones résiduels entraînés convergent dans la limite en large profondeur vers des EDO neuronales de la forme (1.25), nous nous intéressons dans le Chapitre 4 à comprendre les propriétés statistiques de cette classe.

dH ds (s) = V ∞ (s)σ(W ∞ (s)H(s)), s ∈ [0, 1]. (1.25)
Nous examinons le cas simplifié où les matrices intérieures W ∞ sont égales à la matrice identité. Même avec cette simplification, le problème reste délicat comme la sortie du modèle est une fonction non-linéaire des paramètres et que ces derniers appartiennent à un espace de dimension infinie. Nous sommes tout de même en mesure de borner la complexité statistique du modèle en tirant parti de résultats sur les nombres de couverture des fonctions bornées Lipschitz.

Plus précisément, pour une fonction V à valeur matricielle, introduisons la norme matricielle V 1,1,∞ = max s∈[0,1] i,j |V ij (s)|. On s'intéresse alors aux EDO neuronales qui s'écrivent sous la forme

H(0) = x dH ds (s) = V (s)σ(H(s))ds, s ∈ [0, 1] F V (x) = H(1),
(1.26) où V appartient à l'ensemble de fonction

V = {V : [0, 1] → R q×q , V 1,1,∞ R V et V ij est K V -Lipschitz pour i, j ∈ {1, . . . , q}}.
(1.27) C'est une simplification du modèle limite obtenu au chapitre précédent. On se place dans le cadre classique de l'apprentissage supervisé avec un échantillon de n données i.i.d., que l'on suppose presque sûrement bornées. On prend une fonction de perte Lipschitz générale. Notre résultat principal est alors de montrer que l'erreur de généralisation est bornée par

O (q + 1) log(R V mn) n + q 2 √ K V n 1/4 + 1 √ n log 1 δ .
Les premier et dernier termes sont classiques (voir par exemple Bach, 2023, Sections 4.4 and 4.5).

Le second terme est plus surprenant à cause de son taux de convergence en O(n -1/4 ). Ce taux plus lent est dû au fait que l'espace de paramètre V est de dimension infinie.

Notre seconde contribution principale dans ce chapitre est de prouver une borne de généralisation similaire pour des réseaux résiduels de profondeur finie, en utilisant une analyse inspirée par le cas de la profondeur infinie.

Certains réseaux de neurones récurrents sont des méthodes à noyaux [Chapitre 5 du manuscrit]

Dans le Chapitre 5, nous nous intéressons à un modèle légèrement différent, les réseaux de neurones récurrents résiduels. Comme dans le chapitre précédent, nous introduisons un facteur d'échelle 1/L, et considérons donc le réseau défini par l'itération suivante

h k+1 = h k + 1 L σ(U h k + V x k+1 ), où (x k ) 1 k L est
(s) = σ(U H(s) + V X(s)), H(0) = 0, (1.28)
où maintenant la sortie du réseau s'écrit comme une projection linéaire de la valeur de la solution de l'EDO au temps t = 1, soit y = BH(1). Notre théorème principal dans ce chapitre s'écrit alors comme l'existence d'un espace de Hilbert H et de fonctions S et ς tels que, sous certaines hypothèses de régularité, y = S(X), ς(B, U, V ) H .

La fonction S est de plus une fonction connue dans la littérature, il s'agit de la signature de la série temporelle X [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF]. Ce résultat nous permet de réinterpréter l'action du réseau de neurones récurrent comme un produit scalaire dans un espace de Hilbert de dimension infinie, c'est-à-dire comme une méthode à noyau. Cela nous permet d'utiliser l'outillage habituel des méthodes à noyaux pour en déduire des bornes de généralisation et de stabilité.

1.4.5 Convergence des réseaux de neurones à une couche cachée dans la limite bi-échelle [Chapitre 6 du manuscrit]

Le chapitre 6 débute la seconde partie du manuscrit. Il s'intéresse à la dynamique d'apprentissage des réseaux de neurones à une couche cachée, dans un régime bi-échelle où les pas de gradients pour la couche intérieure sont négligeables devant ceux pour la couche extérieure. On considère un cas univarié où le réseau de neurones s'écrit

f v,b (x) = q k=1 v k σ(x -b k ), pour une entrée x ∈ R et des paramètres v k , b k ∈ R.
De façon analogue au chapitre 3, nous faisons l'hypothèse que le réseau de neurones est entraîné par flot de gradient, cette fois directement sur le risque théorique

R(v, b) = E(|f v,b (X) -f * (X)| 2 ),
où X suit une loi uniforme sur [0, 1]. Minimiser directement le risque théorique plutôt que le risque empirique nous permet de mettre de côté les problèmes statistiques pour se concentrer sur le problème d'optimisation, qui reste un problème délicat puisque le risque est non convexe en b.

Le flot de gradient s'écrit

∂v ∂t (t) = - ∂R ∂v (t), ∂b ∂t (t) = -ε ∂R ∂b (t), t 0, (1.29)
où ε paramétrise le ratio entre les pas sur la couche intérieure et ceux sur la couche extérieure. En prenant ε 1, le réseau de neurones peut être vu comme une régression linéaire ajustée, avec des facteurs σ(•b k ), k = 1, . . . , q qui évoluent lentement. Cette simplification nous permet de décrire précisément la dynamique des paramètres. Ainsi, dans ce régime bi-échelle, nous prouvons la convergence du flot de gradient dans le cas où f * est une fonction constante par morceaux, et pour un choix approprié de non-linéarité σ. En outre, le nombre minimal de neurones pour obtenir la convergence ne dépend pas de la précision souhaitée, mais seulement des propriétés de la fonction cible f * . Cela nous distingue des approches courantes pour l'analyse de la dynamique d'apprentissage des réseaux de neurones, le noyau tangent et le régime à champ moyen, qui requièrent toutes deux que le nombre de neurones grandisse afin d'obtenir une précision arbitraire.

1.4.6 Contexte structuré et grammaire à haut taux de couverture pour les questions-réponses conversationnelles basées sur un graphe de connaissance [Chapitre 7 du manuscrit] Nous présentons dans le Chapitre 7 des résultats dédiés à l'architecture Transformer, cette fois avec une approche plus algorithmique. Notre objectif est de construire un réseau de neurones de type Transformer pour la tâche de question-réponse conversationnelle basée sur un graphe de connaissance. Décrivons brièvement cette tâche : un graphe de connaissance est un graphe qui encode des informations factuelles, où les noeuds représentent des concepts et les arêtes (dirigées et labellisées) représentent des relations entre les concepts. L'objectif de la tâche est donc de répondre à une série de questions factuelles en cherchant la réponse dans le graphe. Une des difficultés est la taille du graphe (qui est de l'ordre du milliard d'arêtes voire davantage).

Nous nous attaquons à cette tâche en construisant un modèle d'analyse sémantique, c'est-àdire un modèle de langue qui transforme une phrase en anglais en une requête formelle formulée dans un langage de programmation, qui peut ensuite être exécutée pour obtenir une réponse. Le modèle fonctionne en deux étapes : dans un premier temps, un contexte de taille raisonnable est extrait du graphe et organisé dans une structure d'arbre, où chaque noeud de l'arbre contient un vecteur d'information. Dans un second temps, un réseau de neurones de type Transformer prend en entrée cet arbre et retourne la requête formelle.

Notre travail présente deux contributions méthodologiques principales : premièrement, nous construisons un langage de programmation (une "grammaire") de taille réduite, mais adapté à la modélisation d'une large gamme de questions sur des arbres de connaissance. Deuxièmement, nous introduisons une variante de Transformer qui peut opérer sur des données organisées sous forme d'arbre. Une des difficultés rencontrées est l'absence de données supervisées pour entraîner notre modèle. Nous procédons donc à la création d'un tel jeu de données, en partant d'ensembles de questions-réponses en anglais, et en trouvant par force brute des requêtes logiques qui correspondent aux questions. Nous pouvons ensuite entraîner notre modèle Transformer à l'aide de ces données.

Nous évaluons notre système sur deux jeux de données de questions-réponses basées sur le graphe Wikidata, qui est le plus grand graphe de connaissance public au monde. Notre approche obtient de meilleures performances en termes de précision des réponses par rapport à l'état de l'art sur les deux jeux de données.

Part I

From discrete to continuous architectures: neural networks in the large-depth regime [START_REF] Wang | Deepnet: Scaling transformers to 1,000 layers[END_REF]. From a high-level perspective, the key feature of ResNets is the presence of skip connections between successive layers. In mathematical terms, this means that the (k + 1)-th hidden state h k+1 ∈ R d follows sequentially from the previous hidden state via the recurrence relation

h k+1 = h k + f (h k , θ k+1 ), 0 k L -1, (2.1)
where f (•, θ k+1 ) : R d → R d is the layer function parameterized by θ k+1 ∈ R p and L is the number of layers. The skip connection corresponds to the addition of h k on the right-hand side of (2.1), which is absent in classical feedforward networks. This refinement prevents instability issues during training when L is large, provided training is performed carefully [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF]. The idea of adding skip connections has become common practice in the field of deep learning, and is today incorporated in many other models such as Transformers in natural language processing (Vaswani et al., 2017). For simplicity, in the rest of the chapter, we continue to use the terminology ResNets to denote any architecture of the form (2.1), keeping in mind that this framework goes beyond the original model of He et al. (2016a).

The most common architectures have 50-150 layers, but ResNets can be trained with depths up to the order of thousand layers (He et al., 2016b). Yet, the training procedure needs to be carefully crafted to avoid vanishing or exploding gradients, particularly as the depth increases. As pointed out by, e.g., Shao et al. (2020), these instabilities are related to a shift in the magnitude of the variance of a signal as it passes through the network. In the original approach of He et al. (2016a), the issue was mitigated by adding a normalization step, called batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], which rescales the output of each layer via centering and unit variance normalization. However, this normalization stage introduces practical and theoretical difficulties, among which computational overhead and strong dependence on the batch size (see Brock et al., 2021, and the references therein). A widespread alternative to stabilize training in deep models, explored for example by Yang and Schoenholz (2017), Arpit et al. (2019), Zhang et al. (2019b), and De and Smith (2020), is to incorporate a scaling factor α L in front of the residual term in (2.1), yielding the model

h k+1 = h k + α L f (h k , θ k+1 ), 0 k L -1.
(

2.2)

There is strong evidence that this scaling factor α L should depend on L, without however any consensus to date on the exact form of this dependence, nor on the mathematical grounding of the approach. Thus, despite progresses on the empirical side, the mathematical forces in action behind the stability of deep ResNets are still poorly understood, although they are key to unlock training at arbitrary depth.

Our goal in the present chapter is to take a step forward towards a better theoretical understanding of deep ResNets by providing a thorough probabilistic analysis of the sequence (h k ) 0 k L at initialization when L is large, and by leveraging a continuous-time interpretation of model (2.2) via the so-called neural ordinary differential equation (neural ODE, Chen et al., 2018a) paradigm. In a nutshell, our results highlight the intimate connection that exists at initialization between stability of the learning process, the regularity of the weights, and the scaling factor α L . We offer in particular a proper mathematical grounding on why and how to choose the parameter α L as a function of the depth L and the distribution of the weights.

Our contributions

Scaling at initialization. The optimal parameters of ResNets are learned by minimizing some empirical risk function via a gradient descent algorithm. As highlighted for example by Yang and Schoenholz (2017), Hanin andRolnick (2018), andArpit et al. (2019), a good parameter initialization of this learning phase plays a major role in the quality of the learned model, in particular to avoid vanishing gradients and deadlock at initialization, or exploding gradients and quick divergence of the model parameters at the beginning of training. Moreover, a good initialization allows the use of larger learning rates, which have been shown to correlate with better generalization [START_REF] Jastrzkebski | Three factors influencing minima in SGD[END_REF]. It is thus of great interest to study and understand the role played by scaling of deep ResNets at initialization. This is the context in which we place ourselves in the sequel.

At initialization stage, the weights (θ k ) 1 k L are usually chosen as (realizations of) independent and identically distributed (i.i.d.) random variables, which typically follow a uniform or Gaussian distribution on R p . Accordingly, the sequence (h k ) 0 k L that results from the recursion (2.2) for a given input to the network takes the form of a sequence of random variables that are not i.i.d. but are actually a martingale. Thus, denoting informally by L the differentiable loss associated with the learning task (classification or regression), the distributions of (h k ) 0 k L and ( ∂L ∂h k ) 0 k L as L becomes large carry useful information on the stability of training. For instance, exploding gradients in the backpropagation phase of learning correspond to the fact that, with high probability, ∂L ∂h 0 ∂L ∂h L , where • denotes the Euclidean norm. Our first contribution, in Section 2.2, is to provide thorough mathematical statements on the behavior of these distributions (both for finite and infinite L), depending on the value of α L . Among other results, we show that only the choice α L ≈ 1 / √ L yields a non-trivial behavior at initialization, thereby confirming empirical findings in the literature (Arpit et al., 2019;De and Smith, 2020). For α L 1 / √ L, the norms explode exponentially fast with L, which is inappropriate for training. For α L 1 / √ L, the network is almost equivalent to identity, that is, h L ≈ h 0 . The analysis of the different cases as a function of α L is mathematically involved and makes extensive use of concentration tools from random matrix theory.

The continuous approach. As noticed by several authors (Chen et al., 2018a;[START_REF] Thorpe | Deep limits of residual neural networks[END_REF]E et al., 2019), model (2.2) with a scaling factor α L = 1 /L (and not 1 / √ L) is formally similar to the discretization of a differential equation. Thus, when L tends to infinity, the weights and hidden states change continuously with the layer according to the equation

dH t dt = f (H t , Θ t ), t ∈ [0, 1]. (2.3)
Here, time t is the continuous analogue of the layer index k, H : [0, 1] → R d is a continuous-time hidden state, and Θ : [0, 1] → R p a continuous-time parameter. This important connection between ResNets and differential equations has been identified in the past years under the umbrella name of neural ODE. Since the original article of Chen et al. (2018a), this point of view has led to the development of a variety of new continuous-time models, together with innovative architectures and efficient training algorithms [START_REF] Chang | AntisymmetricRNN: A dynamical system view on recurrent neural networks[END_REF][START_REF] Grathwohl | Scalable reversible generative models with free-form continuous dynamics[END_REF][START_REF] Kidger | Signatory: Differentiable computations of the signature and logsignature transforms, on both CPU and GPU[END_REF]. The neural ODE paradigm also enabled to leverage the rich theory of differential equations to better understand the mechanisms at work behind deep ResNets (E et al., 2019). However, there is a debated question in the neural ODE community about the choice α L = 1 /L, which guarantees convergence of the discrete model (2.2) to its continuous-time counterpart (2.3). As a matter of fact, it seems that this choice is guided by more mathematical than practical considerations, and several authors have suggested that it is inconsistent with what is done in practice [START_REF] Cohen | Scaling properties of deep residual networks[END_REF][START_REF] Bayer | Stability of deep neural networks via discrete rough paths[END_REF]. Moreover, letting α L = 1 /L is somewhat contradictory with the results discussed above, which highlighted that the only non-trivial limit at initialization is α L = 1 / √ L. Thus, as a second contribution, we clarify the problem in Section 2.3 by leveraging our previous results on stability. We show that the value α L = 1 / √ L corresponds in the continuous world to a neural stochastic differential equation (SDE) of the form (2.3), where now Θ : [0, 1] → R p takes the form of a continuous-time stochastic process, typically a Brownian motion. By contrast, we also prove that the neural ODE regime with α L = 1 /L corresponds to the limit of a ResNet, not with i.i.d. weights as considered before, but with more complex and correlated weight distributions. For these weight distributions, the scaling α L = 1 /L is also a critical value between explosion and identity.

Going further, our third contribution is to exhibit in Section 2.4 a continuous range of regimes that are controlled by the choice of α L (beyond the cases 1 / √ L and 1 /L) and the distribution of (θ k ) 1 k L at initialization, derived from a continuous-time process Θ with a regularity different from a Brownian motion. More precisely, we show experimentally that there is a strong interplay (with the same three cases-explosion, identity mapping, non-trivial behavior) between the choice of α L and the regularity of (θ k ) 1 k L as a function of the layer index k, and this will be further investigated in Chapter 3. In addition, empirical evidence suggests that this interplay impacts both the behavior and performance of the networks during training, beyond initialization.

Organization of the chapter. The proofs of the results are postponed to the end of the chapter in Section 2.A. Sections 2.B to 2.D contain results that are useful for the proofs. Finally, Section 2.E details our experimental setting.

Related work

The choice of scaling for ResNets has been discussed in many papers, without however reaching a clear consensus on the form this scaling factor should take. For instance, Hanin and Rolnick (2018) state that stability requires α L 1 /L, while Zhang et al. (2019b) show that α L 1 / √ L is enough to ensure stability. On the other hand, [START_REF] Cohen | Scaling properties of deep residual networks[END_REF] claim that the scaling factor observed in practice in trained ResNets is of the form 1 /L β with β ≈ 0.7. Other authors have proposed more complex choices for α L (e.g., Zhang et al., 2019a;Shao et al., 2020). Taking another point of view, De and Smith (2020) observe that batch normalization is empirically equivalent to taking a 1 / √ L normalization factor. [START_REF] Bachlechner | ReZero is all you need: Fast convergence at large depth[END_REF] suggest learning a scaling parameter α k that is allowed to vary from one layer to another, whereas, in (2.4), α L is kept constant across layers. These authors observe a great acceleration for training compared to traditional ResNets with no scaling. They also suggest a similar architecture for Transformers and then notice that α k ≈ 1 /L at the end of training.

Closest to our analysis at initialization are the papers of Arpit et al. (2019) and Zhang et al. (2019b). Arpit et al. (2019) develop a theoretical analysis based on mean field approximation that suggests that a scaling factor α L = 1 / √ L prevents vanishing/exploding gradients at initialization, and provide experimental evidence that this approach is competitive with batch normalization. However, the authors do not provide rigorous mathematical statements for the three different cases α

L 1 / √ L, α L ≈ 1 / √ L, and α L 1 / √ L,
nor do they highlight the connection with the continuous-time interpretation. Interestingly, the idea of exploiting the martingale structure to analyze the magnitude of the hidden states is present in Zhang et al. (2019b), who study the convergence of gradient descent for over-parameterized ResNets with different values of α L . Nevertheless, they consider a specific model with Gaussian weights, and only provide asymptotic results when both width and depth tend to infinity.

The connection between the choice of scaling and the continuous-time point of view has previously been noticed by Zhang et al. (2019c), then studied in detail by [START_REF] Cohen | Scaling properties of deep residual networks[END_REF]. The latter show that, under assumptions on the form of the weights, it is possible to derive limiting (stochastic or ordinary) differential equations for the hidden states. However, they do not discuss the transition between these two regimes, nor do they link differential equations regimes with the stability of the network.

Scaling at initialization

Our goal in this section is to study the effect of the scaling factor α L on the stability of ResNets at initialization, assuming that the weights are i.i.d. random variables. We start by making more precise the model and the learning problem introduced in (2.1).

Model and assumptions

Model. The data is a sample of n pairs (x i , y i ) 1 i n , where x i is the input vector in R n in and y i ∈ R nout is the output vector to be predicted. This setting includes regression and classification (after one-hot encoding of the labels). Specifying the informal recurrence (2.1), for any input x ∈ R n in , we consider the output F π (x) ∈ R nout of the ResNet model defined by (2.4) where α L > 0 is the scaling factor of the ResNet and π = (A, B,

h 0 = Ax, h k+1 = h k + α L V k+1 g(h k , θ k+1 ), 0 k L -1, F π (x) = Bh L ,
(θ k ) 1 k L , (V k ) 1 k L ) are its parameters, with A ∈ R d×n in , B ∈ R nout×d , θ k ∈ R p and V k ∈ R d×d for k = 1, . . . , L.
The almost-everywhere differentiable function g : R d × R p → R d encodes the choice of architecture. We note that the model includes initial and final linear layers in order to map the input space R n in into the space of hidden states R d , and symmetrically to map the last hidden state h L into the output space R nout . These two transformations are of little interest to us, since we mostly focus on the behavior of the sequence of hidden states (h k ) 0 k L . Let us finally notice that the results of this section can be adapted to hidden layers that do not have the same width, at the cost of increased technicality.

An important feature of model (2.4) is that the layer function takes the form of a matrix-vector multiplication, which will prove crucial to make use of concentration results on random matrices. We stress that this setting is standard in practice and that it encompasses many types of ResNets. It includes for example simple ResNets where g(h, θ) = σ(h) with σ the activation function, and the original ResNets from He et al. (2016a), which have

g(h, θ) = ReLU(W h + b),
where the parameter is a pair θ = (W, b) with W ∈ R d×d a weight matrix and b ∈ R d a bias, and ReLU: x → max(x, 0) is applied element-wise. This setting also includes attention layers, where g corresponds to the scaled dot-product between keys and queries, as well as convolutional layers. Although the assumptions we make later have to be slightly modified to cover this context, the rationale should extend. We leave this extension for future work.

Throughout the chapter, we let : R nout × R nout → R + be a loss function, differentiable w.r.t. its first parameter, for example the squared loss or the cross-entropy loss. The objective of learning is to find the optimal parameter π that minimizes the empirical risk L (π) = n i=1 (F π (x i ), y i ).

Probabilistic setting at initialization. The minimization of the empirical risk is usually performed by stochastic gradient descent or one of its variants (Goodfellow et al., 2016, Chapter 8). The gradient descent is initialized by choosing the weights as (realizations of) i.i.d. random variables. The parameters θ 1 , V 1 , . . . , θ L , V L in model (2.4) are therefore assumed to be an i.i.d. collection of random variables, where we recall that θ k ∈ R p and V k ∈ R d×d parameterize the k-th layer of the network. In this stochastic context, the successive hidden states h 0 , . . . , h L given a fixed input x are also random variables, but their distribution is not i.i.d.-in fact, under our assumptions, this sequence is a martingale. To avoid unnecessary technicalities, we assume that the sequence (h k ) 0 k L is non-atomic. This is for example the case if the distribution of the parameters is absolutely continuous w.r.t. the Lebesgue measure. In particular, this ensures that the sequence (h k ) 0 k L almost surely does not hit the non-differentiability points of g.

It is stressed that the distribution of the parameters are assumed to be independent of the depth, so that all the dependence on L is captured in the scaling factor α L . This model enables us to consider multiple architectures at once, via the function g. By contrast, some authors formulate the problem of scaling as a choice of the variance at initialization (e.g., Yang and Schoenholz, 2017;[START_REF] Wang | Deepnet: Scaling transformers to 1,000 layers[END_REF], which makes the analysis architecture-dependent. However, for a given architecture, these two approaches are essentially equivalent since Var(α L V k ) = α 2 L Var(V k ). The quantity h Lh 0 / h 0 carries key information on the behavior of the network at initialization. On the one hand, if h Lh 0 h 0 , the network is essentially equal to the identity function. On the other hand, if h Lh 0 h 0 , the output of the network explodes. An intermediate situation is when h Lh 0 ≈ h 0 . In addition, another source of information is provided by the gradients of the hidden states with respect to the empirical risk L . If ∂L ∂h 0 -∂L ∂h L ∂L ∂h L , the gradients do not change as they flow through the network, which means that the exact same information is backpropagated throughout the network. Conversely, if ∂L ∂h 0 -∂L ∂h L ∂L ∂h L , the gradients explode during backpropagation. By exploiting the martingale structure of ( h k ) 0 k L , as well as state-of-the-art concentration inequalities for random matrices with sub-Gaussian entries, we provide in this section probabilistic bounds on the magnitude of these various quantities.

Assumptions. Some assumptions are needed on the choices of architecture and initialization. Recall that a centered real-valued random variable X is said to be s 2 sub-Gaussian (van Handel, 2016, Chapter 3) if for all λ ∈ R, E(exp(λX)) exp( λ 2 s 2 /2). The sub-Gaussian property is a constraint on the tail of the probability distribution. As an example, Gaussian random variables on the real line are sub-Gaussian and so are bounded random variables.

The following assumptions will be needed throughout the section: for any 1 k L, (A 1 ) For some s 1, the entries of √ dV k are centered i.i.d. s 2 sub-Gaussian random variables, independent of d and L, with unit variance.

(A 2 ) For some C > 0, independent of d and L, and for any h ∈ R d ,

h 2 2 E g(h, θ k ) 2 h 2 and E g(h, θ k ) 8 C h 8 .
Assumption (A 1 ) is mild and satisfied by all initializations used in practice. For example, the classical Glorot initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]-which is the default implementation in the Keras package [START_REF] Chollet | [END_REF]-takes the entries of V k as uniform U(-3/d, 3/d) variables. This means that √ dV k is initialized with U(-√ 3, √ 3) random variables, which satisfy (A 1 ). Other examples include the Gaussian N (0, 1/d) initialization of [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF] and, for example, initialization with Rademacher variables.

The first part of Assumption (A 2 ) ensures that g(•, θ k ) is not too far away from being an isometry in expectation. The second part is more technical and, roughly, allows to upper bound the deviations of the norm of g(h k-1 , θ k ). Our next Proposition 2.1 shows that most classical ResNet architectures verify Assumption (A 2 ). For the sake of readability, these models, together with their parameters, are summarized in Table 2.1 below.

Name

Recurrence relation Parameters a < b 1. In the last two cases, W k+1 ∈ R d×d . Proposition 2.1. Let res-1, res-2, and res-3 be the models defined in Table 2.1. Then (i) Assumption (A 2 ) is satisfied for res-1.

res-1 Simple ResNet h k+1 = h k + α L V k+1 σ(h k ) θ k+1 = ∅ res-2 Parametric ResNet h k+1 = h k + α L V k+1 σ(W k+1 h k ) θ k+1 = W k+1 res-3 Original ResNet h k+1 = h k + α L V k+1 ReLU(W k+1 h k ) θ k+1 = W k+1
(ii) Assumption (A 2 ) is satisfied for res-2 and res-3, as soon as the entries of √ dW k+1 , 0 k L -1, are centered i.i.d. sub-Gaussian random variables, independent of d and L, with unit variance.

Proof. See Section 2.A.1.

In the models res-1 and res-2, σ can be, for instance, taken as the parametric ReLU function, i.e., σ(x) = x + + sx -, where x + (resp. x -) denotes the positive (resp. negative) part and the slope s ∈ [ 1 / √ 2, 1] is a parameter of the model. Observe also that res-2 differs from res-3 since the classical ReLU function is defined by ReLU(x) = x + and thus does not satisfy the condition |σ(x)| a|x|. Note that there is no bias term in these three models, as this term is commonly initialized to zero, and we are interested in the behavior at initialization.

Probabilistic bounds on the norm of the hidden states

The next two propositions describe how the quantity h Lh 0 / h 0 changes as a function of Lα 2 L . Proposition 2.2 provides a high-probability bound of interest when Lα 2 L 1. In this case, we see that, with high probability, the network acts as the identity function, directly mapping h 0 to h L . On the other hand, Proposition 2.3 provides information in the two cases Lα

-δ, h L -h 0 2 h 0 2 > exp 3Lα 2 L 8 - 11Lα 2 L dδ -1, provided that 2L exp - d 64α 2 L s 2 δ 11 . (2.5) (ii) Assume that α 2 L 1 √ C(d+128s 4 )
. Then, for any δ ∈ (0, 1), with probability at least 1δ,

h L -h 0 2 h 0 2 < exp Lα 2 L + 5Lα 2 L dδ + 1.
Proof. See Section 2.A.3.

Note that the assumptions of Proposition 2.3 on d and α L are mild, since in the learning tasks where deep ResNets are involved, one typically has α L = 1 /L β with β > 0, d 10 2 and L 10 2 . Note also that condition (2.5) is not severe since, when d and L are large, it encompasses all reasonable values of δ. Propositions 2.2 and 2.3 are interesting in the sense that they provide finite-depth high-probability bounds on the behavior of the hidden states, depending on the magnitude of Lα 2 L . The results become clearer by letting α L = 1 /L β , with β > 0, as shown in the following corollary.

Corollary 2.4. Consider a ResNet (2.4) such that Assumptions (A 1 ) and (A 2 ) are satisfied, and let α L = 1 /L β , with β > 0.

(i) If β > 1 /2, then h L -h 0 h 0 P ----→ L→∞ 0. (ii) If β < 1 /2 and d 9, then h L -h 0 h 0 P ----→ L→∞ ∞. (iii) If β = 1 /2, d 64, L ( 1 2 √ Cs 4 + 2 √ C + 8s 4 )d + 96 √ Cs 4
, then, for any δ ∈ (0, 1), with probability at least 1δ, Proof. See Section 2.A.4.

exp 3 8 - 22 dδ -1 < h L -h 0 2 h 0 2 < exp 1 + 10 dδ + 1, provided that 2L exp - Ld 64s 2 δ 11 .
Corollary 2.4 highlights three different asymptotic behaviors for h L , depending on the values of β. For β > 1 /2, statement (i) tells that h L converges towards h 0 in probability, as L tends to infinity, which means that the neural network is essentially equivalent to an identity mapping. On the other hand, for β < 1 /2, the norm of h L explodes with high probability. Finally, for the critical value β = 1 /2, we see that h L fluctuates around h 0 , with a fluctuation size independent of L. Observe that the lower bound in (iii) is not trivial as soon as exp( 3 /8 -11 /dδ) > 1, i.e., d > 99 /64δ. The message of Corollary 2.4 is that the only scaling leading to a non-degenerate distribution at initialization is for β = 1 /2.

The three statements of Corollary 2.4 are illustrated in Figure 2.1. In this experiment, we consider model res-3, a random Gaussian observation x in dimension n in = 64, and parameters initialized with a uniform distribution U(-3/d, 3/d). We refer to Appendix 2.E for a detailed setup of all the experiments of the chapter. Figure 2.2a shows the empirical distribution of h L / h 0 when β = 1 /2 for a large number of realizations. This figure illustrates in particular that our bounds are reasonably sharp, since the bounds indicate that the first quartile of the distribution is larger than 0.87 (whereas the first quartile of the empirical histogram is equal to 1.21) and the third quartile is less than 2.06 (whereas the third quartile of the empirical histogram is equal to 1.34). Determining the exact distribution of h L / h 0 is an interesting avenue for future research that is beyond the scope of the present chapter. There is however a strong indication that the ratio follows a log-normal distribution, as confirmed by a normality test on (the log of) the empirical distribution.

In a nutshell, the proofs of Propositions 2.2 and 2.3 rest upon controlling of the norm of the hidden states, which obeys the recurrence

h k+1 2 = h k 2 + α 2 L V k+1 g(h k , θ k+1 ) 2 + 2α L h k , V k+1 g(h k , θ k+1 ) , (2.6)
where •, • denotes the standard scalar product in R d . Taking the expectations on both side, one deduces with Assumptions (A 1 ) and (A 2 ) that

E V k+1 g(h k , θ k+1 ) 2 = E E V k+1 g(h k , θ k+1 ) 2 h k , θ k+1 = E g(h k , θ k+1 ) 2 ≈ h k 2 (2.7)
and

E h k , V k+1 g(h k , θ k+1 ) = E E h k , V k+1 g(h k , θ k+1 ) h k , θ k+1 = 0. (2.8)
The equalities (2.7) and (2.8) allow deriving without further work bounds in expectation on h L , as already observed by Arpit et al. (2019). However, the results we are after are stronger since they involve high-probability bounds. A finer control of the deviations of V k+1 g(h k , θ k+1 ) 2 and h k , V k+1 g(h k , θ k+1 ) is then needed. This involves concentration inequalities on random matrices with sub-Gaussian entries.

Probabilistic bounds on the gradients

Propositions 2.2 and 2.3 provide insights on the output of the network when L is large. However, they do not carry information on the backwards dynamics of propagation of the gradients of the loss p k = ∂L ∂h k ∈ R d . Assessing the dynamics of the (p k ) 0 k L as a function of L is important since the behavior of this sequence impacts trainability of the network at initialization. Thus, in this subsection, we are interested in quantifying the magnitude of p 0p L / p L , when L is large. Notice that, contrarily to the previous subsection where we were mostly interested in the last hidden state h L , the quantity of interest is now p 0 (not p L ), the gradient at index 0. The reason is that the sequence (p k ) 0 k L is defined backwardly, as we will see below. We also stress that (p k ) 0 k L is the sequence of derivatives of the loss w.r.t. the hidden states h k , and not w.r.t. the parameters. The reason for considering this sequence is that the p k are involved in the backpropagation algorithm and are therefore essential for assessing the stability of the gradient descent (see, e.g., Arpit et al., 2019).

Analyzing the behavior of the sequence (p k ) 0 k L is challenging since, according to the backpropagation (or reverse-mode differentiation) formula, one has

p k = p k+1 + α L ∂g(h k , θ k+1 ) ∂h V k+1 p k+1 .
Taking the norm,

p k 2 = p k+1 2 + α 2 L ∂g(h k , θ k+1 ) ∂h V k+1 p k+1 2 + 2α L p k+1 , ∂g(h k , θ k+1 ) ∂h V k+1 p k+1 .
Although the equation looks qualitatively similar to (2.6), it has the unpleasant feature that ∂g(h k ,θ k+1 ) ∂h depends on h k , hence on θ 1 , V 1 , . . . , θ k , V k , while p k+1 depends on θ k+2 , V k+2 , . . . , θ L , V L . This forbids applying directly the same proof techniques as for the hidden states. Therefore, to extract useful information from this recurrence equation, one needs to characterize the dependence of the distribution of ∂g(h k ,θ k+1 ) ∂h with respect to h k . To do so, it is sometimes assumed that these two quantities are independent (see, e.g., Yang and Schoenholz, 2017). However, assuming independence remains a strong requirement, which is not verified for many network architectures (for example model res-1). We tackle the problem from a different point of view and propose an alternative approach based on forward-mode differentiation, valid under a much weaker assumption. The cost we pay is that we obtain results in expectation and not in high probability.

Let us sketch our approach before stating the results more formally. We denote by z ∈ R d an independent random variable that will be used as a tool to assess the magnitude of the gradients. For any 0 i, j L, let ∂h j ∂h i ∈ R d×d be the Jacobian matrix of h j with respect to h i . Recall that the (m, n)-th entry of this matrix equals the derivative of the m-th coordinate of h j w.r.t. the n-th coordinate of h i . Then, letting q k (z) = ∂h k ∂h 0 z, we have, by the chain rule,

q k+1 (z) = ∂h k+1 ∂h k q k (z) = q k (z) + α L V k+1 ∂g(h k , θ k+1 ) ∂h q k (z).
(2.9) Identity (2.9), which is similar to (2.4), expresses q k+1 (z) as a function of q k (z), and therefore respects the flow of information. Next, assuming that z is random with a Gaussian distribution, it is possible to express one of our quantities of interest, p 0 / p L , as a function of the last vector q L (z), by taking the expectation over z. Indeed,

p 0 2 p L 2 = 1 p L 2 E z∼N (0,I d ) p 0 z 2 = E z∼N (0,I d ) p L p L q L (z) 2 , (2.10)
where I d is the identity matrix in R d and the second equality is a consequence of

p 0 z = ∂L ∂h 0 z = ∂L ∂h L ∂h L ∂h 0 z = p L q L (z).
In summary, the recurrence (2.9) allows us to derive bounds on the norm of q L (z), which can then transfer to p 0 / p L via (2.10). For this, it is necessary to make the following assumption on the ratio p L / p L :

(A 3 ) Let b = p L / p L . Then E(b|h L ) = 0 and E(b b|h L ) = I d /d.
It is a mild assumption, which is verified for instance if n out = 1 with squared error (for regression) or cross-entropy (for binary classification). In these cases, p L / p L = B / B F , where

• F is the Frobenius norm and B is the weight matrix of the last layer. We finally need the following assumption, which is the equivalent of Assumption (A 2 ) for the gradients.

(A 4 ) One has, almost surely,

q k 2 2 E ∂g(h k , θ k+1 ) ∂h q k 2 h k , q k q k 2 .
Assumption (A 4 ) is satisfied by all the standard architectures listed in Table 2.1, as shown by the next proposition.

Proposition 2.5. Let res-1, res-2, and res-3 be the models defined in Table 2.1. Assume that (A 1 ) is satisfied and σ is almost everywhere differentiable, with a σ b. Then (i) Assumption (A 4 ) is satisfied for res-1.

(ii) Assumption (A 4 ) is satisfied for res-2 and res-3, when the entries of 1, then, for any δ ∈ (0, 1), with probability at least 1δ,

√ dW k , 1 k L,
p 0 -p L 2 p L 2 2Lα 2 L δ .
Proof. See Section 2.A.6.

Proposition 2.7. Consider a ResNet (2.4) such that Assumptions (A 1 )-(A 4 ) are satisfied. Then

1 + 1 2 α 2 L L -1 E p 0 -p L 2 p L 2 (1 + α 2 L ) L -1.
Proof. See Section 2.A.7.

A simple corollary of the propositions above is as follows.

Corollary 2.8. Consider a ResNet (2.4) such that Assumptions (A 1 )-(A 4 ) are satisfied, and take α L = 1 /L β , with β > 0. Then

(i) If β > 1 /2, p 0 -p L p L P ----→ L→∞ 0. (ii) If β < 1 /2, E p 0 -p L 2 p L 2 L→∞ ----→ ∞. (ii) If β = 1 /2, exp 1 2 -1 E p 0 -p L 2 p L 2 exp(4) -1.
Proof. See Section 2.A.8.

Corollary 2.8 is illustrated in Figure 2.3. The experimental protocol is the same as in Figure 2.1, but we now track p 0 and p L , the gradients of the loss L with respect to the first and the last hidden states. In accordance with our results, when β > 1 /2, the gradient remains the same from one layer to another (left plot). On the other hand, the middle plot clearly shows that when β < 1 /2 the gradient explodes. Once again, the case β = 1 /2 (right plot) is the only one for which the distribution of gradients at initialization is non-trivial. Figure 2.2b illustrates that the empirical distribution of gradients in this case also seems to be log-normal.

In summary, this and the previous subsection both point towards the same conclusion: there are three different cases, depending on the value of β-explosion when β < 1 /2, non-degenerate limit when β = 1 /2, and identity when β > 1 /2. In the explosion case, it is well known that the network cannot be trained (Yang and Schoenholz, 2017). The theory thus points out that the value 1 /2 plays a pivotal role. Remarkably, this value has a specific interpretation in the continuous-time point of view of ResNets, in terms of SDE. This is the topic that we address in the next section.

Scaling in the continuous-time setting

Starting with the discrete ResNet (2.4), it is tempting to let L go to infinity and consider the network as the discretization of a differential equation where the layer index k ∈ {0, . . . , L} is replaced by the time index t ∈ [0, 1]. This interpretation of deep neural networks has been popularized by Chen et al. (2018a) and is often referred to as the neural ODE paradigm. Notice that this setting is different from the so-called mean-field analysis, where the width of the network is assumed to be infinite beforehand. In our setting, the width d is assumed to be finite and fixed.

Convergence towards a SDE in the large-depth regime

One of the main messages of Section 2.2 is that the standard initialization with i.i.d. parameters leads to a non-degenerate model for large values of L only if Lα 2 L ≈ 1 (Propositions 2.2 and 2.3), or, equivalently, if β = 1 /2 when α L = 1 /L β (Corollary 2.4). Remarkably, in the continuous-time limit, this regime corresponds to the discretization of a SDE. Indeed, consider for simplicity the (discrete) ResNet model res-1 (2.11) where the entries of all (V k ) 1 k L are assumed to be i.i.d. N (0, 1 /d). Recall the following definition:

h 0 = Ax, h k+1 = h k + 1 √ L V k+1 σ(h k ), 0 k L -1,
Definition 2.9. A one-dimensional Brownian motion (B t ) t∈[0,1] is a continuous-time stochastic process with B 0 = 0, almost surely continuous, with independent increments, and such that for any 0 s < t 1, B t -B s ∼ N (0, ts).

Now, let B : [0, 1] → R d×d be a (d × d)-dimensional Brownian motion, in the sense that the (B ij ) 1 i,j d are independent one-dimensional Brownian motions. Thus, for any 0 k L -1 and any 1 i, j d, we have

B(k+1) /L,i,j -Bk /L,i,j ∼ N 0, 1 L ,
and the increments for different values of (i, j, k) are independent. As a consequence, the recurrence (2.11) is equivalent in distribution to the recurrence

h k+1 = h k + 1 d σ(h k )(B(k+1) /L -Bk /L ), 0 k L -1.
(Note that this is true because V k+1 has the same distribution as V k+1 .) We recognize the Euler-Maruyama discretization [START_REF] Kloeden | Numerical Solution of Stochastic Differential Equations[END_REF] on the {k/L, 0 k L} mesh of the SDE

H 0 = Ax, dH t = 1 d σ(H t )dB t , t ∈ [0, 1], ( 2
E Hk /L -h k c √ L ,
for some c > 0.

Proof. See Section 2.A.9.

Notice that the requirement that σ is Lipschitz continuous is satisfied by most classical activation functions, including ReLU. This proposition is interesting for several reasons. First, the scaling β = 1 /2, which is exactly the one that yields a non-trivial dynamics at initialization, corresponds in the continuous world to a remarkably 'simple' model of diffusion. This shows that very deep neural networks properly initialized with i.i.d. weights are equivalent to solutions of SDE. This analogy opens interesting perspectives for training deep networks using automatic differentiation for solutions of neural SDE (Li et al., 2020b).

Second, we stress that the emergence of a SDE instead of an ODE carries an important message. Several authors (including, e.g., [START_REF] Thorpe | Deep limits of residual neural networks[END_REF] have shown that, under appropriate assumptions, a deep ResNet converges in the large depth limit to an ODE and not a SDE. The reason why we obtain a SDE here is intrinsically connected with the choice of i.i.d. initialization for the weights, which makes a Brownian motion appear at the limit, as highlighted above. In other words, the i.i.d. initialization, the choice β = 1 /2 (the relevant critical value exhibited in Section 2.2), and the emergence of a SDE are intimately linked together. On the other hand, the case β = 1 matches with an ODE if the initialization is not i.i.d., as we will see in Subsection 2.3.2.

Finally, we point out that Proposition 2.10 states the convergence of a ResNet towards a SDE for the basic architecture res-1 and for Gaussian initialization. The extension to more general settings is an interesting direction of research, although clearly beyond the scope of the present chapter (see, e.g., [START_REF] Peluchetti | Infinitely deep neural networks as diffusion processes[END_REF][START_REF] Peluchetti | Infinitely deep neural networks as diffusion processes[END_REF][START_REF] Cohen | Scaling properties of deep residual networks[END_REF], for results in this direction).

Scaling in the neural ODE setting

Convergence towards an ODE. The basic message of our Proposition 2.10 is that an i.i.d. initialization, together with β = 1 /2, leads to a SDE rather than an ODE. A natural question is then whether a different choice of weight distributions (at initialization) and scaling can lead to a classical neural ODE.

To answer this question and leave the world of i.i.d. initialization, we assume that the weights (V k ) 1 k L and (θ k ) 1 k L are discretizations of smooth functions V : [0, 1] → R d×d and Θ : [0, 1] → R p . We then consider the general iteration (2.4) with α L = 1 /L, that is, (2.13) where

h 0 = Ax, h k+1 = h k + 1 L V k+1 g(h k , θ k+1 ), 0 k L -1,
V k = V k/L and θ k = Θ k/L . Of course, it is still possible to consider (V k ) 1 k L (resp. (θ k ) 1 k L )
as random variables, by letting (V t ) t∈[0,1] (resp. (Θ t ) t∈[0,1] ) be a continuous-time stochastic process. In this model, we shall need the following assumption:

(A 5 ) For any 1 k L, one has V k = V k/L and θ k = Θ k/L
, where the stochastic processes V and Θ are almost surely Lipschitz continuous and bounded.

More precisely, almost surely, there exist

K V , K Θ , C V , C Θ > 0, such that, for any s, t ∈ [0, 1], V t -V s K V |t -s|, Θ t -Θ s K Θ |t -s|, V t C V , Θ t C Θ .
A typical model that satisfies Assumption (A 5 ) is obtained by letting the entries of V and Θ be independent Gaussian processes with expectation zero and squared exponential covariance

K(x, x ) = exp(-(x-x ) 2 2 2 )
, where > 0. We shall also need the following requirement on g, which is satisfied by all our models as soon as σ is Lipschitz continuous:

(A 6 )
The function g is Lipschitz continuous on compact sets, in the sense that for any compact P ⊆ R p , there exists

K P > 0 such that, for all h, h ∈ R d , θ ∈ P, g(h, θ) -g(h , θ) K P h -h ,
and for any compact D ⊆ R d , there exists K D,P > 0 such that, for all h ∈ D, θ, θ ∈ P, g(h, θ)g(h, θ ) K D,P θθ .

Under Assumptions (A 5 ) and (A 6 ), the recurrence (2.13) almost surely converges towards the neural ODE given by

H 0 = Ax, dH t = V t g(H t , Θ t )dt, t ∈ [0, 1],
( 2.14) as shown by the proposition below. Proof. See Section 2.A.10.

It should be stressed that the transition from the discrete recurrence (2.13) to the continuoustime differential equation (2.14) relies on the assumptions that the weight sequences (θ k ) 1 k L and (V k ) 1 k L are the discretizations of smooth limiting processes Θ and V on the one hand, and that the scaling α L is chosen as 1 /L on the other hand. From a practical perspective, Proposition 2.11 shows that it is possible to initialize ResNets in the ODE regime, by choosing a smooth stochastic process, discretizing it at each layer, and taking a 1 /L scaling. This is in sharp contrast with the results of Sections 2.2 and 2.3.1, which show that the usual i.i.d. procedure leads to a neural SDE. Stability and scaling. Assuming that the weights of the network are discretizations of a smooth function (Assumption (A 5 )), it is possible to obtain stability results, depending on the value of β, similarly to what has been done in Section 2.2. We show below that β = 1 is a critical value, by examining the hidden states, in the same way as β = 1 /2 is a critical value in the i.i.d. setting. Similar results can be shown for the gradients. We begin by a proposition handling the cases β > 1 and β = 1.

Proposition 2.12. Consider a ResNet (2.4) such that Assumptions (A 5 ) and (A 6 ) are satisfied.

Let α L = 1 /L β , with β > 0. (i) If β > 1, then, almost surely, h L -h 0 h 0 L→∞ ----→ 0.
(ii) If β = 1, then, almost surely, there exists some c > 0 such that

h L -h 0 h 0 c.
Proof. See Section 2.A.11.

The explosion case (β < 1) is more delicate to deal with. We prove it for a linear model, and leave for future work the extension to more general cases. Proposition 2.13. Consider the res-1 model, taking σ as the identity function. Assume that Assumption (A 5 ) is satisfied and that V T 0 has a positive eigenvalue. Let α L = 1 /L β , with β ∈ (0, 1). Then, almost surely,

max k h k -h 0 h 0 L→∞ ----→ ∞.
Proof. See Section 2.A.12.

The assumption of the existence of a positive eigenvalue for V 0 is mild. For instance, if the entries of V 0 are i.i.d. random variables with finite moments of all order, [START_REF] Götze | Rate of convergence to the circular law via smoothing inequalities for log-potentials[END_REF] show that such an eigenvalue exists with probability at least 1 -1 /d for d large enough.

In this setting, we observe experimentally a behavior of the output and of the gradients when L grows large similar to the one explored in Section 2.2. This is illustrated in Figures 2.4 which mirror Figures 2.1 and 2.3 in Section 2.2. The figures clearly show that there exist three cases for the output and for the gradients: an identity case (left plots), an explosion case (middle), and a non-trivial case separating explosion and identity (right). However, the remarkable point is that the separation occurs for β = 1, and not β = 1 /2, as predicted by Propositions 2.12 and 2.13.

Experiments

We experimentally investigate in this section two questions. The first one is to know whether there exists a range of scaling factors β > 0 and weight initializations, beyond the i.i.d. and the smooth regimes. The second question is whether our analysis, which pertains to the initialization phase, provides insights into the training phase, beyond initialization.

Intermediate regimes

In order to describe the transition between the i.i.d. and smooth cases, a possible route is to consider that the weights are increments of a γ-Hölder stochastic process. This model is interesting insofar as the Brownian motion (SDE regime) is ( 1 /2ε)-Hölder (ε > 0) and a Lipschitz continuous stochastic process (ODE regime) is 1-Hölder. In line with the above, in a series of experiments, we initialize the weights as increments of a fractional Brownian motion (B H t ) t∈[0,1] . Recall that B H is a continuous-time Gaussian process, starting at zero, with zero expectation for all t ∈ [0, 1], and covariance function

E(B H s B H t ) = 1 2 (|s| 2H + |t| 2H -|t -s| 2H ), 0 s, t 1,
where H ∈ (0, 1) is called the Hurst index. This index describes the raggedness of the process, with a higher value leading to a smoother process. When H = 1 /2, the process is a standard Brownian motion (Definition 2.9), whose increments are independent by construction. When H > 1 /2, the increments of the process are positively correlated, while if H < 1 /2 the increments are negatively correlated. Importantly, a fractional Brownian motion with Hurst index H is (Hε)-Hölder continuous for any ε > 0. In the limit when H → 1, the trajectories converge to linear functions (whose increments satisfy (A 5 )). As an illustration, Figure 2.6 depicts three realizations of a fractional Brownian motion with H = 0. The orange zone corresponds to the explosion regime, i.e., h Lh 0 h 0 and p 0p L p L . The blue zone corresponds to the identity regime, i.e., h Lh 0 h 0 and p 0p L p L . Finally, the black zone is an intermediate regime, where h Lh 0 ≈ h 0 and p 0p L ≈ p L .

In order to assess the effect of the scaling factor β and the Hurst index H, we initialize a neural network res-3 with d = 40, L = 1000, various values of β ∈ [0.2, 1.3], and with weights taken as increments of fractional Brownian motions with various Hurst indices H ∈ (0, 1). Figure 2.7 depicts the empirical magnitude of the output and the gradients at initialization as a function of the Hurst index H and the scaling factor β. First note that we recover the two regimes (i.i.d. and smooth) discussed so far. For H = 1 /2, the i.i.d. regime kicks in, with explosion (β < 1 /2, orange zone), non-trivial behavior (β = 1 /2, black zone), and identity (β > 1 /2, blue zone). Likewise, we see at H = 1 a similar pattern in the smooth regime, with, as predicted by Proposition 2.12, a critical value β = 1. Beyond these two specific cases, we observe for an index H varying in ( 1 /2, 1) a whole range of intermediate situations, where the transition between identity and explosion seems to happen for a critical β = H. Interestingly, for H < 1 /2, the transition seems to saturate at the value β = 1 /2.

The take-home message is that the choice of the scaling of a ResNet seems to be closely linked to the regularity of the weights as a function of the layer. More precisely, for all regimes, the critical scaling factor between explosion and identity seems to have a natural interpretation as the (Hölder) regularity of the underlying continuous-time stochastic process. We believe that the mathematical understanding of this connection, beyond the fractional Brownian motion case, is a promising research direction for the future. Finally, these experiments suggest that it is sensible to initialize a ResNet for any value of the scaling β ∈ ( 1 /2, 1), while avoiding the identity and explosion situations, by simulating a fractional Brownian motion of Hurst index H = β and initializing the weights as the increments of this process.

Beyond initialization

At initialization, before the gradient descent, the distribution of the weights (θ k ) 1 k L and (V k ) 1 k L is chosen by the practitioner. By contrast, during and after training, control is lost on these distributions, making the picture more complex. In particular, the existence and characterization of a continuous-time stochastic process whose discretization matches the trained ResNet is an interesting but difficult problem. Attacking this question requires a fine understanding of the interaction between training dynamics and the regularity of the sequence of the weights during the gradient descent. However, there is experimental evidence that the trained weights exhibit strong structure as a function of the layer index k [START_REF] Cohen | Scaling properties of deep residual networks[END_REF][START_REF] Bayer | Stability of deep neural networks via discrete rough paths[END_REF], and that their regularity strongly depends on the choice of initialization. Figure 2.8 depicts this mechanism by plotting a given coordinate of θ k as a function of the layer index k ranging from 1 to the depth L = 1000, after training. This will be further investigated in Chapter 3. To investigate the link between regularity of the weights at initialization, scaling, and performance after training, we train ResNets on the datasets MNIST [START_REF] Deng | The MNIST database of handwritten digit images for machine learning research[END_REF] and CIFAR-10 (Krizhevsky, 2009). As in Subsection 2.4.1, we initialize the ResNets with various scaling factors and weights that are increments of fractional Brownian motions with different regularities. Then, for each combination of weight initialization and scaling factor, the ResNet is trained using the Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] for 10 epochs. The results in terms of accuracy are presented in Figure 2.9 (light orange = good performance, blue = bad performance). We observe a pattern similar to the one of Figure 2.7, however shifted downwards. This means that, for a given regularity, the network is unable to learn if it is initialized with a scaling too far below the critical value, which of course is connected with the gradient explosion issue discussed previously. On the other hand, and perhaps more surprisingly, the performance seems to be more or less stable in the identity region, with perhaps a small degradation in the case of CIFAR-10. This somewhat contrasts with the results from Yang and Schoenholz (2017), who exhibit a decrease in performance for i.i.d. initialization and a large scaling factor β. Note however that, in the experiments reported in Figure 2.7, we adapt the learning rate of the gradient descent on a grid by cross-validation. When taking a fixed learning rate, we also observe a decrease in performance for large scaling factor β. The interplay between the learning rate and the scaling factor is one of the keys to better assess how the performance of the trained network is connected with the scaling.

2.A Proofs

Throughout the proofs, the i-th coordinate of a vector v is denoted by v i . Similarly, the i-th row of a matrix M is denoted by M i , and its (i, j)-th entry by M ij .

2.A.1 Proof of Proposition 2.1

Statement (i) is clear (with C = 1) since, for any h ∈ R d , σ(h) 2 ∈ a 2 h 2 , b 2 h 2 ⊆ 1 2 h 2 , h 2 .
With respect to statement (ii), it is enough to show that for any h ∈ R d and any random matrix W satisfying the assumptions of the proposition, one has

h 2 2 E σ(W h) 2 h 2 and E σ(W h) 8 C h 8 ,
as well as

E ReLU(W h) 2 = h 2 2 and E ReLU(W h) 8 C h 8 .
The two claims with the squared norms are consequences of Lemmas 2.16 and 2.17 in Appendix 2.B, together with the fact that the variance of the entries of W equals 1/d. In order to prove the other two statements, first note that

E( σ(W h) 8 ) E( W h 8 ) and E( ReLU(W h) 8 ) E( W h 8 ).
The results are then consequences of Lemma 2.21 in Appendix 2.C, which states that

E W h 8 h 8 1 + 384s 4 d + 3072s 6 d 2 1 + 384s 4 + 3072s 6 .

2.A.2 Proof of Proposition 2.2

According to Lemma 2.14 below, one has

E h L -h 0 2 h 0 2 1 + α 2 L L -1 . But, for Lα 2 L 1, we have (1 + α 2 L ) L -1 exp(Lα 2 L ) -1 2Lα 2 L . Therefore, E h L -h 0 2 h 0 2 2Lα 2 L ,
and the result follows from Markov's inequality.

Lemma 2.14. Consider a ResNet (2.4) such that Assumptions (A 1 ) and (A 2 ) are satisfied. Then

1 + α 2 L 2 L -1 E h L -h 0 2 h 0 2 1 + α 2 L L -1 .
Proof (Lemma 2.14) Taking the squared norm of the forward update rule (2.4) and dividing by h 0 2 yields

h k+1 2 h 0 2 = 1 h 0 2 h k 2 + α 2 L V k+1 g(h k , θ k+1 ) 2 + 2α L h k , V k+1 g(h k , θ k+1 ) . (2.15)
We deduce by Assumptions (A 1 ) and

(A 2 ) that 1 + α 2 L 2 E h k 2 h 0 2 E h k+1 2 h 0 2 (1 + α 2 L )E h k 2 h 0 2 .
Therefore, by recurrence, we are led to

1 + α 2 L 2 k E h k 2 h 0 2 (1 + α 2 L ) k . (2.16) Now, observe that h L = h 0 + α L L-1 k=0 V k+1 g(h k , θ k+1 )
. Thus, we have

E h L -h 0 2 h 0 2 = α 2 L L-1 k,k =0 E g(h k , θ k+1 ) V k+1 V k +1 g(h k , θ k +1 ) h 0 2 .
By conditioning on all random variables except V k +1 for k < k (and V k+1 for k > k ), it is easy to see that the only non-zero terms are when k = k . This yields

E h L -h 0 2 h 0 2 = α 2 L L-1 k=0 E V k+1 g(h k , θ k+1 ) 2 h 0 2 α 2 L L-1 k=0 E h k 2 h 0 2
(by Assumptions A 1 and A 2 )

α 2 L L-1 k=0 (1 + α 2 L ) k (by (2.16)) = 1 + α 2 L L -1 .
Similarly,

E h L -h 0 2 h 0 2 α 2 L 2 L-1 k=0 E h k 2 h 0 2 = 1 + α 2 L 2 L -1 .

2.A.3 Proof of Proposition 2.3

Dividing (2.15) by h k 2 and taking the logarithm leads to

ln( h k+1 2 ) = ln( h k 2 ) + ln 1 + α 2 L V k+1 g(h k , θ k+1 ) 2 h k 2 + 2α L h k h k , V k+1 g(h k , θ k+1 ) h k . Let Y k,1 = α 2 L V k+1 g(h k , θ k+1 ) 2 h k 2 , Y k,2 = 2α L h k h k , V k+1 g(h k , θ k+1 ) h k ,
and

Y k = Y k,1 + Y k,2 .
The proof of Proposition 2.3 strongly relies on the following lemma, which provides technical information on the moments of Y k,1 and Y k,2 . For the sake of clarity, its proof is postponed to Appendix 2.B.

Lemma 2.15. Assume that Assumptions (A 1 ) and (A 2 ) are satisfied.

Then

(E 1 ) E(Y k,2 |h k ) = 0. (E 2 ) α 2 L 2 E(Y k,1 |h k ) α 2 L . (E 3 ) E(Y k,1 Y k,2 |h k ) = 0. (E 4 ) E(Y 2 k,2 |h k ) 4 α 2 L d . (E 5 ) E(Y 4 k,2 |h k ) 2048 s 4 α 4 L d 2 . (E 6 ) E(Y 4 k,1 |h k ) C 3072 s 6 d 2 + 384 s 4 d + 1 α 8 L . (E 7 ) E(Y 2 k,1 |h k ) √ C 128 s 4 d + 1 α 4 L .
For c > 0, we have

P h L 2 h 0 2 c = P ln( h L 2 ) -ln( h 0 2 ) ln(c) = P L-1 k=0 ln(1 + Y k ) ln(c) P L-1 k=0 Y k ln(c) (using ln(1 + x) x for x > -1). Let S = L-1 k=0 Y k -E(Y k |h k ). By (E 1 ) and (E 2 ), L-1 k=0 E(Y k |h k ) Lα 2 L . So, for c > exp(Lα 2 L ), P h L 2 h 0 2 c P S ln(c) - L-1 k=0 E(Y k |h k ) P(S ln(c) -Lα 2 L ) P S 2 ln(c) -Lα 2 L 2 E(S 2 ) (ln(c) -Lα 2 L ) 2 (2.17)
(by Markov's inequality.)

It remains to upper bound E(S 2 ). To this aim, note that

E(S 2 ) = L-1 k=0 E Y k -E(Y k |h k ) 2 L-1 k=0 E Y 2 k 4 Lα 2 L d + 128 √ C Lα 4 L s 4 d + √ CLα 4 L (by (E 3 ), (E 4 ), and (E 7 )) 5 Lα 2 L d .
The last inequality is true for α 2 L 1 √ C(d+128s 4 ) . Therefore, by inequality (2.17), we obtain, for c > exp(Lα 2 L ),

P h L 2 h 0 2 c 5Lα 2 L d ln(c) -Lα 2 L 2 .
We conclude that, for any δ ∈ (0, 1), with probability at least 1δ,

h L 2 h 0 2 < exp Lα 2 L + 5Lα 2 L dδ .
This shows statement (ii) of the proposition.

Next, to prove statement (i), observe that c > 0,

P h L 2 h 0 2 c = P ln( h L 2 ) -ln( h 0 2 ) ln(c) = P L-1 k=0 ln(1 + Y k ) ln(c) = P L-1 k=0 ln(1 + Y k ) ln(c) and ∀k, Y k - 1 2 + P L-1 k=0 ln(1 + Y k ) ln(c) and ∃k, Y k < - 1 2 .
Using the inequality ln(1 + x) xx 2 for x -1 /2, we obtain

P h L 2 h 0 2 c P L-1 k=0 Y k -Y 2 k ln(c) and ∀k, Y k - 1 2 + P L-1 k=0 ln(1 + Y k ) ln(c) and ∃k, Y k < - 1 2 .
Thus,

P h L 2 h 0 2 c P L-1 k=0 Y k -Y 2 k ln(c) + L-1 k=0 P Y k,2 < - 1 2 . (2.18)
We handle the two terms above on the right-hand side separately. For the first term, let

Z k = Y k -Y 2 k and S = L-1 k=0 Z k -E(Z k |h k ). Observe that, by (E 1 )-(E 4 ) and (E 7 ), L-1 k=0 E(Z k |h k ) Lα 2 L 2 -4 Lα 2 L d -128 √ C Lα 4 L s 4 d - √ CLα 4 L 3 8 Lα 2 L , (2.19)
where the last inequality is valid for d 64 and α 2

L 1 16 √ C(2s 4 +1) . Hence, for 0 < c < exp( 3Lα 2 L/8), P L-1 k=0 Y k -Y 2 k ln c = P S ln(c) - L-1 k=0 E(Z k |h k ) P S ln(c) - 3Lα 2 L 8 P S 2 ln(c) - 3Lα 2 L 8 2 E(S 2 ) ln(c) - 3Lα 2 L 8 2 (by Markov's inequality.) Using the c r -inequality (a + b) n 2 n-1 (a n + b n ) respectively for n = 2 and n = 4, we see that E(S 2 ) = L-1 k=0 E Z k -E(Z k |h k ) 2 L-1 k=0 E Z 2 k 2 L-1 k=0 E Y 2 k + E Y 4 k 2 L-1 k=0 E(Y 2 k,1 ) + E(Y 2 k,2 ) + 2E(Y k,1 Y k,2 ) + 8E(Y 4 k,1 ) + 8E(Y 4 k,2 ).
By (E 3 )-(E 7 ), it is easy to verify that, for d 64 and

α 2 L 1 ( √ Cs 4 /16+2 √ C+8s 4 )d , E(S 2 ) 10 Lα 2 L d .
This shows that, for c < exp( 3Lα 2 L/8),

P L-1 k=0 Y k -Y 2 k ln(c) 10Lα 2 L d ln(c) - 3Lα 2 L 8 2 .
To conclude the proof, it remains to upper bound the second term of inequality (2.18). According to inequality (2.22) in the proof of Lemma 2.15 (with t = 1 /2), one has

L-1 k=0 P Y k,2 < - 1 2 2L exp - d 64α 2 L s 2 .
Putting everything together, we are led to

P h L 2 h 0 2 c 10Lα 2 L d ln(c) - 3Lα 2 L 8 2 + 2L exp - d 64α 2 L s 2 . Take δ ∈ (0, 1). Then, if 2L exp -d 64α 2 L s 2 δ 11 , with probability at least 1 -δ, h L 2 h 0 2 > exp 3Lα 2 L 8 - 11Lα 2 L dδ .
Notice that this inequality is valid under the assumption

α 2 L 2 ( √ Cs 4 +4 √ C+16s 4 )d .

2.A.4 Proof of Corollary 2.4

Statement (i) is a consequence of Proposition 2.2, whereas (ii) is a consequence of Proposition 2.3 (i). The latter is valid under the conditions d 64 and α L 2 ( √ Cs 4 +4 √ C+16s 4 )d , which is automatically satisfied for all L large enough. Furthermore, an inspection of the proof of Proposition 2.3 reveals that the divergence in high probability of h L can be proved under the relaxed assumption d 9. Indeed, the main constraint on d comes from the lower bound (2.19), where one needs to make sure that

Lα 2 L 2 -4 Lα 2 L d > 0, which is the case for d = 9.
To prove (iii), we use a union bound on both statements of Proposition 2.3.

2.A.5 Proof of Proposition 2.5

The first claim follows from the observation that

∂g(h k , θ k+1 ) ∂h q k =      σ (h k,1 ) 0 . . . 0 0 σ (h k,2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . σ (h k,d )      q k , from (A 1 )
, and from the assumption on σ .

Let us now prove (ii). In the rest of the proof, the subscript k is ignored to lighten the notation. Observe that

∂g(h, θ) ∂h q = V      σ ( W 1 , h ) 0 . . . 0 0 σ ( W 2 , h ) . . . 0 . . . . . . . . . . . . 0 0 . . . σ ( W d , h )      W q.
Denote by D the matrix in the middle of the right-hand side. Then

E ∂g(h, θ) ∂h q 2 h, q = E V DW q 2 |h, q = E DW q 2 |h, q (by (A 1 ))
For model res-2, we have

E ∂g(h, θ) ∂h q 2 h, q = E d i=1 d j=1 W ij q j 2 σ ( W i , h ) h, q .
The conclusion follows from the hypothesis that a σ b and E W q 2 |q = q 2 . For model res-3, we have

E ∂g(h, θ) ∂h q 2 h, q = E d i=1 d j=1 W ij q j 2 1 d j=1 W ij h j 0 h, q .
Since the (W ij ) 1 i,j d are centered random variables, we conclude that

E ∂g(h, θ) ∂h q 2 h, q = 1 2 E d i=1 d j=1 W ij q j 2 q = 1 2 E W q 2 |q = q 2 2 .

2.A.6 Proof of Proposition 2.6

Letting b = p L / p L , as in Assumption (A 3 ), and taking expectation in (2.10), we obtain

E p 0 2 p L 2 = E(|b q L (z)| 2 ) = 1 d E( q L (z) 2 ) (2.20) (by (A 3 )).
The rest of the proof is similar to the proof of Proposition 2.2. From (2.9), we have

q k+1 (z) 2 = q k (z) 2 + α 2 L V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) 2 + 2α L q k (z), V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) .
By independence of V k+1 from q k (z) and

∂g(h k ,θ k+1 ) ∂h , E q k (z), V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) = 0.
Next,

E V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) 2 = E E V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) 2 h k , θ k+1 , q k (z) = E ∂g(h k , θ k+1 ) ∂h q k (z) 2 (by (A 1 )) = E E ∂g(h k , θ k+1 ) ∂h q k (z) 2 h k , q k (z) .
By Assumption (A 3 ), we are led to

1 + 1 2 α 2 L E( q k (z) 2 ) E( q k+1 (z) 2 ) (1 + α 2 L )E( q k (z) 2 ),
and thus, by induction, since q 0 (z) = z and E( z 2 ) = d,

d 1 + 1 2 α 2 L k E( q k (z) 2 ) d(1 + 4α 2 L ) k .
In particular, for k = L,

d 1 + 1 2 α 2 L L E( q L (z) 2 ) d(1 + α 2 L ) L .
Therefore, by (2.20),

1 + 1 2 α 2 L L E p 0 2 p L 2 (1 + α 2 L ) L .
To finish the proof, observe that

1 p L (p 0 -p L ) z = b (q L (z) -z).
Using arguments similar to (2.20), we may write

E p 0 -p L 2 p L 2 = 1 d E q L (z) -z 2 z 2 . Now, upon noting that q L (z) -z = q L (z) -q 0 (z) = α L L-1 k=0 V k+1 ∂g(h k ,θ k+1 ) ∂h q k (z), E( q L (z) -z 2 ) = α 2 L L-1 k,k =0 E q k (z) ∂g(h k , θ k+1 ) ∂h V k+1 V k +1 ∂g(h k , θ k +1 ) ∂h q k (z) = α 2 L L-1 k=0 E V k+1 ∂g(h k , θ k+1 ) ∂h q k (z) 2 dα 2 L L-1 k=0 (1 + α 2 L ) k = d (1 + α 2 L ) L -1 d exp(Lα 2 L ) -1 2dLα 2 L ,
for Lα 2 L 1. Note that the second equality is obtained by conditioning on every random variable except V k +1 for k < k (and V k+1 for k > k ). Finally, by using Markov's inequality, we conclude that, for any ε > 0,

P p 0 -p L 2 ε p L 2 2Lα 2 L ε .

2.A.7 Proof of Proposition 2.7

The proof of Proposition 2.6 reveals that

E p 0 -p L 2 p L 2 (1 + α 2 L ) L -1.
Using similar arguments, one has

E p 0 -p L 2 p L 2 = 1 d E q L (z) -z 2 z 2 α 2 L L-1 k=0 1 + 1 2 α 2 L k = 1 + 1 2 α 2 L L -1.

2.A.8 Proof of Corollary 2.8

The first statement is an immediate consequence of Proposition 2.6. The second one is a consequence of Proposition 2.7 and the fact that, for β < 1,

1 + 1 L β L = exp L ln 1 + 1 L β ∼ exp L 1-β → ∞.
Finally, (iii) follows from Proposition 2.7.

2.A.9 Proof of Proposition 2.10

The proposition is a consequence of Kloeden and Platen (1992, Theorems 4.5.3 and 10.2.2) for the SDE

dH t = 1 d σ(H t )dB t .
Letting a(h, t) = 0 and b(h, t) = 1 d σ(h), we need to check the following assumptions:

(H 1 ) The functions a(•, •) and b(•, •) are jointly measurable on R d × [0, 1]. (H 2 ) There exists a constant C 1 > 0 such that, for any x, y ∈ R d , t ∈ [0, 1], a(x, t) -a(y, t) + b(x, t) -b(y, t) C 1 x -y . (H 3 ) There exists a constant C 2 > 0 such that, for any x ∈ R d , t ∈ [0, 1], a(x, t) + b(x, t) C 2 (1 + x ). (H 4 ) E H 0 2 < ∞. (H 5 ) There exists a constant C 3 > 0 such that, for any x ∈ R d , s, t ∈ [0, 1], a(x, t) -a(x, s) + b(x, t) -b(x, s) C 3 (1 + x )|t -s| 1 /2 .
Assumptions (H 1 ), (H 4 ), and (H 5 ) readily follow from the definitions. Assumption (H 2 ) is true since σ is Lipschitz continuous, and (H 3 ) follows from

σ(x) b x x 1 + x .

2.A.10 Proof of Proposition 2.11

Let

ψ : R d × [0, 1] → R d be defined for any h ∈ R d , t ∈ [0, 1], by ψ(h, t) = V t g(h, Θ t ).
With this notation, the ODE (2.14) is equivalent to the initial value problem

dH t = ψ(H t , t)dt, H 0 = Ax.
By Assumptions (A 5 ) and (A 6 ), ψ is Lipschitz continuous in its first argument, in the sense that there exists

K > 0 such that, for all h, h ∈ R d , t ∈ [0, 1], ψ(h, t) -ψ(h , t) K h -h .
In addition, it is continuous in its second one. Thus, according to the Picard-Lindelöf theorem (Theorem 2.22 in Appendix 2.D), this is enough to show that the neural ODE (2.14) has a unique solution on [0, 1]. Note that the solution H is continuous on [0, 1] and is therefore bounded by a constant M > 0.

In order to prove the approximation bound of Proposition 2.11, we start by proving that both ψ and H are Lipschitz continuous in t. Under (A 5 ) and (A 6 ), this is clear for ψ since H is bounded. Moreover, for any [s, t] ⊂ [0, 1], we have

H t -H s = t s ψ(H u , u)du t s ψ(H u , u) du (t -s) sup u∈[0,1] h∈R d , h M ψ(h, u) .
Now, let K 1 and K 2 denote the Lipschitz constants of ψ (in both arguments) and H respectively, and, for any 0 k L, let t k = k /L. Then we have, for k 1,

H t k -h k = H t k-1 + t k t k-1 ψ(H u , u)du -h k-1 - 1 L ψ(h k-1 , t k-1 ) H t k-1 -h k-1 + t k t k-1 ψ(H u , u) -ψ(h k-1 , t k-1 ) du H t k-1 -h k-1 + K 1 t k t k-1 H u -h k-1 du + K 1 t k t k-1 |u -t k-1 |du 1 + K 1 L H t k-1 -h k-1 + K 1 t k t k-1 H u -H t k-1 du + K 1 t k t k-1 |u -t k-1 |du 1 + K 1 L H t k-1 -h k-1 + (K 2 + 1)K 1 t k t k-1 |u -t k-1 |du = 1 + K 1 L H t k-1 -h k-1 + (K 2 + 1)K 1 2L 2 .
By recurrence, we obtain

H t k -h k k-1 j=0 1 + K 1 L j × (K 2 + 1)K 1 2L 2 L 1 + K 1 L L × (K 2 + 1)K 1 2L 2 e K 1 (K 2 + 1)K 1 2L ,
which concludes the proof.

2.A.11 Proof of Proposition 2.12

Starting from (2.4) and using Assumption (A 6 ), one easily obtains the existence of C 1 and C 2 (whose values depend on the realization of V and Θ) such that

h k+1 (1 + C 1 α L ) h k + C 2 α L .
By recurrence,

h k+1 (1 + C 1 α L ) k h 0 + C 2 C 1 . Hence, using α L 1 /L, h k+1 exp(C 1 ) h 0 + C 2 C 1 .
Since g is Lipschitz continuous on compact sets, it is bounded on every ball of R d × R p . The result is then a consequence of the identity

h L -h 0 = α L L-1 k=0 V k+1 g(h k , θ k+1 ),
since we showed that each term in the sum is bounded by some constant C 3 > 0, independent of L and k. Hence we have that

h L -h 0 C 3 Lα L = C 3 L 1-β ,
yielding the results depending on the value of β.

2.A.12 Proof of Proposition 2.13

In the linear case, (2.4) can be written

h k+1 = h k + α L V k+1 h k , 0 k L -1.
Take y a unit-norm eigenvector of V 0 with associated eigenvalue λ > 0. Then

h k+1 , y = h k + α L V k+1 h k , y = h k , y + α L h k , V k+1 y = h k , y + λα L h k , y + α L h k , (V k+1 -V 0 ) y . Since V is Lipschitz and V k+1 = V k+1/L , there exists c such that V k+1 -V 0 c k+1 L . Hence | h k+1 , y | (1 + λα L )| h k , y | -cα L k + 1 L h k .
Then, by recurrence,

| h L , y | (1 + λα L ) L | h 0 , y | -c α L L L-1 k=0 (k + 1)(1 + λα L ) k h k (1 + λα L ) L | h 0 , y | -cα L (1 + λα L ) L max k h k . Let M = | h 0 ,y | 2cα L , and suppose that h k M for all 0 k L. Then h L | h L , y |
(by the Cauchy-Schwartz inequality)

(1 + λα L ) L | h 0 , y | -cM α L .
Then, for λα L 1,

h L 1 2 (1 + λα L ) L | h 0 , y | 1 2 exp λLα L 2 | h 0 , y |.
Thus, since Lα L = L 1-β , we have that h L → ∞, which contradicts our assumption that h k M for all 0 k L. We deduce that, for all L large enough,

max k h k > | h 0 , y | 2cα L L→∞ ----→ ∞.
Furthermore,

max k h k -h 0 h 0 > | h 0 , y | 2c h 0 α L -1 L→∞ ----→ ∞.

2.B Technical results

2.B.1 Lemmas 2.16 and 2.17

x ∈ R, a|x| |σ(x)| b|x|, 1 / √ 2 a < b 1. Then, for any x ∈ R d , 1 2 E W x 2 E σ(W x) 2 E W x 2 and E ReLU(W x) 2 = 1 2 E W x 2 .
Proof. The first part is a consequence of the assumption on σ. To prove the equality, let

X i = d j=1 W ij x j . Then E ReLU(W x) 2 = E d i=1 d j=1 W ij x j 2 1 d j=1 W ij x j 0 = E d i=1 X 2 i 1 X i 0 .
Since the (W ij ) 1 j d are centered and independent random variables, X i is also centered. Hence

E(X 2 i 1 X i 0 ) = 1/2E(X 2 i ), which concludes the proof.
Lemma 2.17. Let W ∈ R d×d be a matrix whose entries are centered i.i.d. random variables, with finite variance s 2 . Then, for any

x ∈ R d , E W x 2 = s 2 d x 2 .
Proof. For any 1 i d,

|W x| 2 i = d j=1 W ij x j 2 = d j,j =1 W ij W ij x j x j .
Thus, by independence,

E |W x| 2 i = E d j,j =1 W ij W ij x j x j = d j=1 E(W 2 ij )x 2 j = s 2 x 2 .
(2.21)

The result follows by summing over all i ∈ {1, . . . , d}.

2.B.2 Proof of Lemma 2.15

(E 1 ) and (E 2 ) are simple consequences of Assumptions (A 1 ) and (A 2 ).

To prove

(E 3 ), let f (h k , θ k+1 ) = V k+1 g(h k , θ k+1 ). Then E(Y k,2 Y k,1 |h k ) = 1 h k 4 E f (h k , θ k+1 ) 2 h k , f (h k , θ k+1 ) h k = E d i=1 d j=1 f (h k , θ k+1 ) 2 i (h k ) j f (h k , θ k+1 ) j h k .
It is easy to verify that, under Assumption (A 1 ), each term of the sum above has zero expectation. This shows (E 3 ).

To establish (E 4 ), we start by noting that

E h k h k , f (h k , θ k+1 ) h k 2 h k = 1 h k 4 E h k f (h k , θ k+1 )f (h k , θ k+1 ) h k |h k = 1 h k 4 h k E f (h k , θ k+1 )f (h k , θ k+1 ) |h k h k .
Clearly,

E(f (h k , θ k+1 ) i f (h k , θ k+1 ) j ) = 0 for i = j. Since, furthermore, the coordinates of f (h k , θ k+1 ) are identically distributed conditionally on h k , we obtain E f (h k , θ k+1 )f (h k , θ k+1 ) |h k = 1 d E( f (h k , θ k+1 ) 2 |h k )I d .
Thus,

E h k h k , f (h k , θ k+1 ) h k 2 h k = 1 d h k 4 E( f (h k , θ k+1 ) 2 |h k )h k h k 1 d , by Assumptions (A 1 ) and (A 2 ). To prove (E 5 ), let ϕ = V k+1 g(h k ,θ k+1 ),h k g(h k ,θ k+1 ) h k .
Then, for any t > 0,

P(|Y k,2 | > t) = P |ϕ| > t h k 2α L g(h k , θ k+1 ) = E P |ϕ| > t h k 2α L g(h k , θ k+1 ) h k , θ k+1 .
So, by Lemma 2.20 in Appendix 2.C,

P(|Y k,2 | > t) E 2 exp - dt 2 h k 2 16α 2 L s 2 g(h k , θ k+1 ) 2 = E E 2 exp - dt 2 h k 2 16α 2 L s 2 g(h k , θ k+1 ) 2 h k E 2 exp - dt 2 h k 2 16α 2 L s 2 E( g(h k , θ k+1 ) 2 |h k )
, by Jensen's inequality. Finally, using Assumption (A 2 ), we deduce that

P(|Y k,2 | > t) E 2 exp - dt 2 16α 2 L s 2 = 2 exp - dt 2 16α 2 L s 2 .
(2.22)

In particular, for all q 1 (see, e.g., [START_REF] Pauwels | Statistics, Optimization and Algorithms in High Dimension[END_REF],

E(Y 2q k,2 ) q! 32s 2 α 2 L d q .
The result is obtained by taking q = 2.

Finally, (E 6 ) and (E 7 ) are consequences of Lemma 2.21 in Appendix 2.C.

2.C Concentration of sub-Gaussian random matrices

In this appendix, we are interested in the concentration of linear and quadratic forms of sub-Gaussian matrices (Lemma 2.20 and Lemma 2.21). These two propositions are byproducts of the main result of [START_REF] Kontorovich | Concentration in unbounded metric spaces and algorithmic stability[END_REF], which generalizes McDiarmid's inequality to sub-Gaussian variables. We start by a technical result regarding the sub-Gaussian diameter introduced by Kontorovich ( 2014), whose definition is recalled below.

Definition 2.18. Let X be a real-valued random variable, X an independent copy of X, and ε a Rademacher random variable, independent of X and X . Then the sub-Gaussian diameter of X is defined as the smallest t such that ε|X -X | is t 2 sub-Gaussian.

Lemma 2.19. Let X be an s 2 sub-Gaussian centered random variable. Then the sub-Gaussian diameter of X is less than √ 2s.

Proof. Let λ ∈ R. Then, using the notation of Definition 2.18, one has

E(exp λε|X-X | ) = E(exp λ(X-X ) 1 ε=1 ) + E(exp λε(X -X) 1 ε=-1 ) = E(exp λ(X-X ) ) = E(exp λX ) 2 exp 2λ 2 s 2 ,
where the last equality is a consequence of the symmetry of X about 0.

We are now ready to prove the two main results of this appendix.

Lemma 2.20 (Bound on the deviation of linear forms). Let V be a R d×d matrix whose entries are i.i.d s 2 /d sub-Gaussian random variables. Then, for any x, y ∈ R d , x, y = 0,

P V x, y x y t 2 exp - dt 2 4s 2 . Proof. For any 1 i, j d, set X ij = x i V ij y j x y . Let X = R d 2 endowed
with the 1 norm, let X be the vector in X whose (id + j)-th coordinate is X ij , and let the function ϕ be defined by

ϕ : X Y -→ d 2 i=1 Y i .
By the triangle inequality, ϕ is a Lipschitz continuous function, with Lipschitz constant equal to 1. Observe also that X ij is a centered x 2 i s 2 y 2 j/d x 2 y 2 sub-Gaussian random variable. Thus, according to Lemma 2.19, the sub-Gaussian diameter of X ij is less than

√ 2x i sy j/ √ d x y
. By Kontorovich (2014, Theorem 1), for any t > 0, one has

P (ϕ(X) t) 2 exp - t 2 2 d i,j=1 2s 2 x 2 i y 2 j d x 2 y 2 , that is P V x, y x y t 2 exp - dt 2 4s 2 .
Lemma 2.21 (Bound of moments of quadratic forms). Let V be a R d×d matrix whose entries are i.i.d s 2 /d sub-Gaussian random variables, with variance 1 /d. Then, for any

x ∈ R d , x = 0, E V x 4 x 4 1 + 128s 4 d and E V x 8 x 8 1 + 384s 4 d + 3072s 6 d 2 .
Proof. The proof is similar to the one of Lemma 2.20, with

X ij = V ij x j
x , X = R d , and

ϕ i : X X → d j=1 X ij .
Each function ϕ i is a Lipschitz continuous function, with Lipschitz constant equal to 1. Observe now that the random variable

X ij is x 2 j s 2
/d x 2 sub-Gaussian and centered. Thus, according to Lemma 2.19, the sub-Gaussian diameter of X ij is less than

√ 2x j s / √ d x .
Therefore, according to Kontorovich (2014, Theorem 1), for any t > 0,

P (ϕ i (X) t) 2 exp - t 2 2 d j=1 2s 2 x 2 j d x 2
, that is

P | V i , x | x t 2 exp - dt 2 4s 2 .
Hence (see, e.g., [START_REF] Pauwels | Statistics, Optimization and Algorithms in High Dimension[END_REF],

E V i , x x 2q q! 8s 2 d q .
(2.23)

From identity (2.21) in the proof of technical Lemma 2.17, given in Appendix 2.B, we obtain that, for q = 1,

E V i , x x 2 = 1 d , (2.24)
which is an improvement by a factor 8s 2 over the previous upper bound. To conclude, it remains to conclude V x 4 and V x 8 with the V i , x . To do so, observe that

V x 4 = d i=1 V i , x 2 2 = d i,j=1 i =j V i , x 2 V j , x 2 + d i=1 V i , x 4 .
Hence, by independence of the (2.23) and(2.24)) Similarly,

(V i ) 1 i d , E V x 4 x 4 = d i,j=1 i =j E V i , x 2 x 2 E V j , x 2 x 2 + d i=1 E V i , x 4 x 4 = d(d -1) 1 d 2 + d 2(8s 2 ) 2 d 2 1 + 128s 4 d (by
V x 8 = d i=1 V i , x 2 3 = d i,j,k=1 i =j =k V i , x 2 V j , x 2 V j , x 2 + d i,j=1 i =j V i , x 2 V j , x 4 + d i=1 V i , x 8 .
Hence,

E V x 8 x 8 = d i,j,k=1 i =j =k E V i , x 2 x 2 E V j , x 2 x 2 E V k , x 2 x 2 + d i,j=1 i =j E V i , x 4 x 4 E V j , x 2 x 2 + d i=1 E V i , x 8 x 8 = d(d -1)(d -2) 1 d 2 + 3d(d -1) 2(8s 2 ) 2 d 3 + d 6(8s 2 ) 3 d 3 1 + 384s 4 d + 3072s 6 d 2 .

2.D A version of the Picard-Lindelöf theorem

Theorem 2.22. Assume that f : R d × [0, 1] → R d is Lipschitz continuous in its first argument and continuous in its second one. Then, for any z ∈ R d , the initial value problem

dH t = f (H t , t)dt, H 0 = z, (2.25
)

admits a unique solution H : [0, 1] → R d . Proof. Let C ([s, t], R d ) be the set of continuous functions from [s, t] to R d . For any [s, t] ⊂ [0, 1], ζ ∈ R d , let Ψ be the function Ψ : C ([s, t], R d ) → C ([s, t], R d ) Y → v → ζ + v s f (Y u , u)du . For any Y, Y ∈ C ([s, t], R d ), v ∈ [s, t], one has, denoting by K f the Lipschitz constant of f in its first argument, Ψ(Y ) v -Ψ(Y ) v v s f (Y u , u) -f (Y u , u) du v s K f Y u -Y u du K f v s Y -Y ∞ du K f Y -Y ∞ (t -s).
This yields

Ψ(Y ) -Ψ(Y ) ∞ K f (t -s) Y -Y ∞ ,
which means that the function Ψ is Lipschitz continuous on C ([s, t], R d ) endowed with the supremum norm, with Lipschitz constant K f (ts). So, on any interval [s, t] of length smaller than δ = 1 /2K f , the function Ψ is a contraction. Thus, by the Banach fixed-point theorem, for any initial value ζ, Ψ has a unique fixed point. Hence, there exists a unique solution to (2.25) on any interval of length δ with any initial condition. To obtain a solution on [0, 1] it is sufficient to concatenate these solutions.

2.E Detailed experimental setting

Our code is available at https://github.com/PierreMarion23/scaling-resnets.

To obtain Figures 2.1 to 2.3, we initialize ResNets from res-3 with the hyperparameters of Table 2.2.

Name

Value

d 40 n in 64 n out 1 L 10 to 1000 β 0.25, 0.5, 1 weight distribution U(-3/d, 3/d) data distribution standard Gaussian Table 2.2: Hyperparameters of Figures 2.1 to 2.3
Each experiment is repeated 50 times, with independent data and weight sampling. For Figures 2.4 and 2.5, we take the same hyperparameters except for β, which now takes values in {0.5, 1, 2}, and for the weight distribution. The weights are now initialized as discretizations of a Gaussian process. More precisely, each entry of V and Θ is an independent Gaussian process with zero mean and an RBF kernel of variance 10 -2 .

To obtain Figure 2.7, we take the hyperparameters of More precisely, for each 1 i, j d, we let (V k+1,i,j ) 0 k L-1 be the increments of a fractional Brownian motion (fBm), where the various fBm involved are independent. The procedure is the same for θ.

In Figure 2.9, we use res-1, with the hyperparameters of Table 2.4. We train on MNIST1 and CIFAR-10 2 using the Adam optimizer (Kingma and Ba, 2015) for 10 epochs. The learning rate is divided by 10 after 5 epochs. The best performance on the learning rate grid is reported in the figure. In this chapter, we take a further step towards establishing a solid mathematical link between residual neural networks and neural ordinary differential equations (ODEs), by proving an implicit regularization of deep residual networks towards neural ODEs. Our result holds for nonlinear networks trained with gradient flow. We prove that if the network is initialized as a discretization of a neural ODE, then such a discretization holds throughout training. Our results are valid for a finite training time, and also as the training time tends to infinity provided that the network satisfies a Polyak-Łojasiewicz condition. Importantly, this condition holds for a family of residual networks where the residuals are two-layer perceptrons with an overparameterization in width that is only linear, and implies the convergence of the gradient flow to a global minimum of the loss. Our results are illustrated by numerical experiments.

Introduction

Residual networks are a successful family of deep learning models popularized by breakthrough results in computer vision (He et al., 2016a). The key idea of residual networks, namely the presence of skip connections, is now ubiquitous in deep learning, and can be found, for example, in Transformer models (Vaswani et al., 2017). The main advantage of skip connections is to allow successful training with depth of the order of a thousand layers, in contrast to vanilla neural networks, leading to significant performance improvements (e.g., [START_REF] Wang | Deepnet: Scaling transformers to 1,000 layers[END_REF]. This has motivated research on the properties of residual networks in the limit where the depth tends to infinity. One of the main explored directions is the neural ordinary differential equation (ODE) limit (Chen et al., 2018a).

To present the neural ODE principle, we first introduce the mathematical formalism of deep residual networks. We consider a single model throughout the chapter to simplify the exposition, but most of our results apply to more general models, as will be discussed later. We consider the formulation

h k+1 = h k + 1 L √ m V k+1 σ 1 √ q W k+1 h k , k ∈ {0, . . . , L -1}, (3.1)
where L is the depth of the network, h k ∈ R q is the output of the k-th hidden layer, V k ∈ R q×m , W k ∈ R m×q are the weights of the k-th layer, and σ is an activation function applied element-wise.

Scaling with the square root of the width is classical, although it often appears as an equivalent condition on the variance at initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF][START_REF] Lecun | Efficient BackProp[END_REF][START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF]. We make the scaling factors explicit to have weights of magnitude O(1) independently of the width and the depth. The 1/L scaling factor is less common, but it is necessary for the correspondence with neural ODEs to hold. More precisely, if there exist Lipschitz continuous functions V and W such that

V k = V(k/L) and W k = W(k/L), then the residual network (3.1) converges, as L → ∞, to the ODE dH ds (s) = 1 √ m V(s)σ 1 √ q W(s)H(s) , s ∈ [0, 1], (3.2)
where s is the continuous-depth version of the layer index. It is important to note that this correspondence holds for fixed limiting functions V and W. This is especially true at initialization, for example by setting the V k to zero and the W k to weight-tied Gaussian matrices. In this case, the initial residual network is trivially equal to the neural ODE dH ds (s) = 0. Of course, more sophisticated initialization choices are possible, as shown, e.g., in Chapter 2 and Sander et al. (2022b). However, regardless of an ODE structure at initialization, a more challenging question is that of the structure of the network after training. Since the weights are updated during training, there is no a priori guarantee that an ODE limit still holds after training, even if it does at initialization.

The question of a possible ODE structure for the trained network is not a mere technical one. In fact, it is important for at least three reasons. First, it gives a precise answer to the question of the connection between (trained) residual networks and neural ODEs, providing more solid ground to a common statement in the community that both can coincide in the large-depth limit (see, e.g., [START_REF] Haber | Stable architectures for deep neural networks[END_REF]E et al., 2019;[START_REF] Dong | Towards adaptive residual network training: A neural-ODE perspective[END_REF]Massaroli et al., 2020;[START_REF] Kidger | On Neural Ordinary Differential Equations[END_REF]. Second, it opens exciting perspectives for understanding residual networks. Indeed, if trained residual networks are discretizations of neural ODEs, then it is possible to apply results from neural ODEs to the large family of residual networks. In particular, from a theoretical point of view, the approximation capabilities of neural ODEs are well understood [START_REF] Teshima | Universal approximation property of neural ordinary differential equations[END_REF]Zhang et al., 2020a) and it is relatively easy to obtain generalization bounds for these models (see Hanson and Raginsky, 2022 and Chapter 4 of this manuscript). From a practical standpoint, advantages of neural ODEs include memory-efficient training (Chen et al., 2018a;Sander et al., 2022b) and weight compression [START_REF] Erichson | Lipschitz recurrent neural networks[END_REF]. This is important because in practice memory is a bottleneck for training residual networks (Gomez et al., 2017). Finally, our analysis is a first step towards understanding the implicit regularization (Neyshabur et al., 2015b;[START_REF] Vardi | On the implicit bias in deep-learning algorithms[END_REF] of gradient descent for deep residual networks, that is, characterizing the properties of the trained network among all minimizers of the empirical risk.

Throughout the document, it is assumed that the network is trained with gradient flow, which is a continuous analog of gradient descent. The parameters V k are updated according to an ODE of the form dV k dt (t) = -L ∂ ∂V k (t) for t ≥ 0, where is an empirical risk (the exact mathematical context and assumptions are detailed in Section 3.3), and similarly for W k . The scaling factor L is the counterpart of the factor 1/L in (3.1), and prevents exploding or vanishing gradients as L tends to infinity. Note that the gradient flow is defined with respect to a time index t different from the layer index s.

Contributions. Our first main contribution (Section 3.4.1) is to show that a neural ODE limit holds after training up to time t, i.e., there exists a function V(s, t) such that the residual network converges, as L tends to infinity, to the ODE

dH ds (s) = 1 √ m V(s, t)σ 1 √ q W(s, t)H(s) , s ∈ [0, 1].
This large-depth limit holds for any finite training time t 0. However, the convergence of the optimization algorithm as t tends to infinity, which we refer to as the long-time limit to distinguish it from the large-depth limit L → ∞, is not guaranteed without further assumptions, due to the non-convexity of the optimization problem. We attack the question (Section 3.4.2) when the width is large enough by proving a Polyak-Łojasiewicz (PL) condition, which is now state of the art in analyzing the properties of optimization algorithms for deep neural networks [START_REF] Liu | Loss landscapes and optimization in over-parameterized non-linear systems and neural networks[END_REF]. The main assumption for our PL condition to hold is that the width m of the hidden layers should be greater than some constant times the number of data n. As a second main contribution, we show that the PL condition yields the long-time convergence of the gradient flow for residual networks with linear overparameterization. Finally, we prove the convergence with high probability in the long-time limit, namely the existence of functions V ∞ and W ∞ such that the discrete trajectory defined by the trained residual network (3.1) converges as both L and t tend to infinity to the solution of the neural ODE (3.2) with V = V ∞ and W = W ∞ . In addition, our approach points out that this limiting ODE interpolates the training data. Finally, our results are illustrated by numerical experiments (Section 3.5).

Organization of the chapter. Section 3.2 presents some related work. We then move on to detail the mathematical context and notation in Section 3.3 before giving our main results in Section 3.4. Section 3.5 is devoted to numerical experiments. We conclude the main part of the chapter in Section 3.6. Then, in Section 3.A, we prove results on a more general residual network model that encompasses the one presented so far. These results are then instantiated in the specific case of the residual network (3.3) in Section 3.B, thus proving the results of the main part of the chapter. Section 3.C contains some lemmas that are useful for the proofs. We present in Section 3.D a counter-example showing that a residual network with the ReLU activation can move away from the neural ODE structure during training. Finally, Section 3.E presents some experimental details.

Related work

Deep residual networks and neural ODEs. Several works study the large-depth convergence of residual networks to differential equations, but without considering the training dynamics, such as Chapter 2 of this manuscript or [START_REF] Cohen | Scaling properties of deep residual networks[END_REF]; Thorpe and van Gennip (2022); [START_REF] Hayou | On the infinite-depth limit of finite-width neural networks[END_REF]. Closer to our setting, [START_REF] Cont | Convergence and implicit regularization properties of gradient descent for deep residual networks[END_REF] and Sander et al. (2022b) analyze the dynamics of gradient descent for deep residual networks, as we do, but with significant differences. [START_REF] Cont | Convergence and implicit regularization properties of gradient descent for deep residual networks[END_REF] consider a 1 / √ L scaling factor in front of the residual branch, resulting in a limit that is not a neural ODE. In addition, only W is trained. Furthermore, to obtain convergence in the long-time limit, it is assumed that the data points are nearly orthogonal. Sander et al. (2022b) prove the existence of an ODE limit for trained residual networks, but in the simplified case of a linear activation and under a more restricted setting.

Long-time convergence of wide residual networks. Polyak-Łojasiewicz conditions are a modern tool to prove long-time convergence of overparameterized neural networks [START_REF] Liu | Loss landscapes and optimization in over-parameterized non-linear systems and neural networks[END_REF]. These conditions are a relaxation of convexity, and mean that the gradients of the loss with respect to the parameters cannot be small when the loss is large. They have been applied to residual networks with both linear [START_REF] Bartlett | Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks[END_REF]Wu et al., 2019;Zou et al., 2020b) and nonlinear activations [START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF]Frei et al., 2019;Barboni et al., 2022;[START_REF] Cont | Convergence and implicit regularization properties of gradient descent for deep residual networks[END_REF][START_REF] Macdonald | A global analysis of global optimisation[END_REF]. Building on the proof technique of Nguyen and Mondelli (2020) for non-residual networks, we need only a linear overparameterization to prove our PL condition, i.e., we require m = Ω(n). This compares favorably with results requiring polynomial overparameterization [START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF]Barboni et al., 2022) or assumptions on the data, either a margin condition (Frei et al., 2019) or a sample size smaller than the dimension of the data space [START_REF] Cont | Convergence and implicit regularization properties of gradient descent for deep residual networks[END_REF][START_REF] Macdonald | A global analysis of global optimisation[END_REF].

Implicit regularization. This chapter can be related to a line of work on the implicit regularization of gradient-based algorithms for residual networks (Neyshabur et al., 2015b). We show that the optimization algorithm does not just converge to any residual network that minimizes the empirical risk, but rather to the discretization of a neural ODE. Note that most implicit regularization results state that the optimization algorithm converges to an interpolator that minimizes some complexity measure, which can be a margin [START_REF] Lyu | Gradient descent maximizes the margin of homogeneous neural networks[END_REF], a norm (Boursier et al., 2022), or a matrix rank (Li et al., 2021b). Thus, an interesting next step is to understand if the neural ODE found by gradient flow actually minimizes some complexity measure, and to characterize its generalization properties.

Definitions and notation

This section is devoted to specifying the setup outlined in Section 3.1.

Residual network. A (scaled) residual network of depth L ∈ N * is defined by h L 0 = A L x h L k+1 = h L k + 1 L √ m V L k+1 σ 1 √ q W L k+1 h L k , k ∈ {0, . . . , L -1}, F L (x) = B L h L L . (3.3)
To allow the hidden layers h L k ∈ R q to have a different dimension than the input x ∈ R d , we first map x to h L 0 with a weight matrix A L ∈ R q×d . We assume that the hidden layers belong to a higher dimensional space than the input and output, i.e., q max(d, d ). The residual transformations are two-layer perceptrons parameterized by the weight matrices V L k ∈ R q×m and W L k ∈ R m×q . This is standard in the literature (e.g., He et al., 2016b;Chen et al., 2018a;Dupont et al., 2019;Barboni et al., 2022). The last weight matrix B L ∈ R d ×q maps the last hidden layer to the output F L (x) in R d . Also, σ : R → R is an element-wise activation function assumed to be C 2 , non-constant, Lipschitz continuous, bounded, and such that σ(0

) = 0. The convenient shorthand Z L k = (V L k , W L k
) is occasionally used, and we denote Z L k F the sum of the Frobenius norms

V L k F + W L k F .
Data and loss. The data is a sample of n pairs

(x i , y i ) 1≤i≤n ∈ (X × Y) n where X × Y is a compact set of R d ×R d . The empirical risk is the mean squared error L = 1 n n i=1 F L (x i )-y i 2 .
Initialization. We initialize

A L = (I R d×d , 0 R (q-d)×d
) as the identity matrix in R d×d concatenated row-wise with the zero matrix in R (q-d)×d , to act as a simple projection of the input onto the higher dimensional space R q , and similarly

B L = (0 R d ×(q-d ) , I R d ×d ).
The weights V L k are initialized to zero and the W L k as weight-tied standard Gaussian matrices, i.e., for all k ∈ {1, . . . , L}, m×q) . Initializing outer matrices to zero is standard practice (Zhang et al., 2019a), while taking weight-tied matrices instead of i.i.d. ones is less common. We show in Section 3.5 that it is still possible to learn with this initialization scheme on real world data. As explained in Section 3.4.3, other initialization choices are possible, provided they correspond to the discretization of a Lipschitz continuous function, but we focus on this one in the main text for simplicity.

W L k = W ∼ N (0, 1) ⊗(
Training algorithm. Gradient flow is the limit of gradient descent as the learning rate tends to zero. The parameters are set at time t = 0 by the initialization, and then evolve according to the ODE

dA L dt (t) = - ∂ L ∂A L (t), dZ L k dt (t) = -L ∂ L ∂Z L k (t), dB L dt (t) = - ∂ L ∂B L (t), t 0, (3.4)
for k ∈ {1, . . . , L}. In the following, the dependence in t is made explicit when necessary, e.g., we write h L k (t) instead of h L k , and F L (x; t) instead of F L (x). It turns out that, without further assumptions, the gradient flow can diverge in finite time. This is because the dynamics are not (globally) Lipschitz continuous, breaking the conditions of the Picard-Lindelöf theorem (see Lemma 3.19) for existence and uniqueness of ODE solutions. A common practice (Goodfellow et al., 2016, Section 10.11.1) is to consider instead a clipped gradient flow

dA L dt (t) = π - ∂ L ∂A L (t) , dZ L k dt (t) = π -L ∂ L ∂Z L k (t) , dB L dt (t) = π - ∂ L ∂B L (t) , (3.5)
where π is a generic notation for a bounded Lipschitz continuous operator. For example, clipping each coordinate of the gradient at some C > 0 amounts to taking π as the projection on the ball centered at 0 of radius C for the ∞ norm. Clipping ensures that the dynamics are well defined, as shown in the next proposition that is a consequence of the Picard-Lindelöf theorem.

Proposition 3.1. The (clipped) gradient flow (3.5) has a unique solution for all t 0.

Proof. See Section 3.B.1.

In Section 3.4.2, we make additional assumptions to prove the long-time convergence of the gradient flow. We then prove that these assumptions ensure that the dynamics of the gradient flow (3.4) are well defined, eliminating the need for clipping.

Neural ODE. The neural ODE corresponding to the residual network (3.3) is defined by

H(0) = Ax dH ds (s) = 1 √ m V(s)σ 1 √ q W(s)H(s) , s ∈ [0, 1], F (x) = BH(1), (3.6)
where x ∈ R d is the input, H ∈ R q is the variable of the ODE, V : [0, 1] → R q×m and W : [0, 1] → R m×q are Lipschitz continuous functions, A ∈ R q×d and B ∈ R d ×q are matrices, and the output is F (x) ∈ R d . The following proposition shows that the neural ODE is well defined. In addition, its output is close to the residual network (3.3) provided the weights are discretizations of V and W.

Proposition 3.2. The neural ODE (3.6) has a unique solution H : [0, 1] → R q . Consider, moreover, the residual network

(3.3) with A L = A, V L k = V(k/L) and W L k = W(k/L) for k ∈ {1, . . . , L}, and B L = B. Then there exists C > 0 such that, for all L ∈ N * , sup x∈X F (x) - F L (x) ≤ C L . Proof. See Section 3.B.2.
Clearly, our choices of V L k and W L k at initialization are discretizations of the Lipschitz continuous (in fact, constant) functions V(s) ≡ 0 and W(s) ≡ W ∼ N (0, 1) ⊗(m×q) . Thus, Proposition 3.2 holds at initialization, and the residual network is equivalent to the trivial ODE dH ds (s) = 0. The next section shows that after training we obtain non-trivial dynamics, which still discretize neural ODEs.

Large-depth limit of residual networks

We study the large-depth limit of trained residual networks in two settings. In Section 3.4.1, we consider the case of a finite training time. We move in Section 3.4.2 to the case where the training time tends to infinity, which is tractable under a Polyak-Łojasiewicz condition.

Clipped gradient flow and finite training time

We first consider the case where the neural network is trained with clipped gradient flow (3.5) on some training time interval [0, T ], T > 0. This allows us to prove large-depth convergence to a neural ODE without further assumptions. We emphasize that stopping training after a finite training time is a common technique in practice, referred to as early stopping (Goodfellow et al., 2016, Section 7.8). It is considered as a form of implicit regularization, and our result sheds light on this intuition by showing that the complexity of the trained networks increases with T .

The following proposition is a key step in proving the main theorem of this section.

Proposition 3.3. There exist M, K > 0 such that, for any t ∈ [0, T ], L ∈ N * , and k ∈ {1, . . . , L},

max A L (t) F , V L k (t) F , W L k (t) F , B L (t) F ≤ M, and, for k ∈ {1, . . . , L -1}, max V L k+1 (t) -V L k (t) F , W L k+1 (t) -W L k (t) F ≤ K L .
Moreover, with probability at least 1exp -3qm 16 , the following expressions hold for M and K:

M = T M π + 2 √ qm, K = βT e αT , (3.7)
where M π is the supremum of π in Frobenius norm, and α and β depend on X , Y, M , and σ.

Proof. See Section 3.B.3.

This proposition ensures that the size of the weights and the difference between successive weights remain bounded throughout training. We can now state the main result, which states the convergence, for any training time in [0, T ], of the neural network to a neural ODE as L → ∞. Recall that a sequence of functions f L of some variable u is said to converge uniformly over

u ∈ U to f if sup u∈U f L (u) -f (u) → 0.
Theorem 3.4. Consider the residual network (3.3) with the training dynamics (3.5). Then the following statements hold as L tends to infinity:

(i) There exist functions A : [0, T ] → R q×d and B : [0, T ] → R d ×q such that A L (t) and B L (t)
converge uniformly over t ∈ [0, T ] to A(t) and B(t).

(ii) There exists a Lipschitz continuous function

Z : [0, 1] × [0, T ] → R q×m × R m×q such that Z L : [0, 1] × [0, T ] → R q×m × R m×q , (s, t) → Z L (s, t) = Z L (L-1)s +1 (t) (3.8) converges uniformly over s ∈ [0, 1] and t ∈ [0, T ] to Z = (V, W).
(iii) Uniformly over s ∈ [0, 1], t ∈ [0, T ], and x ∈ X , the hidden layer h L Ls (t) converges to the solution at time s of the neural ODE

H(0, t) = A(t)x ∂H ∂s (s, t) = 1 √ m V(s, t)σ 1 √ q W(s, t)H(s, t) , s ∈ [0, 1]. (3.9) (iv) Uniformly over t ∈ [0, T ] and x ∈ X , the output F L (x; t) converges to B(t)H(1, t).
Proof. See Section 3.B.4.

Let us sketch the proof of statement (ii), which is the cornerstone of the theorem. A first key idea is to introduce in (3.8) the piecewise-constant continuous-depth interpolation Z L of the weights, whose ambient space does not depend on L, in contrast to the discrete weight sequence Z L k . Since the weights remain bounded during training by Proposition 3.3, the Arzelà-Ascoli theorem guarantees the existence of an accumulation point for Z L . We show that the accumulation point is unique because it is the solution of an ODE satisfying the conditions of the Picard-Lindelöf theorem. The uniqueness of the accumulation point then implies the existence of a limit for the weights.

There are two notable byproducts of our proof. The first one is an explicit description of the training dynamics of the limiting weights A, B, and Z, as the solution of an ODE system, as presented in Appendix 3.A.5. The second one, which we now describe, consists of norm bounds on the weights. Proposition 3.3 bounds the discrete weights and the difference between two consecutive weights respectively by some M, K > 0. The proof of Theorem 3.4 shows that this bound carries over to the continuous weights, in the sense that A(t), V(s, t), W(s, t), and B(t) are uniformly bounded by M , and V(•, t) and W(•, t) are uniformly Lipschitz continuous with Lipschitz constant K. Formally, this last property means that, for any s, s ∈ [0, 1] and t ∈ [0, T ],

V(s , t) -V(s, t) F ≤ K|s -s| and W(s , t) -W(s, t) F ≤ K|s -s|.
The boundedness and Lipschitz continuity of the weights are important features because they limit the statistical complexity of neural ODEs (see Chapter 4). More generally, norm-based bounds are a common approach in the statistical theory of deep learning (see, e.g., Bartlett et al., 2017, and references therein). Looking at the formula (3.7) for M and K, one can see in particular that the bounds diverge exponentially with T , providing an argument in favor of early stopping.

Our approach so far characterizes the large-depth limit of the neural network for a finite training time T , but two questions remain open. A first challenge is to characterize the value of the loss after training. A second one is to provide insight into the convergence of the optimization algorithm in the long-time limit, i.e., as T tends to infinity. To answer these questions, we move to the setting where the width of the network is large enough, which allows us to prove a Polyak-Łojasiewicz condition and thereby the long-time convergence of the training loss to zero.

Convergence in the long-time limit for wide networks

Proving convergence of gradient-based optimization algorithms for neural networks is a major difficulty in deep learning theory. One direction recently explored considers sufficiently wide neural networks, with the Polyak-Łojasiewicz (PL) condition. In our setting, they are written as follows (with the notation

Z L = (V L k , W L k ) k∈{1,...,L} ):
Definition 3.5. For M, µ > 0, the residual network (3.3) is said to satisfy the (M, µ)-local PL condition around a set of parameters ( ĀL , ZL , BL ) if, for every set of parameters

(A L , Z L , B L ) such that A L -ĀL F M, sup k∈{1,...,L} Z L k -ZL k F M, B L -BL F M, one has ∂ L ∂A L 2 F + L L k=1 ∂ L ∂Z L k 2 F + ∂ L ∂B L 2 F ≥ µ L ,
where the loss L is evaluated at the set of parameters (A L , Z L , B L ).

The next important point is to observe that, under the setup of Section 3.3 and some additional assumptions, the residual network satisfies the local PL condition of Definition 3.5. Proposition 3.6. Assume that the sample points (x i , y i ) are i.i.d. such that x i 2 = √ q. Then there exist c 1 , . . . , c 4 > 0 (depending only on σ) and δ > 0 such that, if

q ≥ d + d , m ≥ c 1 n, L ≥ c 2 √ nq,
then, with probability at least 1δ, the residual network

(3.3) satisfies the (M, µ)-local PL condition around its initialization, with M = c 3 √ nq and µ = c 4 n √ nq .
Proof. See Section 3.B.5.

We emphasize that Proposition 3.6 requires the width m to scale only linearly with the sample size n, which improves on the literature (see Section 3.2). The other assumptions are mild. Note that our proof shows that the parameter δ is small if n grows at most polynomially with d (see Appendix 3.B.5).

We are now ready to state convergence in the long-time and large-depth limits to a global minimum of the empirical risk, when the local PL condition holds and the norm of the targets y i is small enough.

Theorem 3.7. Consider the residual network (3.3) with the training dynamics (3.4), and assume that the assumptions of Proposition 3.6 hold. Then there exist C, δ > 0 such that, if

1 n n i=1 y i 2 ≤ C
, then, with probability as least 1δ, the gradient flow is well defined on R + , and, for t ∈ R + and L ∈ N * ,

L (t) ≤ exp - C t n √ nq L (0), (3.10)
for some C > 0 depending on σ. Moreover, the following statements hold as t and L tend to infinity:

(i) There exist matrices A ∞ ∈ R q×d and B ∞ ∈ R d ×q such that A L (t) and B L (t) converge to A ∞ and B ∞ .
(ii) There exists a Lipschitz continuous function

Z ∞ : [0, 1] → R q×m × R m×q such that Z L : [0, 1] × R + → R q×m × R m×q , (s, t) → Z L (s, t) = Z L (L-1)s +1 (t) converges uniformly over s ∈ [0, 1] to Z ∞ = (V ∞ , W ∞ ).
(iii) Uniformly over s ∈ [0, 1] and x ∈ X , the hidden layer h L Ls (t) converges to the solution at time s of the neural ODE

H(0) = A ∞ x dH ds (s) = 1 √ m V ∞ (s)σ 1 √ q W ∞ (s)H(s) , s ∈ [0, 1]. (iv) Uniformly over x ∈ X , the output F L (x; t) converges to F ∞ (x) = B ∞ H(1). Furthermore, F ∞ (x i ) = y i for all i ∈ {1, . . . , n}.
Proof. See Section 3.B.6.

This theorem proves two important results of separate interest. On the one hand, equation (3.10) shows the long-time convergence of the gradient flow for deep residual networks under the linear overparameterization assumption m ≥ c 1 n of Proposition 3.6. On the other hand, when both t and L tend to infinity, the network converges to a neural ODE that further interpolates the training data. Note that the order in which t and L tend to infinity does not matter by uniform convergence properties.

Generalizations to other architectures and initialization

To simplify the exposition, we have so far considered a particular residual architecture defined in (3.3). However, most of our results hold for a more general residual network of the form

h L k+1 = h L k + 1 L f (h L k , Z L k+1 ), k ∈ {0, . . . , L -1}, (3.11)
where f : R q × R p → R q is a C 2 function such that f (0, •) ≡ 0 and f (•, z) is uniformly Lipschitz for z in any compact. All our results are shown in the appendix for this general model, except the PL condition of Proposition 3.6, which we prove only for the specific setup of Section 3.3. In particular, the conclusions of Theorem 3.4 hold for the general model (3.11), as well as those of Theorem 3.7 if the network satisfies a (M, µ)-local PL condition with µ sufficiently large (see Appendix 3.B for details).

It is easy to see that our residual network of interest (3.3) is a special case of the general model (3.11) if σ satisfies the assumptions of Section 3.3. However, other choices are possible, such as convolutional layers or a Lipschitz continuous version of Transformer [START_REF] Kim | The lipschitz constant of self-attention[END_REF]. This latter application is particularly interesting in the light of the literature analyzing the Transformer architecture from a neural ODE point of view [START_REF] Lu | Understanding and improving transformer from a multi-particle dynamic system point of view[END_REF]Sander et al., 2022a;[START_REF] Geshkovski | The emergence of clusters in self-attention dynamics[END_REF].

Moreover, the initialization assumption made in Section 3.3 can also be relaxed to include any so-called smooth initialization of the weights (see Chapter 2). A smooth initialization corresponds to taking V L k (0) and W L k (0) as discretizations of some Lipschitz continuous functions

V 0 : [0, 1] → R q×m and W 0 : [0, 1] → R m×q , that is, for k ∈ {1, . . . , L}, V L k (0) = V 0 ( k L ) and W L k (0) = W 0 ( k L ).
A typical concrete example is to let the entries of V 0 and W 0 be independent Gaussian processes with expectation zero and squared exponential covariance

K(x, x ) = exp(-(x-x ) 2
2 2 ), for some > 0. As shown by Proposition 3.2, a smooth initialization means that the network discretizes a neural ODE.

Numerical experiments

We now present numerical experiments to validate our theoretical findings, using both synthetic and real-world data. Experimental details are given in Appendix 3.E. Our code will be open sourced.

Synthetic data

We consider the residual network (3.3) with the initialization scheme of Section 3.3. So, the V L k are initialized to zero and the W L k to weight-tied standard Gaussian matrices. To ease the presentation, we consider the case where q = d = d , and we do not train the weights A L and B L , which therefore stay equal to the identity. The activation function is GELU [START_REF] Hendrycks | Gaussian Error Linear Units (GELUs)[END_REF], which is a smooth approximation of ReLU: x → max(x, 0). The sample points (x i , y i ) 1≤i≤n follow independent standard Gaussian distributions. Note that it does not hurt to take x and y independent since, in this subsection, our focus is on optimization results only and not on statistical aspects. The mean-squared error is minimized using full-batch gradient descent.

The following experiments exemplify the large-depth (t ∈ [0, T ], L → ∞) and long-time (t → ∞, L finite) limits.

( Z L k (t) -Z L k+1 (t) F ).
Right: uniform convergence of Z L to its largedepth limit Z. Here, for a matrix-valued function f , f denotes (

1 0 f (s) 2 F ds) 1/2 .
Large-depth limit. We illustrate key insights of Proposition 3.3 and Theorem 3.4, with T = 500. In Figure 3.1 (left), we plot the maximum distance between two successive weight matrices, i.e., max 1≤k≤L,t∈[0,T ] ( Z L k (t) -Z L k+1 (t) F ), for different values of L and training time t ∈ [0, T ]. We observe a 1/L convergence rate, as predicted by Proposition 3.3. Moreover, for a fixed L, the distance between two successive weight matrices increases with the training time, however at a much slower pace than the exponential upper bound on K given in identity (3.7). Figure 3.1 (right) depicts the uniform convergence of Z L to its large-depth limit Z, illustrating statement (ii) of Theorem 3.4. The function Z is computed using Z L for L = 2 14 . Note that the convergence is slower for larger training times.

Real-world data

We now investigate the properties of deep residual networks on the CIFAR 10 dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. We deviate from the mathematical model (3.3) by using convolutions instead of fully connected layers. More precisely, A L is replaced by a trainable convolutional layer, and the residual layers write

h L k+1 = h L k + 1 L bn L 2,k (conv L 2,k (σ(bn L 1,k (conv L 1,k (h L k ))))), k ∈ {0, . . . , L -1},
where conv L i,k are convolutions and bn L i,k are batch normalizations. The output of the residual layers is mapped to logits through a linear layer B L . We initialize bn L 2,k to 0, and bn L 1,k and conv L i,k either to weight-tied or to i.i.d. Gaussian. Table 3.1 reports the accuracy of the trained network, and whether it has Lipschitz continuous (or smooth) weights after training, depending on the activation function σ and on the initialization scheme. To assess the smoothness of the weights, we simply resort to visual inspection. For example, Figure 3.3 (left) shows two random entries of the convolutions across layers with GELU and a weight-tied initialization: the smoothness is preserved after training. Smooth weights indicate that the residual network discretizes a neural ODE (see, e.g., Proposition 3.2). On the contrary, if an i.i.d. initialization is used, smoothness is not preserved after training, as shown in Figure 3.3 (right), and the residual network does not discretize a neural ODE. 98.4 ± 0.1 88.4 ± 0.5 × Table 3.1: Accuracy and smoothness of the trained weights depending on the choice of activation function σ and initialization scheme. We display the median over 5 runs and the interquartile range between the first and third quantile. Smooth weights correspond to a neural ODE structure.

A coeff. of conv L i,k i = 1 i = 2
Table 3.1 conveys several important messages. First, in accordance with our theory (Theorem 3.4), we obtain a neural ODE structure when using a smooth activation function and weight-tied initialization (lines 1 and 3 of Table 3.1). This is not the case when using the non-smooth ReLU activation and/or i.i.d. initialization. In fact, we prove in Appendix 3.D that the smoothness of the weights is lost when training with ReLU in a simple setting, confirming this experimental observation. Furthermore, the third line of Table 3.1 shows that it is possible to obtain a reasonable accuracy with a neural ODE structure, which, as emphasized in Section 3.1, also comes with theoretical and practical advantages. Nevertheless, we obtain an improvement in accuracy in the cases corresponding to non-smooth weights, i.e., to a residual network that does not discretize an ODE. Extending our theory to such cases is left for future work.

Conclusion

We study the convergence of deep residual networks to neural ODEs. When properly scaled and initialized, residual networks trained with fixed-horizon gradient flow converge to neural ODEs as the depth tends to infinity. This result holds for very general architectures. In the case where both training time and depth tend to infinity, convergence holds under a local Polyak-Łojasiewicz condition. We prove such a condition for a family of deep residual networks with linear overparameterization.

The setting of neural ODE-like networks comes with strong guarantees, at the cost of some performance gap when compared with i.i.d. initialization as highlighted by the experimental section. Extending the mathematical large-depth study to the i.i.d. case is an interesting problem for future research. Previous work suggests that the correct limit object is then a stochastic differential equation instead of an ODE, such as Chapter 2 of this manuscript or [START_REF] Cohen | Scaling properties of deep residual networks[END_REF]; [START_REF] Cont | Convergence and implicit regularization properties of gradient descent for deep residual networks[END_REF].

3.A Some results for general residual networks

Lipschitz continuity. Let (U , • ), (V , • ), and (W , • ) be generic normed spaces. Then a function of two variables g : U × V → W is:

(i) (Globally) Lipschitz continuous if there exists K ≥ 0 such that, for (u, v), (u , v ) ∈ U × V , g(u, v) -g(u , v) ≤ K u -u + K v -v . (ii) Locally Lipschitz continuous in its first variable if, for any compacts E ⊂ U , E ⊂ V , there exists K ≥ 0 such that, for (u, v), (u , v) ∈ E × E , g(u, v) -g(u , v) ≤ K u -u .
Equivalent definitions hold for a function of one variable. Moreover, g(•, v) is said to be uniformly Lipschitz continuous for v in V if there exists

K ≥ 0 such that, for (u, v), (u , v) ∈ U × V , g(u, v) -g(u , v) ≤ K u -u ,
and uniformly Lipschitz continuous for v in any compact if, for any compact E ⊂ V , there exists

K ≥ 0 such that, for (u, v), (u , v) ∈ U × E , g(u, v) -g(u , v) ≤ K u -u .
Throughout, we refer to a Lipschitz continuous function with Lipschitz constant K ≥ 0 as K-Lipschitz.

Model. As explained in Section 3.4.3, most of our results are proven for the general residual network

h L 0 (t) = A L (t)x h L k+1 (t) = h L k (t) + 1 L f (h L k (t), Z L k+1 (t)), k ∈ {0, . . . , L -1}, (3.12) F L (x; t) = B L (t)h L L (t),
where

Z L (t) = (Z L 1 (t), . . . , Z L L (t)) ∈ (R p ) L and f : R q × R p → R q is a C 2 function such that f (0, •) ≡ 0 and f (•, z
) is uniformly Lipschitz for z in any compact. Let us introduce the backpropagation equations, which are instrumental in the study of the gradient flow dynamics. These equations define the backward state p L k (t) ∈ R q through the backward recurrence

p L L (t) = 2B L (t) (F L (x; t) -y) p L k (t) = p L k+1 (t) + 1 L ∂ 1 f (h L k (t), Z L k+1 (t))p L k+1 (t), k ∈ {0, . . . , L -1}, (3.13)
where ∂ 1 f ∈ R q×q stands for the Jacobian matrix of f with respect to its first argument. Similarly, we let ∂ 2 f ∈ R q×p be the Jacobian matrix of f with respect to its second argument. For a sample (x i , y i ) 1≤i≤n ∈ (X × Y) n , we let h L k,i (t) and p L k,i (t) be, respectively, the hidden layer h L k (t) and the backward state p L k (t) associated with the i-th input x i . Denoting the mean squared error associated with the sample by L , we have, by the chain rule,

∂ L ∂A L (t) = 1 n n i=1 p L 0,i (t)x i (3.14) ∂ L ∂Z L k (t) = 1 nL n i=1 ∂ 2 f (h L k-1,i (t), Z L k (t)) p L k-1,i (t), k ∈ {1, . . . , L}, (3.15) ∂ L ∂B L (t) = 2 n n i=1 (F L (x i ; t) -y i )h L L,i (t) . (3.16) Initialization. The parameters (Z L k (t)) 1≤k≤L are initialized to Z L k (0) = Z init k L , where Z init : [0, 1] → R p is a Lipschitz continuous function. Furthermore, we initialize A L (0) to some matrix A init ∈ R q×d and B L (0) = B init ∈ R d ×q .
Note that this initialization scheme is a generalization of the one presented in Section 3.3.

Additional notation. For a vector x, x denotes the Euclidean norm. For a matrix A, the operator norm induced by the Euclidean norm is denoted by A 2 , and the Frobenius norm is denoted by A F . Finally, we use the notation

A L (resp. Z L k , B L ) to denote the function t → A L (t) (resp. t → Z L k (t), t → B L (t))
, since the parameters are considered as functions of the training time throughout this appendix.

Overview of Appendix A. First, in Section 3.A.1, we study the case of the (clipped) gradient flow (3.5). We show that the weights and the difference between successive weights are bounded during the entire training. Section 3.A.2 shows a similar result for the standard gradient flow (3.4) under a PL condition. In Section 3.A.3, we show a generalized version of the Arzelà-Ascoli theorem, which allows us to prove the existence of a converging subsequence of the weights in the large-depth limit. Section 3.A.4 is devoted to the convergence of the Euler scheme for parameterized ODEs. We then proceed to prove in Section 3.A.5 our main result, i.e., the large-depth convergence of the gradient flow. The key step is to establish the uniqueness of the adherence point of the weights. Finally, in Section 3.A.6, we prove the existence of a double limit for the weights and the hidden states when both the depth and the training time tend to infinity.

3.A.1 The trained weights are bounded in the finite training-time setup

Before stating the result, let us introduce the notation ∂ 22 f (h, z) ∈ R q×p×p , which is the thirdorder tensor of second partial derivatives of f with respect to z. We endow the space R q×p×p with the operator norm • 2 induced by the Euclidean norm in R p and the • 2 norm in R q×p . In other words,

∂ 22 f (h, z) 2 = sup u∈R p , u =1 ∂ 22 f (h, z)u 2 ,
where

∂ 22 f (h, z)u ∈ R q×p is the tensor product of ∂ 22 f (h, z) against u. Similarly, ∂ 21 f (h, z) ∈
R q×p×q denotes the third-order tensor of cross second partial derivatives of f , and the space R q×p×q is endowed with the operator norm • 2 induced by the Euclidean norm in R q and the • 2 norm in R q×p .

Proposition 3.8. Consider the residual network (3.12) initialized as explained in Appendix 3.A and trained with the gradient flow (3.5) on [0, T ], for some T ∈ (0, ∞). Let

M π = max max A∈R q×d π(A) F , max Z∈R p π(Z) , max B∈R d ×q π(B) F , M 0 = max A init F , sup s∈[0,1] Z init (s) , B init F and M = M 0 + T M π .
Then the gradient flow is well defined on [0, T ], and, for t ∈ [0, T ], L ∈ N * , and k ∈ {1, . . . , L},

A L (t) F ≤ M, Z L k (t) ≤ M, and B L (t) F ≤ M.
(3.17)

Moreover, there exist α, β > 0 such that, for t ∈ [0, T ] and k ∈ {1, . . . , L -1},

Z L k+1 (t) -Z L k (t) ≤ Z L k+1 (0) -Z L k (0) + βT L e αT .
The following expressions for α and β hold:

α = 2e K K M (e K M 2 M X + M Y ) and β = 2Ke K M (K + e K K M M X )(e K M 2 M X + M Y ),
where

M X = sup x∈X x , M Y = sup y∈Y y , K 1 = sup z ≤M ∂ 1 f (h, z) 2 (3.18) E = {(h, z) ∈ R d × R p , h ≤ e K 1 M M X , z ≤ M } (3.19) K 2 = sup (h,z)∈E ∂ 2 f (h, z) 2 , K = max(K 1 , K 2 ) K = sup (h,z)∈E max ∂ 22 f (h, z) 2 , ∂ 21 f (h, z) 2 .
Proof. The time-independent dynamics

(A L , Z L k , B L ) → π - ∂ L ∂A L , π -L ∂ L ∂Z L k , π - ∂ L ∂B L
defining the gradient flow (3.5) are locally Lipschitz continuous, hence the gradient flow is defined on a maximal interval [0, T max ) by the Picard-Lindelöf theorem (see Lemma 3. [START_REF] Woodworth | Kernel and rich regimes in overparametrized models[END_REF]). Let us show by contradiction that T max = T . Assume that T max < T . If this is true, again by the Picard-Lindelöf theorem, we know that the parameters diverge to infinity at T max . However, for any t ∈ [0, T max ), we have

A L (t) F ≤ A L (0) F + t 0 dA L dt (τ ) F dτ ≤ M 0 + t 0 M π dτ ≤ M 0 + T M π = M.
Bounds on B L and Z L k by M can be shown similarly. This contradicts the divergence of the parameters at t = T max . We conclude that the gradient flow is well defined on [0, T ] and that the bounds (3.17) hold.

It remains to bound the difference

Z L k+1 (t) -Z L k (t) . We have, for t ∈ [0, T ] and k ∈ {1, . . . , L -1}, dZ L k+1 dt (t) - dZ L k dt (t) = L ∂ L ∂Z L k+1 (t) - ∂ L ∂Z L k (t) ≤ n i=1 1 n ∂ 2 f (h L k,i (t), Z L k+1 (t)) p L k,i (t) -∂ 2 f (h L k-1,i (t), Z L k (t)) p L k-1,i (t) ≤ 1 n n i=1 ∂ 2 f (h L k,i (t), Z L k+1 (t)) 2 p L k,i (t) -p L k-1,i (t) + p L k-1,i (t) ∂ 2 f (h L k,i (t), Z L k+1 (t)) -∂ 2 f (h L k-1,i (t), Z L k (t)) 2 (3.20) Furthermore, for t ∈ [0, T ], k ∈ {0, . . . , L -1}, and i ∈ {1, . . . , n}, h L k+1,i (t) = h L k,i (t) + 1 L f (h L k,i (t), Z L k+1 (t)) ≤ (1 + K 1 L ) h L k,i (t) , since f (•, Z L k+1 (t)) is K 1 -Lipschitz,
where K 1 is defined by (3.18), and f (0, Z L k+1 (t)) = 0. Therefore, for any k ∈ {1, . . . , L},

h L k,i (t) ≤ e K 1 h L 0,i (t) = e K 1 A L (t)x i ≤ e K 1 M M X . (3.21)
This bound shows that the pair

(h L k,i (t), Z L k+1 (t)) belongs to the compact E defined in (3.19) for every t ∈ [0, T ], k ∈ {1, . . . , L}, and i ∈ {1, . . . , n}. In particular, ∂ 2 f (h L k-1,i (t), Z L k (t)) 2 ≤ K, and
∂ 2 f (h L k,i (t), Z L k+1 (t)) -∂ 2 f (h L k-1,i (t), Z L k (t)) 2 ≤ K h L k,i (t) -h L k-1,i (t) + K Z L k+1 (t) -Z L k (t) .
Returning to (3.20), we obtain

dZ L k+1 dt (t) - dZ L k dt (t) ≤ 1 n n i=1 K p L k,i (t) -p L k-1,i (t) + K p L k-1,i (t) h L k,i (t) -h L k-1,i (t) + Z L k+1 (t) -Z L k (t) .
For k ∈ {1, . . . , L} and i ∈ {1, . . . , n},

p L k,i (t) -p L k-1,i (t) = 1 L ∂ 1 f (h L k-1,i (t), Z L k (t))p L k,i (t) ≤ K L p L k,i (t) ,
and, similarly,

h L k,i (t) -h L k-1,i (t) = 1 L f (h L k-1,i (t), Z L k (t)) ≤ K L h L k-1,i (t) ≤ Ke K M M X L .
Thus,

dZ L k+1 dt (t) - dZ L k dt (t) ≤ 1 n n i=1 p L k,i (t) K 2 L + K K L e K M M X + K Z L k+1 (t) -Z L k (t) .
Moreover, for k ∈ {0, . . . , L} and i ∈ {1, . . . , n},

p L k,i (t) ≤ p L k+1,i (t) + 1 L ∂ 1 f (h L k,i (t), Z L k+1 (t))p L k+1,i (t) ≤ p L k+1,i (t) + K L p L k+1,i (t) . Hence p L k,i (t) ≤ e K p L L,i (t) = 2e K B L (t) (F L (x i ; t) -y i ) ≤ 2e K M B L (t)h L L,i (t) + y i ≤ 2e K M (e K M 2 M X + M Y ),
where we use (3.17) and (3.21) for the last inequality. Putting all the pieces together, we obtain

dZ L k dt (t) - dZ L k+1 dt (t) ≤ α Z L k (t) -Z L k+1 (t) + β L .
Integrating between 0 and t, we see that

Z L k+1 (t) -Z L k (t) ≤ Z L k+1 (0) -Z L k (0) + βt L + t 0 α Z L k (τ ) -Z L k+1 (τ ) dτ.
Applying Grönwall's inequality (see, e.g., [START_REF] Dragomir | Some Gronwall Type Inequalities and Applications[END_REF], we conclude that

Z L k+1 (t) -Z L k (t) ≤ ( Z L k+1 (0) -Z L k (0) + βT L )
e αT , as desired.

3.A.2 The trained weights are bounded under the local PL condition

Proposition 3.9. Consider the residual network (3.12) initialized as explained in Appendix 3.A and trained with the gradient flow (3.4) on [0, ∞]. Then, for M > 0, there exists µ > 0 such that, if the residual network satisfies the (M, µ)-local PL condition (3.5) around its initialization for any L ∈ N * , then:

(i) The gradient flow is well defined on R + , and, for t ∈ R + , L ∈ N * , and k ∈ {1, . . . , L},

A L (t) F ≤ M A , Z L k (t) ≤ M Z , and B L (t) F ≤ M B ,
where

M A = A init 2 + M, M Z = sup s∈[0,1] Z init (s) + M, and M B = B init 2 + M.
(ii) There exists K > 0 such that, for t ∈ R + , L ∈ N * , and k ∈ {1, . . . , L},

Z L k (t) -Z L k+1 (t) ≤ K L .
(iii) There exists a bounded integrable function b : R + → R such that, for t ∈ R + , L ∈ N * , and k ∈ {1, . . . , L},

max dA L dt (t) , dZ L k dt (t) , dB L dt (t) ≤ b(t) (iv) A L (t), B L (t)
, and Z L k (t) admit a limit uniformly over L ∈ N * and k ∈ {1, . . . , L} as t → ∞.

(v) For t ∈ R + and L ∈ N * , L (t) ≤ e -µt L (0).
Moreover, the following expression for µ hold:

µ = max(M B K, M B M X , M A M X ) 8e K M sup L∈N * L (0), (3.22)
where

M X = sup x∈X x , K 1 = sup z ≤M Z ∂ 1 f (h, z) E = {(h, z) ∈ R d × R p , h ≤ e K 1 M A M X , z ≤ M Z } K 2 = sup (h,z)∈E ∂ 2 f (h, z) , K = max(K 1 , K 2 ).
Proof. Let M > 0, µ defined by (3.22), and assume that the residual network satisfies the (M, µ)-local PL condition (3.5) around its initialization for any L ∈ N * . The time-independent dynamics

(A L , Z L k , B L ) → - ∂ L ∂A L , -L ∂ L ∂Z L k , - ∂ L ∂B L
defining the gradient flow (3.5) are locally Lipschitz continuous, hence the gradient flow is defined on a maximal interval [0, T max ) by the Picard-Lindelöf theorem (see Lemma 3. [START_REF] Woodworth | Kernel and rich regimes in overparametrized models[END_REF]). Let us show by contradiction that T max = ∞. Assume that T max < ∞. If this is true, again by the Picard-Lindelöf theorem, we know that the parameters diverge to infinity at T max . In particular, there exist t ∈ (0, T max ) and k ∈ {1, . . . , L} such that

A L (t) -A L (0) F > M or Z L k (t) -Z L k (0) > M or B L (t) -B L (0) F > M.
Let t * ∈ (0, T max ) be the infimum of such times t. Then, for t < t * and k ∈ {1, . . . , L},

A L (t) -A L (0) F ≤ M and Z L k (t) -Z L k (0) ≤ M and B L (t) -B L (0) F ≤ M, (3.23)
and, by continuity of A L , B L , and Z L k , these inequalities also hold for t = t * . By definition, this means that the (M, µ)-local PL condition is satisfied for t ≤ t * , and ensures that

∂ L ∂A L (t) 2 F + L L k=1 ∂ L ∂Z L k (t) 2 + ∂ L ∂B L (t) 2 F ≥ µ L (t).
Therefore, by definition of the gradient flow (3.4),

d L dt (t) = ∂ L ∂A L (t), dA L dt (t) + L k=1 ∂ L ∂Z L k (t), dZ L k dt (t) + ∂ L ∂B L (t), dB L dt (t) = - ∂ L ∂A L (t) 2 F -L L k=1 ∂ L ∂Z L k (t) 2 - ∂ L ∂B L (t) 2 F ≤ -µ L (t).
Thus, by Grönwall's inequality, for t ≤ t * ,

L (t) ≤ e -µt L (0). (3.24)
Furthermore, by (3.23) and the definition of M A , M B , M Z , we have, for t ≤ t * and k ∈ {1, . . . , L},

A L (t) F ≤ M A , Z L k (t) ≤ M Z , and B L (t) F ≤ M B .
A quick scan through the proof of Proposition 3.8 reveals that by similar arguments, we have, for t ≤ t * , k ∈ {1, . . . , L}, and i ∈ {1, . . . , n},

(h L k-1,i (t), Z L k (t)) ∈ E and p L k-1,i (t) ≤ 2e K p L L,i (t) ≤ 2e K M B F L (x i ; t) -y i .
Thus, for k ∈ {0, . . . , L},

1 n n i=1 p L k,i (t) ≤ 2e K M B n n i=1 F L (x i ; t) -y i ≤ 2e K M B L (t) ≤ 2e K M B e -µt 2 L (0), (3.25)
where the second inequality is a consequence of the Cauchy-Schwartz inequality. Let us now bound

Z L k (t * ) -Z L k (0) . We have, for k ∈ {1, . . . , L}, Z L k (t * ) -Z L k (0) ≤ t * 0 dZ L k dt (t) dt ≤ 1 n n i=1 t * 0 ∂ 2 f (h L k-1,i (t), Z L k (t)) p L k-1,i (t) dt (by (3.15)). ≤ K n n i=1 t * 0 p L k-1,i (t) dt, since (h L k-1,i (t), Z L k (t)) ∈ E and ∂ 2 f (h, z) ≤ K for (h, z) ∈ E. Therefore, by (3.25), Z L k (t * ) -Z L k (0) ≤ 2Ke K M B t * 0 e -µt 2 L (0)dt ≤ 4Ke K M B µ L (0) ≤ M 2 ,
where the last inequality is a consequence of the definition of µ. Similarly, by (3.14) and (3.25),

A L (t * ) -A L (0) F ≤ t * 0 dA L dt (t) F dt ≤ t * 0 1 n n i=1 p L 0,i (t)x i F dt ≤ 2e K M B M X L (0) t * 0 e -µt 2 dt ≤ 4e K M B M X µ L (0) ≤ M 2 .
Finally, by (3.16),

B L (t * ) -B(0) F ≤ t * 0 dB L dt (t) F dt ≤ t * 0 2 n n i=1 (F L (x i ; t) -y i )h L L,i (t) F dt ≤ 2e K M A M X L (0) t * 0 e -µt 2 dt ≤ 4e K M A M X µ L (0) ≤ M 2 ,
where the third inequality is a consequence of the Cauchy-Schwartz inequality and of the fact that h L L,i (t) ≤ e K M A M X . By continuity of A L , Z L k , and B L , these three bounds contradict the definition of t * . We conclude that T max = ∞ and that the parameters stay within a ball of radius M of their initialization, yielding the inequalities, for t ∈ R + , L ∈ N * , and k ∈ {1, . . . , L},

A L (t) F ≤ M A , B L (t) F ≤ M B , Z L k (t) ≤ M Z .
This proves statement (i) of the proposition. Moreover, the analysis above show that the derivatives of A L , Z L k , and B L are bounded by a bounded integrable function independent of L and k. This shows (iii), together with the fact that the functions A L (t), Z L k (t), and B L (t) admit limits as t → ∞. Furthermore, the convergence towards their limit is uniform over L and k, as we show for example for A L (t). If we denote by A L ∞ its limit, and apply the same steps as for bounding A L (t * ) -A L (0) F , we obtain, for any t ≥ 0,

A L ∞ -A L (t) F ≤ ∞ t dA L dτ (τ ) F dτ ≤ 2e K M B M X L (0) ∞ t e -µτ 2 dτ = 4e K M B M X µ e -µt 2
L (0) ≤ M 2 e -µt 2 ,
where the last inequality comes from the definition of µ. The bound is independent of L, proving statement (iv). Statement (v) readily follows from (3.24).

To complete the proof, it remains to prove statement (ii) by bounding the differences

Z L k+1 (t) -Z L k (t)
. Now that we know that the weights are bounded, we can follow the same steps as in the proof of Proposition 3.8 and show the existence of

C 1 , C 2 > 0 such that dZ L k+1 dt (t) - dZ L k dt (t) ≤ 1 n n i=1 p L k,i (t) C 1 L + C 2 Z L k+1 (t) -Z L k (t) .
Using (3.25), we obtain

dZ L k+1 dt (t) - dZ L k dt (t) ≤ 2e K M B e -µt 2 L (0) C 1 L + C 2 Z L k+1 (t) -Z L k (t) .
Integrating between 0 and t, we obtain

Z L k+1 (t) -Z L k (t) ≤ Z L k+1 (0) -Z L k (0) + t 0 2e K M B e -µτ 2 L (0) C 1 L dτ + t 0 2e K M B e -µτ 2 L (0)C 2 Z L k+1 (τ ) -Z L k (τ ) dτ ≤ Z L k+1 (0) -Z L k (0) + C 1 M 2M X L + t 0 2e K M B e -µτ 2 L (0)C 2 Z L k+1 (τ ) -Z L k (τ ) dτ,
where the second inequality uses the definition of µ. By Grönwall's inequality,

Z L k+1 (t) -Z L k (t) ≤ Z L k+1 (0) -Z L k (0) + C 1 M 2M X L exp t 0 2e K M B e -µτ 2 L (0)C 2 dτ ≤ Z L k+1 (0) -Z L k (0) + C 1 M 2M X L exp C 2 M 2M X ,
again by definition of µ. Finally, since Z L k (0) = Z init ( k L ) and Z init is Lipschitz continuous, this proves the existence of K > 0 (independent of L, t and k) such that Z L k+1 (t) -Z L k (t) ≤ K L , which yields statement (ii).

3.A.3 Generalized Arzelà-Ascoli theorem

Proposition 3.10 (Generalized Arzelà-Ascoli theorem). Let I ⊆ R + be an interval. We denote by (Z L k ) L∈N * ,1≤k≤L be a family of C 1 functions from I to R p . Define

Z L : [0, 1] × I → R p , (s, t) → Z L (s, t) = Z L (L-1)s +1 (t).
Assume that there exist a constant C > 0 and a bounded integrable function b : I → R such that the following statements hold for any t ∈ I and L ∈ N * :

(i) For k ∈ {1, . . . , L -1}, Z L k+1 (t) -Z L k (t) ≤ C L , (ii) For k ∈ {1, . . . , L}, Z L k (t) ≤ C and dZ L k dt (t) ≤ b(t).
Then there exist a subsequence (Z φ(L) ) L∈N * of (Z L ) L∈N * and a Lipschitz continuous function

Z φ : [0, 1] × I → R p such that Z φ(L) (s, t) tends to Z φ (s, t)
uniformly over s and t.

Note that if I is a compact interval, then the existence of a (uniformly) convergent subsequence is guaranteed by the standard Arzelà-Ascoli theorem. Indeed, the uniform equicontinuity is a consequence of assumptions (i) and (ii), while (ii) provides a uniform bound. However, if I is not compact, more involved arguments are needed.

Proof. Assume, without loss of generality, that b is also bounded by C. According to assumption (i), for t ∈ I and i, j ∈ {1, . . . , L},

Z L i (t) -Z L j (t) ≤ C|i -j| L .
Also, according to (ii), for t, t ∈ I and k ∈ {1, . . . , L},

Z L k (t) -Z L k (t ) = t t dZ L k dτ (τ )dτ ≤ C|t -t |.
It follows that, for s, s ∈ [0, 1] and t, t ∈ I,

Z L (s, t) -Z L (s , t ) ≤ Z L (s, t) -Z L (s, t ) + Z L (s, t ) -Z L (s , t ) ≤ C|t -t | + C| (L -1)s -(L -1)s | L .
Therefore, with some simple algebra, we obtain

Z L (s, t) -Z L (s , t ) ≤ C|t -t | + C|s -s | + C L . (3.26)
The statement of the proposition is then a consequence of the next three steps.

There exists a convergent subsequence of (Z L (s, t)

) L∈N * . First, let ((s i , t i )) i∈N = (Q ∩ [0, 1]) × (Q ∩ I)
. By (ii), the sequence (Z L (s i , t i )) L∈N * ,i∈N is bounded. It is therefore possible to construct by a diagonal procedure a subsequence (Z φ(L) ) L∈N * such that, for each i ∈ N, (Z φ(L) (s i , t i )) L∈N * is a convergent sequence.

Let us now show that (Z φ(L) (s, t)) L∈N * converges for any s ∈ [0, 1] and t ∈ I, by proving that it is a Cauchy sequence in the complete metric space R p . Let ε > 0, s ∈ [0, 1], and t ∈ I. Since ((s i , t i )) i∈N is dense in [0, 1] × I, there exists some j ∈ N such that |s j -s| ≤ ε and |t j -t| ≤ ε. Then, for L, M ∈ N * , we have

Z φ(L) (s, t) -Z φ(M ) (s, t) ≤ Z φ(L) (s, t) -Z φ(L) (s j , t j ) + Z φ(L) (s j , t j ) -Z φ(M ) (s j , t j ) + Z φ(M ) (s j , t j ) -Z φ(M ) (s, t) ≤ 2Cε + C φ(L) + Z φ(L) (s j , t j ) -Z φ(M ) (s j , t j ) + 2Cε + C φ(M )
,

where we used inequality (3.26) twice. Since (Z φ(L) (s j , t j )) L∈N * is a convergent sequence, it is a Cauchy sequence. Thus, the bound can be made arbitrarily small for L, M large enough. This shows that (Z φ(L) (s, t)) L∈N * is also a Cauchy sequence. It is therefore convergent, and we denote by Z φ (s, t) its limit.

The function Z φ is Lipschitz continuous. By considering (3.26) for the subsequence φ(L) and letting L → ∞, we have that, for any s, s ∈ [0, 1] and t, t ∈ I,

Z φ (s, t) -Z φ (s , t ) ≤ C(|s -s | + |t -t |). (3.27)
The convergence of (Z φ(L) (s, t)) L∈N * to Z φ (s, t) is uniform over s and t. Let ε > 0, s ∈ [0, 1], and t ∈ I. Then, by (3.26) and (3.27), it is possible to find δ > 0 such that, for any s , s ∈ [0, 1] and t , t ∈ I satisfying |ss | ≤ δ and |tt | ≤ δ,

Z φ(L) (s , t ) -Z φ(L) (s , t ) ≤ ε + C φ(L)
and Z φ (s , t ) -Z φ (s , t ) ≤ ε, (3.28) and

Z φ(L) (s , t ) -Z φ(L) (s , t ) ≤ ε + C φ(L) and Z φ (s , t ) -Z φ (s , t ) ≤ ε. (3.29)
Furthermore, there exists a finite set {s 1 , . . . , s S } ⊂ [0, 1] such that

[0, 1] ⊂ S i=1 (s i -δ, s i + δ).
In the sequel, we denote by s * an element of {s 1 , . . . , s S } that is at distance at most δ from s.

If I is unbounded, then, by assumption (ii) and since b is integrable, there exists some t 0 > 0 such that, for t ≥ t 0 ,

Z φ(L) (s, t) -Z φ(L) (s, t 0 ) ≤ t t 0 d dt Z φ(L) (φ(L)s-1) +1 (τ ) dτ ≤ t t 0 b(τ )dτ ≤ ε. (3.30)
The same inequality holds for Z φ by letting L tend to infinity. If I is bounded, we simply let t 0 = sup I.

We may then pick a finite set {t 1 , . . . , t T } ⊂ [0, t 0 ] such that

[0, t 0 ] ⊂ T i=1 (t i -δ, t i + δ).
Two cases may arise depending on the value of t. If t ∈ [0, t 0 ], then there exists an element of the set {t 1 , . . . , t T } at distance at most δ from t, and we denote it by t * . If t > t 0 , we let t * = t 0 .

According to (3.29) and (3.30), we then have in both cases that

Z φ(L) (s, t) -Z φ(L) (s, t * ) ε + C φ(L)
and Z φ (s, t) -Z φ (s, t * ) ε.

(3.31)

To conclude, we have to bound the term Z φ(L) (s, t) -Z φ (s, t) uniformly over s and t. We first have

Z φ(L) (s, t) -Z φ (s, t) ≤ Z φ(L) (s, t) -Z φ(L) (s, t * ) + Z φ(L) (s, t * ) -Z φ (s, t * ) + Z φ (s, t * ) -Z φ (s, t) ≤ 2ε + C φ(L) + Z φ(L) (s, t * ) -Z φ (s, t * ) ,
where the last inequality is a consequence of (3.31). The last term can be bounded as follows:

Z φ(L) (s, t * ) -Z φ (s, t * ) ≤ Z φ(L) (s, t * ) -Z φ(L) (s * , t * ) + Z φ(L) (s * , t * ) -Z φ (s * , t * ) + Z φ (s * , t * ) -Z φ (s, t * ) ≤ 2ε + C φ(L) + max i∈{1,...,S} Z φ(L) (s i , t * ) -Z φ (s i , t * ) ,
by using (3.28) and the fact that s * ∈ {s 1 , . . . , s S }. Putting all the pieces together, we finally obtain

Z φ(L) (s, t) -Z φ (s, t) ≤ 4ε + 2C φ(L) + max i∈{1,...,S},j∈{1,...,T } Z φ(L) (s i , t j ) -Z φ (s i , t j ) .
By taking L large enough, independent of s and t, the sum of the last two terms can be made less than ε. Since ε is arbitrary, this concludes the proof.

A consequence of this result is a simplified version for sequences of functions only indexed by L and not k, as follows.

Corollary 3.11. Let I ⊆ R + be an interval, and (Z L ) L∈N * be a family of C 1 functions from I to R p . Assume that there exist a constant C > 0 and a bounded integrable function b : I → R such that, for any t ∈ I and L ∈ N * , Z L (t) ≤ C and dZ L dt (t) ≤ b(t). Then there exist a subsequence (Z φ(L) ) L∈N * of (Z L ) L∈N * and a function Z φ : I → R p such that Z φ(L) (t) tends to Z φ (t) uniformly over t.

3.A.4 Consistency of the Euler scheme for parameterized ODEs

Proposition 3.12 (Consistency of the Euler scheme for parameterized ODEs.). We denote by (θ L k ) L∈N * ,1≤k≤L be a bounded family of vectors of R p , and let

Θ L : [0, 1] → R p , s → θ L (L-1)s +1 .
Assume that there exists Θ : [0, 1] → R p a Lipschitz continuous function such that Θ L (s) tends to Θ(s) uniformly over s. Let (a L ) L∈N * be a sequence of vectors in some compact E ⊂ R d converging to a ∈ E. Let g : R d × R p → R d be a C 1 function such that g(0, •) ≡ 0 and g(•, θ) is uniformly Lipschitz continuous for θ in any compact of R p . Consider the discrete scheme

u L 0 = a L u L k+1 = u L k + 1 L g(u L k , θ L k+1 ), k ∈ {0, . . . , L -1}.
(3.32)

Then u L Ls tends to U (s) uniformly over s ∈ [0, 1], where U is the unique solution of the ODE

U (0) = a dU ds (s) = g(U (s), Θ(s)), s ∈ [0, 1]. (3.33)
Moreover, the convergence only depends on the sequence (a L ) L∈N * and on its limit a ∈ E through ( a La ) L∈N * .

Proof. Let M be a bound of the sequence (θ L k ) L∈N * ,1≤k≤L . By definition of Θ L , the sequence (Θ L ) L∈N * is also uniformly bounded by M , and the same is true for Θ. Then the function Therefore, by Grönwall's inequality,

g(•, Θ(s)) is uniformly Lipschitz for s ∈ [0, 1]. Furthermore, (U, s) → g(U, Θ(s))
U (s) ≤ U (0) exp(C) = a exp(C) ≤ D E exp(C),
where D E = sup x∈E x < ∞. A similar reasoning applies to the discrete scheme (3.32), using the discrete version of Grönwall's inequality. More precisely, for any k ∈ {0, . . . , L -1},

u L k+1 ≤ u L k + 1 L g(u L k , θ L k+1 ) ≤ 1 + C L u L k . Thus, u L k ≤ u L 0 exp(C) = a L exp(C) ≤ D E exp(C).
Overall, we can consider a restriction of g to a compact set depending only on M , C, and E, which we will still denote by g with a slight abuse of notation.Since g is C 1 , it is therefore bounded and Lipschitz continuous, and we still let C be its Lipschitz constant.

For L ∈ N * and k ∈ {0, . . . , L}, we denote by ∆ L k the gap between the continuous and the discrete schemes, i.e.,

∆ L k = U k L -u L k .
The next step is to recursively bound the size of this gap, first observing that

∆ L 0 = a L -a . We have that s → dU ds (s) = g(U (s), Θ(s)) (3.
U k + 1 L = U k L + 1 L dU ds k L + (k+1)/L k/L k + 1 L -s d 2 U ds 2 (s)ds,
where the norm of the remainder term is less than C/L 2 . Therefore,

∆ L k+1 = U k + 1 L -u L k+1 = U k L + 1 L g U k L , Θ k L + (k+1)/L k/L k + 1 L -s d 2 U ds 2 (s)ds -u L k - 1 L g(u L k , θ L k+1 ) ≤ U k L -u L k + 1 L g U k L , Θ k L - 1 L g(u L k , θ L k+1 ) + (k+1)/L k/L k + 1 L -s d 2 U ds 2 (s) ds ∆ L k + C L ∆ L k + C L Θ k L -θ L k+1 + C L 2 .
In the last inequality, we used the fact that g is C-Lipschitz. Since,by definition, θ L k+1 = Θ L ( k L-1 ), we obtain, for k ∈ {0, . . . , L -1},

∆ L k+1 1 + C L ∆ L k + C L Θ k L -Θ L k L -1 + C L 2 1 + C L ∆ L k + C L sup s∈[0,1] Θ(s) -Θ L (s) + C L Θ k L -Θ k L -1 + C L 2 1 + C L ∆ L k + C L sup s∈[0,1] Θ(s) -Θ L (s) + CC Θ L 2 + C L 2 ,
where C Θ is the Lipschitz constant of Θ. By the discrete Grönwall's inequality, we deduce that, for k ∈ {0, . . . , L -1}, (3.35) This shows that the gaps ∆ L k converge to zero uniformly over k ∈ {0, . . . , L} as L tends to infinity. We conclude by observing that, for any s ∈ [0, 1],

∆ L k+1 ∆ L 0 + sup s∈[0,1] Θ(s) -Θ L (s) + C Θ L + C LC e C = a L -a + sup s∈[0,1] Θ(s) -Θ L (s) + C Θ L + C LC e C .
U (s) -u L Ls U (s) -U Ls L + U Ls L -u L Ls ≤ C U L + ∆ L Ls , (3.36)
where C U is the Lipschitz constant of U . Both terms converge to zero uniformly over s as L tends to infinity. Finally, an inspection of our bounds shows that the convergence only depends on (a L ) L∈N * ∈ E N * through a La .

The results of Proposition 3.12 can be extended without much effort to two other related cases. First, the parameters θ L k may depend on some other variable t, as long as all assumptions are verified uniformly over t. Second, these parameters may converge to some limit parameters as both L and t go to infinity. This is encapsulated in the following two corollaries.

Corollary 3.13. Let I ⊆ R + be an interval. Let (θ L k ) L∈N * ,1≤k≤L be a uniformly bounded family of functions from I to R p , and let

Θ L : [0, 1] × I → R p , (s, t) → θ L (L-1)s +1 (t).
Assume that there exists a function Θ : [0, 1] × I → R p such that Θ L (s, t) tends to Θ(s, t) uniformly over s and t, and Θ(•, t) is uniformly Lipschitz continuous for t ∈ I. Let (a L ) L∈N * be a family of functions from I to some compact E ⊂ R d , uniformly converging to a : I → E. Let g : R d × R p → R d be a C 1 function such that g(0, •) ≡ 0 and g(•, θ) is uniformly Lipschitz continuous for θ in any compact of R p . Consider the discrete scheme, for t ∈ I,

u L 0 (t) = a L (t) u L k+1 (t) = u L k (t) + 1 L g(u L k (t), θ L k+1 (t)), k ∈ {0, . . . , L -1}.
Then u L Ls (t) tends to U (s, t) uniformly over s ∈ [0, 1] and t ∈ I, where U (•, t) is the unique solution of the ODE

U (0, t) = a(t) ∂U ∂s (s, t) = g(U (s, t), Θ(s, t)), s ∈ [0, 1].
Moreover, the convergence only depends on the sequence (a L ) L∈N * and on its limit a ∈ E I through (sup t∈I a L (t)a(t) ) L∈N * .

Corollary 3.14. Let I ⊆ R + be an interval. Let (θ L k ) L∈N * ,1≤k≤L be a uniformly bounded family of functions from I to R p , and let

Θ L : [0, 1] × R + → R p , (s, t) → θ L (L-1)s +1 (t).
Assume that there exists a function Θ ∞ : [0, 1] → R p such that Θ L (s, t) tends to Θ ∞ (s) uniformly over s as L, t → ∞, and Θ ∞ is Lipschitz continuous. Let (a L ) L∈N * be a family of functions from I to some compact E ⊂ R d , and converging to a ∞ ∈ E as L, t → ∞. Let g : R d × R p → R d be a C 1 function such that g(0, •) ≡ 0 and g(•, θ) is uniformly Lipschitz continuous for θ in any compact of R p . Consider the discrete scheme, for t ∈ I,

u L 0 (t) = a L (t) u L k+1 (t) = u L k (t) + 1 L g(u L k (t), θ L k+1 (t)), k ∈ {0, . . . , L -1}.
Then u L Ls (t) tends to U (s) uniformly over s ∈ [0, 1] as L, t → ∞, where U is the unique solution of the ODE

U (0) = a ∞ dU ds (s) = g(U (s), Θ ∞ (s)), s ∈ [0, 1].
Moreover, the convergence only depends on the sequence (a L ) L∈N * and on its limit a ∈ E I through (sup t∈I a L (t)a(t) ) L∈N * .

3.A.5 Large-depth convergence of the gradient flow

This section is devoted to proving the main result of Appendix 3.A, namely the large-depth convergence of the gradient flow. The setting we consider encompasses both Section 3. (ii) There exists a Lipschitz continuous function

Z : [0, 1] × I → R p such that Z L : [0, 1] × I → R p , (s, t) → Z L (s, t) = Z L (L-1)s +1 (t)
converges uniformly over s ∈ [0, 1] and t ∈ I to Z(s, t).

(iii) Uniformly over s ∈ [0, 1], t ∈ I, and x ∈ X , the hidden layer h L Ls (t) converges to the solution at time s of the neural ODE

H(0, t) = A(t)x ∂H ∂s (s, t) = f (H(s, t), Z(s, t)), s ∈ [0, 1].
(iv) Uniformly over t ∈ I and x ∈ X , the output F L (x; t) converges to B(t)H(1, t).

Proof. According to Proposition 3.10, there exists a subsequence (Z φ(L) ) L∈N * of (Z L ) L∈N * and a Lipschitz continuous function

Z φ : [0, 1] × I → R p such that Z φ(L) (s, t) tends to Z φ (s, t)
uniformly over s and t. Similarly, by Corollary 3.11, there exists subsequences of (A L ) L∈N * and (B L ) L∈N * that converge uniformly. With a slight abuse of notation, we still denote these subsequences by φ, and the corresponding limits by A φ and B φ .

In the remainder, we prove the uniqueness of the accumulation point (Z φ , A φ , B φ ) by showing that it is the solution of an ODE that satisfies the assumptions of the Picard-Lindelöf theorem. The statements (i) to (iv) then follow easily.

Consider a general input (x, y) ∈ X × Y, and let H L (s, t) = h L Ls (t) (recall that h L k (t) is defined by the forward propagation (3.12)). Corollary 3.13, with θ ) x, g = f , ensures that H φ(L) (s, t) converges uniformly (over s and t) to H φ (s, t) that is the solution at time s of the ODE

L k = Z φ(L) k , Θ = Z φ , a L = A φ(L
H φ (0, t) = A φ (t)x ∂H φ ∂s (s, t) = f (H φ (s, t), Z φ (s, t)), s ∈ [0, 1].
By inspecting the proof of the corollary, we also have that (h

φ(L) k
) L∈N * ,1≤k≤φ(L) and (H φ(L) ) L∈N * are uniformly bounded and that H φ (•, t) is uniformly Lipschitz continuous for t ∈ I.

We now turn our attention to the backpropagation recurrence (3.13), which defines the backward state p L k (t). First observe that the convergence of H φ(L) implies that

p φ(L) φ(L) (t) = 2B φ(L) (t) (B φ(L) (t)h φ(L) φ(L) (t) -y) = 2B φ(L) (t) (B φ(L) (t)H φ(L) (1, t) -y) converges uniformly to 2B φ (t) (B φ (t)H φ (1, t) -y) ∈ R d . Now, let P L (s, t) = p L Ls (t)
. We apply again Corollary 3.13, this time to the backpropagation recurrence (3.13), with θ L k = (h

φ(L) k , Z φ(L) k ), Θ = (H φ , Z φ ), g : (p, (h, Z)) → ∂ 1 f (h, Z)p, and a L = 2(B φ(L) ) (B φ(L) H φ(L) (1, •) -y).
Let us quickly check that the conditions of the corollary are met:

• The sequence (h

φ(L) k
) L∈N * ,1≤k≤φ(L) is bounded, as noted previously, and the same holds for

(Z φ(L) k
) L∈N * ,1≤k≤φ(L) by the assumptions of Theorem 3.15.

• The function H φ (•, t) is uniformly Lipschitz continuous for t ∈ I, as noted previously, and the same is true for Z φ (•, t) since Z φ is Lipschitz continuous.

• The function h φ(L)

(φ(L)-1)s +1 (t) tends to H φ (s, t) uniformly over s and t, as seen in the beginning of the proof. More precisely, we know that H φ(L) (s, t) = h φ(L) φ(L)s (t) tends to H φ (s, t). Simple algebra and the fact that two successive iterates of (3.12) are separated by a distance proportional to 1/L show that both statements are equivalent. Furthermore, Z φ(L) (s, t) tends to Z φ (s, t) uniformly over s and t as noted above.

• The sequence (a L ) L∈N * is uniformly bounded, since B φ(L) and H φ(L) (1, •) are. It also converges uniformly to a : t → 2B φ (t) (B φ (t)H φ (1, t)y).

• The function g is C 1 since f is C 2 . We clearly have g(0, •) ≡ 0. Finally, g(•, (h, Z)) is uniformly Lipschitz continuous for (h, Z) in any compact since ∂ 1 f is continuous.

Overall, we obtain that P φ(L) (s, t) converges uniformly (over s and t) to P φ (s, t), the solution at time s of the backward ODE

P φ (1, t) = 2B φ (t) (B φ (t)H φ (1, t) -y) ∂P φ ∂s (s, t) = ∂ 1 f (H φ (s, t), Z φ (s, t))P φ (s, t), s ∈ [0, 1].
Furthermore, the proof of the corollary shows that (P φ(L) ) L∈N * is uniformly bounded. Now, recall that the gradient flow for Z φ(L) k

(t), given by (3.5) and (3.15), takes the following form, for

t ∈ I and k ∈ {1, . . . , φ(L)}, ∂Z φ(L) k (t) ∂t = π - 1 n n i=1 ∂ 2 f (h φ(L) k-1,i (t), Z φ(L) k (t)) p φ(L) k-1,i (t) ,
where the i subscript corresponds to the i-th input x i . By definition, for s ∈

[0, 1], Z φ(L) (s, t) = Z φ(L)
(φ(L)-1)s +1 (t). Thus, the equation above can be rewritten, for s ∈ [0, 1] and t ∈ I,

∂Z φ(L) (s, t) ∂t = π - 1 n n i=1 ∂ 2 f (h φ(L) (φ(L)-1)s ,i (t), Z φ(L) (φ(L)-1)s +1 (t)) p φ(L) (φ(L)-1)s ,i (t) . (3.37)
The term inside π can be rewritten as (s, t) converge uniformly to H φ i (s, t) and P φ i (s, t). Therefore, the right-hand side of (3.37) converges uniformly over s and t to

- 1 n n i=1 ∂ 2 f H φ(L) i (φ(L) -1)s φ(L) , t , Z φ(L) (s, t) P φ(L) i (φ(L) -1)s φ(L) , t . Since f is C 2 , ∂ 2 f
π - 1 n n i=1 ∂ 2 f (H φ i (s, t), Z φ (s, t)) P φ i (s, t) .
We have just shown the uniform convergence of the derivative in t of Z φ(L) (s, t). Furthermore, we know that, for s ∈ [0, 1], the sequence (t → Z φ(L) (s, t)) L∈N * converges to Z φ (s, •). These two statements imply that Z φ is differentiable with respect to t and that, for s ∈ [0, 1], its derivative satisfies the ordinary differential equation

∂Z φ (s, t) ∂t = π - 1 n n i=1 ∂ 2 f (H φ i (s, t), Z φ (s, t)) P φ i (s, t) . (3.38)
Moreover, by our initialization scheme,

Z φ (s, 0) = Z init (s). (3.39)
A similar approach reveals that A φ (t) and B φ (t) are differentiable and that they verify the equations

dA φ dt (t) = π - 1 n n i=1 P φ i (0, t)x i , A φ (0) = A init , (3.40) dB φ dt (t) = π - 2 n n i=1 (B φ (t)H φ i (1, t) -y i )H φ i (1, t) , B φ (0) = B init . (3.41)
The equations (3.38) to (3.41) can be seen as an initial value problem whose variables are the function Z φ (•, t) : [0, 1] → R p and the matrices A φ (t) ∈ R q×d , B φ (t) ∈ R d ×q . To complete the proof, it remains to show, using the Picard-Lindelöf theorem (see Lemma 3. [START_REF] Woodworth | Kernel and rich regimes in overparametrized models[END_REF], that there exists a unique solution to this problem. First, note that the space B([0, 1], R p ) of bounded functions from [0, 1] to R p endowed with the supremum norm is a Banach space, which is the proper space in which to apply the Picard-Lindelöf theorem. We therefore endow the space of parameters

B([0, 1], R p ) × R q×d × R d ×q with the norm (Z, A, B) := sup s∈[0,1] Z(s) + A 2 + B 2 ,
which makes it a Banach space. We have to show that the mapping

(Z, A, B) → s → π - 1 n n i=1 ∂ 2 f (H i (s), Z(s)) P i (s) , π - 1 n n i=1 P i (0)x i , π - 2 n n i=1 (BH i (1) -y i )H i (1) (3.42)
is locally Lipschitz continuous with respect to this norm, where we recall that H i (s) in (3.42) is the solution at time s of the initial value problem

H i (0) = Ax i dH i ds (s) = f (H i (s), Z(s)), s ∈ [0, 1], (3.43)
and P i (s) is the solution at time s of the initial value problem

P i (1) = 2B (BH i (1) -y i ) dP i ds (s) = ∂ 1 f (H i (s), Z(s))P i (s), s ∈ [0, 1]. (3.44)
To prove that the mapping (3.42) is locally Lipschitz continuous, we first check that it is well defined. Since Z is assumed to be only bounded (and not continuous), the solutions of the initial value problems (3.43) and (3.44) are well defined in the sense of the Caratheodory conditions, which are given in Lemma 3.20.

Next, we can show that (Z, A, B) → H i is locally Lipschitz continuous for i ∈ {1, . . . , n}. To do this, consider two sets of parameters (Z, A, B) and ( Z, Ã, B) belonging to a compact set D. Let H i and Hi denote the corresponding hidden states. As in the proof of Proposition 3.12, it holds that H i and Hi belong to some compact set E that depends only on D and f . Let K f be the Lipschitz constant of the C 1 function f on E × D. Then,

Hi (s) -H i (s) ≤ Hi (0) -H i (0) + s 0 d Hi dr (r) - dH i dr (r) dr ≤ Hi (0) -H i (0) + s 0 f ( Hi (r), Z(r)) -f (H i (r), Z(r)) dr.
The norm inside the integral can be bounded by

f ( Hi (r), Z(r)) -f ( Hi (r), Z(r)) + f ( Hi (r), Z(r)) -f (H i (r), Z(r)) ≤ K f sup r∈[0,1] Z(r) -Z(r) + K f Hi (r) -H i (r) .
Therefore,

Hi (s) -H i (s) ≤ Ã -A 2 x i + K f sup r∈[0,1] Z(r) -Z(r) + s 0 K f Hi (r) -H i (r) dr.
Using Grönwall's inequality, we obtain, for any s ∈ [0, 1],

Hi (s) -H i (s) ≤ Ã -A 2 x i + K f sup r∈[0,1] Z(r) -Z(r) exp(K f ).
This shows that the function (Z, A, B) → H i is locally Lipschitz continuous. One proves by similar arguments that the function (Z, A, B) → P i is locally Lipschitz continuous. Thus, overall, the mapping (3.42) is locally Lipschitz continuous as a composition of locally Lipschitz continuous functions.

The Picard-Lindelöf theorem guarantees the uniqueness of the maximal solution of the initial value problem (3.38)-(3.41) in the space B([0, 1], R p ) × R d×q × R d ×q . Since any accumulation point (Z φ , A φ , B φ ) is a solution belonging to this space, this proves the uniqueness of the accumulation point, which we therefore denote as (Z, A, B).

The uniform convergence of (Z L , A L , B L ) to (Z, A, B) is then easily shown by contradiction. Suppose that uniform convergence does not hold. If this is true, then there exists a subsequence that stays at distance ε > 0 from (Z, A, B) (in the sense of the uniform norm). Then arguments similar to the beginning of the proof show the existence of a second accumulation point, which is a contradiction. This shows the uniform convergence, yielding statements (i) and (ii) of the theorem.

Finally, reapplying Corollary 3.13 with θ L k = Z L k , Θ = Z, a L = A L x, g = f , completes the proof by proving statements (iii) and (iv).

Training dynamics of the limiting weights. Interestingly, the proof of Theorem 3.15 provides us with an explicit description of the evolution of the continuous-depth limiting weights during training. With the notation of the proof, the continuous weights satisfy the training dynamics:

dA dt (t) = π - 1 n n i=1 P i (0, t)x i ∂Z ∂t (s, t) = π - 1 n n i=1 ∂ 2 f (H i (s, t), Z(s, t)) P i (s, t) dB dt (t) = π - 2 n n i=1 (B(t)H i (1, t) -y i )H i (1, t) ,
where we recall that H i (s, t) is the solution at time s of the initial value problem

H i (0, t) = A(t)x i ∂H i ∂s (s, t) = f (H i (s, t), Z(s, t)), s ∈ [0, 1],
and P i (s, t) is the solution at time s of the problem

P i (1, t) = 2B(t) (B(t)H i (1, t) -y i ) ∂P i ∂s (s, t) = ∂ 1 f (H i (s, t), Z(s, t))P i (s, t), s ∈ [0, 1].
These equations can be thought of as the continuous-depth equivalent of the backpropagation equations.

3.A.6 Existence of the double limit when L, t tend to infinity Proposition 3.16. Consider the residual network (3.12), and assume that:

(i) A L (t), Z L Ls (t)
, and B L (t) converge uniformly over L ∈ N * and s ∈ [0, 1] as t → ∞.

(ii) A L (t), Z L Ls (t), and B L (t) converge uniformly over t ∈ R + and s ∈ [0, 1] as L → ∞.

(iii) The loss L (t) converges to 0 uniformly over L ∈ N * as t → ∞.

Then the following four statements hold as t and L tend to infinity:

(i) There exist matrices A ∞ ∈ R q×d and B ∞ ∈ R d ×q such that A L (t) and B L (t) converge to A ∞ and B ∞ .
(ii) There exists a Lipschitz continuous function

Z ∞ : [0, 1] → R p such that Z L Ls (t) converges to Z ∞ (t) uniformly over s ∈ [0, 1].
(iii) Uniformly over s ∈ [0, 1] and x ∈ X , the hidden layer h L Ls (t) converges to the solution at time s of the ODE

H(0) = A ∞ x dH ds (s) = f (H(s), Z ∞ (s)), s ∈ [0, 1]. (3.45) (iv) Uniformly over x ∈ X , the output F L (x; t) converges to F ∞ (x) = B ∞ H(1). Furthermore, F ∞ (x i ) = y i for i ∈ {1, . . . , n}.
Proof. The existence of limits A ∞ and B ∞ to A L (t) and B L (t) as L and t tend to infinity is given by Lemma 3.22. The same argument applies to Z L sL (t), which provides a limit Z ∞ (s) to the sequence. Furthermore, following the proof of the lemma, we see that the convergence of

Z L sL (t) to Z ∞ (s) is uniform over s ∈ [0, 1]. Corollary 3.14, applied with θ L k = Z L k , Θ ∞ = Z ∞ , a L = A L x, g = f , then ensures that h L
Ls (t) converges uniformly (over s ∈ [0, 1] and x ∈ X ) to H(s) that is the solution at time s of (3.45), as L and t tend to infinity. As a consequence, F L (x; t) converges uniformly over x to F ∞ (x) as L, t → ∞. Furthermore, recall that

L (t) = 1 n n i=1 F L (x i ; t) -y i 2 2 .
The left-hand side converges as L, t → ∞ to 0 by assumption of the proposition, while the right-hand side converges to

1 n n i=1 F ∞ (x i ) -y i 2 2 .
Therefore, F ∞ (x i ) = y i for i ∈ {1, . . . , n}, and the proof is complete.

3.B Proofs of the results of the main part of the chapter

Most of the results follow from those presented in Section 3.A. The only substantial proof is that of Proposition 3.6, which shows the local PL condition. It uses a result of Nguyen and Mondelli (2020) involving the Hermite transform and the sub-Gaussian variance proxy, which we define briefly. We refer to Debnath and Bhatta (2014, Chapter 17) and Vershynin (2018, Sections 2.5.2 and 3.4.1), respectively, for more detailed explanations.

Hermite transform. The r-th normalized probabilist's Hermite polynomial is given by

h r (x) = 1 √ r! (-1) r e x 2 /2 d r dx r e -x 2 /2 , r ≥ 0.
This family of polynomials forms an orthonormal basis of square-integrable functions for the inner product

f 1 , f 2 = 1 √ 2π ∞ -∞ f 1 (x)f 2 (x)e -x 2 /2 dx.
Therefore, any function σ such that 1 √ 2π ∞ -∞ σ 2 (x)e -x 2 /2 dx < ∞ can be decomposed on this basis. The r-th coefficient of this decomposition is denoted by η r (σ). Sub-Gaussian random vector. A random vector x ∈ R d is sub-Gaussian with variance proxy v x > 0 if, for every y ∈ R d of unit norm,

P(| x, y | ≥ t) ≤ 2 exp - t 2 2v 2 x .
Additional notation. For a matrix A, we let s min and s max its minimum and maximum singular values, and similarly, λ min and λ max its minimum and maximum eigenvalues (whenever they exist).

Before delving into the proofs, we briefly describe the parts of this section that make use of the specific model (3.3). The most important one is the proof of Proposition 3.6, i.e., the proof that the residual network satisfies the (M, µ)-local PL condition. Additionally, in the proof of Proposition 3.3, the expressions for M and K are valid only for the specific model (3.3). Finally, in the proof of Theorem 3.7, the beginning of the proof reveals that condition (3.22) of Proposition 3.9 on µ can be expressed as a condition on the norm of the labels y i . This applies only to the specific model (3.3). Observe that, if one assumes that the general residual network of Section 3.A satisfies the (M, µ)-local PL condition with µ given by (3.22), then the rest of the proof of Theorem 3.7 unfolds, and the conclusions of the theorem hold for the general model.

3.B.1 Proof of Proposition

3.1 Proposition 3.1 is a consequence of Proposition 3.8 with f (h, (V, W )) = 1 √ m V σ( 1 √ q W h).

3.B.2 Proof of Proposition 3.2

Proposition 3.12, with θ

L k = (V L k , W L k ), Θ = (V, W), a L = Ax, g(h, (V, W )) = 1 √ m V σ( 1 √ q W h),
gives the existence and uniqueness of the solution of the neural ODE (3.6). Moreover, inspecting the proof of Proposition 3.12, equations (3.35) gives that, for any input x ∈ X , the difference between the last hidden layer h L L of the discrete residual network (3.3) and its continuous counterpart H(1) in the neural ODE (3.6) is bounded by

C 1 L + sup s∈[0,1] Θ(s) -Θ L (s) ,
where C > 0 is independent of L and x ∈ X , and Θ L (s) = θ L (L-1)s +1 . The function Θ L is a piecewise-constant interpolation of Θ with pieces of length 1 L-1 . Since Θ is Lipschitz continuous, the distance between Θ and Θ L decreases as C /L for some C > 0 depending on Θ but not on L.

This yields h L L -H(1) ≤ C (1++C )

L

, where C and C are independent of L and x ∈ X . Since F L (x) = Bh L L and F (x) = BH(1), the result is proven.

3.B.3 Proof of Proposition 3.3

We apply Proposition 3.8 with f (h, (V,

W )) = 1 √ m V σ( 1 √ q W h).
Recall that the parameters Z = (V, W ) are considered in Proposition 3.8 as a vector. In particular, Z = V F + W F . Therefore, Proposition 3.8 shows that, for t ∈ [0, T ], L ∈ N * , and k ∈ {1, . . . , L},

A L (t) F ≤ M, V L k (t) F + W L k (t) F ≤ M, and B L (t) F ≤ M,
where

M = M 0 + T M π M 0 = max A L (0) F , V L 0 (0) F + W L 0 (0) F , B L (0) F M π = max max A∈R q×d π(A) F , max Z∈R q×m ×R m×q π(Z) , max B∈R d ×q π(B) F .
Furthermore, due to our initialization scheme described in Section 3.3,

A L (0) F = √ d, V L 0 (0) F = 0, W L 0 (0) F ≤ 2 √ qm, B L (0) F = √ d ,
where the third inequality holds with probability at least 1-exp(-3qm 16 ) by Lemma 3.23. Since we take q ≥ max(d, d ), this implies that, with high probability, M 0 ≤ 2 √ qm, yielding the formula for M in Proposition 3.3. Finally, the existence of K = βT e αT such that the difference between two successive weight matrices is bounded by K /L, as well as the dependence of α and β on X , Y, M , and σ, follows easily from Proposition 3.8, given that our initialization scheme ensures that Z L k (0) = Z L k+1 (0) for all L ∈ N * and k ∈ {1, . . . , L}.

3.B.4 Proof of Theorem 3.4

By Proposition 3.3 and the fact that π is bounded, the sequences (A L ) L∈N * and (B L ) L∈N * each satisfy the assumptions of Corollary 3.11, and (Z L k ) L∈N * ,1≤k≤L satisfies the assumptions of Proposition 3.10. Theorem 3.4 then follows directly from Theorem 3.15, by taking, as previously,

f (h, (V, W )) = 1 √ m V σ( 1 √ q W h).

3.B.5 Proof of Proposition 3.6

We drop the L superscripts for this proof, since L is fixed. Denote by Ā, B, Vk , Wk parameters sampled according to the initialization scheme of Section 3.3, which means in particular that Vk = 0 and Wk = W ∼ N ⊗(m×q) . Since, by assumption, the activation function σ is bounded and not constant, it cannot be a polynomial function. As a consequence, there are infinitely many non-zero coefficients η r (σ) in its Hermite expansion (defined at the beginning of Section 3.B). Throughout, we let r ≥ 2 be an integer such that η r (σ) is nonzero. We also let K σ be the Lipschitz constant of σ and M σ its supremum norm. Now, let A, B, V k , W k be parameters at distance at most M = min( ηr(σ) 32Kσ √ 2nq , 1 2 ) from Ā, B, Vk , Wk in the sense of Definition 3.5. It is useful for this proof to introduce a matrix-valued version of the residual network (3.3). More specifically, given data matrices x ∈ R d×n and y ∈ R d ×n , the matrix-valued residual network writes

h 0 = Ax h k+1 = h k + 1 L √ m V k+1 σ 1 √ q W k+1 h k , k ∈ {0, . . . , L -1}, (3.46)
where now h k ∈ R q×n . The loss is equal to = 1 n Bh Ly 2 F and we let p k = ∂ ∂h k ∈ R q×n be the matrix-valued backward state. Observe that the columns of x are bounded and thus sub-Gaussian. In the sequel, we denote by v x the sub-Gaussian variance proxy of the columns of d /qx. Now that we have introduced the necessary notation, we can proceed to prove some preliminary estimates. Since M ≤ 1 2 ≤ √ 2qm, we have, for k ∈ {1, . . . , n},

A -Ā F ≤ M, B -B F ≤ 1 2 , V k F ≤ 1, W k -W F ≤ 1 2 ≤ 2qm. ( 3
A 2 ≤ 2, s min (B) ≥ 1 2 , B 2 ≤ 3 2 , V k F ≤ 1, W k F ≤ 2 2qm, (3.48)
where the second inequality is a consequence of Lemma 3.21, as follows:

s min (B) ≥ s min ( B) -B -B F = 1 -B -B F ≥ 1 2 .
Let us now bound h k F and p k F . We have

h 0 F = Ax F ≤ A 2 x F ≤ 2 √ qn. (3.49)
Moreover, by (3.46), for any k ∈ {0, . . . , L -1},

h k+1 F ≤ h k F + K σ L √ m √ q V k+1 F W k+1 F h k F ≤ 1 + 2 √ 2K σ L h k F ,
where the second inequality is a consequence of (3.48). Therefore, by (3.49),

h k F ≤ exp(2 √ 2K σ ) h 0 F ≤ 2 exp(2 √ 2K σ ) √ qn. (3.50)
Moving on to p k F , the chain rule leads to

p k = p k+1 + 1 L √ qm W k+1 (V k+1 p k+1 ) σ 1 √ q W k+1 h k , k ∈ {0, . . . , L -1},
where denotes the element-wise product. Noting that |σ | ≤ K σ and using (3.48), we obtain

p k F ≥ p k+1 F - K σ L √ qm W k+1 F V k+1 F p k+1 F ≥ 1 - 2 √ 2K σ L p k+1 F . It follows that p k F ≥ exp(-2 √ 2K σ ) p L F . In addition, p L = ∂ ∂h L = 2 n B (Bh L -y).
Therefore, by Lemma 3.21, since d ≤ q,

p L F ≥ 2 n s min (B) Bh L -y F ≥ 1 √ n √ .
Collecting bounds, we conclude that, for k ∈ {0, . . . , L},

p k F ≥ 1 √ n exp(-2 √ 2K σ ) √ .
(3.51)

A similar proof reveals that, for k ∈ {0, . . . , L},

p k F ≤ 3 √ n exp(2 √ 2K σ ) √ .
Having established these preliminary estimates, our goal in the remainder of the proof is to lower bound the quantity ∂ ∂V k+1 F . First note that, by the chain rule, for any k ∈ {0, . . . , L -1},

∂ ∂V k+1 = 1 L √ m p k+1 σ 1 √ q W k+1 h k .
As a consequence, when m ≥ n, by Lemma 3.21,

∂ ∂V k+1 F ≥ 1 L √ m p k+1 F • s min σ 1 √ q W k+1 h k ≥ 1 L √ mn exp(-2 √ 2K σ ) √ • s min σ 1 √ q W k+1 h k , (3.52)
using (3.51). Next, by Lemma 3.21,

s min σ 1 √ q W k+1 h k ≥ s min σ 1 √ q W Āx -σ 1 √ q W k+1 h k -σ 1 √ q W Āx F .
Let us first lower bound the first term. Since, by our choice of initialization, Ā = (I R d×d , 0 R (q-d)×d ), we have

s min σ 1 √ q W Āx = s min (σ( W x)),
where W ∼ N (0, 1) ⊗(m×d) and x = 1 √ q x ∈ R d×n has i.i.d. unitary columns independent of W .

Therefore, by Lemma 3.24, with probability at least

1 -exp -3mη 2 r (σ) 64M 2 σ n -2n 2 exp -d 2vxn 2/r , s min σ 1 √ q W Āx ≥ √ mη r (σ) 4 .
Next,

σ 1 √ q W k+1 h k -σ 1 √ q W Āx F ≤ K σ √ q W k+1 -W F h k F + W F h k -Ax F + W F Ax -Āx F .
Clearly,

h k -Ax F = k j=1 1 L √ m V j σ 1 √ q W j h j-1 F ≤ 4 √ 2K σ k L exp(2 √ 2K σ ) √ qn,
by (3.48) and (3.50). Also,

Ax -Āx F ≤ A -Ā F x F ≤ η r (σ) 32 √ 2K σ ,
by (3.47) and by definition of M . Putting together the two bounds above as well as (3.47), (3.48), and (3.50), we obtain

σ 1 √ q W k+1 h k -σ 1 √ q W Āx F ≤ K σ exp(2 √ 2K σ ) √ n 1 + √ qm 8K σ k L + √ m η r (σ) 32 ≤ C 1 √ n + C 2 √ nqmk 16L + √ m η r (σ) 32 ,
where

C 1 = K σ exp(2 √ 2K σ ) and C 2 = 128C 1 K σ . Thus, when C 1 √ n ≤ 1 32 √ mη r (σ), we have s min σ 1 √ q W k+1 h k ≥ √ m 3 16 η r (σ) - C 2 16 √ nq k L ≥ 1 8 √ mη r (σ) for k ≤ Lηr(σ) C 2 √ nq . As a consequence, for k ≤ Lηr(σ) C 2 √ nq , returning to (3.52), ∂ ∂V k+1 F ≥ 1 8L √ n η r (σ) exp(-2 √ 2K σ ) √ = C 3 η r (σ) L √ n √ , letting C 3 = exp(-2 √ 2Kσ) 8
. Therefore,

∂ ∂A 2 F + L L k=1 ∂ ∂Z k+1 2 F + ∂ ∂B 2 F ≥ L Lηr (σ) C 2 √ nq k=1 ∂ ∂V k+1 2 F ≥ L Lη r (σ) C 2 √ nq C 2 3 η r (σ) 2 L 2 n ≥ C 2 3 η r (σ) 3 2C 2 n √ nq ,
where we used the inequality x ≥ x/2 for x ≥ 1. This proves the result, with

c 1 = max 2 10 C 2 1 η r (σ) 2 , 1 = max 2 10 K 2 σ exp(4 √ 2K σ ) η r (σ) 2
, 1

c 2 = C 2 η r (σ) = 128K 2 σ exp(2 √ 2K σ ) η r (σ) c 3 = min η r (σ) 32 √ 2K σ , 1 2
c 4 = C 2 3 η r (σ) 3 2C 2 = η r (σ) 3 2 14 K 2 σ exp(6 √ 2K σ ) δ = exp - qm 16 + n exp - 3mη 2 r (σ) 64M 2 σ n + 2n 2 exp - d 2v x n 2/r .
Remark 3.17. With appropriate values of r and m, the probability of failure δ can be made as small as

ε + 2n 2 exp - d 2v x n ε , (3.53)
for any ε > 0. This is possible first by choosing r such that 2/r ≥ ε, then by choosing m such that the first two terms are less than ε. Moreover, we refer the interested reader to Goel et al. (2020, Lemmas A.2 and A.9) for quantitative estimates of η r (σ) for ReLU and sigmoid activations.

Finally, the expression (3.53) is essentially the same as the one appearing in Nguyen and Mondelli (2020, Theorem 3.3). As in this chapter, we note that this expression is small if n grows at most polynomially with d, in which case the exponential term in d dominates the polynomial term in n.

3.B.6 Proof of Theorem 3.7

By Proposition 3.6, there exists δ > 0 such that, with probability at least 1δ, the residual network (3.3) satisfies the (M, µ)-local PL condition around its initialization, with

M = c 3 √ nq and µ = c 4 n √ nq ,
for c 3 and c 4 depending on σ. We now apply Proposition 3.9 with f (h, (V,

W )) = 1 √ m V σ( 1 √ q W h).
The only assumption of Proposition 3.9 that requires some care to check is that the PL condition holds for the value of µ given by equation (3.22). Since the (M, µ)-local PL condition implies the (M, μ)-local PL condition for any μ ∈ (0, µ), it is the case if

c 4 n √ nq ≥ max(M B K, M B M X , M A M X ) 8e K M sup L∈N * L (0),
with M X , M A , M B , and K defined in Proposition 3.9. Due to the initialization scheme of Section 3.3, we have, for any input

x ∈ X , h L L (0) = h L 0 (0), hence F L (x) = B L (0)A L (0)x = 0 since q ≥ d + d . As a consequence, L (0) = 1 n n i=1 y i 2 .
Therefore, the condition becomes

1 n n i=1 y i 2 ≤ c 2 3 c 2 4 64n 4 q 3 max(M B K, M B M X , M A M X ) 2 e 2K ,
where we replaced M by its value. Define C to be equal to the constant on the right-hand side. Then, according to the above, as soon as 1 n n i=1 y i 2 ≤ C, we can apply Proposition 3.9, which gives several guarantees. First, the gradient flow is well defined on R + . Moreover, the proposition and the expression of µ given above yield the bound on the empirical risk. In particular, the empirical risk converges uniformly to zero. Furthermore, Proposition 3.9 shows the uniform convergence of the weights as t → ∞. Finally, the proposition ensures that the sequences (A L ) L∈N * and (B L ) L∈N * each satisfy the assumptions of Corollary 3.11, and that (Z L k ) L∈N * ,1≤k≤L satisfies the assumptions of Proposition 3.10. We can therefore apply Theorem 3.15, with f defined above and π equal to the identity. This gives the uniform convergence of the weights as L → ∞. The four asymptotic statements of Theorem 3.7 are then a consequence of Proposition 3.16.

Remark 3.18. A close examination of the quantities involved in the definition of C reveals that it depends only on X , σ, n, and q. In particular, it does not depend on the dimension m.

3.C Some technical lemmas

We start by recalling the Picard-Lindelöf theorem (see, e.g., Luk, 2017, for a self-contained presentation, and Arnold, 1992, for a textbook). where g : R d × I → R d is continuous and locally Lipschitz continuous in its first variable. Then the initial value problem is well defined on an interval [0, T max ) ⊂ I, i.e., there exists a unique maximal solution on this interval. Moreover, if T max < T , then U (s) tends to infinity when s tends to T max . Finally, if g(•, r) is uniformly Lipschitz continuous for r in any compact, then T max = T .

We define time-dependent dynamics (3.54) for generality, but the time-independent case U (s) = U 0 + s 0 g(U (r))dr is also of interest. In this case, the existence and uniqueness of the maximal solution holds if g is locally Lipschitz continuous, and the solution is defined on I if g is Lipschitz continuous. Besides, the first statement of Lemma 3.19 (existence and uniqueness of the maximal solution) also holds if R d is replaced by any (potentially infinite-dimensional) Banach space.

The next lemma gives conditions for the existence and uniqueness of the global solution of the initial value problem (3.54) when the assumption of continuity of g in its second variable is removed, thereby generalizing the Picard-Lindelöf theorem.

Lemma 3.20 (Caratheodory conditions for the existence and uniqueness of the global solution of an initial value problem). Consider the initial value problem

U (s) = U 0 + s 0 g(U (r), r)dr, s ∈ [0, 1],
where g : R d × [0, 1] → R d is measurable and the integral is understood in the sense of Lebesgue integration. Assume that g(•, r) is uniformly Lipschitz continuous for almost all r ∈ [0, 1], and that g(0, r) ≡ 0. Then there exists a unique solution to the initial value problem, defined on [0, 1].

Proof. The proof is a consequence [START_REF] Filippov | Differential Equations with Discontinuous Righthand Sides[END_REF] Theorems 1, 2, and4). More specifically, denote by C > 0 the uniform Lipschitz constant of g(•, r). According to Filippov (1988, Theorems 1 and 2), under the conditions of the lemma, there exists a unique maximal solution to the initial value problem. Let us now consider a restricted version of the problem, where g is defined on D × [0, 1], with D a compact of R d large enough to contain in its interior the ball of center 0 and radius U 0 exp(C). There exists a unique maximal solution to this problem as well, also according to Filippov (1988, Theorems 1 and 2), and, according to Filippov (1988, Theorem 4), it is defined until it reaches the boundary of D × [0, 1], which it reaches at some point (U * , s * ). If s * < 1, it means that U * is on the boundary of D, and in particular that U * > U 0 exp(C). But, on the other hand, for almost every r ∈ [0, 1], g(U (r), r) ≤ g(0, r) + g(U (r), r)g(0, r) ≤ C U (r) .

Hence, by Grönwall's inequality, for s ≤ s * ,

U (s) ≤ U 0 exp(C).
Thus, U * ≤ U 0 exp(C), which is impossible. Hence the maximal solution of the restricted problem is defined on [0, 1]. Furthermore, the maximal solution of the original problem coincides with the restricted one whenever U (s) ∈ D, which is the case for every s ∈ [0, 1], hence the maximal solution is defined on [0, 1].

The next three lemmas recall well-known results from linear algebra, analysis, and random matrix theory. Recall that s min and λ min denote respectively the minimum singular value and eigenvalue of a matrix.

Lemma 3.21. Let A, A ∈ R m×r and B ∈ R r×n . Then

s min (A + A ) ≥ s min (A) -A F . If m ≥ r, then AB F ≥ s min (A) B F . Furthermore, if n ≥ r, then AB F ≥ A F s min (B).
Proof. The first statement is a consequence of, e.g., [START_REF] Loyka | On singular value inequalities for the sum of two matrices[END_REF], which establishes that s min (A + A ) ≥ s min (A)s max (A ), yielding the first inequality since s max (A ) = A 2 ≤ A F . As for the second one, we have

AB 2 F = Tr(ABB A ) = Tr(BB A A) ≥ λ min (A A)Tr(BB ) = λ min (A A) B 2 F .
Since m ≥ r, the rightmost quantity is equal to s min (A) B F , proving the second statement of the lemma. The third statement is similar.

Lemma 3.22. Let (e x,y ) x∈R + ,y∈R + ⊂ E, where E is a Banach space, such that e x,y converges uniformly to e ∞,y when x → ∞, and converges uniformly to e x,∞ when y → ∞. Then there exists e ∞ ∈ E such that Proof. Let ε > 0. Since e x,y converges uniformly to e ∞,y as x → ∞, there exists x 0 ∈ R + such that, for x 1 , x 2 > x 0 and y ∈ R + , e x 1 ,ye x 2 ,y ≤ ε 2 .

Similarly, there exists y 0 ∈ R + such that, for x ∈ R + and y 1 , y 2 > y 0 , e x,y 1e x,y 2 ≤ ε 2 .

Hence, for x 1 , x 2 > x 0 and y 1 , y 2 > y 0 , e x 1 ,y 1e x 2 ,y 2 ≤ e x 1 ,y 1e x 1 ,y 2 + e x 1 ,y 2e x 2 ,y 2 ≤ ε.

We conclude that (e x,y ) x∈R + ,y∈R + is a Cauchy sequence, which therefore converges to some limit e ∞ ∈ E.

Lemma 3.23. Let W ∈ R q×m be a standard Gaussian random matrix. Then, for M W √ 2,

with probability at least 1exp(-

(M 2 W -1)qm 16
), one has

W F ≤ M W √ q √ m.
Proof. The quantity W 2 F follows a chi-squared distribution with qm degrees of freedom. Hence, according to Laurent and Massart (2000, Lemma 1), for x ≥ 0,

P( W 2 F -qm ≥ 2 √ qmx + 2x) ≤ exp(-x). Taking x = (M 2 W -1)qm 16
, we see that

2 √ qmx = 1 2 M 2 W -1qm ≤ 1 2 (M 2 W -1)qm,
where the bound follows from M W √ 2. Since furthermore 2x ≤ 1 2 (M 2 W -1)qm, we obtain

2 √ qmx + 2x ≤ (M 2 W -1)qm,
and thus

P( W 2 F > M 2 W qm) ≤ P( W 2 F -qm ≥ 2 √ qmx + 2x) ≤ exp(-x),
yielding the result.

Finally, the last lemma of the section gives a lower bound on the smallest singular value of a matrix of the form σ(A), where σ is a bounded function applied element-wise and A belongs to a family of random matrix. The lower bound involves the Hermite transform of σ, which is defined in Section 3.B. Lemma 3.24. Let σ be a function bounded by some M σ > 0. Let W ∈ R m×d be a standard Gaussian random matrix, and X ∈ R d×n a random matrix with i.i.d. unitary columns independent of W . Then, for any integer r ≥ 2, there exists δ > 0 such that, with probability at least 1δ, the smallest singular value of σ(W X) is greater than 1 4 √ mη r (σ), where η r (σ) is the r-th coefficient in the Hermite transform of σ. Furthermore, the following expression for δ holds:

δ = n exp - 3mη 2 r (σ) 64M 2 σ n + 2n 2 exp - d 2Cn 2/r ,
where C is the sub-Gaussian variance proxy of the columns of √ dX.

Proof. Denoting by w i the i-th row of W and letting

M i = σ(X w i )σ(w i X),
our goal is to lower bound the smallest eigenvalue value λ min (M ) of M = m i=1 M i . Observe that

E(M |X) = mE w∼N (0,I d ) σ(X w )σ( wX) X = mE w∼N (0, 1 d I d ) σ ( √ dX) w σ w( √ dX) X .
Letting λ min (E(M |X)) be the smallest eigenvalue of this matrix and r ≥ 2 be an integer, Nguyen and Mondelli (2020, Lemma 3.4) show that, with probability at least 1 -2n 2 exp(-d 2Cn 2/r ) over the matrix X,

λ min (E(M |X)) ≥ mη 2 r (σ) 8 . (3.55)
We now apply a matrix Chernoff's bound to lower bound with high probability the smallest eigenvalue λ min (M |X) of M conditionally on X, as a function of λ min (E(M |X)). By Tropp (2012, Remark 5.3), we have, for t ∈ [0, 1],

P(λ min (M ) ≤ tλ min (E(M |X))|X) ≤ n exp - (1 -t 2 )λ min (E(M |X)) 2R(X) ,
where R(X) is an almost sure upper bound on the largest eigenvalue of M i |X, which we can take equal to M 2 σ n since the largest eigenvalue of

M i is equal to σ(w i X) 2 2 ≤ M 2 σ n. Taking t = 1/2, we obtain, on the event [λ min (E(M |X)) ≥ mη 2 r (σ) 8 ], P λ min (M ) ≥ λ min (E(M |X)) 2 X ≥ 1 -n exp - 3mη 2 r (σ) 64M 2 σ n , thus, on the event [λ min (E(M |X)) ≥ mη 2 r (σ) 8
],

P λ min (M ) ≥ mη 2 r (σ) 16 ≥ 1 -n exp - 3mη 2 r (σ) 64M 2 σ n .
Using (3.55), we obtain

P λ min (M ) ≥ mη 2 r (σ) 16 ≥ 1 -n exp - 3mη 2 r (σ) 64M 2 σ n P λ min (E(M |X)) ≥ mη 2 r (σ) 8 ≥ 1 -n exp - 3mη 2 r (σ) 64M 2 σ n 1 -2n 2 exp - d Cn 2/r ≥ 1 -n exp - 3mη 2 r (σ) 64M 2 σ n -2n 2 exp - d 2Cn 2/r .

3.D Counter-example for the ReLU case.

This section gives a proof sketch to illustrate that, with the ReLU activation σ : x → max(0, x), the smoothness of the weights can be lost during training. More precisely, we show a case where successive weights are at distance O( 1 L ) at initialization and at distance Ω(1) after training.

For the sake of simplicity, we will assume that the depth is even, and denote it as 2L. We place ourselves in a one-dimensional setting (i.e., d = 1). The parameters are (w 1 , • • • , w 2L ) ∈ R 2L , and the residual network writes as follows, for an input x ∈ R:

h 0 (t) = x h k+1 (t) = h k (t) + 1 2L σ(w k+1 (t)h k (t)), k ∈ {0, . . . , 2L -1}.
We consider a sample consisting of a single point (x, Cx) ∈ R 2 + , with C > 1 (independent of L), and define the empirical risk as (t) = (h 2L (t) -Cx) 2 . The risk is minimized by gradient flow.

The weights are initialized to w k (0) = (-1) k 2L . For x ∈ R + we have that h k (t) ≥ 0 for all k ∈ {0, . . . , 2L}. Note that the argument of σ on the odd layers is negative. Therefore, by definition of σ, the gradient of the loss with respect to the odd layers is zero and we have, for k ∈ {0, . . . , L -1}, w 2k+1 (t) = w 2k+1 (0). On the other hand, the argument of σ is positive on the even layers, and thus,

h 2L (t) = L j=1 1 + w 2j (t) 2L x.
As a consequence, the gradient flow equation for the even layers is, for k ∈ {1, . . . , L},

dw 2k dt (t) = - ∂ ∂w 2k (t) = 2x C - L j=1 1 + w 2j (t) 2L L j=1,j =k 1 + w 2j (t) 2L .
Due to the symmetry of these equations for k ∈ {1, . . . , L} and the fact that all the w 2k (0) are equal, the parameters on each even layer coincide at all times and are equal to w(t) such that

dw dt (t) = 2x C -1 + w(t) 2L L 1 + w(t) 2L L-1
.

An analysis of this ODE reveals that w(t) tends as t → ∞ to w > 0 satisfying that

1 + w 2L L = C. (3.56)
This can be seen by letting y(t) = C -(1 + w(t) 2L ) L , and applying Grönwall's inequality to y. Therefore, as t → ∞, one has w 2k+1 (t) → -1 2L and w 2k (t) → w , where (3.56) implies that w ≥ 2 log(C). This shows that the final weights are not smooth in the sense that the distance between two successive weights is Ω(1).

This result contrasts sharply with Proposition 3.8, which shows that successive weights remain at a distance O( 1 L ) throughout training, when initialized as a discretization of a Lipschitz continuous function, and with a smooth activation function. In fact, Proposition 3.8 can be generalized to any initialization such that successive weights are at distance O( 1 L ) at initialization, which is the case in the counter-example. This means that the only broken assumption in our counter-example is the non-smoothness of the activation function. This non-smoothness causes the gradient flow dynamics for two successive weights to deviate, even though the weights are initially close to each other, because they are separated by the kink of ReLU at zero.

3.E Experimental details

We use PyTorch (Paszke et al., 2019).

Large-depth limit. We take n = 100, d = 16, m = 32. We train for 500 iterations, and set the learning rate to L × 10 -2 . The scaling of the learning rate with L is the equivalent of the L factor in the gradient flow (3.4).

Long-time limit. We take n = 50, d = 16, m = 64, L = 64, and train for 80,000 iterations with a learning rate of 5L × 10 -3 .

Real-world data. We take L = 256. The first layer is a trainable convolutional layer with a kernel size of 5 × 5, a stride of 2, a padding of 1, and 16 out channels. We then iterate the residual layers

h L k+1 = h L k + 1 L bn L 2,k (conv L 2,k (σ(bn L 1,k (conv L 1,k (h L k ))))), k ∈ {0, . . . , L -1},
where conv L i,k are convolutions with kernel size 3, stride of 2, and padding of 1, and bn L i,k are batch normalizations, as is standard in residual networks (He et al., 2016a). The model is trained using stochastic gradient descent on the cross-entropy loss for 180 epochs. The initial learning rate is 4 × 10 -2 and is gradually decreased using a cosine learning rate scheduler. 119

Introduction

Neural ordinary differential equations (neural ODEs, Chen et al., 2018a) are a flexible family of neural networks used in particular to model continuous-time phenomena. Along with variants such as neural stochastic differential equations (neural SDEs, Tzen and Raginsky, 2019) and neural controlled differential equations (Kidger et al., 2020), they have been used in diverse fields such as pharmokinetics [START_REF] Lu | Neural-ODE for pharmacokinetics modeling and its advantage to alternative machine learning models in predicting new dosing regimens[END_REF]Qian et al., 2021), finance [START_REF] Gierjatowicz | Robust pricing and hedging via neural sdes[END_REF], and transportation [START_REF] Zhou | Urban flow prediction with spatial-temporal neural ODEs[END_REF]. We refer to Massaroli et al. (2020) for a self-contained introduction to this class of models.

Despite their empirical success, the statistical properties of neural ODEs have not yet been fully investigated. What is more, neural ODEs can be thought of as the infinite-depth limit of (properly scaled) residual neural networks (He et al., 2016a), a connection made by, e.g., E (2017); Haber and Ruthotto (2017); [START_REF] Lu | Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations[END_REF]. Since standard measures of statistical complexity of neural networks grow with depth (see, e.g., [START_REF] Bartlett | Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks[END_REF], it is unclear why infinite-depth models, including neural ODEs, should enjoy favorable generalization properties.

To better understand this phenomenon, our goal in this chapter is to study the statistical properties of a class of time-dependent neural ODEs that write

dH t = W t σ(H t )dt, (4.1)
where W t ∈ R d×d is a weight matrix that depends on the time index t, and σ : R → R is an activation function applied component-wise. Time-dependent neural ODEs were first introduced by Massaroli et al. ( 2020) and generalize time-independent neural ODEs

dH t = W σ(H t )dt, (4.2)
as formulated in Chen et al. (2018a), where W ∈ R d×d now denotes a weight matrix independent of t. There are two crucial reasons to consider time-dependent neural ODEs rather than the more restrictive class of time-independent neural ODEs. On the one hand, the time-dependent formulation is more flexible, leading to competitive results on image classification tasks [START_REF] Queiruga | Continuous-in-depth neural networks[END_REF][START_REF] Erichson | Lipschitz recurrent neural networks[END_REF]. As a consequence, obtaining generalization guarantees for this family of models is a valuable endeavor by itself. On the other hand, time dependence is required for the correspondence with general residual neural networks to hold. More precisely, the time-dependent neural ODE (4.1) is the limit, when the depth L goes to infinity, of the deep residual network

H k+1 = H k + 1 L W k+1 σ(H k ), 0 k L -1, (4.3)
where (W k ) 1≤k≤L ∈ R d×d are weight matrices and σ is still an activation function. We refer to Chapter 2 and to Sander et al. (2022b); Thorpe and van Gennip (2022) for statements that make precise under what conditions and in which sense this limit holds, as well as its consequences for learning. These two key reasons compel us to consider the class of time-dependent ODEs (4.1) for our statistical study, which in turn will inform us on the properties of the models (4.2) and ( 4.3).

In fact, we extend our study to the larger class of parameterized ODEs, which we define as the mapping from x ∈ R d to the value at time t = 1 of the solution of the initial value problem

H 0 = x, dH t = m i=1 θ i (t)f i (H t )dt, (4.4)
where H t is the variable of the ODE, θ i are functions from [0, 1] into R that parameterize the ODE, and f i are fixed functions from R d into R d . Time-dependent neural ODEs (4.1) are obtained by setting a specific entrywise form for the functions f i in (4.4), with m = d 2 (see Section 4.3.3 for details).

Since the parameters θ i belong to an infinite-dimensional space, in practice they need to be approximated in a finite-dimensional basis of functions. For example, the residual neural networks (4.3) can be seen as an approximation of the neural ODEs (4.1) on a piecewise constant basis of function. But more complex choices are possible, such as B-splines [START_REF] Yu | Neural generalized ordinary differential equations with layer-varying parameters[END_REF]. However, the formulation (4.4) is agnostic from the choice of finite-dimensional approximation. This more abstract point of view is fruitful to derive generalization bounds, for at least two reasons. First, the statistical properties of the parameterized ODEs (4.4) only depend on the characteristics of the functions θ i and not on the specifics of the approximation scheme, so it is more natural and convenient to study them at the continuous level. Second, their properties can then be transferred to any specific discretization, such as the deep residual networks (4.3), resulting in generalization bounds for the latter.

Regarding the characteristics of the functions θ i , we make the structural assumption that they are Lipschitz-continuous and uniformly bounded. This is a natural assumption to ensure that the initial value problem (4.4) has a unique solution in the usual sense of the Picard-Lindelöf theorem. Remarkably, this assumption on the parameters also enables us to obtain statistical guarantees despite the fact that we are working with an infinite-dimensional set of parameters.

Contributions. We provide a generalization bound for the large class of parameterized ODEs (4.4), which include time-dependent and time-independent neural ODEs (4.1) and ( 4.2). To the best of our knowledge, this is the first available bound for neural ODEs. By leveraging the connection between (time-dependent) neural ODEs and deep residual networks, our approach allows us to provide a depth-independent generalization bound for the class of deep residual networks (4.3). The bound is precisely compared with earlier results. Our bound depends in particular on the magnitude of the difference between successive weight matrices, which is, to our knowledge, a novel way of controlling the statistical complexity of neural networks. Numerical illustration is provided to show the relationship between this quantity and the generalization ability of neural networks.

Organization of the chapter. Section 4.2 presents additional related work. In Section 4.3, we specify our class of parameterized ODEs, before stating the generalization bound for this class and for neural ODEs as a corollary. The generalization bound for residual networks is presented in Section 4.4 and compared to other bounds, before some numerical illustration. Section 4.5 concludes the chapter. The proof technique is discussed in the main part of the chapter, but the core of the proofs is relegated to Section 4.A. Finally, Section 4.B contains the details of the numerical illustrations presented in Section 4.4.3.

Related work

Hybridizing deep learning and differential equations. The fields of deep learning and dynamical systems have recently benefited from sustained cross-fertilization. On the one hand, a large line of work is aimed at modeling complex continuous-time phenomena by developing specialized neural architectures. This family includes neural ODEs, but also physics-informed neural networks [START_REF] Raissi | Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations[END_REF], neural operators (Li et al., 2021a) and neural flows (Biloš et al., 2021). On the other hand, successful recent advances in deep learning, such as diffusion models, are theoretically supported by ideas from differential equations (Huang et al., 2021).

Generalization for continuous-time neural networks. Obtaining statistical guarantees for continuous-time neural networks has been the topic of a few recent works. For example, we consider in Chapter 5 a class of continuous-time recurrent neural networks (RNNs) that can be written as input-driven ODEs. They show that these models are actually kernel methods, which entails a generalization bound. [START_REF] Lim | Understanding recurrent neural networks using nonequilibrium response theory[END_REF] also show a generalization bound for ODE-like RNNs, and argue that adding stochasticity (that is, replacing ODEs with SDEs) helps with generalization. Yin et al. (2021) propose a neural ODE model to enable transfer learning across multiple environments and provide a generalization bound in this setting.

Lipschitz-based generalization bounds for deep neural networks. From a high-level perspective, our proof technique is similar to previous works (Bartlett et al., 2017;[START_REF] Neyshabur | A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks[END_REF] that show generalization bounds for deep neural networks, which scale at most polynomially with depth. More precisely, these authors show that the network satisfies some Lipschitz continuity property (either with respect to the input or to the parameters), then exploit results on the statistical complexity of Lipschitz function classes. Under stronger norm constraints, these bounds can even be made depth-independent [START_REF] Golowich | Size-independent sample complexity of neural networks[END_REF]. However, their approach differs from ours insofar as we consider neural ODEs and the associated family of deep neural networks, whereas they are solely interested in finite-depth neural networks. As a consequence, their hypotheses on the class of neural networks differ from ours. Section 4.4 develops a more thorough comparison. Similar Lipschitz-based techniques have also been applied to obtain generalization bounds for deep equilibrium networks [START_REF] Pabbaraju | Estimating lipschitz constants of monotone deep equilibrium models[END_REF]. Going beyond statistical guarantees, Béthune et al. ( 2022) study approximation and robustness properties of Lipschitz neural networks.

Generalization bounds for parameterized ODEs

We start by recalling the usual supervised learning setup and introduce some notation in Section 4.3.1, before presenting our parameterized ODE model and the associated generalization bound in Section 4.3.2. We then apply the bound to the specific case of time-invariant neural ODEs in Section 4.3.3.

Learning procedure

We place ourselves in a supervised learning setting. Let us introduce the notation that is used throughout the chapter (up to Section 4.4.1). The input data is a sample of n i.i.d. pairs (x i , y i ) with the same distribution as some generic pair (x, y), where x (resp. y) takes its values into some bounded ball

X = B(0, R X ) (resp. Y = B(0, R Y )) of R d , for some R X , R Y > 0.
This setting encompasses regression but also classification tasks by (one-hot) encoding labels in R d . Note that we assume for simplicity that the input and output have the same dimension, but our analysis easily extends to the case where they have different dimensions by adding (parameterized) projections at the beginning or at the end of our model. Given a parameterized class of models F Θ = {F θ , θ ∈ Θ}, the parameter θ is fitted by empirical risk minimization using a loss function : R d × R d → R + that we assume to be Lipschitz with respect to its first argument, with a Lipschitz constant K > 0. In the following, we write for the sake of conciseness that such a function is K -Lipschitz. We also assume that (x, x) = 0 for all x ∈ R d . The theoretical and empirical risks are respectively defined, for any θ ∈ Θ, by

R(θ) = E[ (F θ (x), y)] and R n (θ) = 1 n n i=1 F θ (x i ), y i ,
where the expectation E is evaluated with respect to the distribution of (x, y). Letting θ n a minimizer of the empirical risk, the generalization problem consists in providing an upper bound on the difference R( θ n ) -R n ( θ n ).

Generalization bound

Model. We start by making more precise the parameterized ODE model introduced in Section 4.1. The setup presented here can easily be specialized to the case of neural ODEs, as we will see in Section 4.3.3. Let f 1 , . . . , f m : R d → R d be fixed K f -Lipschitz functions for some K f > 0.

Denote by M their supremum on X (which is finite since these functions are continuous). The parameterized ODE F θ is defined by the following initial value problem that maps some

x ∈ R d to F θ (x) ∈ R d : H 0 = x dH t = m i=1 θ i (t)f i (H t )dt F θ (x) = H 1 , (4.5)
where the parameter θ = (θ 1 , . . . , θ m ) is a function from [0, 1] to R m . We have to impose constraints on θ for the model F θ to be well-defined. To this aim, we endow (essentially bounded) functions from [0, 1] to R m with the following (1, ∞)-norm

θ 1,∞ = sup 0 t 1 m i=1 |θ i (t)|. (4.6)
We can now define the set of parameters

Θ = {θ : [0, 1] → R m , θ 1,∞ R Θ and θ i is K Θ -Lipschitz for i ∈ {1, . . . , m}}, (4.7)
for some R Θ > 0 and K Θ 0. Then, for θ ∈ Θ, the following Proposition, which is a consequence of the Picard-Lindelöf Theorem, shows that the mapping x → F θ (x) is well-defined.

Proposition 4.1 (Well-posedness of the parameterized ODE). For θ ∈ Θ and x ∈ R d , there exists a unique solution to the initial value problem (4.5).

Proof. See Section 4.A.1.

An immediate consequence of Proposition 4.1 is that it is legitimate to take F Θ = {F θ , θ ∈ Θ} for our model class.

When K Θ = 0, the parameter space Θ is finite-dimensional since each θ i is constant. This setting corresponds to the time-independent neural ODEs of Chen et al. (2018a). In this case, the norm (4.6) reduces to the • 1 norm over R m . Note that, to fit exactly the formulation of Chen et al. (2018a), the time t can be added as a variable of the functions f i , which amounts to adding a new coordinate to H t . This does not change the subsequent analysis. In the richer time-dependent case where K Θ > 0, the set Θ belongs to an infinite-dimensional space and therefore, in practice, θ i is approximated in a finite basis of functions, such as Fourier series, Chebyshev polynomials, and splines. We refer to Massaroli et al. (2020) for a more detailed discussion, including formulations of the back-propagation algorithm (a.k.a. the adjoint method) in this setting.

Note that we consider the case where the dynamics at time t are linear with respect to the parameter θ i (t). Nevertheless, we emphasize that the mapping x → F θ (x) remains a highly nonlinear function of each θ i (t). To fix ideas, this setting can be seen as analogous to working with pre-activation residual networks instead of post-activation (see He et al., 2016b, for definitions of the terminology), which is a mild modification.

Statistical analysis. Since Θ is a subset of an infinite-dimensional space, complexity measures based on the number of parameters cannot be used. Instead, our approach is to resort to Lipschitz-based complexity measures. More precisely, to bound the complexity of our model class, we propose two building blocks: we first show that the model F θ is Lipschitz-continuous with respect to its parameters θ. This allows us to bound the complexity of the model class depending on the complexity of the parameter class. In a second step, we assess the complexity of the class of parameters itself.

Starting with our first step, we show the following estimates for our class of parameterized ODEs. Here and in the following, • denotes the 2 norm over R d . Proposition 4.2 (The parameterized ODE is bounded and Lipschitz). Let θ and θ ∈ Θ. Then, for any x ∈ X ,

F θ (x) R X + M R Θ exp(K f R Θ ) and F θ (x) -F θ(x) 2M K f R Θ exp(2K f R Θ ) θ -θ 1,∞ .
Proof. See Section 4.A.2.

The proof, given in the Appendix, makes extensive use of Grönwall's inequality, a standard tool to obtain estimates in the theory of ODEs, in order to bound the magnitude of the solution H t of (4.5).

The next step is to assess the magnitude of the covering number of Θ. Recall that, for ε > 0, the ε-covering number of a metric space is the number of balls of radius ε needed to completely cover the space, with possible overlaps. Proposition 4.3 (Covering number of the ODE parameter class). For ε > 0, let N (ε) be the ε-covering number of Θ endowed with the (1, ∞)-norm (4.6). Then

log N (ε) m log 16mR Θ ε + m 2 K Θ log(4) ε .
Proof. See Section 4.A.3.

Proposition 4.3 is a consequence of a classical result, see, e.g., Kolmogorov and Tikhomirov (1959, example 3 of paragraph 2). A self-contained proof is given in the Appendix for completeness. We also refer to [START_REF] Gottlieb | Efficient regression in metric spaces via approximate lipschitz extension[END_REF] for more general results on covering numbers of Lipschitz functions.

The two propositions above and an ε-net argument allow to prove the first main result of the chapter.

Theorem 4.4 (Generalization bound for parameterized ODEs). Consider the class of parameterized ODEs F Θ = {F θ , θ ∈ Θ}, where F θ is given by (4.5) and Θ by (4.7). Let δ > 0.

Then, for n ≥ 9 max(m -2 R -2 Θ , 1), with probability at least 1δ,

R( θ n ) R n ( θ n ) + B (m + 1) log(R Θ mn) n + B m √ K Θ n 1/4 + B √ n log 1 δ , where B is a constant depending on K , K f , R Θ , R X , R Y , M . More precisely, B = 6K K f exp(K f R Θ ) R X + M R Θ exp(K f R Θ ) + R Y .
Proof. See Section 4.A.4.

Three terms appear in our upper bound of R( θ n ) -R n ( θ n ). The first and the third ones are classical (see, e.g. Bach, 2023, Sections 4.4 and 4.5). On the contrary, the second term is more surprising with its convergence rate in O(n -1/4 ). This slower convergence rate is due to the fact that the space of parameters is infinite-dimensional. In particular, for K Θ = 0, corresponding to a finite-dimensional space of parameters, we recover the usual O(n -1/2 ) convergence rate, however at the cost of considering a much more restrictive class of models. Finally, it is noteworthy that the dimensionality appearing in the bound is not the input dimension d but the number of mappings m.

Note that this result is general and may be applied in a number of contexts that go beyond deep learning, as long as the instantaneous dependence of the ODE dynamics to the parameters is linear. One such example is the predator-prey model, describing the evolution of two populations of animals, which reads dx t = x t (αβy t )dt and dy t = -y t (γδx t )dt, where x t and y t are real-valued variables and α, β, γ and δ are model parameters. This ODE falls into the framework of this section, if one were to estimate the parameters by empirical risk minimization. We refer to Deuflhard and Röblitz (2015, section 3) for other examples of parameterized biological ODE dynamics and methods for parameter identification.

Nevertheless, for the sake of brevity, we focus on applications of this result to deep learning, and more precisely to neural ODEs, which is the topic of the next section.

Application to neural ODEs

As explained in Section 4.1, parameterized ODEs include both time-dependent and timeindependent neural ODEs. Since the time-independent model is more common in practice, we develop this case here and leave the time-dependent case to the reader (the result is actually given as Theorem 1.5 in Chapter 1). We thus consider the following neural ODE:

H 0 = x dH t = W σ(H t )dt F W (x) = H 1 , (4.8)
where W ∈ R d×d is a weight matrix, and σ : R → R is an activation function applied componentwise. We assume σ to be K σ -Lipschitz for some K σ > 0. This assumption is satisfied by all common activation functions. To put the model in the form of Section 4.3.2, denote e 1 , . . . , e d the canonical basis of R d . Then the dynamics (4.8) can be reformulated as

dH t = d i,j=1 W ij σ ij (H t )dt,
where σ ij (x) = σ(x j )e i . Each σ ij is itself K σ -Lipschitz, hence we fall in the framework of Section 4.3.2. In other words, the functions f i of our general parameterized ODE model form a shallow neural network with pre-activation. Denote by W 1,1 the sum of the absolute values of the elements of W . We consider the following set of parameters, which echoes the set Θ of Section 4.3.2:

W = {W ∈ R d×d , W 1,1 R W }, (4.9)
for some R W > 0. We can then state the following result as a consequence of Theorem 4.4.

Corollary 4.5 (Generalization bound for neural ODEs). Consider the class of neural ODEs F W = {F W , W ∈ W}, where F W is given by (4.8) and W by (4.9). Let δ > 0.

Then, for n ≥ 9R -1 W max(d -4 R -1 W , 1), with probability at least 1δ,

R( W n ) R n ( W n ) + B(d + 1) log(R W dn) n + B √ n log 1 δ ,
where B is a constant depending on K , K σ , R W , R X , R Y , M . More precisely,

B = 6 √ 2K K σ exp(K σ R W ) R X + M R W exp(K σ R W ) + R Y .
Proof. See Section 4.A.5.

Note that the term in O(n -1/4 ) from Theorem 4.4 is now absent. Since we consider a time-independent model, we are left with the other two terms, recovering a standard O(n -1/2 ) convergence rate.

Generalization bounds for deep residual networks

As highlighted in Section 4.1, there is a strong connection between neural ODEs and discrete residual neural networks. The previous study of the continuous case in Section 4.3 paves the way for deriving a generalization bound in the discrete setting of residual neural networks, which is of great interest given the pervasiveness of this architecture in modern deep learning.

We begin by presenting our model and result in Section 4.4.1, before detailing the comparison of our approach with other papers in Section 4.4.2 and giving some numerical illustration in Section 4.4.3.

Model and generalization bound

Model. We consider the following class of deep residual networks:

H 0 = x H k+1 = H k + 1 L W k+1 σ(H k ), 0 k L -1 F W (x) = H L , (4.10)
where the parameter W = (W k ) 1≤k≤L ∈ R L×d×d is a set of weight matrices and σ is still a K σ -Lipschitz activation function. To emphasize that W is here a third-order tensor, as opposed to the case of time-invariant neural ODEs in Section 4.3.3, where W was a matrix, we denote it with a bold notation. We also assume in the following that σ(0) = 0. This assumption could be alleviated at the cost of additional technicalities. Owing to the 1 /L scaling factor, the deep limit of this residual network is a (time-dependent) neural ODE of the form studied in Section 4.3. We refer to Chapter 2 for further discussion on the link between scaling factors and deep limits. We simply note that this scaling factor is not common practice, but preliminary experiments show it does not hurt performance and can even improve performance in a weight-tied setting (Sander et al., 2022b). The space of parameters is endowed with the following (1, 1, ∞)-norm

W 1,1,∞ = sup 1 k L d i,j=1 |W k,i,j |. (4.11)
Also denoting • ∞ the element-wise maximum norm for a matrix, we consider the class of matrices

W = W ∈ R L×d×d , W 1,1,∞ R W and W k+1 -W k ∞ K W L for 1 ≤ k ≤ L -1 , (4.12)
for some R W > 0 and K W 0, which is a discrete analogous of the set Θ defined by (4.7). In particular, the upper bound on the difference between successive weight matrices is to our knowledge a novel way of constraining the parameters of a neural network. It corresponds to the discretization of the Lipschitz continuity of the parameters introduced in (4.7). By analogy, we refer to it as a constraint on the Lipschitz constant of the weights. Note that, for standard initialization schemes, the difference between two successive matrices is of the order O(1) and not O(1/L), or, in other words, K W scales as O(L). This issue can be solved by adding correlations across layers at initialization by taking, for k ∈ {1, . . . , L} and i, j ∈ {1, . . . , d},

W k,i,j = 1 √ d f i,j ( k L )
, where f i,j a smooth function, for example a Gaussian process with an RBF kernel. Note that such a non-i.i.d. initialization scheme is necessary for the correspondence between deep residual networks and neural ODEs to hold, as shown in Chapter 2. Furthermore, Sander et al. (2022b) prove that, with such an initialization scheme, the constraint on the Lipschitz constant also holds for the trained network, with K W independent of L.

Statistical analysis. At first sight, a reasonable strategy would be to bound the distance between the model (4.10) and its limit L → ∞ that is a parameterized ODE, then apply Theorem 4.4. This idea is straightforward, but comes at the cost of an additional O(1/L) term in the generalization bound, as a consequence of the discretization error between the discrete iterations (4.10) and their continuous limit. For example, this strategy is used in Chapter 5 to prove a generalization bound for discrete RNNs and the additional error term is incurred. We follow another way by mimicking all the proof with a finite L. This is a longer approach but it yields a sharper result since we avoid the O(1/L) discretization error. The proof structure is similar to Section 4.3: the following two Propositions are the discrete counterparts of Propositions 4.2 and 4.3.

Proposition 4.6 (The residual network is bounded and Lipschitz). Let W and W ∈ W. Then, for any x ∈ X ,

F W (x) R X exp(K σ R W ) and F W (x) -F W(x) R X R W exp(2K σ R W ) W -W 1,1,∞ .
Proof. See Section 4.A.6.

Proposition 4.7 (Covering number of the residual network parameter class). Let N (ε) be the covering number of W endowed with the (1, 1, ∞)-norm (4.11). Then

log N (ε) d 2 log 16d 2 R W ε + d 4 K W log(4) ε .
Proof. See Section 4.A.7.

The proof of Proposition 4.6 is a discrete analogous of Proposition 4.2. On the other hand, Proposition 4.7 can be proven as a consequence of Proposition 4.3, by showing the existence of an injective isometry from W into a set of the form (4.7). Equipped with these two propositions, we are now ready to state the generalization bound for our class of residual neural networks. Theorem 4.8 (Generalization bound for deep residual networks). Consider the class of neural networks F W = {F W , W ∈ W}, where F W is given by (4.10) and W by (4.12). Let δ > 0.

Then, for n ≥ 9R -1 W max(d -4 R -1 W , 1), with probability at least 1δ,

R( W n ) R n ( W n ) + B(d + 1) log(R W dn) n + B d 2 √ K W n 1/4 + B √ n log 1 δ , (4.13)
where B is a constant depending on K , K σ , R W , R X , R Y . More precisely,

B = 6 √ 2K max exp(K σ R W ) R W , 1 (R X exp(K σ R W ) + R Y ).
Proof. See Section 4.A.8.

We emphasize that this result is non-asymptotic and valid for any width d and depth L. Furthermore, the depth L does not appear in the upper bound (4.13). This should not surprise the reader since Theorem 4.4 can be seen as the deep limit L → ∞ of this result, hence we expect that our bound remains finite when L → ∞ (otherwise the bound of Theorem 4.4 would be infinite). However, L appears as a scaling factor in the definition of the neural network (4.10) and of the class of parameters (4.12). This is crucial for the depth independence to hold, as we will comment further on in the next section.

Furthermore, the depth independence comes at the price of a O(n -1/4 ) convergence rate. Note that, by taking K W = 0, we obtain a generalization bound for weight-tied neural networks with a faster convergence rate in n, since the term in O(n -1/4 ) vanishes.

Comparison with other bounds

As announced in Section 4.2, we now compare Theorem 4.8 with the results of Bartlett et al. (2017) and [START_REF] Golowich | Size-independent sample complexity of neural networks[END_REF]. Beginning by Bartlett et al. (2017), we first state a slightly weaker version of their result to match our notations and facilitate comparison. Corollary 4.9 (of Theorem 1.1 of Bartlett et al. (2017)). Consider the class of neural networks

F W = {F W , W ∈ W}, where F W is given by (4.10) and W = {W ∈ R L×d×d , W 1,1,∞ R W }.
Assume that L ≥ R W and K σ = 1, and let γ, δ > 0. Consider (x, y), (x 1 , y 1 ), . . . , (x n , y n ) drawn i.i.d. from any probability distribution over R d × {1, . . . d} such that a.s. x ≤ R X .

Then, with probability at least 1δ, for every W ∈ W,

P argmax 1≤j≤d F W (x) j = y ≤ R n (W) + C R X R W exp(R W ) log(d) √ L γ √ n + C √ n log 1 δ , (4.14)
where R n (W) ≤ n -1 n i=1 1 F W (x i )y i ≤γ+max j =y i f (x i ) j and C is a universal constant.

Proof. See Section 4.A.9.

We first note that the setting is slightly different from ours: they consider a large margin predictor for a multi-class classification problem, whereas we consider a general Lipschitzcontinuous loss . This being said, the model class is identical to ours, except for one notable difference: the constraint on the Lipschitz constant of the weights appearing in equation (4.12) is not required here.

Comparing (4.13) and (4.14), we see that our bound enjoys a better dependence on the depth L but a worse dependence on the width d. Regarding the depth, our bound (4.13) does not depend on L, whereas the bound (4.14) scales as O( √ L). This comes from the fact that we consider a smaller set of parameters (4.12), by adding the constraint on the Lipschitz norm of the weights. This constraint allows us to control the complexity of our class of neural networks independently of depth, as long as K W is independent of L. If K W scales as O(L), which is the case for i.i.d. initialization schemes, our result also features a scaling in O( √ L). As for the width, Bartlett et al. (2017) achieve a better dependence by a subtle covering numbers argument that takes into account the geometry induced by matrix norms. Since the chapter focuses on a depth-wise analysis by leveraging the similarity between residual networks and their 130 infinite-depth counterpart, improving the scaling of our bound with width is left for future work. Finally, note that both bounds have a similar exponential dependence in R W .

As for [START_REF] Golowich | Size-independent sample complexity of neural networks[END_REF], they consider non-residual neural networks of the form x → M L σ(M L-1 σ(. . . σ(M 1 x))). These authors show that the generalization error of this class scales as

O R X Π F log Π F π S n 1/4 ,
where Π F is an upper-bound on the product of the Frobenius norms L k=1 M k F and π S is a lower-bound on the product of the spectral norms L k=1 M k . Under the assumption that both Π F and Π F/π S are bounded independently of L, their bound is indeed depth-independent, similarly to ours. Interestingly, as ours, the bound presents a O(n -1/4 ) convergence rate instead of the more usual O(n -1/2 ). However, the assumption that Π F is bounded independently of L does not hold in our residual setting, since we have M k = I + 1 L W k and thus we can lower-bound

L k=1 M k F L k=1 I F - 1 L M k F ≥ √ d - R W L L ≈ d L 2 e - R W √ d .
In our setting, it is a totally different assumption, the constraint that two successive weight matrices should be close to one another, which allows us to derive depth-independent bounds.

Numerical illustration

The bound of Theorem 4.8 features two quantities that depend on the class of neural networks, namely R W that bounds a norm of the weight matrices and K W that bounds the maximum difference between two successive weight matrices, a.k.a. the Lipschitz constant of the weights.

The first one belongs to the larger class of norm-based bounds that has been extensively studied (see, e.g., Neyshabur et al., 2015a). We are therefore interested in getting a better understanding of the role of the second quantity, which is much less common, in the generalization ability of deep residual networks.

To this aim, we train deep residual networks (4.10) (of width d = 30 and depth L = 1000) on MNIST. We prepend the network with an initial weight matrix to project the data x from dimension 768 to dimension 30, and similarly postpend it with another matrix to project the output M W (x) into dimension 10 (i.e. the number of classes in MNIST). Finally, we consider two training settings: either the initial and final matrices are trained, or they are fixed random projections. We use the initialization scheme outlined in Section 4.4.1. Further experimental details are postponed to the Appendix.

We report in Figure 4.1a the generalization gap of the trained networks, that is, the difference between the test and train errors (in terms of cross entropy loss), as a function of the maximum Lipschitz constant of the weights sup 0≤k≤L-1 ( W k+1 -W k ∞ ). We observe a positive correlation between these two quantities. To further analyze the relationship between the Lipschitz constant of the weights and the generalization gap, we then add the penalization term λ

• L-1 k=0 W k+1 - W k 2 F
1/2 to the loss, for some λ 0. The obtained generalization gap is reported in Figure 4.1b as a function of λ. We observe that this penalization allows to reduce the generalization gap. These two observations go in support of the fact that a smaller Lipschitz constant improves the generalization power of deep residual networks, in accordance with Theorem 4.8.

However, note that we were not able to obtain an improvement on the test loss by adding the penalization term. This is not all too surprising since previous work has investigated a related penalization, in terms of the Lipschitz norm of the layer sequence (H k ) 0≤k≤L , and was similarly not able to report any improvement on the test loss (Kelly et al., 2020).

Conclusion

We provide a generalization bound that applies to a wide range of parameterized ODEs. As a consequence, we obtain the first generalization bounds for time-independent and time-dependent neural ODEs. By discretizing our reasoning, we also provide a bound for a class of deep residual networks. In the future, it should also be interesting to extend our result to the more involved case of neural SDEs, which have also been found to be deep limits of a large class of residual neural networks, as shown in Chapter 2 and in [START_REF] Cohen | Scaling properties of deep residual networks[END_REF].

4.A Proofs

4.A.1 Proof of Proposition 4.1

The function

(t, h) → m i=1 θ i (t)f i (h)
is locally Lipschitz-continuous with respect to its first variable and globally Lipschitz-continuous with respect to its second variable. Therefore the existence and uniqueness of the solution of the initial value problem (4.5) for t ≥ 0 comes as a consequence of the Picard-Lindelöf theorem (see, e.g., Luk, 2017 for a self-contained presentation and Arnold, 1992 for a textbook).

4.A.2 Proof of Proposition 4.2

For x ∈ X , let H be the solution of the initial value problem (4.5) with parameter θ and with the initial condition H 0 = x. Let us first upper-bound f i (H t ) for all i ∈ {1, . . . , m} and t > 0.

To this aim, for t ≥ 0, we have

H t -H 0 = t 0 m i=1 θ i (s)f i (H s )ds t 0 m i=1 |θ i (s)| f i (H 0 ) ds + t 0 m i=1 |θ i (s)| f i (H s ) -f i (H 0 ) ds M t 0 m i=1 |θ i (s)|ds + K f t 0 H s -H 0 m i=1 |θ i (s)| ds tM R Θ + K f R Θ t 0 H s -H 0 ds.
Next, Grönwall's inequality yields, for t ∈ [0, 1],

H t -H 0 tM R Θ exp(tK f R Θ ) M R Θ exp(K f R Θ ). Hence H t H 0 + H t -H 0 R X + M R Θ exp(K f R Θ ),
yielding the first result of the proposition. Furthermore, for any i ∈ {1, . . . , m},

f i (H t ) f i (H t ) -f i (H 0 ) + f i (H 0 ) M K f R Θ exp(K f R Θ ) + 1 =: C.
Now, let H be the solution of the initial value problem (4.5) with another parameter θ and with the same initial condition H0 = x. Then, for any t 0,

H t -Ht = t 0 m i=1 θ i (s)f i (H s )ds - t 0 m i=1 θi (s)f i ( Hs )ds.
Hence

H t -Ht = t 0 m i=1 (θ i (s) -θi (s))f i (H s )ds + t 0 m i=1 θi (s)(f i (H s ) -f i ( Hs ))ds t 0 m i=1 |θ i (s) -θi (s)| f i (H s ) ds + t 0 m i=1 | θi (s)| f i (H s ) -f i ( Hs ) ds t 0 m i=1 |θ i (s) -θi (s)| f i (H s ) ds + K f t 0 H s -Hs m i=1 | θi (s)| ds tC θ -θ 1,∞ + K f R Θ t 0 H s -Hs ds.
Then Grönwall's inequality implies that, for t ∈ [0, 1],

H t -Ht tC θ -θ 1,∞ exp(tK f R Θ ) M (K f R Θ exp(K f R Θ ) + 1) exp(K f R Θ ) θ -θ 1,∞ 2M K f R Θ exp(2K f R Θ ) θ -θ 1,∞ since 1 K f R Θ exp(K f R Θ ) because K f 1, R Θ 1.

4.A.3 Proof of Proposition 4.3

We first prove the result for m = 1. Let G x be an ε/2K Θ -grid of [0, 1] and G y an ε/2-grid of [-R Θ , R Θ ]. Formally, we can take

G x = kε 2K Θ , 0 k 2K Θ ε and G y = -R Θ + kε 2 , 1 k 4R Θ ε
Our cover consists of all functions that start at a point of G y , are piecewise linear with kinks in G x , where each piece has slope +K Θ or -K Θ . Hence our cover is of size

N 1 (ε) = |G y |2 |Gx| 4R Θ ε 2 2K Θ ε +2 = 16R Θ ε 4 K Θ ε .
Now take a function f : [0, 1] → R that is uniformly bounded by R Θ and K Θ -Lipschitz. We construct a cover member at distance ε from f as follows. Choose a point y 0 in G y at distance at most ε/2 from f (0

). Since f (0) ∈ [-R Θ , R Θ ],
this is clearly possible, except perhaps at the end of the interval. To verify that it is possible at the end of the interval, note that R Θ is at a distance less than ε/2 of the last element of the grid, since

R Θ --R Θ + 4R Θ ε ε 2 = 2R Θ - 4R Θ ε ε 2 ∈ 2R Θ - 4R Θ ε ε 2 , 2R Θ - 4R Θ ε -1 ε 2 = 0, ε 2 .
Then, among the cover members that start at y 0 , choose the one which is closest to f at each point of G x (in case of equality, pick any one). Let us denote this cover member as f . Let us show recursively that f is at ∞ -distance at most ε from f . More precisely, let us first show by induction on k that for all k ∈ {0, . . . ,

2K Θ ε }, f kε 2K Θ -f kε 2K Θ ε 2 . (4.15) First, |f (0) -f (0)| ε 2 .
Then, assume that (4.15) holds for some k. Then we have the following inequalities:

f kε 2K Θ -ε f kε 2K Θ - ε 2 (by induction) f (k + 1)ε 2K Θ (f is K Θ -Lipschitz) f kε 2K Θ + ε 2 (f is K Θ -Lipschitz) f kε 2K Θ + ε (by induction).
Moreover, by definition,

f (k+1)ε K Θ is the closest point to f (k+1)ε K Θ among f kε K Θ - ε 2 , f kε K Θ + ε 2 .
The bounds above show that, among those two points, at least one is at distance no more than ε/2 from f (k+1)ε K Θ

. This shows (4.15) at rank k + 1. To conclude, take now x ∈ [0, 1]. There exists k ∈ {0, . . . , 2K Θ ε } such that x is at distance at most ε/4K Θ from kε 2K Θ . Again, this is clear except perhaps at the end of the interval, where it is also true since

1 - 2K Θ ε ε 2K Θ ≤ 1 - 2K Θ ε ε 2K Θ = 0,
meaning that 1 is located between two elements of the grid G x , showing that it is at distance at most ε/4K Θ from one element of the grid. Then, we have

|f (x) -f (x)| f (x) -f kε 2K Θ + f kε 2K Θ -f kε 2K Θ + f kε 2K Θ -f (x) ε 4 + ε 2 + ε 4 ,
where the first and third terms are upper-bounded because f and f are K Θ -Lip, while the second term is upper bounded by (4.15). Hence f -f ∞ ε, proving the result for m = 1. Finally, to prove the result for a general m, note that the Cartesian product of ε/m-covers for each coordinate of θ gives an ε-cover for θ. Indeed, consider such covers and take θ ∈ Θ. Since each coordinate of θ is uniformly bounded by R Θ and K Θ -Lipschitz, the proof above shows the existence of a cover member θ such that, for all i ∈ {1, . . . , m}, θ i -θi ∞ ≤ ε/m. Then

θ -θ 1,∞ = sup 0 t 1 m i=1 |θ i (t) -θi (t)| ≤ sup 0 t 1 m i=1 θ i -θi ∞ ≤ ε.
As a consequence, we conclude that

N (ε) ≤ N 1 ε m m = 16mR Θ ε m 4 m 2 K Θ ε .
Taking the logarithm yields the result.

4.A.4 Proof of Theorem 4.4

First note that, for any θ ∈ Θ, x ∈ X and y ∈ Y,

| (F θ (x), y)| | (F θ (x), y) -(y, y)| + | (y, y)| K F θ (x) -y ,
since, by assumption, is K -Lipschitz with respect to its first variable and (y, y) = 0. Thus

| (F θ (x), y)| K ( F θ (x) + y ) K R X + M R Θ exp(K f R Θ ) + R Y =: M ,
by Proposition 4.2. Now, taking δ > 0, a classical computation involving McDiarmid's inequality (see, e.g., Wainwright, 2019, proof of Theorem 4.10) yields that, with probability at least 1δ,

R( θ n ) R n ( θ n ) + E sup θ∈Θ |R(θ) -R n (θ)| + M √ 2 √ n log 1 δ . Denote C = 2M K f R Θ exp(2K f R Θ ). Then we show that R and R n are CK -Lipschitz with respect to (θ, • 1,∞ ): for θ, θ ∈ Θ, |R(θ) -R( θ)| E | (F θ (x), y) -(F θ(x), y)| K E F θ (x) -F θ(x) CK θ -θ 1,∞ ,
according to Proposition 4.2. The proof for the empirical risk is very similar. Let now ε > 0 and N (ε) be the covering number of Θ endowed with the (1, ∞)-norm. By Proposition 4.3,

log N (ε) m log 16mR Θ ε + m 2 K Θ log(4) ε .
Take θ (1) , . . . , θ (N (ε)) the associated cover elements. Then, for any θ ∈ Θ, denoting θ (i) the cover element at distance at most ε from θ,

|R(θ) -R n (θ)| |R(θ) -R(θ (i) )| + |R(θ (i) ) -R n (θ (i) )| + | R n (θ (i) ) -R n (θ)| 2CK ε + sup i∈{1,...,N (ε)} |R(θ (i) ) -R n (θ (i) )|. Hence E sup θ∈Θ |R(θ) -R n (θ)| 2CK ε + E sup i∈{1,...,N (ε)} |R(θ (i) ) -R n (θ (i) )| .
Since R n (θ) is the average of n independent random variables, which are each almost surely bounded by M , it is M / √ n sub-Gaussian, hence we have the classical inequality on the expectation of the maximum of sub-Gaussian random variables (see, e.g., Rigollet and Hütter, 2017, Theorem 1.14)

E sup i∈{1,...,N (ε)} |R(θ (i) ) -R n (θ (i) )| M 2 log(2N (ε)) n .
The remainder of the proof consists in computations to put the result in the required format.

More precisely, we have

E sup θ∈Θ |R(θ) -R n (θ)| 2CK ε + M 2 log(2N (ε)) n 2CK ε + M 2 log(2) + 2m log 16mR Θ ε + 2m 2 K Θ ε log(4) n 2CK ε + M 2(m + 1) log 16mR Θ ε + 2m 2 K Θ ε log(4) n .
The third step is valid if 16mR Θ ε 2. We will shortly take ε to be equal to 1 √ n , thus this condition holds true under the assumption from the Theorem that mR Θ √ n ≥ 3. Hence we obtain

R( θ n ) R n ( θ n ) + 2CK ε + M 2(m + 1) log 16mR Θ ε + 2m 2 K Θ ε log(4) n + M √ 2 √ n log 1 δ . (4.16) Now denote B = 2M K f exp(K f R Θ ). Then CK B and 2M B. Taking ε = 1 √ n , we obtain R( θ n ) R n ( θ n ) + 2 B √ n + B 2 2(m + 1) log(16mR Θ √ n) n + 2m 2 K Θ log(4) √ n + B √ n log 1 δ R n ( θ n ) + 2 B √ n + B 2 2(m + 1) log(16mR Θ √ n) n + B 2 m 2K Θ log(4) n 1/4 + B √ n log 1 δ R n ( θ n ) + 3 B 2 2(m + 1) log(16mR Θ √ n) n + B m √ K Θ n 1/4 + B √ n log 1 δ , since 2 ≤ 2 log(2) ≤ 2(m + 1) log(16mR Θ √ n) since 16mR Θ √ n ≥ 2 by the Theorem's
assumptions, and 2 log(4) 2. We finally obtain that

R( θ n ) R n ( θ n ) + 3 B (m + 1) log(mR Θ n) n + B m √ K Θ n 1/4 + B √ n log 1 δ , by noting that n ≥ 9 max(m -2 R -2 Θ , 1) implies that log(16mR Θ √ n) ≤ 2 log(mR Θ n).
The result unfolds since the constant B in the Theorem is equal to 3 B.

4.A.5 Proof of Corollary 4.5

The corollary is an immediate consequence of Theorem 4.4. To obtain the result, note that m = d 2 , thus in particular

√ m + 1 = √ d 2 + 1 ≤ d + 1, and besides log(R W d 2 n) ≤ 2 log(R W dn) since R W n ≤ R 2
W n 2 by assumption on n.

4.A.6 Proof of Proposition 4.6

For x ∈ X , let (H k ) 0≤k≤L be the values of the layers defined by the recurrence (4.10) with the weights W and the input H 0 = x. We denote by • the 2 -norm for vectors and the spectral norm for matrices. Then, for k ∈ {0, . . . , L -1}, we have

H k+1 ≤ H k + 1 L W k σ(H k ) H k + 1 L W k σ(H k ) 1 + K σ R W L H k ,
where the last inequality uses that the spectral norm of a matrix is upper-bounded by its (1, 1)-norm and that σ(0) = 0. As a consequence, for any k ∈ {0, . . . , L},

H k ≤ 1 + K σ R W L k H 0 ≤ exp(K σ R W )R X =: C,
yielding the first claim of the Proposition. Now, let H be the values of the layers (4.10) with another parameter W and with the same input H0 = x. Then, for any k ∈ {0, . . . , L -1},

H k+1 -Hk+1 = H k -Hk + 1 L (W k σ(H k ) -Wk σ( Hk )).
Hence, using again that the spectral norm of a matrix is upper-bounded by its (1, 1)-norm and that σ(0) = 0,

H k+1 -Hk+1 ≤ H k -Hk + 1 L W k (σ(H k ) -σ( Hk )) + 1 L (W k -Wk )σ( Hk ) 1 + K σ R W L H k -Hk + K σ L W k -Wk Hk 1 + K σ R W L H k -Hk + CK σ L W k -Wk .
Then, dividing by (1 + K σ R W L ) k+1 and using the method of differences, we obtain that

H k -Hk (1 + K σ R W L ) k ≤ H 0 -H0 + CK σ L k-1 j=0 W j -Wj (1 + K σ R W L ) j+1 ≤ CK σ L W -W 1,1,∞ k-1 j=0 1 (1 + K σ R W L ) j+1 . Finally note that k-1 j=0 (1 + K σ R W L ) k (1 + K σ R W L ) j+1 = k-1 j=0 (1 + K σ R W L ) j = L K σ R W (1 + K σ R W L ) k -1 ≤ L K σ R W (exp(K σ R W ) -1).
We conclude that

H k -Hk ≤ C R W (exp(K σ R W ) -1) W -W 1,1,∞ ≤ R X R W exp(2K σ R W ) W -W 1,1,∞ .
φ(W) i k L = W k L ,(i//d)+1,(i%d)+1 , and φ(W) i (0) = φ(W) i (1/L). Then φ(W) satisfies two properties. First, it is a linear function of W. Second, for W ∈ R L×d×d , φ(W) 1,∞ = W 1,1,∞ , because, for x ∈ [0, 1], φ(W)(x) is a convex combination of two vectors that are bounded in 1 - norm by W 1,1,∞ , so it is itself bounded in 1 -norm by W 1,1,∞ , implying that φ(W) 1,∞ ≤ W 1,1,∞ . Reciprocally, φ(W) 1,∞ = sup 0 t 1 φ(W)(x) 1 sup 1≤k≤L φ(W) k L 1 = W 1,1,∞ . Now, take W ∈ W. The second property of φ implies that φ(W) 1,∞ ≤ R W . Moreover, each coordinate of φ(W) is K W -Lipschitz, since the slope of each piece of φ(W) i is at most K W . As a consequence, φ(W) belongs to Θ W = {θ : [0, 1] → R d 2 , θ 1,∞ R W and θ i is K W -Lipschitz for i ∈ {1, . . . , d 2 }}.
Therefore φ(W) is a subset of Θ W , thus its covering number is less than the one of Θ W . Moreover, φ is clearly injective, thus we can define φ -1 on its image. Consider an ε-cover (θ 1 , . . . , θ N ) of (φ(W), • 1,∞ ). Let us show that (φ -1 (θ 1 ), . . . , φ -1 (θ N )) is an ε-cover of (W, • 1,1,∞ ): take W ∈ W and consider θ i a cover member at distance less than ε from φ(W). Then

W -φ -1 (θ i ) 1,1,∞ = φ(W -φ -1 (θ i )) 1,∞ = φ(W) -θ i 1,∞ ≤ ε,
where the second equality holds by linearity of φ. Therefore, the covering number of (W, • 1,1,∞ ) is upper bounded by the one of (φ(W), • 1,∞ ), which itself is upper bounded by the one of (Θ W , • 1,∞ ), yielding the result by Proposition 4.3.

4.A.8 Proof of Theorem 4.8

The proof structure is the same as the one of Theorem 4.4, but some constants change. Similarly to (4.16), we obtain that, if 16d 2 R W ε 2 (which holds true for ε = 1 / √ n and under the assumption of the Theorem),

R( W n ) R n ( W n ) + 2CK ε + M 2(d 2 + 1) log 16d 2 R W ε + 2d 4 K W ε log(4) n + M √ 2 √ n log 1 δ , with M = K (R X exp(K σ R W ) + R Y ) and C = R X R W exp(2K σ R W ).
Finally denote

B = 2M max exp(K σ R W ) R W , 1 .
Then CK B and 2M B. Taking ε = 1 √ n , we obtain as in the proof of Theorem 4.4 that

R( W n ) R n ( W n ) + 3 B (d 2 + 1) log(d 2 R W n) n + B d 2 √ K W n 1/4 + B √ n log 1 δ for n 9R -1 W max(d -4 R -1 W , 1). Thus R( W n ) R n ( W n ) + 3 √ 2 B(d + 1) log(dR W n) n + B d 2 √ K W n 1/4 + B √ n log 1 δ , since √ d 2 + 1 ≤ d + 1 and R W n ≤ R 2 W n 2
by assumption on n. The result unfolds since the constant B in the Theorem is equal to 3 √ 2 B.

4.

A.9 Proof of Corollary 4.9

Let

A(W) = L k=1 I + 1 L W k L k=1 W T k 2/3 2,1 L 2/3 I + 1 L W k 2/3 3/2
, where • 2,1 denotes the (2, 1)-norm defined as the 1 -norm of the 2 -norms of the columns, and I is the identity matrix (and we recall that • denotes the spectral norm). We apply Theorem 1.1 from Bartlett et al. (2017) by taking as reference matrices the identity matrix. The theorem shows that, under the assumptions of the corollary,

P argmax 1≤j≤d F W (x) j = y ≤ R n (W) + C R X A(W) log(d) γ √ n + C √ n log 1 δ ,
where, as in the corollary, R n (W) ≤ n -1 n i=1 1 F W (x i )y i ≤γ+max j =y i f (x i ) j and C is a universal constant. Let us upper bound A(W) to conclude. On the one hand, we have

L k=1 I + 1 L W k ≤ L k=1 I + 1 L W k ≤ L k=1 1 + 1 L W k 1,1 ≤ L k=1 1 + 1 L R W ≤ exp(R W )
On the other hand, for any k ∈ {1, . . . , L},

W T k 2,1 ≤ W T k 1,1 ≤ R W , while I + 1 L W k ≥ 1 - 1 L W k ≥ 1 - R W L ≥ 1 2 ,
under the assumption that L ≥ R W . All in all, we obtain that

A(W) ≤ exp(R W ) 2 2/3 L 1/3 R 2/3 W 3/2 = 2R W exp(R W ) √ L,
which yields the result.

4.B Experimental details

Our code is available at https://github.com/PierreMarion23/generalization-ode-resnets.

We use the following model, corresponding to model (4.10) with additional projections at the beginning and at the end: We use the initialization scheme outlined in Section 4.4.1: we initialize, for k ∈ {1, . . . , L} and i, j ∈ {1, . . . , d},

H 0 = Ax H k+1 = H k + 1 L W k+1 σ(H k ), 0 k L -1 F W (x) = BH L , where x ∈ R 768 is a vectorized MNIST image, A ∈ R d×768 ,
W k,i,j = 1 √ d f i,j k L ,
where f i,j are independent Gaussian processes with an RBF kernel (with bandwidth equal to 0.1).

We refer to 2 and to Sander et al. (2022b) for further discussion on this initialization scheme. However, A and B are initialized with a more usual scheme, namely with i.i.d. N (0, 1/c) random variables, where c denotes the number of columns of A (resp. B).

In Figure 4.1a, we repeat training 10 times independently. Each time, we perform 30 epochs, and compute after each epoch both the Lipschitz constant of the weights and the generalization gap. This gives 300 pairs (Lipschitz constant, generalization gap), which each corresponds to one dot in the figure. Furthermore, we report results for two setups: when A and B are trained or when they are fixed random matrices.

In Figure 4.1b, A and B are not trained. The reason is to assess the effect of the penalization on W for a fixed scale of A and B. If we allow A and B to vary, then it is possible that the effect of the penalization might be neutralized by a scale increase of A and B during training.

For all experiments, we use the standard MNIST data split (60k training samples and 10k testing samples). We train using the cross entropy loss, mini-batches of size 128, and the optimizer Adam [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with default parameters and a learning rate of 0.02.

We use PyTorch (Paszke et al., 2019) and PyTorch Lightning for our experiments.

The code takes about 60 hours to run on a standard laptop (no GPU).

Introduction

Recurrent neural networks (RNN) are among the most successful methods for modeling sequential data. They have achieved state-of-the-art results in difficult problems such as natural language processing (e.g., [START_REF] Mikolov | Recurrent neural network based language model[END_REF][START_REF] Collobert | Natural language processing (almost) from scratch[END_REF] or speech recognition (e.g., Hinton et al., 2012a;[START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF]. This class of neural networks has a natural interpretation in terms of (discretization of) ordinary differential equations (ODE), which casts them in the field of neural ODE (Chen et al., 2018a). This observation has led to the development of continuousdepth models for handling irregularly-sampled time-series data, including the ODE-RNN model (Rubanova et al., 2019), GRU-ODE-Bayes (De Brouwer et al., 2019), or neural CDE models (Kidger et al., 2020;[START_REF] Morrill | Neural rough differential equations for long time series[END_REF]. In addition, the time-continuous interpretation of RNN allows to leverage the rich theory of differential equations to develop new recurrent architectures [START_REF] Chang | AntisymmetricRNN: A dynamical system view on recurrent neural networks[END_REF][START_REF] Herrera | Theoretical guarantees for learning conditional expectation using controlled ODE-RNN[END_REF][START_REF] Erichson | Lipschitz recurrent neural networks[END_REF], which are better at learning long-term dependencies.

On the other hand, the development of kernel methods for deep learning offers theoretical insights on the functions learned by the networks (Cho and Saul, 2009;[START_REF] Belkin | To understand deep learning we need to understand kernel learning[END_REF]Jacot et al., 2018). Here, the general principle consists of defining a reproducing kernel Hilbert space (RKHS)-that is, a function class H -, which is rich enough to describe the architectures of networks. A good example is the construction of Bietti andMairal (2017, 2019), who exhibit an RKHS for convolutional neural networks. This kernel perspective has several advantages. First, by separating the representation of the data from the learning process, it allows studying invariances of the representations learned by the network. Next, by reducing the learning problem to a linear one in H , generalization bounds can be more easily obtained. Finally, the Hilbert structure of H provides a natural metric on neural networks, which can be used for example for regularization (Bietti et al., 2019).

Contributions. By taking advantage of the neural ODE paradigm for RNN, we show that RNN are, in the continuous-time limit, linear predictors over a specific space associated with the signature of the input sequence [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF]. The signature transform, first defined by [START_REF] Chen | Integration of paths-a faithful representation of paths by non-commutative formal power series[END_REF] and central in rough path theory [START_REF] Lyons | Differential Equations Driven by Rough Paths[END_REF][START_REF] Friz | Multidimensional Stochastic Processes as Rough Paths: Theory and Applications[END_REF], summarizes sequential inputs by a graded feature set of their iterated integrals. Its natural environment is a tensor space that can be endowed with an RKHS structure [START_REF] Király | Kernels for sequentially ordered data[END_REF]. We exhibit general conditions under which classical recurrent architectures such as feedforward RNN, Gated Recurrent Units (GRU, [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF], or Long Short-Term Memory networks (LSTM, [START_REF] Hochreiter | Long short-term memory[END_REF], can be framed as a kernel method in this RKHS. This enables us to provide generalization bounds for RNN as well as stability guarantees via regularization. The theory is illustrated with some experimental results.

Related works. The neural ODE paradigm was first formulated by Chen et al. (2018a) for residual neural networks. It was then extended to RNN in several articles, with a focus on handling irregularly sampled data (Rubanova et al., 2019;Kidger et al., 2020) and learning long-term dependencies [START_REF] Chang | AntisymmetricRNN: A dynamical system view on recurrent neural networks[END_REF]. The signature transform has recently received the attention of the machine learning community [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF]Kidger et al., 2019;[START_REF] Liao | Learning stochastic differential equations using RNN with log signature features[END_REF][START_REF] Toth | Bayesian learning from sequential data using Gaussian processes with signature covariances[END_REF][START_REF] Fermanian | Embedding and learning with signatures[END_REF] and, combined with deep neural networks, has achieved state-of-the-art performance for several applications [START_REF] Yang | DeepWriterID: An end-to-end online text-independent writer identification system[END_REF][START_REF] Yang | Developing the path signature methodology and its application to landmark-based human action recognition[END_REF][START_REF] Arribas | Derivatives pricing using signature payoffs[END_REF][START_REF] Wang | A path signature approach for speech emotion recognition[END_REF][START_REF] Morrill | Utilization of the signature method to identify the early onset of sepsis from multivariate physiological time series in critical care monitoring[END_REF]. [START_REF] Király | Kernels for sequentially ordered data[END_REF] use the signature transform to define kernels for sequential data and develop fast computational methods. The connection between continuous-time RNN and signatures has been pointed out by [START_REF] Lim | Understanding recurrent neural networks using nonequilibrium response theory[END_REF] for a specific model of stochastic RNN. Deriving generalization bounds for RNN is an active research area [START_REF] Zhang | Stabilizing gradients for deep neural networks via efficient SVD parameterization[END_REF][START_REF] Akpinar | Sample complexity bounds for recurrent neural networks with application to combinatorial graph problems[END_REF][START_REF] Tu | Understanding generalization in recurrent neural networks[END_REF]. By leveraging the theory of differential equations, our approach encompasses a large class of RNN models, ranging from feedforward RNN to LSTM. This is in contrast with most existing generalization bounds, which are architecture-dependent. Close to our point of view is the work of Bietti and Mairal (2017) for convolutional neural networks.

Mathematical context. We place ourselves in a supervised learning setting. The input data is a sample of n i.i.d. vector-valued sequences {x (1) , . . . , x (n) }, where x (i) = (x

(i) 1 , . . . , x (i) T ) ∈ (R d ) T , T ≥ 1.
The outputs of the learning problem can be either labels (classification setting) or sequences (sequence-to-sequence setting). Even if we only observe discrete sequences, each x (i) is mathematically considered as a regular discretization of a continuous-time process

X (i) ∈ BV c ([0, 1], R d ), where BV c ([0, 1], R d )
is the space of continuous functions from [0, 1] to R d of finite total variation. Informally, the total variation of a process corresponds to its length. Formally, for any [s, t] ⊂ [0, 1], the total variation of a process

X ∈ BV c ([0, 1], R d ) on [s, t] is defined by X T V ;[s,t] = sup (t 0 ,...,t k )∈Ds,t k j=1 X t j -X t j-1 ,
where D s,t denotes the set of all finite partitions of [s, t] and • the Euclidean norm. We therefore have that

x (i) j = X (i) j /T , 1 ≤ j ≤ T , where X (i) t := X (i) (t)
. We make two assumptions on the processes X (i) . First, they all begin at zero, and second, their total variations are bounded by L ∈ (0, 1). These assumptions are not too restrictive, since they amount to data translation and normalization, common in practice. Accordingly, we denote by X the subset of

BV c ([0, 1], R d ) defined by X = X ∈ BV c ([0, 1], R d ) | X 0 = 0 and X T V ;[0,1] ≤ L
and assume therefore that X (1) , . . . , X (n) are i.i.d. according to some X ∈ X . The norm on all spaces R m , m ≥ 1, is always the Euclidean one. Observe that assuming that X ∈ X implies that, for any t ∈ [0, 1],

X t = X t -X 0 ≤ X T V ;[0,1] ≤ L < 1.
Recurrent neural networks. Classical RNN are defined by a sequence of hidden states h 1 , . . . , h T that all belong to R e , where, for x = (x 1 , . . . , x T ) a generic data sample,

h 0 = 0 and h j+1 = f (h j , x j+1 ) for 0 ≤ j ≤ T -1.
At each time step 1 ≤ j ≤ T , the output of the network is z j = ψ(h j ), where ψ is a linear function. In this chapter, we rather consider the following residual version, which is a natural adaptation of classical RNN in the neural ODE framework (see, e.g., [START_REF] Yue | Residual recurrent neural networks for learning sequential representations[END_REF]:

h 0 = 0 and h j+1 = h j + 1 T f (h j , x j+1 ) for 0 ≤ j ≤ T -1.
(5.1)

The simplest choice for the function f is the feedforward model, say f RNN , defined by

f RNN (h, x) = σ(U h + V x + b), (5.2)
where σ is an activation function, U ∈ R e×e and V ∈ R e×d are weight matrices, and b ∈ R e is the bias. The function f RNN , equipped with a smooth activation σ (such as the logistic or hyperbolic tangent functions), will be our leading example throughout the chapter. However, the GRU and LSTM models can also be rewritten under the form (5.1), as shown in Appendix 5.A.1. Thus, model (5.1) is flexible enough to encompass most recurrent networks used in practice.

Overview. Section 5.2 is devoted to framing RNN as linear functions in a suitable RKHS. We start by embedding iteration (5.1) into a continuous-time model, which takes the form of a controlled differential equation (CDE). This allows, after introducing the signature transform, to define the appropriate RKHS, and, in turn, to show that model (5.1) boils down, in the continuous-time limit, to a linear problem on the signature. This framework is used in Section 5.3 to derive generalization bounds and stability guarantees. We provide some experiments in Section 5.4 before discussing our results in Section 5.5. All proofs are postponed to the end of the chapter. In Section 5.A, we present some useful additional definitions and lemmas. The proofs are given in Section 5.B, and rely on some algebra rules over tensor spaces that are given in Section 5.C. Finally, the experimental details are presented in Section 5.D.

Framing RNN as a kernel method

Roadmap. First, we quantify the difference between the discrete recurrent network (5.1) and its continuous-time counterpart (Proposition 5.1). Then, we rewrite the corresponding ODE as a CDE (Proposition 5.2). Under appropriate conditions, Proposition 5.9 shows that the solution of this equation is a linear function of the signature of the driving process. Importantly, these assumptions are valid for a feedforward RNN, as stated by Proposition 5.10. We conclude in Theorem 5.11.

From discrete to continuous time

Recall that h 0 , . . . , h T denote the hidden states of the RNN (5.1), and let H : [0, 1] → R e be the solution of the ODE

dH t = f (H t , X t )dt, H 0 = h 0 . (5.3)
By bounding the difference between Hj /T and h j , the following proposition shows how to pass from discrete to continuous time, provided f satisfies the following assumption:

(A 1 ) The function f is Lipschitz continuous in h and x, with Lipschitz constants K h and K x .

We let K f = max(K h , K x ).

Proposition 5.1. Assume that (A 1 ) is verified. Then there exists a unique solution H to (5.3) and, for any 0 ≤ j ≤ T ,

Hj /T -h j ≤ c 1 T ,
where

c 1 = K f e K f L + sup h ≤M, x ≤L f (h, x) e K f and M = sup x ≤L f (h 0 , x) e K f . Moreover, for any t ∈ [0, 1], H t ≤ M .
Proof. See Section 5.B.1.

Then, following Kidger et al. (2020), we show that the ODE (5.3) can be rewritten under the form of a CDE. At the cost of increasing the dimension of the hidden state from e to e + d, this allows us to reframe model (5.3) as a linear model in dX, in the sense that X has been moved 'outside' of f . Proposition 5.2. Assume that (A 1 ) is verified. Let H : [0, 1] → R e be the solution of (5.3), and let X : [0, 1] → R d+1 be the time-augmented process Xt = (X t , 1-L 2 t) . Then there exists a tensor field

F : R ē → R ē× d, ē = e + d, d = d + 1, such that if H : [0, 1] → R ē is the solution of the CDE d Ht = F( Ht )d Xt , H0 = (H 0 , X 0 ) , (5.4)
then its first e coordinates are equal to H.

Although this definition is technical, the signature should simply be thought of as a feature map that embeds a bounded variation process into an infinite-dimensional tensor space. The signature has several good properties that make it a relevant tool for machine learning (e.g., [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF][START_REF] Chevyrev | A primer on the signature method in machine learning[END_REF][START_REF] Fermanian | Embedding and learning with signatures[END_REF]. In particular, under certain assumptions, S(X) characterizes X up to translations and reparameterizations, and has good approximation properties. We also highlight that fast libraries exist for computing the signature [START_REF] Reizenstein | Algorithm 1004: The iisignature library: Efficient calculation of iterated-integral signatures and log signatures[END_REF][START_REF] Kidger | Signatory: Differentiable computations of the signature and logsignature transforms, on both CPU and GPU[END_REF].

The expert reader is warned that this definition differs from the usual one by the normalization of X k [0,t] by k!, which is more adapted to our context. In the sequel, for any index (i 1 , . . . , i k ) ∈ {1, . . . , d} k , S (i 1 ,...,i k ) [0,t] (X) denotes the term associated with the coordinates (i 1 , . . . , i k ) of X k [0,t] . When the signature is taken on the whole interval [0, 1], we simply write S(X), S (i 1 ,...,i k ) (X), and X k .

Example 5.5. Let X be the d-dimensional linear path defined by

X t = (a 1 + b 1 t, . . . , a d + b d t) , a i , b i ∈ R. Then S (i 1 ,...,i k ) (X) = b i 1 • • • b i k and X k = b ⊗k .
The next proposition, which ensures that S [0,t] ( X) ∈ T , is an important step.

Proposition 5.6. Let X ∈ X and Xt = (X t , 1-L 2 t) as in Proposition 5.2. Then, for any

t ∈ [0, 1], S [0,t] ( X) T ≤ 2(1 -L) -1 .
Proof. See Section 5.B.3.

The signature kernel. By taking advantage of the structure of Hilbert space of T , it is natural to introduce the following kernel:

K : X × X → R (X, Y ) → S( X), S( Ȳ ) T ,
which is well-defined according to Proposition 5.6. We refer to [START_REF] Király | Kernels for sequentially ordered data[END_REF] for a general presentation of kernel methods with signatures and to [START_REF] Salvi | The signature kernel is the solution of a goursat pde[END_REF] for a kernel trick. The RKHS associated with K is the space of functions

H = ξ α : X → R | ξ α (X) = α, S( X) T , α ∈ T , (5.7)
with scalar product ξ α , ξ β H = α, β T (see, e.g., [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF].

From the CDE to the signature kernel

An important property of signatures is that the solution of the CDE (5.4) can be written, under certain assumptions, as a linear function of the signature of the driving process X. This operation can be thought of as a Taylor expansion for CDE. More precisely, let us rewrite (5.4) as (5.8) where X t = (X 1 t , . . . , X d t ) , F : R e → R e×d , and F i : R e → R e are the columns of F-to avoid heavy notation, we momentarily write e, d, H, and X instead of ē, d, H, and X. Throughout, the bold notation is used to distinguish tensor fields and vector fields. We recall that a vector field F : R e → R e or a tensor field F : R e → R e×d are said to be smooth if each of their coordinates is C ∞ . Definition 5.7. Let F, G : R e → R e be smooth vector fields and denote by J(•) the Jacobian matrix. Their differential product is the smooth vector field F G : R e → R e defined, for any h ∈ R e , by

dH t = F(H t )dX t = d i=1 F i (H t )dX i t ,
(F G)(h) = e j=1 ∂G ∂h j (h)F j (h) = J(G)(h)F (h).
In differential geometry, F G is simply denoted by F G. Since the operation is not associative, we take the convention that it is evaluated from right to left, i.e., F 1 F 2 F 3 := F 1 (F 2 F 3 ).

Taylor expansion. Let H be the solution of (5.8), where F is assumed to be smooth. We now show that H can be written as a linear function of the signature of X, which is the crucial step to embed the RNN in the RKHS H . The step-N Taylor expansion of H [START_REF] Friz | Euler estimates for rough differential equations[END_REF] is defined by

H N t = H 0 + N k=1 1 k! 1≤i 1 ,...,i k ≤d S (i 1 ,...,i k ) [0,t] (X)F i 1 • • • F i k (H 0 ). Throughout, we let Λ k (F) = sup h ≤M,1≤i 1 ,...,i k ≤d F i 1 • • • F i k (h) .
Example 5.8. Let F = F RNN defined by (5.5) with an identity activation. Then, for any h ∈ R ē,

1 ≤ i ≤ d + 1, F i RNN ( h) = W i h + b i
, where b i is the (i + d)th vector of the canonical basis of R ē, and

W i = 0 ē×ē , W d+1 = 2 1-L W 0 d×ē , and b d+1 = 2 1-L b 0 d .
The vector fields F i RNN are then affine, J(F i RNN ) = W i , and the iterated star products have a simple expression: for any

1 ≤ i 1 , . . . , i k ≤ d, F i 1 RNN • • • F i k RNN ( h) = W i k • • • W i 2 (W i 1 h + b i 1 ).
The next proposition shows that the step-N Taylor expansion H N is a good approximation of H. Proposition 5.9. Assume that the tensor field F is smooth. Then, for any t ∈ [0, 1],

H t -H N t ≤ d N +1 (N + 1)! Λ N +1 (F).
(5.9)

Proof. See Section 5.B.4.

Thus, provided that Λ N (F) is not too large, the right-hand side of (5.9) converges to zero, hence

H t = H 0 + ∞ k=1 1 k! 1≤i 1 ,...,i k ≤d S (i 1 ,...,i k ) [0,t] (X)F i 1 • • • F i k (H 0 ).
(5.10)

We conclude from the above representation that the solution H of (5.8) is in fact a linear function of the signature of X. A natural concern is to know whether the upper bound of Proposition 5.9 vanishes with N for standard architectures. This property is encapsulated in the following more general assumption:

(A 2 )
The tensor field F is smooth and

∞ k=0 d k k! Λ k (F) 2 < ∞.
Clearly, if (A 2 ) is verified, then the right-hand side of (5.9) converges to 0. The next proposition states formally the conditions under which (A 2 ) is verified for F RNN . It is further illustrated in Figure 5.1, which shows that the convergence is fast with two common activation functions. We let

σ ∞ = sup h ≤M, x ≤L σ(U h + V x + b) and σ (k) ∞ = sup h ≤M, x ≤L σ (k) (U h + V x + b)
, where σ (k) is the derivative of order k of σ.

Proposition 5.10. Let F RNN be defined by (5.5). If σ is the identity function, then (A 2 ) is satisfied. In the general case, (A 2 ) holds if σ is smooth and there exists a > 0 such that, for any k ≥ 0,

σ (k) ∞ ≤ a k+1 k! and W F < 1 -L 8a 2 d , (5.11)
where

• F is the Frobenius norm. Moreover, Λ N (F RNN ) ≤ √ 2a 8a 2 W F 1-L N -1 N ! .
Proof. See Section 5.B.5.

The proof of Proposition 5.10, based on the manipulation of higher-order derivatives of tensor fields, is highly non-trivial. We highlight that the conditions on σ are mild and verified for common smooth activations. For example, they are verified for the logistic function (with a = 2) and for the hyperbolic tangent function (with a = 4)-see Appendix 5.A.5. The second inequality of (5.11) puts a constraint on the norm of the weights, and can be regarded as a radius of convergence for the Taylor expansion.

Putting everything together. We now have all the elements at hand to embed the RNN into the RKHS H . To fix the idea, we assume in this paragraph that we are in a ±1 classification setting. In other words, given an input sequence x, we are interested in the final output z T = ψ(h T ) ∈ R, where h T is the solution of (5.1). The predicted class is 2 • 1(z T > 0) -1.

By Propositions 5.1 and 5.2, z T is approximated by the first e coordinates of the solution of the CDE (5.4), which outputs a R e+d -valued process H. According to Proposition 5.9, H is a linear function of the signature of the time-augmented process X. Thus, on top of H, it remains to successively apply the projection Proj on the e first coordinates followed by the linear function ψ to obtain an element of the RKHS H . This mechanism is summarized in the following theorem.

Theorem 5.11. Assume that (A 1 ) and (A 2 ) are verified. Then there exists a function

ξ α ∈ H such that |z T -ξ α (X)| ≤ ψ op c 1 T , (5.12) where ξ α (X) = α, S( X) T and Xt = (X t , 1-L 2 t) . We have α = (α k ) ∞ k=0 , where each α k ∈ (R d ) ⊗k is defined by α (i 1 ,...,i k ) k = 1 k! ψ • Proj F i 1 • • • F i k ( H0 ) . Moreover, α 2 T ≤ ψ 2 op ∞ k=0 d k k! Λ k (F) 2 .
Proof. See Section 5.B.6.

We conclude that in the continuous-time limit, the output of the network can be interpreted as a scalar product between the signature of the (time-augmented) process X and an element of T . This interpretation is important for at least two reasons: (i) it facilitates the analysis of generalization of RNN by leveraging the theory of kernel methods, and (ii) it provides new insights on regularization strategies to make RNN more robust. These points will be explored in the next section. Finally, we stress that the approach works for a large class of RNN, such as GRU and LSTM. The derivation of conditions (A 1 ) and (A 2 ) beyond the feedforward RNN is left for future work.

Generalization and regularization

Generalization bounds

Learning procedure. A first consequence of framing an RNN as a kernel method is that it gives natural generalization bounds under mild assumptions. In the learning setup, we are given an i.i.d. sample D n of n random pairs of observations (x (i) ,

y (i) ) ∈ (R d ) T × Y , where x (i) = (x (i) 1 , . . . , x (i)
T ). We distinguish the binary classification problem, where Y = {-1, 1}, from the sequential prediction problem, where Y = (R p ) T and y (i) = (y

(i) 1 , . . . , y (i) T ).
The RNN is assumed to be parameterized by θ ∈ Θ ⊂ R q , where Θ is a compact set. To clarify the notation, we use a θ subscript whenever a quantity depends on θ (e.g., f θ for f , etc.). In line with Section 5.2, it is assumed that the tensor field F θ associated with f θ satisfies (A 1 ) and (A 2 ), keeping in mind that Proposition 5.10 guarantees that these requirements are fulfilled by a feedforward recurrent network with a smooth activation function.

Let g θ : (R d ) T → Y denote the output of the recurrent network. The parameter θ is fitted by empirical risk minimization using a loss function : Y × Y → R + . The theoretical and empirical risks are respectively defined, for any θ ∈ Θ, by

R(θ) = E[ (y, g θ (x))] and R n (θ) = 1 n n i=1 y (i) , g θ (x (i) ) ,
where the expectation E is evaluated with respect to the distribution of the generic random pair (x, y). We let θ n ∈ argmin θ∈Θ R n (θ) and aim at upper bounding P(y = g θn (x)) in the classification regime (Theorem 5.12) and R( θ n ) in the sequential regime (Theorem 5.14). To reach this goal, our strategy is to approximate the RNN by its continuous version and then use the RKHS machinery of Section 5.2.

Binary classification. In this context, the network outputs a real number g θ (x) = ψ(h T ) ∈ R and the predicted class is 2 • 1(g θ (x) > 0) -1. The loss : R × R → R + is assumed to satisfy the assumptions of Bartlett and Mendelson (2002, Theorem 7), that is, for any y ∈ {-1, 1}, (y, g θ (x)) = φ(yg θ (x)), where φ(u) ≥ 1(u ≤ 0), and φ is Lipschitz-continuous with constant K . For example, the logistic loss satisfies such assumptions. We let ξ α θ ∈ H be the function of Theorem 5.11 that approximates the RNN with parameter θ. Thus,

z T ≈ ξ α θ ( X) = α θ , S( X) T , up to a O( 1 /T ) term.
Theorem 5.12. Assume that for all θ ∈ Θ, (A 1 ) and (A 2 ) are verified. Assume, in addition, that there exists a constant B > 0 such that for any θ ∈ Θ, ξ α θ H ≤ B. Then with probability at least 1δ,

P y = g θn (x)|D n ≤ R n ( θ n ) + c 2 T + 8BK (1 -L) √ n + 2BK 1 -L log( 1 /δ) 2n , (5.13)
where

c 2 = K sup θ ψ op K f θ e K f θ L + f θ ∞ e K f θ .
Proof. See Section 5.B.7.

Close to our result are the bounds obtained by [START_REF] Zhang | Stabilizing gradients for deep neural networks via efficient SVD parameterization[END_REF], [START_REF] Tu | Understanding generalization in recurrent neural networks[END_REF][START_REF] Chen | On generalization bounds of a family of recurrent neural networks[END_REF]. The main difference is that the term in 1 /T does not usually appear, since it comes from the Euler discretization error, whereas the speed in 1 / √ n is the same. For instance, [START_REF] Chen | On generalization bounds of a family of recurrent neural networks[END_REF] show that, under some assumptions, the excess risk is of order √ de + e 2 T α K n -1/2 . We refer to Section 5.5 for further discussion on the dependency of the different bounds to the parameter T . The take-home message is that the detour by continuous-time neural ODE provides a theoretical framework adapted to RNN, at the modest price of an additional O( 1 /T ) term. Moreover, we note that the bound (5.13) is 'simple' and holds under mild conditions for a large class of RNN. More precisely, for any recurrent network of the form (5.1), provided (A 1 ) and (A 2 ) are satisfied, then (5.13) is valid with constants c 2 and B depending on the architecture. Such constants are given below in the example of a feedforward RNN. We stress that Theorem 5.12 can be extended without significant effort to the multi-class classification task, with an appropriate choice of loss function.

Example 5.13. Take a feedforward RNN with logistic activation, and

Θ = {(W, b, ψ) | W F ≤ K W < (1 -L)/32d, b ≤ K b , ψ op ≤ K ψ }.
Then, Proposition 5.10 states that (A 2 ) is satisfied and, with Theorem 5.11, ensures that

sup θ∈Θ ξ α θ H ≤ √ 2K ψ (1 -L) 1 -L -32dK W := B, K f θ = max( U op , V op ), and
f θ ∞ = 1.
Sequence-to-sequence learning. We conclude by showing how to extend both the RKHS embedding of Theorem 5.11 and the generalization bound of Theorem 5.12 to the setting of sequence-to-sequence learning. In this case, the output of the network is a sequence

g θ (x) = (z 1 , . . . , z T ) ∈ (R p ) T .
An immediate extension of Theorem 5.11 ensures that there exist p elements α 1,θ , . . . , α p,θ ∈ T such that, for any 1 ≤ j ≤ T ,

z j -α 1,θ , S [0, j /T ] ( X) T , . . . , α p,θ , S [0, j /T ] ( X) T ≤ ψ op c 1 T .
(5.14)

The properties of the signature guarantee that S [0, j /T ] (X) = S( X[j] ) where X[j] is the process equal to X on [0, j /T ] and then constant on [ j /T , 1]-see Appendix 5.A.6. With this trick, we have, for any 1 ≤ ≤ p, α ,θ , S [0, j /T ] ( X) T = α ,θ , S( X[j] ) T , so that we are back in H . Observe that the only difference with (5.12) is that we consider vector-valued sequential outputs, which requires to introduce the process X[j] , but that the rationale is exactly the same. We let : (R p ) T × (R p ) T → R + be the L 2 distance, that is, for any y = (y 1 , . . . , y T ), y = (y 1 , . . . , y T ), (y, y ) = 1 T T j=1 y jy j 2 . It is assumed that y takes its values in a compact subset of R q , i.e., there exists K y > 0 such that y j ≤ K y .

Theorem 5.14. Assume that for all θ ∈ Θ, (A 1 ) and (A 2 ) are verified. Assume, in addition, that there exists a constant B > 0 such that for any

1 ≤ ≤ p, θ ∈ Θ, ξ α ,θ H ≤ B . Then with probability at least 1 -δ, R( θ n ) ≤ R n ( θ n ) + c 3 T + 4pc 4 B(1 -L) -1 √ n + 2c 5 log( 1 /δ) n , (5.15)
where

c 3 = sup θ c 1,θ + ψ op f θ ∞ + 2 √ pB(1 -L) -1 + 2K y , c 4 = B(1 -L) -1 + K y , and
c 5 = 4pB(1 -L) -1 c 4 + K 2 y . Proof. See Section 5.B.8.

Regularization and stability

In addition to providing a sound theoretical framework, framing deep learning in an RKHS provides a natural norm, which can be used for regularization, as shown for example in the context of convolutional neural networks by Bietti et al. (2019). This regularization ensures stability of predictions, which is crucial in particular in a small sample regime or in the presence of adversarial examples [START_REF] Gao | Black-box generation of adversarial text sequences to evade deep learning classifiers[END_REF][START_REF] Ko | POPQORN: Quantifying robustness of recurrent neural networks[END_REF]. In our binary classification setting, for any inputs x, x ∈ (R d ) T , by the Cauchy-Schwartz inequality, we have

z T -z T ≤ 2 ψ op c 1 T + ξ α θ ( X) -ξ α θ ( X ) ≤ 2 ψ op c 1 T + ξ α θ H S( X) -S( X ) T .
If x and x are close, so are their associated continuous processes X and X (which can be approximated for example by taking a piecewise linear interpolation), and so are their signatures.

The term S( X)-S( X ) T is therefore small (Friz and Victoir, 2010, Proposition 7.66). Therefore, when T is large, we see that the magnitude of ξ α θ H determines how close the predictions are.

A natural training strategy to ensure stable predictions, for the types of networks covered in this chapter, is then to penalize the problem by minimizing the loss R n (θ) + λ ξ α θ 2 H . From a computational point of view, it is possible to compute the norm in H , up to a truncation at N of the Taylor expansion, which we know by Proposition 5.9 to be reasonable. It remains that computing this norm is a non-trivial task, and implementing smart surrogates is an interesting problem for the future. Note however that computing the signature of the data is not necessary for this regularization strategy.

Numerical illustrations

This section is here for illustration purposes. Our objective is not to achieve competitive performance, but rather to illustrate the theoretical results. We refer to Appendix 5.D for implementation details.

Convergence of the Taylor expansion towards the solution of the ODE. We illustrate Proposition 5.9 on a toy example. The process X is a 2-dimensional spiral, and we take feedforward RNN with 2 hidden units. Repeating this procedure with 10 3 uniform random weight initializations, we observe in Figure 5.1a that the signature approximation converges exponentially fast in N . As seen in Figure 5.1b, the rate of convergence depends in particular on the norm of the weight matrices, as predicted by Proposition 5.10. However, condition (5.11) seems to be over-restrictive, since convergence happens even for weights with norm larger than the bound (we have 1/(8a 2 d) 0.01 here). Adversarial robustness. We illustrate the penalization proposed in Section 5.3.2 on a toy task that consists in classifying the rotation direction of 2-dimensional spirals. We take a feedforward RNN with 32 hidden units and hyperbolic tangent activation. It is trained on 50 examples, with and without penalization, for 200 epochs. Once trained, the RNN is tested on adversarial examples, generated with the projected gradient descent algorithm with Frobenius norm [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF], which modifies test examples to maximize the error while staying in a ball of radius ε. We observe in Figure 5.2 that adding the penalization seems to make the network more stable.

Comparison of the trained networks. The evolution of the Frobenius norm of the weights W F and the RKHS norm ξ α θ H during training is shown in Figure 5.3. This points out that the penalization, which forces the RNN to keep a small norm in H , leads indeed to learning different weights than the non-penalized RNN. The results also suggest that the Frobenius and RKHS norms are decoupled, since both networks have Frobenius norms of similar magnitude but very different RKHS norms. The figures show one random run, but we observe similar qualitative behavior on others.

Discussion and conclusion

Role of the discretization procedure. The starting point of the chapter was motivated by the fact that the classical residual RNN formulation coincides with an Euler discretization of the ODE (5.3). This choice of discretization translates into a 1 /T term in Theorems 5.12 and 5.14. However, we could have considered higher-order discretization schemes, such as Runge-Kutta schemes, for which the discretization error decreases as 1 /T p . Such schemes correspond to alternative architectures, which were already proposed by [START_REF] Wang | Runge-Kutta neural network for identification of dynamical systems in high accuracy[END_REF], among others.

At the limit, we could also consider directly the continuous model (5.3), as proposed by Chen et al. (2018a), in which case the discretization error term vanishes. Of course, such an option requires to be able to sample the continuous-time data at arbitrary times.

Long-term stability. RNN are known to be poor at learning long-term dependencies [START_REF] Bengio | The problem of learning long-term dependencies in recurrent networks[END_REF][START_REF] Hochreiter | Long short-term memory[END_REF]. This is reflected in the literature by performance bounds increasing in T , which is not the case of our results (5.13) and (5.15), seemingly indicating that we fail to capture this phenomenon. This apparent paradox is related to our assumption that the total variation of X is bounded. Indeed, if a time series is observed for a long time, then its total variation may become large. In this case, it is no longer valid to assume that X TV is bounded by L. In other words, in our context, the parameter encapsulating the notion of "long-term" is not T but the regularity of X measured by its total variation. Note that the choice of defining X on [0, 1] and not another interval [0, U ] is arbitrary and does not carry any meaning on the problem of learning long-term dependencies. A thorough analysis of these questions is an interesting research direction for future work.

Radius of convergence. The assumptions

X T V ;[0,1] ≤ L < 1 and W F ≤ K W < (1 - L)/32d
can be seen as radii of convergence of the Taylor expansion (5.10). They allow using the Taylor approximation-which is of a local nature-to prove a global result, the RKHS embedding. In return, the condition on the Frobenius norm of the weights puts restrictions on the admissible parameters of the neural network. However, this bound can be improved, in particular by considering more exotic norms, which we did not make explicit for clarity purposes.

Conclusion.

By bringing together the theory of neural ODE, the signature transform, and kernel methods, we have shown that a recurrent network can be framed in the continuous-time limit as a linear function in a well-chosen RKHS. In addition to giving theoretical insights on the function learned by the network and providing generalization guarantees, this framing suggests regularization strategies to obtain more robust RNN. We have only scratched the surface of the potentialities of leveraging this theory to practical applications, which is a subject of its own and will be tackled in future work.

5.A Some additional definitions and lemmas

5.

A.1 Writing the GRU and LSTM in the neural ODE framework GRU. Recall that the equations of a GRU take the following form: for any 1 ≤ j ≤ T ,

r j+1 = σ(W r x j+1 + b r + U r h j ) z j+1 = σ(W z x j+1 + b z + U z h j ) n j+1 = tanh W n x j+1 + b n + r j+1 * (U n h j + c n ) h j+1 = (1 -z j+1 ) * h j + z j+1 * n j+1 ,
where σ is the logistic activation, tanh the hyperbolic tangent, * the Hadamard product, r j the reset gate vector, z j the update gate vector, W r , U r , W z , U z , W n , U n weight matrices, and b r , b z , b n , c n biases. Since r j+1 , z j+1 , and n j+1 depend only on x j+1 and h j , it is clear that these equations can be rewritten in the form

h j+1 = h j + f (h j , x j+1 ).
We then obtain equation (5.1) by normalizing f by 1 /T .

LSTM. The LSTM networks are defined, for any 1 ≤ j ≤ T , by

i j+1 = σ(W i x j+1 + b i + U i h j ) f j+1 = σ(W f x j+1 + b f + U f h j ) g j+1 = tanh(W g x j+1 + b g + U g h j ) o j+1 = σ(W o x j+1 + b o + U o h j ) c j+1 = f j+1 * c j + i j+1 * g j+1 h j+1 = o j+1 * tanh(c j+1 ),
where σ is the logistic activation, tanh the hyperbolic tangent, * the Hadamard product, i j the input gate, f j the forget gate, g j the cell gate, o j the output gate, c j the cell state,

W i , U i , W f , U f , W g , U g W o , U o weight matrices, and b i , b f , b g , b o biases. Since i j+1 , f j+1 , g j+1
, o j+1 depend only on x j+1 and h j , these equations can be rewritten in the form

h j+1 = f 1 (h j , x j+1 , c j+1 ) c j+1 = f 2 (h j , x j+1 , c j ).
Let hj = (h j , c j ) be the hidden state defined by stacking the hidden and cell state. Then, clearly, h follows an equation of the form

hj+1 = f ( hj , x j+1 ).

We obtain (5.1) by subtracting hj and normalizing by 1 /T .

5.A.2 Picard-Lindelöf theorem

Consider a CDE of the form (5.8). We recall the Picard-Lindelöf theorem as given by Lyons et al. (2007, Theorem 1.3), and provide a proof for the sake of completeness.

Theorem 5.15 (Picard-Lindelöf theorem). Assume that X ∈ BV c ([0, 1], R d ) and that F is Lipschitz-continuous with constant K F . Then, for any H 0 ∈ R e , the differential equation (5.8) admits a unique solution

H : [0, 1] → R e . Proof. Let C ([s, t]), R e ) be the set of continuous functions from [s, t] to R e . For any [s, t] ⊂ [0, 1],
ζ ∈ R e , let Ψ be the function

Ψ : C ([s, t]), R e ) → C ([s, t], R e ) Y → v → ζ + v s F(Y u )dX u . For any Y, Y ∈ C ([s, t]), R e ), v ∈ [s, t], Ψ(Y ) v -Ψ(Y ) v ≤ v s F(Y u ) -F(Y u ) dX u ≤ v s F(Y u ) -F(Y u ) op dX u ≤ v s K F Y u -Y u dX u ≤ K F Y -Y ∞ v s dX u ≤ K F Y -Y ∞ X T V ;[s,t] .
This shows that the function Ψ is Lipschitz-continuous on C ([s, t]), R e ) endowed with the supremum norm, with Lipschitz constant K F X T V ; [s,t] . Clearly, the function t → X T V ;[0,t] is non-decreasing and uniformly continuous on the compact interval [0, 1]. Therefore, for any ε > 0, there exists δ > 0 such that

|t -s| < δ ⇒ X T V ;[0,t] -X T V ;[0,s] < ε. Take ε = 1 /K F . Then on any interval [s, t] of length smaller than δ, one has X T V ;[s,t] = X T V ;[0,t] -X T V ;[0,s] < 1 /K F ,
so that the function Ψ is a contraction. By the Banach fixedpoint theorem, for any initial value ζ, Ψ has a unique fixed point. Hence, there exists a solution to (5.8) on any interval of length δ with any initial condition. To obtain a solution on [0, 1] it is sufficient to concatenate these solutions.

A corollary of this theorem is a Picard-Lindelöf theorem for initial value problems of the form

dH t = f (H t , X t )dt, H 0 = ζ, (5.16)
where

f : R e × R d → R e , ζ ∈ R e .
Corollary 5.16. Assume that f is Lipschitz continuous in its first variable. Then, for any ζ ∈ R e , the initial value problem (5.16) admits a unique solution.

Proof. Let f X : (h, t) → f (h, X t ). Then the solution of (5.16) is solution of the differential equation

dH t = f X (H t , t)dt.
Let d = 1, ē = e + 1, and F be the vector field defined by

F : h → f X (h 1:e , h e+1 ) 1 ,
where h 1:e denotes the projection of h on its first e coordinates. Then, since f X is Lipschitz, so is the vector field F. Theorem 5.15 therefore applies to the differential equation

dH t = F(H t )dt, H 0 = (ζ , 0) .
Projecting this differential equation on the last coordinate gives dH e+1 t = dt, that is, H e+1 t = t. Projecting on the first e coordinates exactly provides equation (5.16), which therefore has a unique solution, equal to H 1:e .

5.A.3 Operator norm

Definition 5.17. Let (E, • E ) and (F, • F ) be two normed vector spaces and let f ∈ L (E, F ), where L (E, F ) is the space of linear functions from E to F . The operator norm of f is defined by

f op = sup u∈E, u E =1 f (u) F .
Equipped with this norm, L (E, F ) is a normed vector space.

This definition is valid when f is represented by a matrix.

5.A.4 Tensor Hilbert space

Let us first briefly recall some elements on tensor spaces. If e 1 , . . . , e d is the canonical basis of

R d , then (e i 1 ⊗ • • • ⊗ e i k ) 1≤i 1 ,...,i k ≤d is a basis of (R d ) ⊗k . Any element a ∈ (R d ) ⊗k can therefore be written as a = 1≤i 1 ,...,i k ≤d a (i 1 ,...,i k ) e i 1 ⊗ • • • ⊗ e i k ,
where

a (i 1 ,...,i k ) ∈ R. The tensor space (R d ) ⊗k is a Hilbert space of dimension d k , with scalar product a, b (R d ) ⊗k = 1≤i 1 ,...,i k ≤d a (i 1 ,...,i k ) b (i 1 ,...,i k )
and associated norm

• (R d ) ⊗k .
We now consider the space T defined by (5.6). The sum, multiplication by a scalar, and scalar product on T are defined as follows: for any a = (a 0 , . . . , a k , . . .

) ∈ T , b = (b 0 , . . . , b k , . . . ) ∈ T , λ ∈ R, a + λb = (a 0 + λb 0 , . . . , a k + λb k , . . . ) and a, b T = ∞ k=0 a k , b k (R d ) ⊗k ,
with the convention (R d ) ⊗0 = R.

Proposition 5.18. (T , +, •, •, • T ) is a Hilbert space.

Proof. By the Cauchy-Schwartz inequality, •, • T is well-defined: for any a, b ∈ T ,

| a, b T | ≤ ∞ k=0 | a k , b k (R d ) ⊗k | ≤ ∞ k=0 a k (R d ) ⊗k b k (R d ) ⊗k ≤ ∞ k=0 a k 2 (R d ) ⊗k 1/2 ∞ k=0 b k 2 (R d ) ⊗k 1/2 < ∞.
Moreover, T is a vector space: for any a, b ∈ T , λ ∈ R, since

a + λb = (a 0 + λb 0 , . . . , a k + λb k , . . . ),
and

∞ k=0 a k + λb k 2 (R d ) ⊗k = ∞ k=0 a k 2 (R d ) ⊗k + λ 2 ∞ k=0 b k 2 (R d ) ⊗k + 2λ ∞ k=0 a k , b k (R d ) ⊗k ≤ ∞ k=0 a k 2 (R d ) ⊗k + λ 2 ∞ k=0 b k 2 (R d ) ⊗k + 2λ a, b T < ∞,
we see that a + λb ∈ T . The operation •, • T is also bilinear, symmetric, and positive definite:

a, a T = 0 ⇔ ∞ k=0 a k 2 (R d ) ⊗k = 0 ⇔ ∀k ∈ N, a k 2 (R d ) ⊗k = 0 ⇔ ∀k ∈ N, a k = 0 ⇔ a = 0.
Therefore •, • T is an inner product on T . Finally, let (a (n) ) n∈N be a Cauchy sequence in T .

Then, for any n, m ≥ 0,

a (n) -a (m) 2 T = ∞ k=0 a (n) k -a (m) k 2 (R d ) ⊗k ,
so for any k ∈ N, the sequence (a

(n) k ) n∈N is Cauchy in (R d ) ⊗k . Since (R d ) ⊗k is a Hilbert space, (a (n) k ) n∈N converges to a limit a (∞) k ∈ (R d ) ⊗k . Let a (∞) = (a (∞) 0 , . . . , a (∞)
k , . . . ). To finish the proof, we need to show that a (∞) ∈ T and that a (n) converges to a (∞) in T . First, note that there exists a constant B > 0 such that for any n ∈ N,

a (n) T ≤ B.
To see this, observe that for ε > 0, there exists N ∈ N such that for any n ≥ N , a (n) -a (N ) T < ε, and so a

(n) T ≤ ε + a (N ) T . Take B = max( a (1) T , . . . , a (N ) T , ε + a (N ) T ). Then, for any K ∈ N, K k=0 a (n) k 2 (R d ) ⊗k ≤ a (n) T ≤ B.
Letting K → ∞, we obtain that a (∞) T ≤ B, and therefore a (∞) ∈ T . Finally, let ε > 0 and let N ∈ N be such that for any n, m ≥ N , a (n)a (m) T < ε. Clearly, for any K ∈ N,

K k=0 a (n) k -a (m) k 2 (R d ) ⊗k < ε 2 .
Letting m → ∞ leads to

K k=1 a (n) k -a (∞) k 2 (R d ) ⊗k < ε 2 ,
and letting K → ∞ gives

a (n) -a (∞) T < ε,
which completes the proof.

5.A.5 Bounding the derivatives of the logistic and hyperbolic tangent activations

Lemma 5.19. Let σ be the logistic function defined, for any x ∈ R, by σ(x) = 1 /(1+e -x ). Then, for any n ≥ 0,

σ (n) ∞ ≤ 2 n-1 n! .
Proof. For any x ∈ R, one has [START_REF] Minai | On the derivatives of the sigmoid[END_REF], Theorem 2)

σ (n) (x) = n+1 k=1 (-1) k-1 (k -1)! n + 1 k σ(x) k ,
where n k stands for the Stirling number of the second kind (see, e.g., [START_REF] Riordan | An Introduction to Combinatorial Analysis[END_REF]. Let

u n = n+1 k=1 (k -1)! n + 1 k for n ≥ 1 and u 0 = 1. Since 0 ≤ σ(x) ≤ 1, it is clear that |σ (n) (x)| ≤ u n .
Using the fact that the Stirling numbers satisfy the recurrence relation

n + 1 k = k n k + n k -1 ,
valid for all 0 ≤ k ≤ n, we have

u n = n k=1 (k -1)! k n k + n k -1 + n! = n k=1 k! n k + n-1 k=0 k! n k + n! = 2 n k=1 k! n k (since n 0 = 0) ≤ 2n n k=1 (k -1)! n k = 2nu n-1 .
Thus, by induction, u n ≤ 2 n-1 n!, from which the claim follows.

Lemma 5.20. Let tanh be the hyperbolic tangent function. Then, for any n ≥ 0,

tanh (n) ∞ ≤ 4 n n! .
Proof. Let σ be the logistic function. Straightforward calculations yield the equality, valid for any x ∈ R, tanh(x) = 2σ(2x) -1.

But, for any n ≥ 1,

tanh (n) (x) = 2 n+1 σ (n) (2x),
and thus, by Lemma 5.19,

tanh (n) ∞ ≤ 2 n+1 σ (n) ∞ ≤ 4 n n! .
The inequality is also true for n = 0 since tanh ∞ ≤ 1.

5.A.6 Chen's formula

First, note that it is straightforward to extend the definition of the signature to any interval [s, t] ⊂ [0, 1]. The next proposition, known as Chen's formula (Lyons et al., 2007, Theorem 2.9), tells us that the signature can be computed iteratively as tensor products of signatures on subintervals.

Proposition 5.21.

Let X ∈ BV c ([s, t], R d ) and u ∈ (s, t). Then S [s,t] (X) = S [s,u] (X) ⊗ S [u,t] (X).
Next, it is clear that the signature of a constant path is equal to 1 = (1, 0, . . . , 0, . . . ) which is the null element in T . Indeed, let Y ∈ BV c ([s, t], R d ) be a constant path. Then, for any k ≥ 1,

Y k [s,t] = k! • • • s≤u 1 <•••<u k ≤t dY u 1 ⊗ • • • ⊗ dY u k = k! • • • s≤u 1 <•••<u k ≤t 0 ⊗ • • • ⊗ 0 = 0. Now let X ∈ BV c ([0, 1], R d
) and consider the path X[j] equal to the time-augmented path X on [0, j /T ] and then constant on [ j /T , 1]-see Figure 5.4. We have by Proposition 5.21 According to Assumption (A 1 ), for any

S [0,1] ( X[j] ) = S [0, j /T ] ( X[j] ) ⊗ S [ j /T ,1] ( X[j] ) = S [0, j /T ] ( X) ⊗ 1 = S [0, j /T ] ( X).
h 1 , h 2 ∈ R e , x 1 , x 2 ∈ R d , one has f (h 1 , x 1 ) -f (h 2 , x 1 ) ≤ K f h 1 -h 2 and f (h 1 , x 1 ) -f (h 1 , x 2 ) ≤ K f x 1 -x 2 .
Under assumption (A 1 ), by Corollary 5.16, the initial value problem (5.3) admits a unique solution H. Let us first show that for any t ∈ [0, 1], H t is bounded independently of X. For any t ∈ [0, 1],

H t -H 0 = t 0 f (H u , X u )du ≤ t 0 f (H u , X u ) du = t 0 f (H u , X u ) -f (H 0 , X u ) + f (H 0 , X u ) du ≤ t 0 f (H u , X u ) -f (H 0 , X u ) + t 0 f (H 0 , X u ) du ≤ K f t 0 H u -H 0 du + t sup x ≤L f (H 0 , x) .
Applying Grönwall's inequality to the function t → H t -H 0 yields

H t -H 0 ≤ t sup x ≤L f (H 0 , x) exp t 0 K f du ≤ sup x ≤L f (H 0 , x) e K f := M.
Given that H 0 = h 0 = 0, we conclude that

H t ≤ M . Next, let f ∞ = sup x ≤L, h ≤M f (h, x).
By similar arguments, for any [s, t] ⊂ [0, 1], Grönwall's inequality applied to the function

t → H t -H s yields H t -H s ≤ (t -s) f ∞ e K f .
Therefore, for any partition (t 0 , . . . , t k ) of [s, t],

k i=1

H t i -H t i-1 ≤ f ∞ e K f k i=1 (t i -t i-1 ) ≤ f ∞ e K f (t -s),
and, taking the supremum over all partitions of [s, t],

H T V ;[s,t] ≤ f ∞ e K f (t -s).
In other words, H is of bounded variation on any interval [s, t] ⊂ [0, 1]. Let (t 0 , . . . , t T ) denote the regular partition of [0, 1] with t j = j /T . For any 1 ≤ j ≤ T , we have

H t j -h j = H t j-1 + t j t j-1 f (H u , X u )du -h j-1 - 1 T f (h j-1 , x j ) ≤ H t j-1 -h j-1 + t j t j-1 f (H u , X u ) -f (h j-1 , x j ) du. Writing f (H u , X u ) -f (h j-1 , x j ) = f (H u , X u ) -f (H u , x j ) + f (H u , x j ) -f (h j-1 , x j ) ≤ f (H u , X u ) -f (H u , x j ) + f (H u , x j ) -f (h j-1 , x j ) ≤ K f X u -x j + K f H u -h j-1 ,
we obtain

H t j -h j ≤ H t j-1 -h j-1 + K f t j t j-1 H u -h j-1 du + K f t j t j-1 X u -x j du ≤ H t j-1 -h j-1 + K f t j t j-1 H u -H t j-1 + H t j-1 -h j-1 du + K f T X T V ;[t j-1 ,t j ] ≤ 1 + K f T H t j-1 -h j-1 + K f T H T V ;[t j-1 ,t j ] + X T V ;[t j-1 ,t j ] .
By induction, we are led to

H t j -h j ≤ K f T j-1 k=0 1 + K f T k H T V ;[t k ,t k+1 ] + X T V ;[t k ,t k+1 ] ≤ K f T 1 + K f T T X T V ;[0,1] + H T V ;[0,1] ≤ K f e K f T L + f ∞ e K f ,
which concludes the proof.

5.B.2 Proof of Proposition 5.2

Let h ∈ R ē and let hi:j = ( hi , . . . , hj ) be its projection on a subset of coordinates. It is sufficient to take F defined by

F( h) = 0 e×d 2
t = 2 1 -L f ( H1:e t , X t ) 1 -L 2 dt = f ( H1:e t , X t )dt, H1:e 0 = h 0 ,
which is exactly (5.3).

5.B.3 Proof of Proposition 5.6

According to Lyons (2014, Lemma 5.1), one has

Xk [0,t] (R d ) ⊗k ≤ X k T V ;[0,t] .
Let (t 0 , . . . , t k ) be a partition of [0, t]. Then

k j=1 Xt j -Xt j-1 = k j=1 X t j -X t j-1 2 + 1 -L 2 2 (t j -t j-1 ) 2 ≤ k j=1 X t j -X t j-1 + 1 -L 2 k j=1 (t j -t j-1 ) = k j=1 X t j -X t j-1 + 1 -L 2 t.
Taking the supremum over any partition of [0, t] we obtain

X T V ;[0,t] ≤ X T V ;[0,t] + 1 -L 2 t ≤ L + 1 -L 2 = 1 + L 2 < 1,
and thus Xk [0,t] (R d ) ⊗k ≤ 1+L 2 k . It is then clear that S [0,t] ( X) T = ∞ k=0 Xk [0,t] 2 (R d ) ⊗k 1 /2 ≤ ∞ k=0 Xk [0,t] (R d ) ⊗k ≤ ∞ k=0 1 + L 2 k = 2(1 -L) -1 .

5.B.4 Proof of Proposition 5.9

We first recall the fundamental theorem of calculus for line integrals (also known as gradient theorem).

Theorem 5.22. Let g : R e → R be a continuously differentiable function, and let γ :

[a, b] → R e be a smooth curve in R e . Then b a ∇g(γ t )dγ t = g(γ b ) -g(γ a ),
where ∇g denotes the gradient of g.

The identity above immediately generalizes to a function g : R e → R e : b a

J(g)(γ t )dγ t = g(γ b ) -g(γ a ),
where J(g) ∈ R e×e is the Jacobian matrix of g. Let us apply Theorem 5.22 to the vector field F i between 0 and t, with γ = H. We have

F i (H t ) -F i (H 0 ) = t 0 J(F i )(H u )dH u = t 0 J(F i )(H u ) d j=1 F j (H u )dX u = d j=1 t 0 J(F i )(H u )F j (H u )dX u = d j=1 t 0 F j F i (H u )dX u .
Iterating this procedure (N -1) times for the vector fields F 1 , . . . , F d yields

H t = H 0 + d i=1 t 0 F i (H u )dX i u = H 0 + d i=1 t 0 F i (H 0 )dX i u + d i=1 t 0 d j=1 u 0 F j F i (H v )dX j v dX i u = H 0 + d i=1 F i (H 0 )S (i) [0,t] (X) + 1≤i,j≤d 0≤v≤u≤t F j F i (H v )dX j v dX i u = • • • = H 0 + N k=1 1≤i 1 ,...,i k ≤d F i 1 • • • F i k (H 0 ) 1 k! S (i 1 ,...,i k ) [0,t] (X) + 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] F i 1 • • • F i N +1 (H u 1 )dX i 1 u 1 • • • dX i N +1 u N +1 , where ∆ N ;[0,t] := {(u 1 , • • • , u N ) ∈ [0, t] N | 0 ≤ u 1 < • • • < u N ≤ t} is the simplex in [0, t] N . The 162 first (N + 1) terms equal H N t . Hence, H t -H N t = 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] F i 1 • • • F i N +1 (H u 1 )dX i 1 u 1 • • • dX i N +1 u N +1 ≤ 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] F i 1 • • • F i N +1 (H u 1 ) |dX i 1 u 1 | • • • |dX i N +1 u N +1 | ≤ 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] sup 1≤i 1 ,...,i N +1 ≤d, h ≤M F i 1 • • • F i N +1 (h) |dX i 1 u 1 | • • • |dX i N +1 u N +1 | ≤ Λ N +1 (F) 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] |dX i 1 u 1 | • • • |dX i N +1 u N +1 |.
Thus,

H t -H N t ≤ Λ N +1 (F) 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] |dX i 1 u 1 | • • • |dX i N +1 u N +1 | ≤ Λ N +1 (F) 1≤i 1 ,...,i N +1 ≤d ∆ N +1;[0,t] dX u 1 • • • dX u N +1 = Λ N +1 (F) d N +1 (N + 1)! [0,t] N +1 dX u 1 • • • dX u N +1 = Λ N +1 (F) d N +1 (N + 1)! t 0 dX u N +1 = Λ N +1 (F) d N +1 (N + 1)! X N +1 T V ;[0,t] ≤ Λ N +1 (F) d N +1 (N + 1)! .

5.B.5 Proof of Proposition 5.10

For simplicity of notation, since the context is clear, we now use the notation • instead of • (R e ) ⊗k . According to Proposition 5.1, the solution H of (5.4) verifies Ht ≤ M + L := M . We therefore place ourselves in the ball B M . Recall that for any

1 ≤ i 1 , . . . , i N ≤ d, h ∈ B M , F i 1 • • • F i N ( h) = J(F i 2 • • • F i N )( h)F i 1 ( h).
(5.17)

Linear case. We start with the proof of the linear case before moving on to the general case. When σ is chosen to be the identity function, each F i RNN is an affine vector field, in the sense that F i RNN ( h) = W i h + b i , where W i = 0 ē×ē , b i is the i + dth vector of the canonical basis of R e+d , and

W d+1 = 2 1-L W 0 d×ē and b d+1 = 2 1-L b 0 d . Since J(F i RNN ) = W i , we have, for any h ∈ R e+d and any 1 ≤ i 1 , . . . , i k ≤ d, F i 1 RNN • • • F i k RNN ( h) = W i k • • • W i 2 (W i 1 h + b i 1 ).
Thus, for any h ∈ B M ,

F i 1 RNN • • • F i k RNN ( h) ≤ W i k op • • • W i 2 op ( W i 1 op M + b i 1 ).
For i = d + 1, W i 1 op = 0, and so

Λ k (F RNN ) ≤ C W d+1 k-1 op , with C = W d+1 op M + max(1, 2(1 -L) -1 b ). Therefore, ∞ k=1 d k k! Λ k (F RNN ) ≤ Cd ∞ k=0 1 k! 2d(1 -L) -1 W op k-1 < ∞.
General case. In the general case, the proof is two-fold. First, we upper bound (5.17) by a function of the norms of higher-order Jacobians of F i 1 , . . . , F i N . We then apply this bound to the specific case F = F RNN . We refer to Appendix 5.C for details on higher-order derivatives in tensor spaces. Let F : R e → R e be a smooth vector field. If F (h) = (F 1 (h), . . . , F e (h)) , each of its coordinates F i is a function from R e to R, C ∞ with respect to all its input variables. We define the derivative of order k of F as the tensor field

J k (F ) : R e → (R e ) ⊗k+1 h → J k (F )(h),
where

J k (F )(h) = 1≤j,i 1 ,...,i k ≤e ∂ k F j (h) ∂h i 1 . . . ∂h i k e j ⊗ e i 1 ⊗ • • • ⊗ e i k .
We take the convention J 0 (F ) = F , and note that J(F ) = J 1 (F ) is the Jacobian matrix, and that J k (J k (F )) = J k+k (F ).

Lemma 5.23. Let A 1 , . . . , A k : R e → R e be smooth vector fields. Then, for any h ∈ R e

A k • • • A 1 (h) ≤ n 1 +•••+n k =k-1 C(k; n 1 , . . . , n k ) J n 1 (A 1 )(h) • • • J n k (A k )(h) ,
where C(k; n 1 , . . . , n k ) is defined by the following recurrence on k: C(1; 0) = 1 and for any n 1 , . . . , n k+1 ≥ 0,

C(k + 1; n 1 , . . . , n k+1 ) = k =1 C(k; n 1 , . . . , n -1, . . . , n k ) if n k+1 = 0, (5.18)
C(k + 1; n 1 , . . . , n k+1 ) = 0 otherwise.
Proof. We refer to Appendix 5.C for the definitions of the tensor dot product and tensor permutations, as well as for computation rules involving these operations. We show in fact by induction a stronger result, namely that there exist tensor permutations π p such that

A k • • • A 1 (h) = n 1 +•••+n k =k-1 1≤p≤C(k;n 1 ,...,n k ) π p J n 1 (A 1 )(h) • • • J n k (A k )(h) . (5.19)
Note that we do not make explicit the permutations nor the axes of the tensor dot operations since we are only interested in bounding the norm of the iterated star products. Also, for simplicity, we denote all permutations by π, even though they may change from line to line.

We proceed by induction on k. For k = 1, the formula is clear. Assume that the formula is true at order k. Then

J(A k • • • A 1 ) = n 1 +•••+n k =k-1 1≤p≤C(k;n 1 ,...,n k ) J π p [ J n 1 (A 1 ) • • • J n k (A k ) ] = n 1 +•••+n k =k-1 1≤p≤C(k;n 1 ,...,n k ) π p J[ J n 1 (A 1 ) • • • J n k (A k ) ] = n 1 +•••+n k =k-1 1≤p≤C(k;n 1 ,...,n k ) k =1 π p • π J n 1 (A 1 ) • • • J n +1 (A ) • • • J n k (A k ) .
In the inner sum, we introduce the change of variable p i = n i for i = and p = n + 1. This yields

J(A k • • • A 1 ) = p 1 +•••+p k =k k =1 1≤p≤C(k;p 1 ,...,p -1,...,p k ) π p • π J n 1 (A 1 ) • • • J n +1 (A ) • • • J n k (A k ) = p 1 +•••+p k+1 =k 1≤q≤C(k+1;p 1 ,...,p k+1 ) π q J n 1 (A 1 ) • • • J p k (A k ) ,
where in the last sum the only non-zero term is for p k+1 = 0. To conclude the induction, it remains to note that

A k+1 • • • A 1 = J(A k • • • A 1 ) A k+1 = J(A k • • • A 1 ) J 0 (A k+1 ).
Hence,

A k+1 • • • A 1 = p 1 +•••+p k+1 =k 1≤q≤C(k+1;p 1 ,...,p k+1 ) π q J n 1 (A 1 ) • • • J p k (A k ) J p k+1 (A k+1 ) = p 1 +•••+p k+1 =k 1≤q≤C(k+1;p 1 ,...,p k+1 ) π q J n 1 (A 1 ) • • • J p k (A k ) J p k+1 (A k+1 ) .
The result is then a consequence of (5.19) and of Lemma 5.29.

We now restrict ourselves to the case F = F RNN as defined by (5.5) and give an upper bound on the higher-order derivatives of the tensor fields F i 1 , . . . , F i N .

Lemma 5.24. For any i ∈ {1, . . . , d + 1}, h ∈ B M , for any k ≥ 0,

J k (F i RNN )( h) ≤ 2 1 -L W F k σ (k) ∞ .
Proof. For any

1 ≤ i ≤ d, F i RNN ( h) is constant, so J k (F 1 RNN ) = • • • = J k (F d RNN ) = 0. For i = d + 1, we have, for any 1 ≤ j ≤ e, ∂ k F d+1 RNN,j ( h) ∂ hi 1 . . . ∂ hi k = 2 1 -L k W ji 1 • • • W ji k σ (k) (W j• h + b),

165

where W j• denotes the jth row of W and for e + 1 ≤ j ≤ ē, F d+1 j = 0. Therefore,

J k (F d+1 RNN )( h) 2 ≤ 2 1 -L 2k 1≤j,i 1 ,...,i k ≤e |W ji 1 • • • W ji k σ (k) (W j• h + b)| 2 = 2 1 -L 2k σ (k) 2 ∞ j i |W ji | 2 k ≤ 2 1 -L 2k σ (k) 2 ∞ W 2k F .
We are now in a position to conclude the proof using condition (5.11). By Lemma 5.23 and 5.24, for any

1 ≤ i 1 , . . . , i N ≤ d + 1, F i 1 RNN • • • F i N RNN ( h) ≤ n 1 +•••+n N =N -1 C(N ; n N , . . . , n 1 ) J n N (F i N RNN )( h) • • • J n 1 (F i 1 RNN )( h) ≤ 2 1 -L W F N -1 n 1 +•••+n N =N -1 C(N ; n N , . . . , n 1 )a n 1 +1 n 1 ! • • • a n N +1 n N ! ≤ a 2 1 -L a 2 W F N -1 n 1 +•••+n N =N -1 C(N ; n N , . . . , n 1 )n 1 ! • • • n N ! .
Assume for the moment that C(N ; n N , . . . , n 1 ) is smaller than the multinomial coefficient N n N ,...,n 1 . Then, using the fact that there are n+k-1 k-1 weak compositions of n in k parts and Stirling's approximation, we have

Λ N (F) ≤ a 2 1 -L a 2 W F N -1 N ! × Card {n 1 + • • • + n N = N -1} ≤ a 2 1 -L a 2 W F N -1 N ! 2N -2 N -1 ≤ a 2 2 1 -L a 2 W F N -1 N ! 2N N ≤ a √ 2e π 8 1 -L a 2 W F N -1 N ! √ N . Hence, provided W F < (1-L) /8a 2 d, ∞ k=1 d k k! Λ k (F) ≤ ad √ 2e π ∞ k=1 8da 2 W F 1 -L k-1 1 √ k < ∞,
and (A 2 ) is verified.

To conclude the proof, it remains to prove the following lemma.

Lemma 5.25. For any k ≥ 1 and n 1 , . . . , n k ≥ 0, C(k; n 1 , . . . , n k ) ≤ k-1 n 1 ,...,n k . Proof. The proof is done by induction, by comparing the recurrence formula (5.18) with the following recurrence formula for multinomial coefficients:

k n 1 , . . . , n k+1 = k+1 =1 k -1 n 1 , . . . , n -1, . . . , n k+1 .
More precisely, for k = 1, C(1; 0) = 1 ≤ 0 0 = 1 and C(1; 1) = 0 ≤ 0 1 = 0. Assume that the formula is true at order k. Then, at order k + 1, there are two cases. If n k+1 = 0, C(k + 1; n 1 , . . . , n k+1 ) = 0, and the result is clear. On the other hand, if n k+1 = 0,

C(k + 1; n 1 , . . . , n k , 0) = k =1 C(k; n 1 , . . . , n -1, . . . , n k ) ≤ k =1 k -1 n 1 , . . . , n -1, . . . , n k ≤ k+1 =1 k -1 n 1 , . . . , n -1, . . . , n k+1 ≤ k n 1 , . . . , n k+1
.

5.B.6 Proof of Theorem 5.11

First, Propositions 5.1 and 5.2 state that if H is the solution of (5.4) and Proj denotes the projection on the first e coordinates, then

z T -ψ Proj( H1 ) = ψ(h T ) -ψ Proj( H1 ]) ≤ ψ op h T -Proj( H1 ) ≤ ψ op c 1 T .
For any 1 ≤ k ≤ N , we let D k ( H0 ) : (R d ) ⊗k → R e be the linear function defined by

D k ( H0 )(e i 1 ⊗ • • • ⊗ e i k ) = F i 1 • • • F i k ( H0 ), (5.20)
where e 1 , . . . , e d denotes the canonical basis of R d. Then, under assumptions (A 1 ) and (A 2 ), if Xk denotes the signature of order k of the path Xt = (X t , 1-L 2 t) , according to Propositions 5.9 and 5.10,

H1 = H0 + ∞ k=1 1 k! 1≤i 1 ,...,i k ≤d S (i 1 ,...,i k ) [0,t] (X)F i 1 • • • F i k ( H0 ) = ∞ k=1 1 k! D k ( H0 )(X k [0,t] ),
and

ψ • Proj( H1 ) = ψ • Proj ∞ k=0 1 k! D k ( H0 )( Xk ) = ∞ k=0 1 k! ψ • Proj D k ( H0 )( Xk ) ,
by linearity of ψ and Proj. Since the maps D k ( H0 ) : (R d ) ⊗k → R e are linear, the above equality takes the form

ψ • Proj( H1 ) = ∞ k=0 α k , Xk (R d ) ⊗k , (5.21)
where

α k ∈ (R d ) ⊗k is the coefficient of the linear map 1 k! ψ • Proj • D k ( H0 ) in the canonical basis. Let α = (α 0 , . . . , α k , . . . ). Under assumption (A 2 ), ∞ k=0 α k 2 (R d ) ⊗k ≤ ∞ k=0 1≤i 1 ,...,i k ≤d 1 k! 2 ψ 2 op F i 1 • • • F i k ( H0 ) 2 ≤ ψ 2 op ∞ k=0 1≤i 1 ,...,i k ≤d 1 k! 2 Λ k (F) 2 ≤ ψ 2 op ∞ k=0 d k k! Λ k (F)
This shows that α ∈ T , and therefore, using (5.21), we conclude

z T -α, S( X) T ≤ ψ op c 1 T .

5.B.7 Proof of Theorem 5.12

Let G = g θ : (R d ) T → R | g θ (x) = z T , θ ∈ Θ
be the function class of (discrete) RNN and

S = ξ α θ : X → R | ξ α θ (X) = α θ , S( X) T , θ ∈ Θ ,
be the class of their RKHS embeddings, where α θ is defined by (5.21). For any θ ∈ Θ, we let

R G (θ) = E[ (y, g θ (x))], and R S (θ) = E[ (y, ξ α θ ( X))],
and denote by R n,G and R n,S the corresponding empirical risks. We also let θ * G , θ * S , θ n,G , and θ n,S be the corresponding minimizers. We have

P y = g θ n,G (x) -R n,G ( θ n,G ) ≤ E (y, g θ n,G (x)) -R n,G ( θ n,G )
where

R B (α) = E[ (Y, ξ α ( X))] and R n,B (α) = 1 n n i=1 (Y (i) , ξ α ( X(i) )).
We now have reached a familiar situation where the supremum is over a ball in an RKHS. A slight extension of Bartlett and Mendelson (2002, Theorem 8) yields that with probability at least 1δ,

sup α∈T , α T ≤B |R B (α) -R n,B (α)| ≤ 4K ERad n (B) + 2BK (1 -L) -1 log( 1 /δ) 2n ,
where Rad n (B) denotes the Rademacher complexity of B. Observe that we have used the fact that the loss is bounded by 2BK (1 -L) -1 since, for any ξ α ∈ B, by the Cauchy-Schwartz inequality,

(y, ξ α ( X)) = φ(y α, S( X) T ) ≤ K |y α, S( X) T | ≤ K α T S( X) T ≤ 2K B(1 -L) -1 .
Finally, the proof follows by noting that Rademacher complexity of B is bounded by

Rad n (B) ≤ B n n i=1 K(X (i) , X (i) ) = B n n i=1 S( X(i) ) 2 T ≤ 2B(1 -L) -1 √ n .

5.B.8 Proof of Theorem 5.14

Let

G = g θ : (R d ) T → (R p ) T | g θ (x) = z 1 , . . . , z T , θ ∈ Θ be the function class of discrete RNN in a sequential setting. Let S = Γ θ : X → (R p ) T | Γ θ (X) = Ξ θ ( X[1] ), . . . , Ξ θ ( X[T ] ) ,
be the class of their RKHS embeddings, where X[j] is the path equal to X on [0, j /T ] and then constant on [ j /T , 1] (see Figure 5.4). For any X ∈ X ,

Ξ θ (a) =    α 1,θ , S( X) T . . . α p,θ , S( X) T    =    ξ α 1,θ (X) . . . ξ α p,θ (X)    ∈ R p ,
where (α 1,θ , . . . , α p,θ ) ∈ (T ) p are the coefficients of the linear maps

1 k! ψ • Proj • D k ( H0 ) : (R d ) ⊗k → R p , k ≥ 0,
in the canonical basis, where D k is defined by (5.20).

We start the proof as in Theorem 5.12, until we obtain

R G ( θ n,G ) -R n,G ( θ n,G ) ≤ sup θ |R G (θ) -R S (θ)| + sup θ |R S (θ) -R n,S (θ)| + sup θ | R n,G (θ) -R n,S (θ)|.
By definition of the loss, for any θ ∈ Θ,

|R G (θ) -R S (θ)| = E y, g θ (x) -y, Γ θ (X) ≤ E 1 T T j=1 y j -z j 2 -y j -Ξ θ ( X[j] ) 2 ≤ E 1 T T j=1 z j + Ξ θ ( X[j] ) -2y j , z j -Ξ θ ( X[j] ) ≤ E 1 T T j=1 z j + Ξ θ ( X[j] ) -2y j × z j -Ξ θ ( X[j] )
(by the Cauchy-Schwartz inequality).

According to inequality (5.14), one has

z j -Ξ θ ( X[j] ) ≤ ψ op c 1,θ T ,
where

c 1,θ = K f θ e K f θ L + f θ ∞ e K f θ . Moreover, Ξ θ ( X[j] ) 2 = p =1 α ,θ , S( X[j] ) T 2 ≤ p =1 α ,θ 2 T S( X[j] ) 2 T ≤ pB 2 2(1 -L) -1 2 , since S( X[j] ) T = S [0, j /T ] ( X) T ≤ S( X) T . This yields z j + Ξ θ ( X[j] ) -2y j ≤ z j + Ξ θ ( X[j] ) + 2 y j ≤ ψ op f θ ∞ + 2 √ pB(1 -L) -1 + 2K y .
Finally,

sup θ |R G (θ) -R S (θ)| ≤ c 3 2T , where c 3 = sup θ c 1,θ + ψ op f θ ∞ + 2 √ pB(1 -L) -1 + 2K y . One proves with similar arguments that sup θ | R n,G (θ) -R n,S (θ)| ≤ c 3 2T .
We now turn to the term sup

θ |R S (θ) -R n,S (θ)|. We have R S (θ) -R n,S (θ) = E[ (y, Γ θ (X))] - 1 n n i=1 (y (i) , Γ θ (X (i) )) = 1 T T j=1 E[ y j -Ξ θ ( X[j] )] 2 - 1 n n i=1 y (i) j -Ξ θ ( X(i) [j] ) 2 .
Therefore,

sup θ |R S (θ) -R n,S (θ)| ≤ 1 T T j=1 sup θ E[ y j -Ξ θ ( X[j] )] 2 - 1 n n i=1 y (i) j -Ξ θ ( X(i) [j] ) 2 .

5.C.2 Computation rules

We need to obtain two computation rules for the tensor dot product: bounding the norm (Lemma 5.29) and differentiating (Lemma 5.30).

Lemma 5.29. Let a ∈ (R e ) ⊗k , b ∈ (R e ) ⊗ . Then, for all p, q,

a p,q b (R e ) ⊗k+ -2d ≤ a (R e ) ⊗k b (R e ) ⊗ .
Proof. By the Cauchy-Schwartz inequality,

a p,q b 2 (R e ) ⊗k+ -2 = 1≤i 1 ,...,i k-1 ,j 1 ,...,j -1 ≤e (a p,q b) 2 (i 1 ,...,i k-1 ,j 1 ,...,j -1 ) = 1≤i 1 ,...,i k-1 ,j 1 ,...,j -1 ≤e 1≤j≤e
a (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) b (j 1 ,...,j q-1 ,j,jq,...,j -1 ) 2 ≤ i 1 ,...,i k-1 ,j 1 ,...,j -1 j a 2

(i 1 ,...,i p-1 ,j,ip,...,i k-1 ) j b 2

(j 1 ,...,j q-1 ,j,jq,...,j -1 )

≤ i 1 ,...,i k-1 ,j a 2
(i 1 ,...,i p-1 ,j,ip,...,i k-1 ) j 1 ,...,j -1 ,j b 2

(j 1 ,...,j q-1 ,j,jq,...,j -1 )

≤ a 2 (R e ) ⊗k b 2 (R e ) ⊗ .
Lemma 5.30. Let A : R e → (R e ) ⊗k , B : R e → (R e ) ⊗ be smooth vector fields, p ∈ {1, . . . , k}, q ∈ {1, . . . , }. Let A p,q B : R e → (R e ) ⊗k+ -2 be defined by A p,q B(h) = A(h) p,q B(h).

Then there exists a permutation π such that J(A p,q B) = π(J(A) p,q B) + A p,q J(B).

Proof. The left-hand side takes the form (J(A p,q B)) i 1 ,...,i k-1 ,j 1 ,...,j -1 ,m = j ∂A ∂h m (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) B (j 1 ,...,j q-1 ,j,jq,...,j -1 )

+ A (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) ∂B ∂h m (j 1 ,...,j q-1 ,j,jq,...,j -1 ) .

The first term of the right-hand side writes (J(A) p,q B) i 1 ,...,i k-1 ,m,j 1 ,...,j -1 = j ∂A ∂h m (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) B (j 1 ,...,j q-1 ,j,jq,...,j -1 ) , and the second one

(A p,q J(B)) i 1 ,...,i k-1 ,j 1 ,...,j -1 ,m = j A (i 1 ,...,i p-1 ,j,ip,...,i k-1 )
∂B ∂h m (j 1 ,...,j q-1 ,j,jq,...,j -1 ) .

Let us introduce the permutation π which keeps the first (k -1) axes unmoved, and rotates the remaining ones such that the last axis ends up in kth position. Then π(J(A) p,q B) i 1 ,...,i k-1 ,j 1 ,...,j -1 ,m = j ∂A ∂h m (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) B (j 1 ,...,j q-1 ,j,jq,...,j -1 ) .

Hence J(A p,q B) = π(J(A) p,q B) + A p,q J(B), which concludes the proof.

The following two lemmas show how to compose the Jacobian and the tensor dot operations with permutations. Their proofs follow elementary operations and are therefore omitted.

Lemma 5.31. Let A : R e → (R e ) ⊗k and π a permutation of {1, . . . , k}. Then there exists a permutation π of {1, . . . , k + 1} such that J(π(A)) = π(J(A)).

Lemma 5.32. Let a ∈ (R e ) ⊗k , b ∈ (R e ) ⊗ , p ∈ {1, . . . , k}, q ∈ {1, . . . , }, π a permutation of {1, . . . , k}. Then there exists p ∈ {1, . . . , k}, q ∈ {1, . . . , }, and a permutation π of {1, . . . , k + -2} such that π(a) p,q b = π(a p,q b).

The following result is a generalization of Lemma 5.30 to the case of a dot product of several tensors.

Lemma 5.33. For ∈ {1, . . . , k}, n ∈ N, let A : R e → (R e ) ⊗n be smooth tensor fields. For any (p ) 1≤ ≤k-1 and (q ) 1≤ ≤k-1 such that p ∈ {1, . . . , n }, q ∈ {1, . . . , n +1 }, there exist k permutations (π ) 1≤ ≤k such that

J(A 1 p 1 ,q 1 A 2 p 2 ,q 2 • • • p k-1 ,q k-1 A k ) = k =1 π [A 1 A 2 • • • J(A ) • • • A k ] ,
where the dot products of the right-hand side are along some axes that are not specify for simplicity.

Proof. The proof is done by induction on k. The formula for k = 1 is straightforward. Assume that the formula is true at order k. As before, we do not specify indexes for tensor dot products as we are only interested in their existence. By Lemma 5.32, we have

J(A 1 • • • A k+1 ) = J((A 1 • • • A k ) A k+1 ) = π(J(A 1 • • • A k ) A k+1 ) + A 1 • • • A k J(A k+1 ) = π k =1 π [A 1 A 2 • • • J(A ) • • • A k ] A k+1 + A 1 • • • A k J(A k+1 ) = π k =1 π [A 1 A 2 • • • J(A ) • • • A k A k+1 ] + A 1 • • • A k J(A k+1 ) = k =1 π [A 1 A 2 • • • J(A ) • • • A k A k+1 ] + A 1 • • • A k J(A k+1 ) (where π = π • π) = k+1 =1 π [A 1 A 2 • • • J(A ) • • • A k A k+1 ] .

5.D Experimental details

All the code to reproduce the experiments is available on GitHub at https://github.com/aferm anian/rnn-kernel. Our experiments are based on the PyTorch (Paszke et al., 2019) framework. When not specified, the default parameters of PyTorch are used.

Convergence of the Taylor expansion. For Figure 5.1, 10 3 random RNN with 2 hidden units are generated, with the default weight initialization. The activation is either the logistic or the hyperbolic tangent. In Figure 5.1b, only the results with the logistic activation are plotted. The process X is taken as a 2-dimensional spiral. The reference solution to the ODE (5.3) is computed with a numerical integration method from SciPy (Virtanen et al., 2020, scipy.integrate.solve_ivp with the 'LSODA' method). The signature in the step-N Taylor expansion is computed with the package Signatory [START_REF] Kidger | Signatory: Differentiable computations of the signature and logsignature transforms, on both CPU and GPU[END_REF].

The step-N Taylor expansion requires computing higher-order derivatives of tensor fields (up to order N ). This is a highly non-trivial task since standard deep learning frameworks are optimized for first-order differentiation only. We refer to, for example, Kelly et al. (2020), for a discussion on higher-order differentiation in the context of a deep learning framework. To compute it efficiently, we manually implement forward-mode higher-order automatic differentiation for the operations needed in our context (described in Appendix 5.C). A more efficient and general approach is left for future work. Our code is optimized for GPU. Penalization on a toy example. For Figure 5.2, the RNN is taken with 32 hidden units and hyperbolic tangent activation. The data are 50 examples of spirals, sampled at 100 points and labeled ±1 according to their rotation direction. We do not use batching and the loss is taken as the cross entropy. It is trained for 200 epochs with Adam [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with an initial learning rate of 0.1. The learning rate is divided by 2 every 40 epochs. For the penalized RNN, the RKHS norm is truncated at N = 3 and the regularization parameter is selected at λ = 0.1. Earlier experiments show that this order of magnitude is sensible. We do not perform hyperparameter optimization since our goal is not to achieve high performance. The initial hidden state h 0 is learned (for simplicity of presentation, our theoretical results were written with h 0 = 0 but they extend to this case). The accuracy is computed on a test set of size 1000. We generate adversarial examples using 50 steps of projected gradient descent (following Bietti et al., 2019). The whole methodology (data generation + training) is repeated 20 times. The average training time on a Tesla V100 GPU for the RNN is 8.5 seconds and for the penalized RNN 12 seconds. Libraries. We use PyTorch (Paszke et al., 2019) as our overall framework, Signatory [START_REF] Kidger | Signatory: Differentiable computations of the signature and logsignature transforms, on both CPU and GPU[END_REF] to compute the signatures, and SciPy [START_REF] Virtanen | SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python[END_REF] for ODE integration. We use Sacred [START_REF] Greff | The Sacred Infrastructure for Computational Research[END_REF] for experiment management.

6

Leveraging the two-timescale regime to demonstrate convergence of neural networks

We study the training dynamics of shallow neural networks, in a two-timescale regime in which the step sizes for the inner layer are much smaller than those for the outer layer. In this regime, we prove convergence of the gradient flow to a global optimum of the non-convex optimization problem in a simple univariate setting. The number of neurons needs not be asymptotically large for our result to hold, distinguishing our result from popular recent approaches such as the neural tangent kernel or mean-field regimes. Experimental illustration is provided, showing that the stochastic gradient descent behaves according to our description of the gradient flow and thus converges to a global optimum in the two-timescale regime, but can fail outside this regime.

Introduction

Artificial neural networks are among the most successful modern machine learning methods, in particular because their non-linear parametrization provides a flexible way to implement feature learning (see, e.g., Goodfellow et al., 2016, chapter 15). Following this empirical success, a large body of work has been dedicated to understanding their theoretical properties, and in particular to analyzing the optimization algorithm used to tune their parameters. It usually consists in minimizing a loss function through stochastic gradient descent (SGD) or a variant [START_REF] Bottou | Optimization methods for large-scale machine learning[END_REF]. However, the non-linearity of the parametrization implies that the loss function is non-convex, breaking the standard convexity assumption that ensures global convergence of gradient descent algorithms.

In this chapter, we study the training dynamics of shallow neural networks, i.e., of the form

f (x; a, u) = a 0 + m j=1 a j g(x; u j ) ,
where m denotes the number of hidden neurons, a = (a 0 , . . . , a m ) and u = (u 1 , . . . , u m ) denote respectively the outer and inner layer parameters, and g(x; u) denotes a non-linear function of x and u. The novelty of this work lies in the use of a so-called two-timescale regime [START_REF] Borkar | Stochastic approximation with two time scales[END_REF] to train the neural network: we set step sizes for the inner layer u to be an order of magnitude smaller than the step sizes of the outer layer a. This ratio is controlled by a parameter ε. In the regime ε 1, the neural network can be thought of as a fitted linear regression with slowly evolving features g(x; u j ), j = 1, . . . , m: this reduction enables us to precisely describe the movement of the inner layer parameters u j .

Our approach proves convergence of the gradient flow to a global optimum of the non-convex landscape with a fixed number m of neurons. The gradient flow can be seen as the simplifying yet insightful limit of the SGD dynamics as the step size h vanishes. Proving convergence with a fixed number of neurons contrasts with two other popular approaches that require to take the limit m → ∞: the neural tangent kernel (Jacot et al., 2018;[START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF]Zou et al., 2020a) and the mean-field approach (Chizat and Bach, 2018;[START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF][START_REF] Rotskoff | Trainability and accuracy of neural networks: An interacting particle system approach[END_REF][START_REF] Sirignano | Mean field analysis of neural networks: a central limit theorem[END_REF]. As a consequence, this chapter is intended as a step towards understanding feature learning with a moderate number of neurons.

While our approach through the two-timescale regime is general, our description of the solution of the two-timescale dynamics and our convergence results are specific to a simple example showcasing the approach. More precisely, we consider univariate data x ∈ [0, 1] and non-linearities of the form g(x; u j ) = σ(η -1 (xu j )), where u j is a variable translation parameter, η is a fixed dilatation parameter, and σ is a sigmoid-like non-linearity. Finally, we restrict ourselves to the approximation of piecewise constant functions.

Organization of this chapter. In Section 6.2, we detail our setting and state our main theorem on the convergence of the gradient flow to a global optimum. Section 6.3 articulates this chapter with related work. Section 6.4 provides a self-contained introduction to the two-timescale limit ε → 0. We explain how it simplifies the analysis of neural networks, and provides heuristic predictions for the movement of neurons in our setting. Section 6.5 gives a rigorous derivation of our result. We prove convergence first in the two-timescale limit ε → 0, then in the two-timescale regime with ε small but positive. Section 6.6 presents numerical experiments showing that the SGD dynamics follow closely those of the gradient flow in the two-timescale regime, and therefore exhibit convergence to a global optimum. On the contrary, SGD can fail to reach a global optimum outside the two-timescale regime. We give a conclusion in Section 6.7 before moving on to the details of the proofs of this chapter. In Section 6.A, we introduce additional notations that will be used throughout in the detailed proofs, then proceed to prove useful technical lemmas. We proceed in Section 6.B to present the detailed proofs of the results of the chapter. Finally, Section 6.C contains details about our experimental settings as well as some additional simulations.

Setting and main result

We present a setting in which a piecewise constant univariate function f * : [0, 1] → R is learned with gradient flow on a shallow neural network. Our notations are summarized on Figure 6.1. We begin by introducing our class of functions of interest. Definition 6.1. Let n 2, ∆v ∈ (0, 1), ∆f > 0 and M 1. We denote F n,∆v,∆f,M the class of functions f * : [0, 1] → R satisfying the following conditions:

• f * is piecewise constant: there exists

0 = v 0 < v 1 < • • • < v n-1 < v n = 1 and f * 0 , . . . , f * n-1 ∈ R such that ∀x ∈ (v i , v i+1 ), f * (x) = f * i ,
• for all i ∈ {1, . . . , n}, v iv i-1 ∆v,

• for all i ∈ {1, . . . , n -1},

|f * i -f * i-1 | ∆f ,
• for all i ∈ {0, . . . , n -1}, |f * i | M . Let us now define our class of neural networks. Consider σ : R → R an increasing, twice continuously differentiable non-linearity such that σ(x) = 0 if x -1/2, σ(x) = 1 if x 1/2, and σ -1/2 is odd. Then, our class of shallow neural networks is defined by

f (x; a, u) = a 0 + m j=1 a j σ η (x -u j ) , σ η (x) = σ(η -1 x) ,
where 0 < η ≤ 1 measure the sharpness of the non-linearity σ η . Note that that inner layer parameter u j determines the translation of the non-linearity; no parameterized multiplicative operation on x is performed in this layer. We refer to the parameter u as the "positions" of the neurons (or, sometimes, simply as the "neurons") and to the parameter a as the "weights" of the neurons. The quadratic loss is defined as

L(a, u) = 1 2 1 0 (f * (x) -f (x; a, u)) 2 dx .
We use gradient flow on L to fit the parameters a and u: they evolve according to the dynamics da dt (t) = -∇ a L(a(t), u(t)) , du dt (t) = -ε∇ u L(a(t), u(t)) , (6.1)

where ε corresponds to the ratio of the step sizes of the two iterations.

Main result. By leveraging the two-timescale regime where ε is small, our theorem shows that, with high probability, a neural network trained with gradient flow is able to recover an arbitrary piecewise constant function to an arbitrary precision. The proof is relegated to the Appendix. Assume that, at initialization, the positions u 1 , . . . , u m of the neurons are i.i.d. uniformly distributed on [0, 1] and their weights a 0 , . . . , a m are equal to zero. Then there exists Q 1 > 0 and Q 2 > 0 depending on ξ, δ, m, ∆f, M such that, if

η Q 1 , ε Q 2 , (6.3)
then, with probability at least 1δ, the solution to the gradient flow (6.1) is defined at least until T = 6 ε(∆f ) 2 , and

1 0 |f * (x) -f (x; a(T ), u(T ))| 2 dx ξ . Further, Q 1 = C 1 M 2 (m + 1) min δ 2 (∆f ) 2 (m + 1) 4 , ξ and Q 2 = C 2 δ 2 M 4 (m + 1) 17/2 min δ(∆f ) 2 m + 1
, ξ for some universal constants C 1 , C 2 > 0.

Proof. See Section 6.B.6.

For this result to hold, the inequality (6.2) requires the number of neurons in the neural network to be large enough. Note that the minimum number of neurons required to approximate the n pieces of f * is equal to n. If the length of all the intervals is of the same order of magnitude, then ∆v = Θ(1/n) and thus the condition is m = Ω(n(1 + log n + log 1/δ)). In this case, condition (6.2) only adds a logarithmic factor in n and δ. Moreover, the lower bound on m does not depend on the target precision ξ. Thus we observe some non-linear feature learning phenomenon: with a fixed number m of neurons, gradient flow on a neural network can approximate any element from the infinite-dimensional space of piecewise constant functions to an arbitrary precision.

The recovery result of Theorem 6.2 is provided under two conditions (6.3). The first one should not surprise the reader: the condition on η enables the non-linearity to be sharp enough in order to approximate well the jumps of the piecewise constant function f * . The novelty of our work lies in the condition on ε, that we refer to as the two-timescale regime. This condition ensures that the step sizes taken in the positions u are much smaller than the step sizes taken in the weights a. As a consequence, the weights a are constantly close to the best linear fit given the current positions u. This property decouples the dynamics of the two layers of the neural network; this enables a sharp description of the gradient flow trajectories and thus the recovery result shown above. This intuition is detailed in Section 6.4.

Related work

Two-timescale regime. Systems with two timescales, or slow-fast systems, have a long history in physics and mathematics, see Berglund and Gentz (2006, Chapter 2) for an introduction. In particular, iterative algorithms with two timescales have been used in stochastic approximation and optimization, see [START_REF] Borkar | Stochastic approximation with two time scales[END_REF] or Borkar (2009, Section 6). For instance, they are used in the training of generative adversarial networks, to decouple the dynamics of the generator from those of the discriminator (Heusel et al., 2017), in reinforcement learning, to decouple the value function estimation from the temporal difference learning [START_REF] Szepesvári | Algorithms for reinforcement learning[END_REF], or more generally in bilevel optimization, to decouple the outer problem dynamics from the inner problem dynamics [START_REF] Hong | A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic[END_REF]. However, to the best of our knowledge, the two-timescale regime has not been used to show convergence results for neural networks.

Layer-wise learning rates. Practitioners are interested in choosing learning rates that depend on the layer index to speed up training or improve performance. Using smaller learning rates for the first layers and higher learning rates for the last layer(s) improves performance for fine-tuning [START_REF] Howard | Universal language model fine-tuning for text classification[END_REF][START_REF] Ro | AutoLR: Layer-wise pruning and auto-tuning of learning rates in finetuning of deep networks[END_REF] and is a common practice for transfer learning (see, e.g., [START_REF] Li | Deep learning for computer vision course[END_REF]. Another line of work proposes to update layer-wise learning rates depending on the norm of the gradients on each layer [START_REF] Singh | Layer-specific adaptive learning rates for deep networks[END_REF][START_REF] You | Scaling SGD batch size to 32k for imagenet training[END_REF][START_REF] Ko | Not all layers are equal: A layer-wise adaptive approach toward large-scale DNN training[END_REF]. However, they aim to compensate the differences across gradient norms in order to learn all the parameters at the same speed, while on the contrary we enjoy the theoretical benefits of learning different speeds.

Theory of neural networks. A key novelty of the analysis of this chapter is that we show recovery with a fixed number of neurons. We now detail the comparison with other analyses.

The neural tangent kernel regime (Jacot et al., 2018;[START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF]Zou et al., 2020a) corresponds to small movements of the parameters of the neural network. In this case, the neural network can be linearized around its initial point, and thus behaves like a linear regression. However, in this regime, the neural network can approximate only a finite dimensional space of functions, and thus it is necessary to take m → ∞ to be able to approximate the infinite-dimensional space of piecewise constant functions to an arbitrary precision.

The mean-field regime (Chizat and Bach, 2018;[START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF][START_REF] Rotskoff | Trainability and accuracy of neural networks: An interacting particle system approach[END_REF][START_REF] Sirignano | Mean field analysis of neural networks: a central limit theorem[END_REF] describes the dynamics of two-layer neural networks in the regime m 1 through a partial differential equation on the density of neurons. This regime is able to describe some non-linear feature learning phenomena, but does not explain the observed behavior with a moderate number of neurons. In this chapter, we show that in the two-timescale regime, only a single neuron aligns with each of the discontinuities of the function f * . However, it should be noted that the neural tangent kernel and mean-field regimes have been applied to show recovery in a wide range of settings, while our work is restricted to the recovery of piecewise constant functions. Extending the application of the two-timescale regime is left for future work.

Our work includes a detailed analysis of the alignment of the positions of the neurons with the discontinuities of the target function f * . This is analogous to a line of work (see, e.g., [START_REF] Saad | On-line learning in soft committee machines[END_REF]; [START_REF] Goldt | Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup[END_REF]; [START_REF] Veiga | Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks[END_REF]) interested in the alignment of a "student" neural network with the features of a "teacher" neural network that generated the data, for high-dimensional Gaussian input. In general, the non-linear evolution equations describing this alignment are hard to study theoretically. On the contrary, thanks to the two-timescale regime and to the simple setting of this chapter, we are able to give a precise description of the movement of the neurons.

Our study bears high-level similarities with the recent work of Safran et al. (2022). In a univariate classification setting, they show that a two-layer neural network achieves recovery with a number of neurons analogous to (6.2): inversely proportional to the length of the smallest constant interval of the target, up to logarithmic terms in the number of constant intervals and in the failure probability. However, the two works are not comparable due to differences in the settings: Safran et al. (2022) consider classification with ReLU activations while we consider regression with sigmoid-like activations. More importantly, the authors do not use the two-timescale regime. Instead, by a specific scale of the initialization, they ensure that the neural network has a first phase in a lazy regime where the positions of the neurons do not move significantly. For the second rich phase, they describe the implicit bias of the limiting point; this approach does not lead to an estimate of the convergence time while the fine description of the two-timescale limit does.

A non-rigorous introduction to the two-timescale limit

This section introduces the core ideas of our analysis in a non-rigorous way. Section 6.4.1 introduces the limit of the dynamics when ε → 0, called the two-timescale limit. Section 6.4.2 applies the two-timescale limit to predict the movement of the neurons.

Introduction to the two-timescale limit

Let us consider the gradient flow equations (6.1) and perform the change of variables τ = εt:

da dτ = - 1 ε ∇ a L(a, u) , du dτ = -∇ u L(a, u) . (6.4)
In the two-timescale regime ε 1, the rate of the gradient flow in the weights a is much larger than then the rate in the positions u. Note that L is marginally convex in a, and thus, for a fixed u, the gradient flow in a must converge to a global minimizer of a → L(a, u). More precisely, assume that {σ η (.u 1 ), . . . , σ η (.u m )} forms an independent set of functions in L 2 ([0, 1]).

Then the global minimizer of a → L(a, u) is unique; we denote it as a * (u). In the limit ε → 0, we expect that a evolves sufficiently quickly with respect to u so that it converges instantaneously to a * (u). In other words, the gradient flow system (6.4) reduces to its so-called two-timescale limit when ε → 0:

a = a * (u) , du dτ = -∇ u L(a * (u), u) .
(6.5)

The two-timescale limit considerably simplifies the study of the gradient flow system because it substitutes the weights a, determined to be equal to a * (u). However, showing that (6.4) reduces to (6.5) requires some mathematical care, including checking that a * (u) is well-defined.

Remark 6.3 (abuse of notation). Equation (6.5) contains the ambiguous notation ∇ u L(a * (u), u): does it denote the gradient in u of the map L * : u → L(a * (u), u) or the gradient (∇ u L)(a, u) taken at a = a * (u)? In fact, by definition of a * (u), both quantities coincide:

(∇ u L * )(u) = da * du (u) (∇ a L)(a * (u), u) + (∇ u L)(a * (u), u) = (∇ u L)(a * (u), u) ,
where da * du denotes the differential of a * in u and, by definition of a * (u), (∇ a L)(a * (u), u) = 0. This is a special case of the envelope theorem (Border, 2015, Sec. 5.10).

The discussion in this section is not specific to the setting of Section 6.2. Using the twotimescale limit to decouple the dynamics of the outer layer a and the inner layer u is a general tool that may be used in the study of any two-layer neural network. We chose the specific setting of this chapter so that the two-timescale limit (6.5) can be easily studied, thereby showcasing the approach. The next section is devoted to a sketch of this study.

Sketch of the dynamics of the two-timescale limit

In this section, in order to simplify the exposition of the behavior of the two-timescale limit (6.5), we consider the limiting case η → 0. Note that this is coherent with Theorem 6.2 that requires η to be small. This limit is a neural network with a non-linearity equal to the Heaviside function

σ 0 (x) = 0 if x < 0 , σ 0 (x) = 1/2 if x = 0 , σ 0 (x) = 1 if x > 0 .
Note that σ 0 would be a poor choice of non-linearity in practice: as its derivative is 0 almost everywhere, the positions u would not move. However, it is a relevant tool to get an intuition about the dynamics of our system for a small η. Moreover, as we will see in Section 6.6, the dynamics sketched here match closely those of the SGD (with η > 0). The set {1, σ 0 (.u 1 ), . . . , σ 0 (.u m )} generates the space of functions that are piecewise constant with respect to the subdivision {u 1 , . . . , u m }. Furthermore, if u 1 , . . . , u m are distinct and in (0, 1), then this set is an independent set of functions in L 2 ([0, 1]). Thus a * (u) is well-defined and represents the coefficients of the best piecewise constant approximation of f * with subdivision {u 1 , . . . , u m }.

This quantity is straightforward to describe under the mild additional assumption that there are at least two neurons u j in each interval (v i-1 , v i ) between two points of discontinuity of f * . For each 1 i n, let u L i denote the largest position of neurons below v i and u R i denote the smallest position above v i (with convention u R 0 = 0 and u L n+1 = 1). By assumption,

0 = u R 0 < u L 1 < u R 1 < • • • < u L n < u R n < u L n+1 = 1 are distinct.
A simple computation then shows the following identities:

• for all i ∈ {1, . . . , n}, for all x ∈ (u

L i , u R i ), f (x; a * (u), u) = v i -u L i u R i -u L i f * i-1 + u R i -v i u R i -u L i f * i ,
• and for all i ∈ {1, . . . , n + 1}, for all x ∈ (u R i-1 , u L i ), f (x; a * (u), u) = f * i-1 , where we recall that f * i denotes the value of f * on the interval (v i , v i+1 ). Figure 6.2 illustrates the situation. Moreover, the loss L(a * (u), u), which is half of the square L 2 -error of this optimal approximation, can be written

L(a * (u), u) = 1 2 n-1 i=1 (v i -u L i )(u R i -v i ) u R i -u L i (f * i -f * i-1 ) 2 . (6.6)
The dynamics of the two-timescale limit (6.5) corresponds to the local optimization of the subdivision u in order to minimize the loss (6.6). A remarkable property of this loss is that it decomposes as a sum of local losses around the jump points v i for i ∈ {1, . . . , n -1}. Each element of the sum involves only the two neurons located at u L i and u R i . As a consequence, the dynamics of the two-timescale limit (6.5) decompose as n independent systems of two neurons u L i and u R i : for all i ∈ {1, . . . , n -1}, du

L i dτ = - dL du L i (a * (u), u) = + 1 2 (u R i -v i ) 2 (u R i -u L i ) 2 (f * i -f * i-1 ) 2 , du R i dτ = - dL du R i (a * (u), u) = - 1 2 (v i -u L i ) 2 (u R i -u L i ) 2 (f * i -f * i-1 ) 2 .
(6.7)

All neurons other than u L 1 , u R 1 , . . . , u L n-1 , u R n-1 do not play a role in the expression (6.6), thus they do not move in the two-timescale limit (6.5). The position u L i moves right and u R i moves left, until one of them hits the point v i . This shows that the positions of the neurons eventually align with the jumps of the function f * , and thus that the function f * is recovered.

Convergence of the gradient flow

In this section, we give precise mathematical statements leading to the convergence of the gradient flow to a global optimum, first in the two-timescale limit ε → 0, then in the two-timescale regime with ε small but positive. All proofs are relegated to the Appendix.

In the two-timescale limit

This section analyzes rigorously the two-timescale limit (6.5), which we recall for convenience:

a * (u) = argmin a L(a, u) , du dτ = -∇ u L(a * (u), u) . (6.8)
We start by giving a rigorous meaning to these equations. First, for L to be differentiable in u, we require the parameter η of the non-linearity to be positive. Second, for a * (u) to be well-defined, we need u → L(a, u) to have a unique minimum. Obviously, if the u i are not distinct, then the features {σ η (.u 1 ), . . . , σ η (.u m )} are not independent and thus the minimum can not be unique. We restrict the state space of our dynamics to circumvent this issue. For u ∈ [0, 1] m , we denote ∆(u) = min

0 j,k m+1, j =k |u j -u k | ,
with the convention that u 0 = -η/2 and u m+1 = 1 + η/2. Furthermore, let us define the set

U = {u ∈ [0, 1] m | ∆(u) > 2η} .
The proposition below shows that U gives a good candidate for a set supporting solutions of (6.8).

Proposition 6.4. For u ∈ U, the Hessian H(u) of the quadratic function L(., u) is positive definite and its smallest eigenvalue is greater than ∆(u) /8. In particular, L(., u) has a unique minimum a * (u).

Proof. See Section 6.B.1.

The bound on the Hessian is useful in the following, in particular in the proof of the following result.

Proposition 6.5. Let G(u) = ∇ u L(a * (u), u) for u ∈ U. Then G : U → R m is Lipschitz- continuous.
Proof. See Section 6.B.2.

Then, the Picard-Lindelöf theorem (see, e.g., Luk, 2017 for a self-contained presentation and Arnold, 1992 for a textbook) guarantees, for any initialization u(0) ∈ U, the existence and uniqueness of a maximal solution of (6.8) taking values in U. This solution is defined on a maximal interval [0, T max ) where it could be that T max < ∞ if u hits the boundary of U. However, the results below show that the target function f * is recovered before this happens (with high probability over the initialization), and thus that this notion of solution is sufficient for our purposes. To this aim, we first define some sufficient conditions that the initialization should satisfy. Definition 6.6. Let D be a positive real. We say that a vector of positions u ∈ [0, 1] m is D-good if (a) for all i ∈ {0, . . . , n -1}, there are at least 6 positions u j in each interval [v i , v i+1 ],

(b) ∆(u) D, and (c) for all i ∈ {1, . . . , n -1}, denoting u L i the position closest to the left of v i and u R i the position closest to the right, we have

|u R i + u L i -2v i | D.
Condition (a) is related to the fact that the derivation in Section 6.4.2 is valid only if there are at least two neurons per piece. Condition (b) indicates that the neurons have to be sufficiently spaced at initialization, which is not surprising since we have to guarantee that ∆(u(τ )) > 2η, that is, u(τ ) ∈ U, for all τ until the recovery of f * happens. Finally, condition (c) also helps to control the distance between neurons: although u L i and u R i move towards each other, as shown by (6.7), their distance can be controlled throughout the dynamics as a function of |u R i + u L i -2v i |. We can now state the Proposition showing the recovery in finite time. The proof resembles the sketch of Section 6.4.2 with additional technical details since we need to control the distance between neurons, and the fact that η > 0 makes the dynamics more delicate to describe. Proposition 6.7. Let f * ∈ F n,∆v,∆f,M . Assume that the initialization u(0) is D-good with D = 2 13/2 (m + 1) 1/2 M η 1/2 (∆f ) -1 . Then the maximal solution of (6.8) taking values in U is defined at least on [0, T ] for T = 6 /(∆f) 2 , and at the end of this time interval, there is a neuron at distance less than η from each discontinuity of f * .

Proof. See Section 6.B.3. This Proposition is the main building block to show recovery in the next Theorem, along with some high-probability bounds to ensure that an i.i.d. uniform initialization is D-good. Theorem 6.8. Let ξ, δ > 0, and f * a piecewise constant function from F n,∆v,∆f,M . Assume that the neural network has m neurons with m 6 ∆v 4 + log n + log 1 δ .

Assume that, at initialization, the positions u 1 , . . . , u m of the neurons are i.i.d. uniformly distributed on [0, 1]. Then there exists Q depending on ξ, δ, m, ∆f, M such that, if η Q , then, with probability at least 1δ, the maximal solution to the two-timescale limit (6.8) is defined at least until T = 6 (∆f ) 2 , and

1 0 |f * (x) -f (x; a * (u(T )), u(T ))| 2 dx ξ .
Furthermore, we have

Q = C M 2 min δ 2 (∆f ) 2 (m + 1) 5 , ξ n
for some universal constant C > 0.

Proof. See Section 6.B.4.

6.5.2 From the two-timescale limit to the two-timescale regime

We now briefly explain how the proof for the two-timescale limit can be adapted for the gradient flow problem in the two-timescale regime (6.1), that is with a small but non-vanishing ε. First note that the existence and uniqueness of the maximal solution to the dynamics (6.1) follow from the local Lipschitz-continuity of ∇ a L and ∇ u L with respect to both their variables. The heuristics of Section 6.4.1 indicate that, for ε small enough, at any time t, the weights a(t) are close to a * (u(t)), the global minimizer of L(•, u(t)). The next Proposition formalizes this intuition. Proposition 6.9. Assume that a(0) = 0 and that, for all s ∈ [0, t], there at least 2 positions u j (s) in each interval [v i , v i+1 ] and ∆(u(s))

D /2 for some D ≥ 32η. Finally, assume that ε 2 -16 D 2 M -2 (m + 1) -5/2 . Then

a(t) -a * (u(t)) 3M √ m + 1 exp -D 16 t + 2 17 M 3 (m + 1) 3 D 2 ε .
Proof. See Section 6.B.5.

The crucial condition in the Proposition is ∆(u(s)) ≥ D /2; it is useful to control the conditioning of the quadratic form L(•, u(s)). The Proposition shows that a(t)a * (u(t)) is upper bounded by the sum of two terms; the first term is a consequence of the initial gap between a(0) and a * (u(0)) and decays exponentially quickly. The second term is negligible in the regime ε 1.

Armed with this Proposition, we show that the two-timescale regime has the same behavior as the two-timescale limit and thereby prove Theorem 6.2.

Numerical experiments

We place ourselves in the setting of Section 6.2. We first compare the dynamics of the gradient flow in the two-timescale limit presented in Section 6.4.2 with the dynamics of SGD. To simulate the SGD dynamics, we assume that we have access to noisy observations of the value of f * ∈ F n,∆v,∆f,M : let (X p , Y p ) p 1 be i.i.d. random variables such that X p is uniformly distributed on [0, 1], and Y p = f * (X p ) + N p where N p is additive noise. The (one-pass) SGD updates are then given by a p+1 = a p -h∇ a (X p+1 , Y p+1 ; a p , u p ) , u p+1 = u p -εh∇ u (X p+1 , Y p+1 ; a p , u p ) , (6.9)

with (X, Y ; a, u) = 1 2 (Yf (X; a, u)) 2 . The experimental settings, as well as additional results, are given in the Appendix.

Remarkably, the dynamics of SGD in the two-timescale regime with η small match closely the gradient flow in the two-timescale limit with η = 0, as illustrated in Figure 6.3. This validates the use of the gradient flow to understand the training dynamics with SGD. Both dynamics are close until the two-timescale limit achieves perfect recovery of the target function, at which point the SGD stabilizes to a small non-zero error. The fact that SGD does not achieve perfect recovery is not surprising, since SGD is performed with η > 0 and f * is not in the span of {1, σ η (xu 1 ), . . . , σ η (xu m )} for any u 1 , . . . , u m and for η > 0. On the contrary, we simulated the dynamics of gradient flow for η = 0, as presented in Section 6.4.2, enabling perfect recovery in that case.

Next, we compare the SGD dynamics in the two-timescale regime (ε 1) and outside this regime (ε ≈ 1). In Figure 6.4, we see that the network trained by SGD (in orange) in the ) and the gradient flow in the two-timescale limit (6.5) with η = 0. In the left-hand plot, to align the SGD and the two-timescale limit, we take τ = εhp. In the right-hand plot, the target function is in blue, the gradient flow in the two-timescale limit is in green, and the SGD is in orange. two-timescale regime ε = 2 • 10 -5 , achieves near-perfect recovery. If we change ε to 1, while keeping all other parameters equal, the algorithm fails to recover the target function (Figure 6.5). This shows that, in our setting with a moderate number of neurons m, recovery can fail away from the two-timescale regime.

Note that the dynamics of the neurons in Figures 6.4 and 6.5 are different. In the two-timescale regime, only the neurons closest to a discontinuity move significantly, while the others do not. These dynamics correspond to the sketch of Section 6.4. Interestingly, it means that in this regime, the neural network learns a sparse representation of the target function, meaning that only n out of the m neurons are active after training. On the contrary, when ε = 1, all neurons move to align with discontinuities of the target function, thus the learned representation is not sparse. Furthermore, since the number of neurons is moderate, one of the discontinuities is left without any neuron.

Conclusion

The two-timescale regime decouples the dynamics of the two layers of the neural network.

As a consequence, it is a useful theoretical tool to simplify the evolution equations. In this chapter showcasing the approach, the two-timescale regime enables to show the alignment of the neurons with the discontinuities of the target function, and thus to prove recovery. We leave the exploration of the benefits of the two-timescale regime in other settings for future work. It would be interesting to prove theoretical results for SGD dynamics, in multivariate settings, or for other types of non-linearities, in particular the recovery of continuous piecewise affine functions by ReLU networks.

6.A Additional notations and technical lemmas

For a vector a, we denote a its 2 -norm, a 1 its 1 -norm and a ∞ its ∞ -norm. For matrices H, H denotes the operator norm associated to the 2 norm and H F denotes the Frobenius norm. Finally, for real-valued functions f , f ∞ denotes the supremum norm.

In all the appendix of this chapter, we denote u 0 = -η/2 and u m+1 = 1 + η/2. Note that σ η (xu 0 ) = 1 for all x ∈ [0, 1], meaning that σ η (•u 0 ) corresponds to the bias term. This notation allows to treat the bias term in a unified fashion with respect to the other terms of f (x; a, u). Since u i ∈ (0, 1) for i ∈ {1, . . . , m}, we assume in the following w.l.o.g. that the (u i ) 0 i m+1 are ordered in increasing order. Note that we prove in the following that the (u i ) 1 i m do not cross during the dynamics, so they remain ordered throughout the dynamics.

The proofs involve comparisons of some quantities when η > 0 and when η = 0. To avoid confusion, we make explicit the dependency of L on η 0, i.e., we let L η (a, u) in place of L(a, u) of the main part of the chapter, and similarly, when the argmin is well-defined and unique,

a * η (u) = argmin a∈R m+1 L η (a, u) ,
in place of a * (u). Similarly, we now make explicit the dependence of f on η 0, i.e., we denote

f η (x; a, u) = a 0 + m j=1 a j σ η (x -u j ) = m j=0 a j σ η (x -u j ) .
The Hessian of the quadratic function L η (•, u) is denoted H η (u) ∈ R (m+1)×(m+1) (in place of H(u)), and satisfies that, for i, j ∈ {0, . . . , m},

H η,ij (u) = 1 0 σ η (x -u i )σ η (x -u j )dx .
Also let, for η 0 and u ∈ R m , b η (u) ∈ R m+1 such that, for j ∈ {0, . . . , m}, b η,j (u) =

1 0 f * (x)σ η (x -u j )dx .
Finally, we let U η in place of U in the main part of the chapter. With these notations, we have, for η 0 and a, u ∈ R m , (6.11) We now move on to a series to lemmas that will be helpful in the proofs of Appendix 6.B.

∂L η ∂u j (a, u) = 1 0 ∂f η (x; a, u) ∂u j (f η (x; a, u) -f * (x)) dx = -a j 1 0 σ η (x -u j ) m k=0 a k σ η (x -u k ) -f * (x) dx . (6.10) and ∂L η ∂a j (a, u) = 1 0 ∂f η (x; a, u) ∂a j (f η (x; a, u) -f * (x)) dx = 1 0 σ η (x -u j ) m k=0 a k σ η (x -u k ) -f * (x) dx = H η,j (u) a -b η,j (u) .
Lemma 6.10. For η 0 and u ∈ R m , we have

b η (u) -b 0 (u) M η √ m + 1 and b η (u) M √ m + 1 .
Proof. For any j ∈ {0, . . . , m},

|b η,j (u) -b 0,j (u)| = 1 0 f * (x)(σ η (x -u j ) -σ 0 (x -u j ))dx f * ∞ 1 0 |σ η (x -u j ) -σ 0 (x -u j )| dx M η ,
where in the last step we use that f

* ∞ M and that σ η (x) = 0 for x -η/2, σ η (x) ∈ [0, 1] for -η/2 < x < η/2 and σ η (x) = 1 for x η/2.
Similarly,

|b η,j (u)| = 1 0 f * (x)σ η (x -u j )dx f * ∞ M .
Lemma 6.11. For η ≥ 0 and u ∈ U η , H η (u) = H 0 (u) + D η , where D η is a diagonal matrix whose elements are independent of u and bounded in absolute value by η/2.

Proof. Let i, j ∈ {0, . . . , m}, and denote c = max(u i , u j , 0). Then

H 0,ij (u) = 1 0 σ 0 (x -u i )σ 0 (x -u j )dx = 1 -c . If i = j = 0, max(u i , u j ) = -η/2, and H η,ij (u) = 1 = H 0,ij (u). If i = j = 0, H η,ij (u) = 1 0 σ η (x -c) 2 dx = 1 -c - η 2 + c+η/2 c-η/2 σ η (x -c) 2 dx = H 0,ij (u) - η 2 + η 1/2 -1/2 σ 2 .
Note that the last integral is non-negative and less than 1, hence

|H η,ij (u) -H 0,ij (u)| ≤ η/2. Finally, if i = j, since |u i -u j | > η, H η,ij (u) = 1 0 σ η (x -u i )σ η (x -u j )dx = 1 0 σ η (x -max(u i , u j ))dx . Furthermore, 0 < max(u i , u j ) < 1 -η 2 , thus H η,ij (u) = 1 0 σ η (x -c)dx = 1 -c - η 2 + c+η/2 c-η/2 σ η (x -c)dx = 1 -c ,
where the last equality comes from the oddness of σ -1/2. Lemma 6.12. For η > 0, let a *

η : u ∈ U η → a * η (u). Then a * η is differentiable and for any u ∈ U η , ∂a * η (u) ∂u 8 ∆(u) 2(m + 1) a * η (u) + M .
Proof. By Proposition 6.4 (whose proof does not rely on this lemma), for u ∈ U η , L η (•, u) has a unique minimizer a * η (u), which is equal to H η (u) -1 b η (u) by (6.11). Furthermore, H η and b η are differentiable with respect to u, hence a * η is also differentiable with respect to u, and we have

∂a * η (u) ∂u k = -H η (u) -1 ∂H η ∂u k (u)a * η (u) + H η (u) -1 ∂b η ∂u k (u) .
Denote w k (u) := ∂Hη ∂u k (u)a * η (u) and W (u) the (m + 1) × (m + 1) matrix formed by stacking column-wise the vectors (w k (u)) 0 k m . Then

∂a * η (u) ∂u = -H η (u) -1 W (u) + H η (u) -1 ∂b η ∂u (u) .
We now estimate the Frobenius norm of the matrix W (u). By Lemma 6.11, for u ∈ U η , H η (u) = H 0 (u) + D η . Take i, j ∈ {0, . . . , m}, then

H η,ij (u) = H 0,ij (u) + D η,ij = 1 0 σ 0 (x -u i )σ 0 (x -u j )dx + D η,ij = 1 -max(u i , u j , 0) + D η,ij . Hence ∂H η,ij ∂u k = 0 if i, j = k. Further, if i = k and j = k, ∂H η,ij ∂u k (u) = ∂ ∂u i (1 -max(u i , u j )) 1 . 190 Of course, the bound | ∂H η,ij
∂u k (u)| 1 also holds when j = k and i = j. Finally, a similar bound shows that

| ∂H η,ij ∂u k (u)| 2 when i = j = k. As a consequence, for k, i ∈ {0, . . . , m}, |w k,i (u)| m j=0 ∂H η,ij ∂u k (u) a * η,j (u) ≤ |a * η,k (u)| if i = k , |a * η,k (u)| + a * η (u) 1 if i = k . Thus W (u) F = m i=0 m k=0 |w k,i (u)| 2 1/2 m i=0 m k=0 |w k,i (u)| 2 1/2 m i=0 2 a * η (u) 1 2 1/2 = 2 √ m + 1 a * η (u) 1 .
With a reasoning similar to the above, note that ∂bη ∂u (u) is a diagonal matrix with diagonal entries in [-M, M ]. Finally, putting these elements together, using Proposition 6.4 and that W (u) W (u) F , we obtain

∂a * η (u) ∂u H η (u) -1 W (u) F + H η (u) -1 ∂b η (u) ∂u 8 ∆(u) 2 √ m + 1 a * η (u) 1 + M .
The following lemma gives exact formulae for the derivative of the loss L η with respect to the positions of the neurons, evaluated for a = a * 0 (u), that is the best piecewise constant approximation of f * with subdivision {u 1 , . . . , u m }. Note that the formulae are the same as in Section 6.4.2, but the derivation is slightly more intricate since we consider here the loss L η and not L 0 . Lemma 6.13. Take η > 0 and u ∈ U η such that there are at least two neurons on each piece

[v i , v i+1 ] of f * . Then, if u j does not flank a discontinuity of f * , ∂L η ∂u j (a * 0 (u), u) = 0.
Furthermore, for a discontinuity v i , denote u L i is the closest neuron to its left and u R i the closest neuron to its right.

If v i -u L i η 2 and u R i -v i η 2 , then ∂L η ∂u L i (a * 0 (u), u) = - 1 2 (u R i -v i ) 2 (u R i -u L i ) 2 (f * i -f * i-1 ) 2 , ∂L η ∂u R i (a * 0 (u), u) = 1 2 (v i -u L i ) 2 (u R i -u L i ) 2 (f * i -f * i-1 ) 2 .
Proof. In this proof, let us denote for simplicity a = a * 0 (u). At the condition that there is at least two neurons on each piece of f * , Section 6.4.2 gives the optimal approximation f 0 (x; a, u) of f * that is piecewise constant with respect to the subdivision {u 1 , . . . , u m }. As a consequence, we easily get the value of a. Namely, if u j does not flank a discontinuity of f * , the value of f 0 (x; a, u) is locally constant around u j , thus a j = 0. Plugging into (6.10), we obtain

∂L η ∂u j (a, u) = 0 .
Further, for a discontinuity v i , denote respectively a L i and a R i the coefficients associated to u

L i and u R i . At u L i , the value of f 0 (x; a, u) jumps from f * i-1 to v i -u L i u R i -u L i f * i-1 + u R i -v i u R i -u L i f * i , thus a L i = v i -u L i u R i -u L i f * i-1 + u R i -v i u R i -u L i f * i -f * i-1 = u R i -v i u R i -u L i (f * i -f * i-1 ) .
Similarly, we have

a R i = v i -u L i u R i -u L i (f * i -f * i-1 ) .
We now compute, using (6.10),

∂L η ∂u L i (a, u) = -a L i 1 0 σ η (x -u L i ) (f η (x; a, u) -f * (x)) dx = -a L i u L i +η/2 u L i -η/2 σ η (x -u L i ) (f η (x; a, u) -f * (x)) dx .
Using that ∆(u) > 2η and that there are at least two neurons on each piece of f * , we have that u

L i -v i-1 2η. Since, in addition, by assumption, v i -u L i η 2 , we get that for x ∈ u L i -η 2 , u L i + η 2 , f * (x) = f * i-1 . Moreover, using again ∆(u) 2η that σ η is equal to σ 0 on (-∞, -η/2] and [η/2, ∞), we have for x ∈ u L i -η 2 , u L i + η 2 , f η (x; a, u) = m k=0 a k σ η (x -u k ) = f 0 u L i - η 2 ; a, u + a L i σ η (x -u L i ) = f * i-1 + a L i σ η (x -u L i ) .
Thus we obtain

∂L η ∂u L i (a, u) = -a L i u L i +η/2 u L i -η/2 σ η (x -u L i )a L i σ η (x -u L i )dx = - (a L i ) 2 2 σ η η 2 2 -σ η - η 2 2 = - (a L i ) 2 2 = - 1 2 (u R i -v i ) 2 (u R i -u L i ) 2 (f * i -f * i-1 ) 2 .
The computation of

∂Lη ∂u R i (a, u) is similar.
Lemma 6.14. Consider η ≥ 0 and u ∈ U η such that there are at least two neurons on each piece

[v i , v i+1 ] of f * . Then, for all x ∈ [0, 1], |f η (x; a * 0 (u), u)| ≤ M .
Proof. In the case where η = 0, the result easily follows from the expressions for f 0 (x; a * 0 (u), u) provided in Section 6.4.2. We now assume η > 0.

Denote A * k (u) = k j=0 a * 0,j (u) (with the convention A * -1 (u) = 0). Recall the convention u 0 = -η/2. We compute

f η (x; a * 0 (u), u) = m k=0 a * 0,k (u)σ η (x -u k ) = m k=0 A * k (u) -A * k-1 (u) σ η (x -u k ) = m-1 k=0 A * k (u) (σ η (x -u k ) -σ η (x -u k+1 )) + A * m (u)σ η (x -u m ) Note that A * k (u) = lim x→u k + f 0 (x; a * 0 (u), u)
, and thus, from the case η = 0, we have |A * k (u)| M . Moreover, σ η is increasing and the u k are in increasing order. We thus get

|f η (x; a * 0 (u), u)| M m-1 k=0 (σ η (x -u k ) -σ η (x -u k+1 )) + σ η (x -u m ) = M σ η (x -u 0 ) M .
Lemma 6.15. Consider η > 0 and u ∈ U η such that there are at least two neurons on each piece

[v i , v i+1 ] of f * . Then, for j ∈ {0, . . . , m}, |a * 0,j (u)| 2M
and, for any a ∈ R m+1 ,

∂L η ∂u j (a, u) - ∂L η ∂u j (a * 0 (u), u) 2M ( √ m + 1 + 1) a -a * 0 (u) + √ m + 1 a -a * 0 (u) 2 .
Proof. The first statement of the Lemma comes from the explicit formulae for a * 0 (u) given in the proof of Lemma 6.13, namely each a * 0,j (u) is either zero or less in magnitude than the gap between two pieces of f * that is less than 2M .

By (6.10), we have

∂L η ∂u j (a, u) - ∂L η ∂u j (a * 0 (u), u) = a j 1 0 σ η (x -u j ) f η (x; a, u) -f * (x) dx -a * 0,j (u) 1 0 σ η (x -u j ) f η (x; a * 0 (u), u) -f * (x) dx |a j -a * 0,j (u)| 1 0 σ η (x -u j ) |f η (x; a * 0 (u), u) -f * (x)| dx + |a j | 1 0 σ η (x -u j ) |f η (x; a, u) -f η (x; a * 0 (u), u)| dx .
We bound the two terms separately. For the first term, we use Lemma 6.14.

1 0 σ η (x -u j ) |f η (x; a * 0 (u), u) -f * (x)| ≤ 1 0 σ η (x -u j ) (|f η (x; a * 0 (u), u)| + |f * (x)|) 2M 1 0 σ η (x -u j )dx 2M .
We now continue with the second term.

|f η (x; a, u) -f η (x; a * 0 (u), u)| = m k=0 (a k -a * 0,k (u))σ η (x -u k ) a -a * 0 (u) 1 ,
and thus

1 0 σ η (x -u j ) |f η (x; a, u) -f η (x; a * 0 (u), u)| dx a -a * 0 (u) 1 1 0 σ η (x -u j )dx a -a * 0 (u) 1 .
Returning to our initial upper bound, we obtain, using the first statement of the Lemma,

∂L η ∂u j (a, u) - ∂L η ∂u j (a * 0 (u), u) 2M a -a * 0 (u) + (|a * 0,j (u)| + |a j -a * 0,j (u)|) a -a * 0 (u) 1 2M a -a * 0 (u) + (2M + a -a * 0 (u) ) √ m + 1 a -a * 0 (u) = 2M ( √ m + 1 + 1) a -a * 0 (u) + √ m + 1 a -a * 0 (u) 2 .
Lemma 6.16. For η ≥ 0 and u ∈ U η ,

a * η (u) -a * 0 (u) 16M √ m + 1η ∆(u)
.

Proof. By (6.11), H η (u)a * η (u) = b η (u) and by (6.11) and by Lemma 6.11,

H η (u)a * 0 (u) = H 0 (u)a * 0 (u) + D η a * 0 (u) = b 0 (u) + D η a * 0 (u).
According to Proposition 6.4 (whose proof does not rely on this lemma), H η (u) is invertible with H η (u) -1 8/∆(u). We thus have

a * η (u) -a * 0 (u) = H η (u) -1 (H η (u)a * η (u) -H η (u)a * 0 (u)) ≤ 8 ∆(u) b η (u) -b 0 (u) -D η a * 0 (u) ≤ 8 ∆(u) b η (u) -b 0 (u) + D η a * 0 (u) ≤ 8 ∆(u) b η (u) -b 0 (u) + η 2 a * 0 (u) .
The result then unfolds from Lemmas 6.10 and 6.15.

Lemma 6.17. Let η > 0, u ∈ R m and a, a ∈ R m+1 . Then

∇ u L η (a, u) √ m + 1 a 2 + M a , ∇ a L η (a, u) √ m + 1( a √ m + 1 + M ) .
As a consequence of the second inequality, by the fundamental theorem of calculus for line integrals,

|L η (a, u) -L η (a , u)| √ m + 1 max( a , a ) √ m + 1 + M a -a .
Proof. Recall that, for all j ∈ {1, . . . , m}, and for all a, u ∈ R m ,

∂L η ∂u j (a, u) = -a j 1 0 σ η (x -u j ) m k=0 a k σ η (x -u k ) -f * (x) dx , ∂L η ∂a j (a, u) = 1 0 σ η (x -u j ) m k=0 a k σ η (x -u k ) -f * (x) dx .
From the first equality, we have

∂L η ∂u j (a, u) ≤ |a j | 1 0 |σ η (x -u j )| m k=1 |a k |σ η (x -u k ) + |f * (x)| dx |a j |( a 1 + M ) 1 0 |σ η (x -u j )|dx |a j |( a 1 + M ) .
As a consequence,

∇ u L η (a, u) a ( a 1 + M ) √ m + 1 a 2 + M a .
Similarly, from the second equality, we have

∂L η ∂a j (a, u) a 1 + M .
As a consequence,

∇ a L η (a, u) √ m + 1( a 1 + M ) = √ m + 1( a √ m + 1 + M ) .
Lemma 6.18. Consider η ≥ 0 and u ∈ U η such that there is a neuron at distance less than η from each discontinuity of f * and 3η ≤ ∆v. Then

1 0 |f η (x; a * η (u), u) -f * (x)| 2 dx 6M 2 ηn . Proof. By definition of a * η (u), 1 0 |f η (x; a * η (u), u) -f * (x)| 2 dx = min a∈R m+1 1 0 |f η (x; a, u) -f * (x)| 2 dx .
Thus it is enough to exhibit some a for which the latter integral is smaller than 6M 2 ηn to conclude.

We construct such an a as follows: set a 0 = f * (0), and for each discontinuity v i , set the coefficient of a neuron at distance less than η to the value f * if * i-1 and set all other neurons to zero. Note that the active neurons are distinct since 3η ≤ ∆v.

Then the neural network is equal to the target function everywhere except on an interval of size 3η/2 around each discontinuity, where they disagree (in infinite norm) by at most 2M . Then, with probability at least 1δ, the vector u is D-good with D = δ 6(m+1) 2 . Proof. We define the following events: (a) A is the event "there are at least 6 positions u j in each interval [v i , v i+1 ] for i ∈ {0, . . . , n-1}", (b) B is the event "∆(u) ≥ D", (c) for all i ∈ {1, . . . , n -1}, E i is the event "there are at least one neuron on the left and on the right of v i " and C i is the event "E i holds and |u R i + u L i -2v i | ≥ D". Note that by Definition 6.6, u is D-good if and only if the event A ∩ B ∩ ( i C i ) holds. To show that this holds with high probability, we bound the probability of the complement

A ∩ B ∩ i C i c = A c ∪ B c ∪ i C c i = A c ∪ B c ∪ i (C c i ∩ A) ⊂ A c ∪ B c ∪ i (C c i ∩ E i ) (as A ⊂ E i ) . Thus P(u is not D-good) ≤ P(A c ) + P(B c ) + n-1 i=1 P(C c i ∩ E i ) .
Below, we bound separately the three terms of the right-hand side.

(a) Denote m = m/6 . For any i ∈ {0, . . . , n -1}, the set

A i = {j ∈ {1, . . . , m } | u j ∈ [v i , v i+1 ]} is empty with probability (1 -(v i+1 -v i )) m
(1 -∆v) m . Thus by the union bound, the probability that at least one of A 1 , . . . , A n is empty is upper bounded by n(1 -∆v) m .

We now check that n(1 -∆v) m δ/18. Indeed,

m = m 6 ≥ m 6 -1 ≥ 3 + log n + log 1 δ ∆v ≥ log n + log 18 δ ∆v ≥ - log n + log 18 δ log(1 -∆v) ,
where we use ∆v ≤ 1, 3 log [START_REF] Belkin | Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation[END_REF], and log(1 -∆v) ≤ -∆v < 0. This gives the desired inequality.

In other words, the probability that at least one of the intervals [v i , v i+1 ] contains none of the u 1 , . . . , u m is bounded by δ/18. As a consequence, by the union bound, the probability that at least one of the intervals [v i , v i+1 ] contains strictly less than 6 of the u 1 , . . . , u m is bounded by δ/3, i.e., P(A c ) ≤ δ/3.

(b) Recall that by convention, u 0 = -η 2 and u m+1 = 1 + η 2 . For all i ∈ {0, . . . , m + 1}, denote I i = (u i -D, u i + D). Denote F j the event "u j ∈ I i for some i ∈ {0, . . . , m + 1}, i = j". Note that B c = ∪ m j=1 F j . Fix j = 1, . . . , m. By conditioning on u i for all i ∈ {0, . . . , m + 1}, i = j, we see that P(F j ) ≤ 2(m + 1)D. By the union bound,

P(B c ) ≤ 2m(m + 1)D ≤ δ 3 .
(c) Take i ∈ {1, . . . , n -1}. For convenience, we define the random variable u L i (resp. u R i ) on the full probability space by setting u L i = 0 (resp. u R i = 1) when there is no neuron on the left (resp. the right) of v i . We compute the joint cumulative distribution function of (u L i , u R i ) (with a convenient change of inequality): for all 0 ≤ y ≤ v i ≤ z ≤ 1,

P(u L i ≤ y, u R i ≥ z) = P(∀j ∈ {1, . . . , m}, u j / ∈ [y, z]) = (1 -(z -y)) m .
We observe that the joint cumulative distribution function of (u L i , u R i ) is a smooth function of (y, z) when (y, z) ∈ (0, v i ) × (v i , 1). Note that the events E i and {(u L i , u R i ) ∈ (0, v i ) × (v i , 1)} are equal up to a null set. Therefore, on this event, (u L i , u R i ) is an absolutely continuous random variable with density g : (0, v

i ) × (v i , 1) → R, g(y, z) = - ∂ 2 ∂y∂z P(u L i ≤ y, u R i ≥ z) = m(m -1) (1 -(z -y)) m-2 .
We compute

P(C c i ∩ E i ) = P({|u R i + u L i -2v i | ≤ D} ∩ E i ) = {0<y<v i <z<1} m(m -1) (1 -(z -y)) m-2 1 {|y+z-2v i |≤D} dydz .
We make the change of variables θ = zy, ν = z + y.

P(C c i ∩ E i ) = m(m -1) 2 {0< ν-θ 2 <v i < ν+θ 2 <1} (1 -θ) m-2 1 |ν-2v i |≤D dθdν ≤ m(m -1) 2 1 0 (1 -θ) m-2 dθ ∞ -∞ 1 |ν-2v i |≤D dν = Dm .
Using m ≥ 24/∆v ≥ 24n, we have

n-1 i=1 P(C c i ∩ E i ) ≤ (n -1)Dm ≤ δ 24 × 6 ≤ δ 3 .
This concludes the proof.

6.B Proofs of the results

6.B.1 Proof of Proposition 6.4

Let us lower-bound the smallest eigenvalue of H η (u) which is equal to

min a =1 a H η (u)a . Now for a ∈ R m+1 such that a = 1, a H η (u)a = m i,j=0 a i a j 1 0 σ η (x -u i )σ η (x -u j )dx = 1 0 m i=0 a i σ η (x -u i ) 2 dx . Since ∆u > 2η (because u ∈ U) and u 0 = -η /2, u m+1 = 1 + η /2, the intervals [u i + η/2, u i+1 -η/2] for i ∈ {0, . . . , m} are disjoint and included in [0, 1]. Thus a H η (u)a m i=0 u i+1 -η/2 u i +η/2 m i=0 a i σ η (x -u i ) 2 dx . Since σ(x) = 0 if x < -1/2 and σ(x) = 1 if x > 1/2, we have that σ η (x) = 0 if x < -η/2
and σ η (x) = 1 if x > η/2. Further recall that the u i are ordered in increasing order. As a consequence, .12) where in the last step, we used that ∆(u) > 2η and thus

a H η (u)a m i=0 u i+1 -η/2 u i +η/2 i k=0 a k 2 dx = m i=0 (u i+1 -u i -η) i k=0 a k 2 ∆(u) 2 m i=0 i k=0 a k 2 , ( 6
u i+1 -u i -η ∆(u) -η ∆(u) - ∆(u)/2 = ∆(u)/2. Now, denote c 0 = 0 and c i = i-1 k=0 a k . Then a = 1 writes m i=0 (c i+1 -c i ) 2 = 1. Furthermore, m i=0 (c i+1 -c i ) 2 = m i=0 c 2 i+1 + m i=0 c 2 i -2 m i=0 c i+1 c i 4 m+1 i=0 c 2 i . Hence m+1 i=0 c 2 i 1 4 ,
which shows in conjunction with (6.12) that the smallest eigenvalue of H η (u) is lower-bounded by ∆u 8 .

6.B.2 Proof of Proposition 6.5

To show that G(u) = (∇ u L η )(a * η (u), u) is Lipschitz-continuous on U η , we show that it is differentiable on U η and that its derivatives are uniformly bounded. The chain rule gives

∂G j ∂u k = m l=0 ∂a * η,l ∂u k (u) ∂ 2 L η ∂u j ∂a l (a * η (u), u) + ∂ 2 L η ∂u j ∂u k (a * η (u), u) .
From (6.10), using that σ is twice continuously differentiable, it can be checked that ∂Lη ∂u j is differentiable in both its arguments and its derivatives are uniformly upper-bounded when a is bounded. Furthermore, for u ∈ U η ,

a * η (u) H η (u) -1 b η (u) 8M √ m + 1 ∆(u)
, by Lemma 6.10 and Proposition 6.4. Finally, according to Lemma 6.12, a * η is differentiable with derivatives uniformly upper-bounded on U η . This concludes the proof.

6.B.3 Proof of Proposition 6.7

In this proof, we denote u L i (τ ) (resp. u R i (τ )) the position at time τ of the neuron that is at initialization closest to v i to the left (resp. the right). Note that because of the movement of the neurons, it could be that u L i (resp. u R i ) does not remain the neuron closest to the left (resp. the right) throughout the dynamics. Our proof discusses when this phenomenon occurs. Similarly, denote u LL i (resp. u RR i ) the neuron second closest to the left (resp. the right) of v i . Since the initialization is D-good, note that all these neurons are distinct.

Denote T the minimal time τ ∈ [0, T max ) such that ∆(u(τ )) D /2 or there are less than two neurons in some piece [v i , v i+1 ] of f * . Note that by assumption, ∆(u(0)) D > D/2 and there are at least 6 neurons in each interval at initialization, thus T > 0. Furthermore, using the trivial inequalities M ∆f /2, m + 1 1 and η 1/2 η, we have

D 2 = 2 11/2 M √ m+1 √ η ∆f 8η > 2η.
Recall that 2η is the quantity defining the set U η supporting the maximal solution of the equation (6.8). As a consequence, we do have T < T max . At the end of the proof, we check that T < T , by controlling carefully the movement of each neuron.

Let us first bound the difference between the dynamics of u and the dynamics that we would have if at each time τ , the weights a were given by a * 0 (u(τ )), the best approximation of f * by a piecewise constant function with subdivision u(τ ). For any τ < T and j ∈ {1, . . . , m}, by Lemma 6.15, we have

du j dτ (τ ) + ∂L η ∂u j (a * 0 (u(τ )), u(τ )) = ∂L η ∂u j (a * η (u(τ )), u(τ )) - ∂L η ∂u j (a * 0 (u(τ )), u(τ )) 2M ( √ m + 1 + 1) a * η (u(τ )) -a * 0 (u(τ )) + √ m + 1 a * η (u(τ )) -a * 0 (u(τ )) 2 . (6.13)
We are therefore led to bounding a * η (u(τ ))a * 0 (u(τ )) , as follows:

a * η (u(τ )) -a * 0 (u(τ )) 2 4 M √ m + 1η ∆(u(τ ))
(by Lemma 6.16)

2 5 M √ m + 1η D (since ∆(u(τ )) D /2) = D(∆f ) 2 2 8 M √ m + 1 (by definition of D).
Then the first term in (6.13) is less than

( √ m + 1 + 1)D(∆f ) 2 2 7 √ m + 1 D(∆f ) 2 2 6 ,
and the second term in (6.13) is less than

D 2 (∆f ) 4 2 16 M 2 √ m + 1 D(∆f ) 2 2 14 , using D ∆(u(0)) 1, ∆f 2M and m + 1 1.
Hence we obtain, for any τ < T and j ∈ {1, . . . , m},

du j dτ (τ ) + ∂L η ∂u j (a * 0 (u(τ )), u(τ )) D(∆f ) 2 60 =: ∆g (6.14)
Now, let us examine how the neurons move, by leveraging Lemma 6.13 that gives exact formulae for ∂Lη ∂u j (a * 0 (u(τ )), u(τ )). First, if u j is not next to a discontinuity, ∂Lη ∂u j (a * 0 (u(τ )), u(τ )) = 0, hence |u j (τ )u j (0)| (∆g)τ .

Let us now study what happens next to a discontinuity

v i . Denote (δf ) i = f * i -f * i-1 . W.l.o.g., assume that u R i (0) -v i > v i -u L i (
0) . In the reverse case, the proof is the same by swapping the roles of u L i and u R i , and of u LL i and u RR i . We are going to show that the dynamics of u L i are divided into two phases. Define T i as the minimal τ ∈ [0, T ] such that u L i (τ ) = v iη/2. In the first phase [0, T i ], we have u

L i (τ ) < v i -η 2
and u L i moves towards v i . In the second phase [T i , T ], we show below that u

L i (τ ) ∈ [v i -η, v i + η]. Note that we can have T i = 0 if u L i (0) v i -η 2 .
It is also possible that T i = ∞ a priori; this means that the second phase does not exist. We show below that this case does not happen. Figure 6.6 depicts the two phases. Beginning by the first phase, we have, while u

L i (τ ) < v i -η 2 and u R i (τ ) > v i + η 2 , according to Lemma 6.13, ∂L η ∂u L i (a * 0 (u(τ )), u(τ )) = - 1 2 (u R i (τ ) -v i ) 2 (δf ) 2 i (u R i (τ ) -u L i (τ )) 2 , ∂L η ∂u R i (a * 0 (u(τ )), u(τ )) = 1 2 (v i -u L i (τ )) 2 (δf ) 2 i (u R i (τ ) -u L i (τ )) 2 .
For ease of computation, let

d L i (τ ) = v i -u L i (τ ) and d R i (τ ) = u R i (τ ) -v i
be the distances between the neurons and v i . Then, by (6.14),

dd R i dτ (τ ) + dd L i dτ (τ ) - 1 2 ((d R i (τ )) 2 + (d L i (τ )) 2 )(δf ) 2 i (d L i (τ ) + d R i (τ )) 2 + 2∆g - (∆f ) 2 4 + 2 D(∆f ) 2 60 - (∆f ) 2 5
since D ∆(u(0)) 1. Thus, in some time less than

T = 6 (∆f ) 2 , d R i (τ ) + d L i (τ ) η, that is, either u L i reaches v i -η 2 or u R i reaches v i + η 2 .
Let us check that the second event cannot actually happen: while u

L i (τ ) < v i -η 2 and u R i (τ ) > v i + η 2 , we also have dd R i dτ (τ ) - dd L i dτ (τ ) ((d R i (τ )) 2 -(d L i (τ )) 2 )(δf ) 2 i (d L i (τ ) + d R i (τ )) 2 -2∆g = (d R i (τ ) -d L i (τ ))(δf ) 2 i d L i (τ ) + d R i (τ ) -2∆g .
By condition (c) of Definition 6.6 and by (6.14), we have

d R i (0) -d L i (0) D = 60∆g (∆f ) 2 ≥ 60∆g (δf ) 2 i , and furthermore d L i (τ ) + d R i (τ ) ≤ 1. An easy reasoning then shows that d R i -d L i is increasing. Therefore u R
i must remain further away from v i than u L i . In summary, we showed that there exists some time

T i T when u L i (T i ) = v i -η 2
, which marks the end of the first phase, and we also have

d R i (T i ) -d L i (T i ) d R i (0) -d L i (0) D .
Moving on to the study of the second phase, let us show that u L i (τ ) stays in the interval

[v i -η, v i + η] for τ ∈ [T i , T ). Consider any τ T such that u L i (τ ) = v i -η.
Then we have by (6.14) and Lemma 6.13 (6.15) where the second upper bound comes from the fact that we have u

du L i dτ (τ ) (u R i (τ ) -v i ) 2 (δf ) 2 i (u R i (τ ) -v i + η) 2 -∆g ∆g ,
R i (τ ) -v i D 2 -η since ∆(u(τ )) D /2, and furthermore, x → x 2 (x+η) 2 is increasing, hence (u R i -v i ) 2 (δf ) 2 i (u R i -v i + η) 2 D 2 -η D 2 2 ∆f 2 (D/2 2η) (∆f ) 2 4 2∆g .
Equation (6.15) implies that u L i (τ ) v iη for all τ ∈ [T i , T ). Similarly, consider any τ T such that u L i (τ ) = v i + η. Note that, for such a τ , u L i (τ ) is now on the right of v i , and the neurons flanking v i are u LL i and u L i . Thus we have by (6.14) and Lemma 6.13

du L i dτ (τ ) - (v i -u LL i (τ )) 2 (δf ) 2 i (v i + η -u LL i (τ )) 2 + ∆g -∆g ,
where the second lower bound unfolds similarly as previously. This shows that u L i (τ ) v i + η for all τ ∈ [T i , T ).

We now check that T < T , that is, for all τ T , ∆(u(τ )) > D/2 and there are at least two neurons in each interval [v i , v i+1 ]. Starting with the first condition, we say that neurons u j and u k collide if |u j (τ )u k (τ )| = D/2 for some τ T . Let us show that no pair of neurons collide.

We start by showing that there is no collision between u LL i and u L i . In the first phase [0, T i ], we have du LL i dτ (τ ) ∆g. Recall that we also have

du L i dτ (τ ) -∆g and thus for τ T i , u L i (τ ) -u LL i (τ ) u L i (0) -u LL i (0) -2T ∆g 4D 5 since u L i (0) -u LL i (
u LL i (T i ) u L i (T i ) - 4D 5 = v i - η 2 - 4D 5 . (6.16) 201
In the second phase, we can have u L i ∈ [v i , v i + η] in which case u LL i becomes the neuron flanking v i to the left and u L i the neuron flanking to the right. Then (6.14) and Lemma 6.13 give

du LL i dτ (u L i (τ ) -v i ) 2 (δf ) 2 i (u L i (τ ) -u LL i (τ )) 2 + ∆g 16η 2 M 2 D 2 + ∆g .
Of course, this bound also holds when u

L i ∈ [v i -η, v i ], because then du LL i dτ
∆g. Thus, in the second phase τ ∈ [T i , T ], by the previous upper bound and the fact that u

L i (τ ) v i -η 2 , u L i (τ ) -u LL i (τ ) v i - η 2 -u LL i (T i ) + (τ -T i ) 16η 2 M 2 D 2 + ∆g 4D 5 -T 16η 2 M 2 D 2 +
= (∆f ) 2 D 2 2 13 (m + 1)M 2 . Thus 16η 2 M 2 T D 2 = 3(∆f ) 2 D 2 2 21 (m + 1) 2 M 2 D 10
using the definition of T , D ∆(u(0)) 1, ∆f 2M and m + 1 1. Finally, T ∆g = D /10. Thus u LL i and u L i do not collide. We now show that u L i and u R i do not collide. In the first phase τ ∈ [0, T i ], we have

u R i (τ ) -u L i (τ ) u R i (τ ) -v i = d R i (τ ) d R i (τ ) -d L i (τ ) D .
As a consequence, u L i and u R i do not collide during the first phase, and we have

u R i (T i ) D + u L i (T i ) = D + v i - η 2 .
(6.17)

In the second phase, u R i plays a role symmetric to u LL i : it can be, or not, the neuron closest to the right of v i , depending on whether u

L i ∈ [v i -η, v i ] or u L i ∈ [v i , v i + η]. As for u LL i , we can show that in any case, for τ ∈ [T i , T ], du R i dτ - 16η 2 M 2 D 2 -∆g .
Thus one concludes as before: for τ ∈ [T i , T ], by the previous lower bound and the fact that

u L i (τ ) v i + η 2 , u R i (τ ) -u L i (τ ) u R i (T i ) -(τ -T i ) 16η 2 M 2 D 2 + ∆g -v i + η 2 .
Then, by (6.17),

u R i (τ ) -u L i (τ ) D -η -T 16η 2 M 2 D 2 + ∆g > D 2 ,
where the last lower-bound unfolds similarly as for u LL i and u L i . Thus there is no collision between u L i and u R i .

202

The reader can check that all other pairs of neurons do not collide, including those involving u 0 = -η/2 and u m+1 = 1 + η/2. In fact, the proof is easier than for u LL i , u L i and u L i , u R i because the discontinuity at v i attracts these neurons together.

Furthermore, we proved that before time T at most one neuron can escape on each side of a piece [v i , v i+1 ] of f . Since we start with at least four (and even six) neurons per piece, there is always before T at least two neurons per piece.

This shows that T < T , and we also proved that at time T , all discontinuities have finished their first phase, hence there is a neuron at distance less than η from each discontinuity of the target function.

6.B.4 Proof of Theorem 6.8

Take C = 2 -19 . Then by assumption of Theorem 6.8,

η δ 2 (∆f ) 2 2 19 M 2 (m + 1) 5 .
Moreover, by the definition of D from Proposition 6.7,

η = (∆f ) 2 D 2 2 13 M 2 (m + 1)
.

This implies that

D 2 δ 2 2 6 (m + 1) 4 ,
and in consequence

D δ 6(m + 1) 2 .
Then Lemma 6.19 shows that the initialization is D-good with probability at least 1δ (since the D-good property is monotonous in D).

Hence, with probability at least 1δ, according to Proposition 6.7, the maximal solution to (6.8) is defined at least until T and at that time, there is a neuron at distance less than η from each discontinuity of the target function. Furthermore, 3η ≤ 1 m+1 ≤ 1 n ≤ ∆v, hence Lemma 6.18 applies. This implies that

1 0 |f * (x) -f (x; a * (u(T )), u(T ))| 2 dx 6M 2 ηn .
The assumption on η allow to conclude that the upper-bound is less than ξ. Remark 6.20. We did not try to optimize the value of C since our goal was to show convergence to a global optimum and the dependency of the dynamics on the parameters (for instance, it is remarkable that T does not depend on ξ).

6.B.5 Proof of Proposition 6.9

For s ≤ t, Proposition 6.4 holds since for ∆(u(s)) ≥ 16η > 2η. Thus a * η (u(s)) is well-defined and verifies ∇ a L η (a * η (u(s)), u(s)) = 0 . Let, for s ≤ t, V (s) = a(s)a * η (u(s)) . Recall that, by (6.11),

∇ a L η (a, u) = H η (u)a -b η (u) . Hence, for s ≤ t, a(s) -a * η (u(s)),∇ a L η (a(s), u(s)) = a(s) -a * η (u(s)), ∇ a L η (a(s), u(s)) -∇ a L η (a * η (u(s)), u(s)) = a(s) -a * η (u(s)), H η (u(s))(a(s) -a * η (u(s))) ∆(u(s)) 8 V (s) 2 ≥ D 16 V (s) 2 ,
where the first lower bound is a consequence of Proposition 6.4. Then we have, for any s ≤ t,

d ds 1 2 V (s) 2 = a(s) -a * η (u(s)), da ds (s) - d ds a * η (u(s)) = a(s) -a * η (u(s)), -∇ a L η (a(s), u(s)) - d ds a * η (u(s)) - D 16 V (s) 2 + d ds a * η (u(s)) V (s) .
Let us now upper bound the norm appearing in the second term. We first have by the chain rule

d ds a * η (u(s)) = ∂a * η ∂u (u(s)) du ds (s) .
By Lemma 6.12 (which holds since for ∆(u(s)) ≥ 16η > 2η),

∂a * η ∂u (u(s)) 8 ∆(u(s)) 2(m + 1) a * η (u(s)) + M .
Besides, du ds (s) ≤ ε ∇ u L η (a(s), u(s)) .

By Lemma 6.17,

∇ u L η (a(s), u(s)) √ m + 1 a(s) 2 + M a(s) . (6.18) Furthermore, a(s) ≤ a * η (u(s)) + a(s) -a * η (u(s)) = a * η (u(s)) + V (s) .
By Lemmas 6.15 and 6.16, which apply since ∆(u(s)) > 2η and since there are at least two positions u

j (s) in each interval [v i , v i+1 ] for s ≤ t, a * η (u(s)) a * 0 (u(s)) + a * 0 (u(s)) -a * η (u(s)) 2M √ m + 1 + 16M √ m + 1η ∆(u(s)) 2M √ m + 1 + 32M √ m + 1η D 3M √ m + 1 ,
where the last upper bound is implied by the assumption D ≥ 32η. Now define T max = inf s 0, V (s) > 3M √ m + 1 and assume s ≤ min(t, T max ) so that V (s) ≤ 3M √ m + 1. Then we proved that a(s) ≤ 6M √ m + 1. Going back to (6.18), we deduce that

∇ u L η (a(s), u(s)) 36M 2 (m + 1) 3/2 + 6M 2 √ m + 1 2 6 M 2 (m + 1) 3/2 . (6.19)
Putting everything together, we obtain

d ds a * η (u(s)) 2 9 M 2 (m + 1) 3/2 ∆(u(s)) 6M (m + 1) 3/2 + M ε ≤ 2 13 M 3 (m + 1) 3 D ε .
All in all, d ds

1 2 V (s) 2 - D 16 V (s) 2 + 2 13 M 3 (m + 1) 3 D εV (s) . Hence d ds (V (s)) = 1 V (s) d ds 1 2 V (s) 2 - D 16 V (s) + 2 13 M 3 (m + 1) 3 D ε .
By Grönwall's inequality, for all s min(t, T max ),

V (s) exp -D 16 s V (0) + 2 17 M 3 (m + 1) 3 D 2 ε(1 -exp -D 16 s ) (6.20) exp -D 16 s V (0) + 2 17 M 3 (m + 1) 3 D 2 ε . (6.21) Finally note that V (0) = a * η (0) 2M √ m + 1 and 2 17 M 3 (m+1) 3 ε D 2
2M √ m + 1 by the assumption of the Proposition on ε. Hence (6.20) implies that for all s min(t, T max ), V (s) is a (weighted) average of two terms less than 2M √ m + 1 hence it is less than 2M √ m + 1. This shows that T max t, which concludes the proof since (6.21) is then valid for s = t.

6.B.6 Proof of Theorem 6.2

In the proof, we take C 1 = 2 -21 and C 2 = 2 -36 . Denote D = δ 6(m + 1) 2 . Lemma 6.19 shows that the initialization is D-good with probability at least 1δ. In the following, we study the case where this event happens.

Denote T the minimal time t > 0 such that ∆(u(t)) D /2 or there are less than two neurons in some piece [v i , v i+1 ] of f * or a(t) > 7M √ m + 1. Note that T > 0 since the initialization is D-good. By Lemma 6.17, ∇ u L η and ∇ a L η are Lipschitz-continuous on compacts, hence the solution of the gradient flow is well-defined for t < T since T defines a compact set of parameters.

Then all the assumptions of Proposition 6.9 are satisfied on the time interval [0, t] for any t < T . More precisely, the assumptions that do not come directly from the definition of T are the lower bound for D and the upper bound for ε. The lower bound for D come from

D = δ 6(m + 1) 2 ≥ 32η (6.22)
by ( 6.3) and the simple bounds δ ≤ 1, ∆f ≤ 2M , m + 1 ≥ 1. The upper bound for ε comes from (6.3) since

ε ≤ δ 3 (∆f ) 2 2 36 M 4 (m + 1) 19/2 ≤ δ 2 36 • 2 16 M 2 (m + 1) 13/2 = D 2 2 16 M 2 (m + 1) 5/2 ,
where the second upper bound uses m ≥ 0, δ ≤ 1 and ∆f ≤ 2M . Therefore, according to Proposition 6.9,

a(t) -a * η (u(t)) 3M √ m + 1 exp -D 16 t + 2 17 M 3 (m + 1) 3 D 2 ε , (6.23)
Furthermore, the proof of Proposition 6.9 actually implies that

a * η (u(t)) 3M √ m + 1 and a(t) 6M √ m + 1. (6.24)
The second bound implies that the condition a(t) > 7M √ m + 1 in the definition of T is actually never active. Let us distinguish between two phases: letting

T 0 = 16 D log 2 16 M 2 (m + 1) 3 δ(∆f ) 2 = 96(m + 1) 2 δ log 2 16 M 2 (m + 1) 3 δ(∆f ) 2 ,
then the first phase corresponds to t ≤ T 0 and the second phase for t ≥ T 0 .

Analysis of the first phase. In the first phase, each neuron moves at most by

εT 0 max j ∂L η ∂u j (a(t), u(t)) εT 0 ∇ u L η (a(s), u(s)) 2 6 εT 0 M 2 (m + 1) 3/2 ,
where the second upper bound comes from (6.19) in the proof of Proposition 6.9. This quantity is less than D 8 if

6144(m + 1) 7/2 M 2 δ log 2 16 M 2 (m + 1) 3 δ(∆f ) 2 ε δ 48(m + 1) 2 .
Let us check this condition: we have

6144(m + 1) 7/2 M 2 δ log 2 16 M 2 (m + 1) 3 δ(∆f ) 2 ε = 16 • 6144(m + 1)
(m + 1) 7/2 M 2 δ log 2 16 M 2 (m + 1) 3 δ(∆f ) 2 ε 64 • 6144(m + 1) 9/2 M 3 δ 2 ∆f ε 6144δ(∆f ) 2 29 M (m + 1) 5 δ 48(m + 1) 2
using ∆f ≤ 2M and m ≥ 0. Note that the upper bound 2 6 εT 0 M 2 (m + 1) 3/2 ≤ D /8 also implies that Similarly, by condition (c) of the definition of a D-good vector, for all discontinuities v i ,

T 0 ≤ D 2 9 εM 2 (m + 1) 3/2 ≤ 1 2ε(∆f ) 2 = T 12 (
|u R i (0) + u L i (0) -2v i | D, thus |u R i (T 0 ) + u L i (T 0 ) -2v i | 3 4 D . (6.27)
Furthermore, there at least four neurons on each piece of f at T 0 , because at most one neuron can move out of each piece by either side between 0 and T 0 .

Analysis of the second phase. Let

∆a = D(∆f ) 2 2 9 M √ m + 1 = δ(∆f ) 2 6 • 2 9 M (m + 1) 5/2 .
In the second phase t ≥ T 0 , we are able to control by ∆a the distance between a(t) and the weights a * 0 (u(t)) that are the best approximation of f * by a piecewise affine function with subdivision u(t). To show this, first note that the first term in (6.23) is smaller than ∆a 4 when

3M √ m + 1 exp -D 16 t ∆a 4 ,
which is equivalent to

t log 12M √ m + 1 ∆a 16 D ,
which is implied by t T 0 . Furthermore, the second term in (6.23) is smaller than ∆a 4 because, by definition of D and by (6.3),

2 17 M 3 (m + 1) 3 D 2 ε = 36 • 2 17 M 3 (m + 1) 7 δ 2 ε ≤ 6 2 δ(∆f ) 2 2 19 M (m + 1) 5/2 = 6 3 ∆a 2 10 ≤ ∆a 4 .
Hence, for all T 0 t < T ,

a(t) -a * η (u(t)) ∆a 2 .
Furthermore, note that the assumption of Lemma 6.16 applies for t < T since ∆(u(t)) ≥ D 2 > 2η by (6.22). Therefore, by Lemma 6.16 and by (6.3),

a * η (u(t)) -a * 0 (u(t)) 2 4 M √ m + 1 ∆(u(t)) η ≤ 2 5 M √ m + 1 D η = 2 5 • 6M (m + 1) 5/2 δ η ≤ 6δ(∆f ) 2 2 16 M (m + 1) 5/2 = 6 2 ∆a 2 7 ∆a 2 .
By the triangular inequality, we deduce the upper bound that we were after, that is

a(t) -a * 0 (u(t)) ∆a .
As in the proof of Proposition 6.7, we can now control the distance between the true dynamics and the one that we would have if the weights were equal to a * 0 (u). Namely, for any T 0 ≤ t ≤ T and j ∈ {1, . . . , m}, by Lemma 6.15 (which applies since ∆(u(t)) > 2η by (6.22)), we have

du j dt (t) + ∂L η ∂u j (a * 0 (u(t)), u(t)) = ∂L η ∂u j (a(t), u(t)) - ∂L η ∂u j (a * 0 (u(t)), u(t)) 2M ( √ m + 1 + 1) a(t) -a * 0 (u(t)) + √ m + 1 a(t) -a * 0 (u(t)) 2 .
The first term is less than

2M ( √ m + 1 + 1)∆a = ( √ m + 1 + 1)D(∆f ) 2 2 8 √ m + 1 D(∆f ) 2 2 7 ,
and the second term is less than We are therefore in a situation very similar to the proof of Proposition 6.7, starting from (6.14).

√ m + 1(∆a) 2 = D 2 (∆f ) 4 2 18 M 2 √ m + 1 D(∆f ) 2 2
One can check that all the arguments used in the proof also apply here. On top of the estimate above that resembles (6.14), the crucial facts that make the argument of Proposition 6.7 work here are the bounds (6.26) and (6.27) as well as the fact that there are at least four neurons on each piece f at T 0 , which together are very similar to the conditions ensuring that u(0) is D-good in the proof of Proposition 6.7. Another key point is (6.25), ensuring that a time at least equal to 11T /12 remains after the first phase of this proof, which is enough time for the dynamics described in the proof of Proposition 6.7 to unfold. This yields that T < T , and that at time T , there is a neuron at distance less than η from each discontinuity of f * . Furthermore, 3η ≤ 1 m+1 ≤ 1 n ≤ ∆v, hence Lemma 6.18 applies. Thus

1 0 (f η (x; a * η (u(T )), u(T )) -f * (x)) 2 dx 6M 2 ηn ξ 2 ,
where the second upper bound comes from n ≤ m + 1 and from (6.3). Furthermore, by (6.24) and by Lemma 6.17,

|L η (a(T ), u(T )) -L η (a * η (u(T )), u(T ))| √ m + 1(6M (m + 1) + M ) a(T ) -a * η (u(T )) ≤ 16M (m + 1) 3/2 a(T ) -a * η (u(T )) .
Let us show that this term is less than ξ /4. Recall that, by (6.23),

a(T ) -a * η (u(T )) 3M √ m + 1 exp -D 16 T + 2 17 M 3 (m + 1) 3 D 2 ε .
By definition of D and T , by using exp(-x) ≤ 1 /x for x ≥ 1 and by (6.3),

16M (m + 1) 3/2 • 3M √ m + 1 exp -D 16 T = 48M 2 (m + 1) 2 exp - δ 16(m + 1) 2 (∆f ) 2 ε ≤ 48 • 16M 2 (m + 1) 4 (∆f ) 2 δ ε ≤ 48(∆f ) 2 δ 2 31 M 2 (m + 1) 9/2 ξ ξ 8
using ∆f 2M , δ ≤ 1, and m + 1 1. Furthermore, by (6.3), we get that

16M (m + 1) 3/2 • 2 17 M 3 (m + 1) 3 D 2 ε = 36 • 2 21 M 4 (m + 1) 17/2 δ 2 ε ξ 8 .
We therefore obtain the sought ξ /4 upper-bound and can conclude that

1 0 (f η (x; a(T ), u(T )) -f * (x)) 2 dx 1 0 (f η (x; a * η (u(T )), u(T )) -f * (x)) 2 dx + 2|L η (a(T ), u(T )) -L η (a * η (u(T )), u(T ))| ξ .

6.C Experimental details

Setting Our code is available at https://github.com/PierreMarion23/two-timescale-nn. To obtain Figures 6.3 and 6.4, we use the parameters of Table 6.1. For Figure 6.5, we use the parameters of Table 6 The number of iterations in Table 6.1 is much larger than the one in Table 6.2, due to the fact that the positions u evolve at a speed εh, which is much smaller in Table 6.1. However, note that it is possible to increase h in Table 6.1 while keeping the same behavior (in our experiment, h is kept to the same value as in Table 6.2 in order to facilitate the comparison). More precisely, taking h = 10 -3 in Table 6.1 yields similar results while dividing the computational cost by 100.

Our target function is defined by f * = 1 on [0., 0.2], [0.35, 0.5], [0.65, 0.8], f * = 2 on [0.5, 0.65] and f * = 4 elsewhere.

Additional plot We re-run the same SGD experiment as above twenty times, and plot the average L2 distance to the target as a function of ε, averaging over the initialization randomness

Introduction

Graphs are a common abstraction of real-world data. Large-scale knowledge bases can be represented as directed labeled graphs, where entities correspond to nodes and subject-predicateobject triplets are encoded by labeled edges. These so-called Knowledge Graphs (KGs) are used both in open knowledge projects (YAGO, Wikidata) and in the industry (Yahoo, Google, Microsoft, etc.). A prominent task on KGs is factual conversational Question Answering (Conversational KG-QA) and it has spurred interest recently, in particular due to the development of AI-driven personal assistants.

The Conversational KG-QA task involves difficulties of different nature: entity disambiguation, long tails of predicates [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF], conversational nature of the interaction. The topology of the underlying graph is also problematic. Not only can KGs be huge (up to several billion entities), they also exhibit hub entities with numerous neighbors.

A recent prominent approach has been to cast the problem as neural semantic parsing [START_REF] Jia | Data recombination for neural semantic parsing[END_REF]Dong andLapata, 2016, 2018;[START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF]. In this setting, a semantic parsing model learns to map a natural language question to a logical form (LF), i.e. a tree of operators over the KG. These operators belong to some grammar, either standard like SPARQL or ad-hoc. The logical form is then evaluated over the KG to produce the candidate answer. In the weak supervision training setup, the true logical form is not available, but only the answer utterance is (as well as annotated entities in some cases, see Section 7.5.4). Hence the training data is not given but it is instead generated, in the format of (question, logical form) pairs. We refer to this data as silver data or silver LFs, as opposed to unknown gold ground truth.

However, this approach has two main issues. First, the silver data generation step is a complex and often resource-intensive task. The standard procedure employs a Breadth-First Search (BFS) exploration (Guo et al., 2018;[START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF], but this simple strategy is prone to failure, especially when naively implemented, for questions that are mapped to nested LFs. This reduces the coverage, i.e. the percentage of training questions associated to a Logical Form. [START_REF] Shen | Effective search of logical forms for weakly supervised knowledge-based question answering[END_REF] proposes to add a neural component for picking the best operator, in order to reduce the computational cost of this task, however complicating the model. [START_REF] Cao | Unsupervised dual paraphrasing for two-stage semantic parsing[END_REF] proposes a two-step semantic parser: the question is first paraphrased into a "canonical utterance", which is then mapped to a LF. This approach simplifies the LF generation by separating it from the language understanding task.

Second, most of the semantic parsing models do not leverage much of the underlying KG structure to predict the LF, as in [START_REF] Dong | Language to logical form with neural attention[END_REF]; Guo et al. (2018). Yet, this contextual graph information is rich [START_REF] Tong | Leveraging Domain Context for Question Answering Over Knowledge Graph[END_REF], and graph-based models leveraging this information yield promising results for KG-QA tasks [START_REF] Vakulenko | Message passing for complex question answering over knowledge graphs[END_REF][START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF]. However these alternative approaches to semantic parsing, that rely on node classification, have their inherent limitations, as they handle less naturally certain queries (see Appendix 7.C.3) and their output is less interpretable. This motivates the desire for semantic parsing models that can make use of the KG context.

Approach, contributions and overview of the chapter. In Section 7.3, we design a new grammar, which can model a large range of queries on the KG, yet is simple enough for BFS to work well. We obtain a high coverage on two KG-QA datasets. On CSQA [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF], we achieve a coverage of 96%, a 16% improvement over the baseline [START_REF] Shen | Effective search of logical forms for weakly supervised knowledge-based question answering[END_REF]. On ConvQuestions [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], a dataset with a large variety of queries, we reach a coverage of 86%.

To leverage the rich information contained in the underlying KG, we propose in Section 7.4 a semantic parsing model that uses the KG contextual data in addition to the utterances. Different options could be considered for the KG context, e.g. lists of relevant entities, annotated with metadata or pre-trained entity embeddings that are graph-aware (Zhang et al., 2020b). The problem is that this information does not come as unstructured textual data, which is common for language models, but is structured.

To enable the use of context together with a strong language model, we propose the Object-Aware Transformer (OAT) model, which can take as input structured data in a JSON-like format. The model then transforms the structured input into embeddings, before feeding them into standard Transformer layers. With this approach, as reported in Section 7.5, we improve the overall execution accuracy on CSQA by 5.0% compared to a strong baseline [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF]. On ConvQuestions, we improve the precision by 4.7% compared to [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF].

Appendices 7.A to 7.D present further description of the experimental setup, comparisons with baselines, and experimental results.

Related work

Neural semantic parsing Our work falls within the neural semantic parsing approaches for Knowledge-Based QA [START_REF] Dong | Language to logical form with neural attention[END_REF][START_REF] Liang | Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision[END_REF][START_REF] Dong | Coarse-to-fine decoding for neural semantic parsing[END_REF]Guo et al., 2019b;[START_REF] Hwang | A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization[END_REF]. The more specific task of conversational KG-QA has been the focus of recent work. Guo et al. (2018) introduces D2A, a neural symbolic model with memory augmentation. This model has been extended by S2A+MAML (Guo et al., 2019a) with a meta-learning strategy to account for context, and by D2A+ES [START_REF] Shen | Effective search of logical forms for weakly supervised knowledge-based question answering[END_REF] with a neural component to improve BFS. [START_REF] Saha | Complex program induction for querying knowledge bases in the absence of gold programs[END_REF] proposes a Reinforcement Learning model to benefit from denser supervision signals. Finally, [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF] introduces MaSP, a multi-task model that performs both entity linking and semantic parsing, with the hope of reducing erroneous entity linking (see Appendix 7.C.2 for a comparison with our setup). Recently, [START_REF] Plepi | Context transformer with stacked pointer networks for conversational question answering over knowledge graphs[END_REF] extended the latter in CARTON. They first predict the LF using a Transformer architecture, then specify the KG items using pointer networks.

Learning on Knowledge Graphs Classical graph learning techniques can be applied to the specific case of KGs. In Convex [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], at each turn, a subgraph is expanded by matching the utterance with neighboring entities. Then a candidate answer is found by a node classifier. Other methods include unsupervised message passing [START_REF] Vakulenko | Message passing for complex question answering over knowledge graphs[END_REF]. However, these approaches lack strong NLP components. Other directions include learning differentiable operators over a KG [START_REF] Cohen | Neural Query Language: A Knowledge Base Query Language for Tensorflow[END_REF], or applying Graph Neural Networks [START_REF] Kipf | Semi-Supervised Classification with Graph Convolutional Networks[END_REF]Hamilton et al., 2017) (GNNs) to the KG, which has been done for entity classification and link prediction tasks [START_REF] Schlichtkrull | Modeling Relational Data with Graph Convolutional Networks[END_REF]. GNNs have also been used to model relationships between utterances and entities [START_REF] Shaw | Generating logical forms from graph representations of text and entities[END_REF].

Structured Input for neural models Our approach of using JSON-like input falls in the line of computing neural embeddings out of structured inputs. [START_REF] Tai | Improved semantic representations from tree-structured long short-term memory networks[END_REF] introduced Tree-LSTM for computing tree embeddings bottom-up. It has then been applied for many tasks, including computer program translation (Chen et al., 2018b), semantic tree structure learning such as JSON or XML [START_REF] Woof | A Framework for End-to-End Learning on Semantic Tree-Structured Data[END_REF], and supervised KG-QA tasks [START_REF] Tong | Leveraging Domain Context for Question Answering Over Knowledge Graph[END_REF][START_REF] Zafar | Deep Query Ranking for Question Answering over Knowledge Bases[END_REF][START_REF] Athreya | Template-based Question Answering using Recursive Neural Networks[END_REF]. In the latter context, Tree-LSTM is used to model the syntactic structure of the question. Other related approaches include tree transformer [START_REF] Harer | Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data[END_REF] and tree attention [START_REF] Ahmed | Improving Tree-LSTM with Tree Attention[END_REF]. Syntactic structures were also modeled as graphs [START_REF] Xu | Exploiting rich syntactic information for semantic parsing with graph-to-sequence model[END_REF]Li et al., 2020a). Specific positional embeddings can also be used to encode structures (Shiv and Quirk, 2019;[START_REF] Herzig | TaPas: Weakly supervised table parsing via pre-training[END_REF].

A grammar for KG exploration

Several previous KG-QA works were based on the grammar from D2A (Guo et al., 2018). We also take inspiration from their grammar, but redesign it to model a wider range of queries. By defining more generic operators, we achieve this without increasing the number of operators nor the average depth of LFs. Section 7.3.4 presents a comparison.

Definitions

An entity (e.g. Marie Curie) is a node in the KG. Two entities can be related through a directed labeled edge called a property (e.g. award received). A property can also relate an entity to a value, which can be a date, a boolean, a quantity or a string. Entities and properties have several attributes, prominently a name and an integer ID. The membership property is treated separately; it relates a member entity (e.g. Marie Curie) to a class entity (e.g. human being).

The objects we will consider in the following are entities, properties, classes, and values. The grammar consists of a list of operators that take objects or sets of objects as arguments. A Logical Form is a binary expression tree of operators.

In several places, we perform Named Entity Linking (NEL), i.e. mapping an utterance to a list of KG objects. Section 7.5.4 details how this is done.

Table 7.1 lists the operators we use, grouped in five categories. Most of them are straightforward, except meta-operators, which we explain next.

Meta-operators

Meta-operators are useful for questions such as: "Which musical instrument is played by the maximum number of persons?". To answer this question, we first compute the set of all musical instruments in the KG. For each entity in this set, we then follow the property played by, producing a set of people who play that instrument. Finally, we compute the max cardinality of all these sets and return the associated instrument.

The corresponding LF is the following: for_each creates a parallel computation over each entity in its argument, which can be terminated by three operators (arg, argmax and argmin). We refer to Appendix 7.B.1 for details.

Silver LF generation

To generate silver LFs, we explore the space of LFs with a BFS strategy, similarly to Guo et al. (2018); [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF]. More precisely, to initialize the exploration, we perform NEL to find relevant entities, values and classes that appear in the question. LFs of depth 0 simply return an annotated object. Then, assume that LFs of depth less or equal to n have been generated and we want to generate those of depth n + 1. We loop through all possible operators; for each operator, we choose each of its arguments among the already-generated LFs. This algorithm brings two challenges, as highlighted in [START_REF] Shen | Effective search of logical forms for weakly supervised knowledge-based question answering[END_REF]: computational cost and spurious LFs. We refer to Appendix 7.B.2 for implementation details that mitigate these difficulties.

Comparison with D2A

Section 7.5.5 shows that our grammar achieves higher coverage with a similar average LF depth. A more thorough quantitative comparison is delicate, as it would require reimplementing D2A within our framework, which is beyond the scope of this chapter. On a qualitative basis, we use more elementary types: in addition to theirs, we introduce set of classes, strings and set of values (which can be strings, numerals or booleans). We use eight fewer operators than D2A; among our operators, six are in common (follow_property, follow_backward, cardinality, union, intersect, difference), four are modified (keep, is_in, argmax, argmin), and the other ten are new. New intents that can be modeled include numerical reasoning (e.g. What actor plays the younger child? ), temporal reasoning (e.g. What is the number of seasons until 2018? ), ordinal reasoning (e.g. What was the first episode date? ), textual form reasoning (e.g. What was Elvis Presley given name? ). We refer to Appendix 7.C.1 for more details and comparison methodology.

Model

Overview

The model, called Object-Aware Transformer (OAT), is a Transformer-based (Vaswani et al., 2017) autoregressive neural semantic parser. As illustrated in Figure 7.1, the model has several steps.

The first step consists of retrieving relevant objects, annotated with metadata, that might appear in the resulting LF. This step is performed using NEL on the utterances and KG lookups to retrieve the graph context information. At this point, the input is composed of lists of objects with their fields. After embedding each field value in a vector space, we perform successive layers of input flattening and Transformer encoding. The Flattener layer is useful to transform the structured input into a list of embeddings. Then a decoder Transformer layer produces a linearized LF, i.e. a list of string tokens. Finally, we evaluate the LF over the KG to produce a candidate answer. In the next sections, we describe each step in details.

ID randomization Directly giving the entity ID to the model would mean training a categorical classifier with millions of possible outcomes, which would lead to poor generalization. To avoid this, we replace the integer ID with a random one, thereby forcing the model to learn to predict the correct entity from the list of entities in input by copying their randomized entity ID to the output.

For numerical values, we associate each value to an arbitrary random ID, that the model should learn to copy in the output. For properties and classes, since there are fewer possibilities in the graph (a few thousand), we do not randomize them.

Embedding

Preprocessing We apply BERT tokenization [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] to textual inputs. A vocabulary V t is generated for each of the non-textual input types t.

Token embedding The goal of this step is to associate an embedding to each field of each object in the input. We do so by using a learned embedding for each input type: BERT embeddings [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] for textual inputs, and categorical embeddings for non-textual inputs. When the input is a list (textual tokens or Entity IDs field), we add to this embedding a positional embedding. To reduce the size of the model, list embeddings are averaged into a single embedding. Formally, the embedding step associates a matrix of embeddings h(x) ∈ R 1×d h to each leaf of the input tree.

Encoding layers

There are two types of encoding layers: Flattener layers and Transformer layers.

Flattener The goal of these layers is to compute the embeddings of tree nodes bottom-up. They are successively applied until we are able to compute the embedding of the root node, i.e. of the whole input. This operation can be seen as flattening the JSON-like structure, hence their name.

Say we want to compute the embedding of some parent node x. An affine projection is first applied to the embedding of each child, then the embedding of the parent node is computed by applying a reduction operation R, which can be either a sum or a concatenation. The weights of the projections are shared between all nodes having the same types T (x). For example, all class name nodes -with types (root, class, name) -share the same weights, but they do not share the weights of entity name nodes -with types (root, entity, name). Hence the embedding of

x is h(x) = R y∈c(x) W T (y) project h(y) + b T (y) project ) .
If the reduction is a sum, all children embeddings need to be matrices of the same dimension, and the dimension of the parent embedding is also the same. If the reduction is a concatenation, the dimension of the parent embedding is y∈c(x) d(y), d h .

Transformer This layer is a classical multi-head Transfomer encoder layer (Vaswani et al., 2017), taking as input a matrix of embeddings of dimensions (n, d h ), performing self-attention between the input embeddings, and outputting another matrix of the same dimensions. Detailed setup can be found in Appendix 7.B.4. We also refer to Section 1.1.3 for an introduction to the mechanics of the Transformer encoder.

Architecture We apply a first Transformer layer only to the utterances, and in parallel a first Flattener layer with sum reduction to all other inputs. The latter computes one embedding for each object. We add a positional embedding to each object, to account for its position in the list of objects of the same type. Then we apply a second Flattener layer with concatenation to all outputs of the first layer. This creates a single matrix of embeddings containing the embeddings of all the objects and utterances. Finally, a second Transformer layer is applied to this matrix.

Decoding layers

The goal of the decoding layers is to produce a list of tokens that corresponds to a prefix representation of the LF tree. Note that the model architecture is grammar-agnostic, in the sense that this output structure is independent of the grammar and we do not use grammar-guided decoding. The tokens can belong to one of the non-textual input types or be a grammar token.

Remember that we computed a vocabulary V t for all token types t ∈ T . We augment the vocabulary with a stop token.

The decoder predicts the output list of tokens iteratively. Assume that we are given the first j tokens y 1 , . . . , y j . We then apply an autoregressive Transformer decoder (see Vaswani et al., 2017 or Phuong and[START_REF] Phuong | Formal algorithms for transformers[END_REF] on the full sequence, and we condition on the last embedding h j of the sequence to predict the next token ŷj+1 . Several categorical classifiers are used to predict ŷj+1 . We first decide whether we should stop decoding: ŝj+1 = argmax p stop,j (7.1) where p stop,j = softmax(W stop h j ) is a distribution over {0, 1} given by a binary classifier. If ŝj+1 = 1, the decoding is finished, and we set ŷj+1 to stop; otherwise, we predict the type of the token: tj+1 = argmax p type,j (7.2) where p type,j = softmax(W type h j ) is a distribution over T given by a |T |-class classifier. Finally, depending on the predicted type, we predict the token itself ŷj+1 = argmax p tj+1 token,j (7.3) where p tj+1 token,j = softmax(W tj+1 token h j ) is a distribution over V tj+1 given by a V tj+1 -class classifier.

Training We train by teacher forcing: for a given training sample (x, [y 1 , • • • , y M ]) and for each step j, the embedding h j is computed using the true output at previous steps: h j = h(y j ; x, y 1 , • • • , y j-1 ). The loss is the cross-entropy between the true output y j+1 and the probability distributions produced by the model. More precisely, let T : t∈T V t → T be the mapping which projects tokens to their type. We denote by p(x) the value of a categorical probability distribution p for the category x. Then, omitting the step subscripts j, the loss equals for all steps except the last, and l(p, y) =log(p stop (1)) for the last step. p stop , p type , and p token are computed as explained above. 7.3: QA performance on CSQA. Our method is the last one (OAT). The metric is the F1 score for question types above the vertical separator, and accuracy for those under. The Total Average score is an average over all question types.

Experiments

Additional comments about the datasets, setups, and additional results can be found in the appendix.

Datasets

We use two weakly supervised conversational QA datasets to evaluate our method, Complex Sequential Question Answering (CSQA)1 [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF] and ConvQuestions2 [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], both grounded in Wikidata3 . CSQA consists of about 1.6M turns in 200k conversations (152k/16k/28k splits), versus 56k turns in 11k conversations (7k/2k/2k splits) for ConvQuestions.

CSQA was created by asking crowd workers to write turns following some predefined patterns, then turns were stitched together into conversations. The questions are organized in different categories, e.g. simple, logical or comparative questions.

For ConvQuestions, crowd workers wrote a 5-turn dialog in a predefined domain (e.g. Books or Movies). The dialogs are more realistic than in CSQA, however at the cost of a smaller dataset.

As presented in Table 7.2, the datasets have different characteristics, which make them an interesting test bed to assess the generality of our approach.

CSQA Experimental Setup

Metrics To evaluate our grammar, we report the coverage, i.e. the percentage of training questions for which we found a candidate Logical Form.

To evaluate the QA capabilities, we use the same metrics as in [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF]. F1 Score is used for questions whose answers are entities, while accuracy is used for questions whose answer is boolean or numerical. We don't report results for "Clarification" questions, as this question type can be accurately modeled with a simple classification task, as reported in Appendix 7.A. Similarly the average metric "Overall" (as defined in [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF] is not reported in Table 7.3, as it depends on "Clarification", but can be found in the Appendix.

Baselines We compare our results with several baselines introduced in Section 7.2: D2A (Guo et al., 2018), D2A+ES [START_REF] Shen | Effective search of logical forms for weakly supervised knowledge-based question answering[END_REF], S2A+MAML (Guo et al., 2019a), and MaSP [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF].

ConvQuestions Experimental Setup

Metrics We use the coverage as above, and the P@1 metric as defined in [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF].

Baseline The only baseline to our knowledge is Convex [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], which casts the problem to a node classification task. For comparison, we tried to make our setup as close as possible to theirs, and refer to Appendix 7.C.3 for details.

Data augmentation Given the small size of the dataset, we merge it with two other data sources: CSQA, and 3.6M examples generated by random sampling. The latter are single-turn dialogs made from graph triplets, e.g. the triplet (Marie Curie, instance of, human) generates the dialog: Q: Marie Curie instance of ? A: Human. More details are given in Appendix 7.B.3. The ConvQuestions dataset is upsampled to match the other data sources sizes.

Named Entity Linking setup

We tried to use a similar setup as baselines for fair comparison. For CSQA, the previous gold answer is given to the model in an oracle-like mode, as in baselines. In addition, we use simple string matching between utterances and names of Wikidata entities to retrieve candidates that are given in input to the model. For ConvQuestions, we use the gold seed entity (as in the Convex baseline we compare with), and the Google Cloud NLP service. We refer to Appendices 7.B.2 and 7.B.4 for details.

Regarding CARTON [START_REF] Plepi | Context transformer with stacked pointer networks for conversational question answering over knowledge graphs[END_REF], their results are not directly comparable as their model uses gold entity annotations as input and hence is not affected by NEL errors. This different NEL setup does have a strong influence on the performance, as running our model on CSQA with a setup similar to CARTON improves our Total Average score by over 10%. We refer to Appendix 7.D.3 for details. More generally, a more thorough study of the impact of the NEL step on the end-to-end performance would be an interesting direction of future work (see also Section 7.5.6).

Results

Our grammar reaches high coverage. With approximately the same numbers of operators as in baselines, we improve the CSQA coverage by 16%, as presented in Table 7.4. The improvement is particularly important for the most complex questions. We reach a coverage of 86.2% on ConvQuestions, whose questions are more varied than in CSQA.

Most queries can be expressed as relatively shallow LFs in our grammar, as illustrated by Table 7.5. This is especially interesting for the ConvQuestions dataset, composed of more realistic dialogs. On CSQA, the average depth of our LFs (2.9) is slightly lower than with D2A grammar (3.2).

We improve the QA performance over baseline on both datasets. For CSQA, our model outperforms baselines for all question types but Direct Simple questions, as shown in Table 7.3. Overall, our model improves the performance by 5%. For ConvQuestions, Table 7.6 shows that our model improves over the baseline for all domains but one, yielding an overall improvement of 4.7%. A precise evaluation of the impact of the various components of our KG-QA approach (grammar, entity linking, model inputs, model architecture, size of the training data, etc.) on the end-to-end performance was out of the scope of this chapter, and is left for future work. Nevertheless, the fact that we are able to improve over baselines for two types of Simple questions and for Logical questions, for which the grammar does not matter so much, as these question types correspond to relatively shallow LFs, suggests that our proposed model architecture is effective.

Error analysis

CSQA By comparing the silver and the predicted LFs on 10k random errors, we could split the errors in two main categories: first, the LF general form could be off, meaning that the model grammar with increased coverage, which can hence be used to model a wider range of queries. These two contributions are fairly independent : on the one hand, since the model predicts LFs as a list of tokens, it is grammar agnostic, and thus it could be used with another grammar. On the other hand, the grammar is not tied to the model, and can be used to generate training data for other model architectures. Experiments on two datasets validate our approach. We plan to extend our model to include a richer KG context, as we believe there is significant headroom for improvements. The second turn is a "Clarification" question: the system asks the user for disambiguation. The disambiguation question usually takes the form "Did you mean", followed by an entity chosen among the previous turn answers. This choice appears to be entirely random. For this reason, we found that it would not be very interesting to try to predict this entity, as baselines propose. Hence we only ask the model to predict that the question is a Clarification (via a special clarification operator).

7.A Clarification Questions in CSQA

We report in Table 7.8 the scores for Clarification questions, as well as the "Overall" score, as defined in [START_REF] Saha | Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph[END_REF]. The results are not directly comparable as the baseline systems report an F1 score, while our approach uses accuracy. 7.8: QA performance on CSQA, including "Clarification" questions. The "Overall" metric is the average F1 scores of the following question types: "Simple (Direct)", "Simple (Coreferenced)", "Simple (Ellipsis)", "Logical", "Quantitative", "Comparative" and "Clarification".

7.B Detailed experimental setup To resolve coreferences, we use entities from the dialog history: all preceding turns for ConvQuestions and only the previous turn for CSQA.

Implementation details We tokenize the input using the BERT-uncased tokenizer. All embeddings in the model have dimension 768. The two transformer encoders share the same configuration: 2 layers each, with output dimension 768, intermediate dimension 2048, 12 attention heads, and dropout probability set to 0.1. The model has 260M parameters. The transformer implementation is based on publicly available BERT code. We initialize the word embedding from a BERT checkpoint, but do not load the transformer layer weights, instead training them from scratch. We train for 600k steps with batch size 128, using the ADAM optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with learning rate 3 × 10 -5 . Training takes around 14 hours on 16 TPU v3 with 32 cores. The D2A (Guo et al., 2018) grammar is the main baseline in previous KG-QA works. We compare with the grammar implemented in their open-sourced code4 , which is a bit different from the published one. Numbers for D2A in Tables 7.4 and 7.5 were computed thanks to the results of the BFS gracefully provided by the authors.

7.C Comparison with baselines

Table 7.9 presents some intents which we are able to model in our grammar and are not straightforward to model with D2A grammar. First, textual form reasoning corresponds to questions about string attributes of entities, which are not included in the D2A grammar. Second, to handle numerical and temporal reasoning, computations based on numerical values are needed, which is not possible with the D2A grammar. Finally, the D2A grammar does not model the order of relations in the graph and the selection of class members, which we start to tackle with respectively the get_first and members operators.

7.C.2 Comparison with MaSP architecture

Similarly to ours, the MaSP [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF] model follow the semantic parsing approach, where the LF is encoded as a sequence of operators and graph items IDs. Regarding the model input, theirs consist only of the utterances, whereas we add additional KG context structured as a JSON tree. The training method is different: MaSP uses multi-task learning to learn jointly entity linking and semantic parsing, whereas we chain both, and trust the model to pick the good entity. Our approach is simpler in this regard, but we pay this by having a slightly lower performance on Simple Direct questions (see Table 7.3). Finally, we do not use beam search for the decoding, contrarily to them.

7.C.3 Comparison with node classification approaches

An alternative to the semantic parsing approach is to train classifiers to predict entities as nodes of the KG. A precise comparison of both approaches is out of the scope of this chapter. Nevertheless, we think that the semantic parsing approach is better suited to our purpose of modeling complex questions. For instance, complex intents involving numerical comparisons can be expressed naturally by a LF, but would be difficult to perform using solely node classifiers. Examples include the numerical and temporal reasoning in Table 7.9. Additional examples include The series consists of which amount of books? (ConvQuestions) or Which television programs have been dubbed by at least 20 people ?, How many episodes is it longer than the second longest season out of the three? (CSQA).

Convex [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF] is an example of such an approach. It is an unsupervised graph exploration method: at each turn, a subgraph is expanded by matching the utterance with neighboring entities. Then a candidate answer is found in the subgraph by a node classifier. On our side, we propose a semantic parsing approach that makes use of entities annotated by an external entity linking service. This is a similar setup to the CSQA baselines [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF], which we re-purposed for ConvQuestions in order to assess the quality of our proposal on another dataset. In order to be closer to the Convex baseline, we changed our CSQA setup by applying the entity linker only to the questions' text and not to the answers' text. In addition, as we use the gold seed entity, we compare with the Oracle+Convex setup of [START_REF] Christmann | Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion[END_REF], which also uses the gold seed entity (and the gold first turn answer entity). Finally, we make use of data augmentation to train our model on ConvQuestions, whereas the baseline does not.

7.C.4 BERT or no BERT, that is the question

The baselines of Table 7.3 do not use BERT. MaSP authors provide an additional BERT variant of their model that uses a fine-tuned BERT base architecture. The Total Average score of this variant is 72.60%, which is 2% above their vanilla variant and 3% under our model. Since we are only using the word embeddings (loaded from a publicly available BERT base checkpoint) and not loading the transformer layer weights, we decided to compare with the vanilla variant of MaSP, and not the BERT one. Finally, CARTON is using a pre-trained BERT base model as a sentence encoder.

7.D Additional results

7.D.1 Coverage results

We present in Table 7.10 the coverage per domain for ConvQuestions. Besides, the evolution of the coverage over turns is stable for both datasets, hence we do not report this result. correctly, given the true previous tokens. Table 7.14 reports the results on the evaluation split of both datasets. The results corroborate the analysis presented in Section 7.5.6. First, the model learns the grammar rules, as it nearly always predicts the good token type. For CSQA, the most frequent errors concern entity ID and class ID. For ConvQuestions, they concern primarily entities and properties.

7.D.5 Case study

Table 7.15 presents examples from ConvQuestions where we are able to predict the good LFs, although there exists very similar properties in the graph. The textual forms of the questions are not sufficient to infer the good property to use, implying that the model had to learn elements from the graph structure in order to answer correctly these questions. Nevertheless, Table 7.14 shows that there is still significant room for improvement in that direction.

Conclusion

Despite its immense successes, paving the way to superhuman performance of computers in numerous intellectual endeavors, deep learning remains a strikingly data-inefficient and energyinefficient method, in particular compared to the human brain. Innovation towards more efficient approaches will come from a combination of various efforts, among which striving for more mathematically-grounded approaches may occupy a foremost role. In this thesis, we presented several contributions to deep learning research, and in particular to the theory of deep learning, which we hope can contribute at their scale to this effort. An important axis in our work was to leverage the large realm of differential equations to study properties of neural networks, be it via the continuous-depth approach in the first part of the thesis, or with the two-timescale (a.k.a. fast-slow) regime for dynamical systems in Chapter 6. Of course, these contributions only scratch the surface of the interplay between deep learning and analysis of differential equations, and there are many more subjects of interest regarding this bridge between two behemoths of modern applied sciences. As a conclusive note, let us sketch a few final comments and perspectives relative to the topics of the thesis.

From neural ODEs to neural SDEs. A major focus in this PhD thesis is the study of the connection between deep residual networks and neural ordinary differential equations. In Chapter 2, we highlight conditions under which this connection holds true at initialization, namely that the residual branch should be rescaled by a factor inversely proportional to the depth of the network, and that the weights should be initialized with correlations across depth. In Chapter 3, we show that the neural ODE limit then also holds during training. Under an additional assumption on the width of the network, we show a Polyak-Łojasiewicz inequality, which allows proving convergence of the training algorithm towards a neural ODE that interpolates the training data. Finally, in Chapter 4, we show that neural ODEs, and subsequently deep ODE-like residual networks, satisfy generalization bounds.

It would be of particular significance to investigate whether these results can extend to neural stochastic differential equations, which we showed in Chapter 2 to be another possible deep limit for residual networks. The extension to neural SDEs is important for at least two reasons. First, as we showed in Chapter 2, standard i.i.d. initialization schemes seem to rather correspond to SDEs than to ODEs. Moving from neural ODEs to neural SDEs would therefore bring our analysis closer to the practice. Second, stochasticity has empirically been shown to play a key role in deep learning training, for instance via dropout (Hinton et al., 2012b) or stochastic gradient descent. Recent theoretical results prove that stochasticity enables the optimization algorithm to reach minima with better generalization properties (Pesme et al., 2021). As a consequence, understanding the effect of stochasticity due to initialization would be very interesting, and the comparison of the neural SDE and neural ODE limits could be a fruitful axis to do so.

Moving away from supervised learning. All our results are presented in a supervised learning setup, either for regression or classification. Could they be extended to other learning contexts? In particular, several recent generative models, among which the acclaimed diffusion models, can be written as a neural ODE [START_REF] Song | Score-based generative modeling through stochastic differential equations[END_REF], where the ODE flow corresponds to a transport equation mapping Gaussian noise to the target distribution. Understanding how to translate to this sampling context the theoretical results obtained in supervised learning would be worthwhile.

Better characterizing the implicit regularization for deep residual networks. Chapters 3 and 4 show a gap between the optimization and statistical points of views: our global convergence result in Chapter 3 requires the width of the residual network to grow linearly with the sample size, while on the contrary our statistical result of Chapter 4 requires the sample size to be larger than the width. The literature on implicit regularization presented in Section 1.1.2.2 suggests that it is unlikely that this gap could be bridged by refinements of the current approach. A more promising direction would be to obtain further results on the implicit regularization of gradient algorithms for deep residual networks. More precisely, we showed in Chapter 3 that the gradient flow for a class of residual networks converges towards a neural ODE, but it remains to understand more finely the properties of this limiting neural ODE, and in particular its generalization abilities.

Extending global convergence results to other settings. Chapters 3 and 6 present global convergence results for neural networks obtained with very different techniques, either through a Polyak-Łojasiewicz inequality or by studying explicitly the gradient flow dynamics. Adapting these techniques to other architectures would be particularly beneficial. The proof technique of Chapter 6, which enables us to prove convergence of the gradient flow for sigmoid networks to approximate one-dimensional piecewise constant target functions, is delicate to adapt to a multidimensional setting, but other one-dimensional settings such as using ReLU networks to learn piecewise affine functions could be considered. On the other side, the proof technique of Chapter 3 via a Polyak-Łojasiewicz inequality is fairly general, and it might be applicable to other residual architectures such as Transformer.

Going beyond on the Transformer architecture. The contributions of Chapter 7 are a first attempt at tackling the numerous questions raised by Transformer. In particular, an overarching question is to understand to what extent the striking results of this architecture are linked to the form of the attention mechanism. More precisely, is there something specific to attention which allows it to model particularly well some sequential data? Or is attention an instance of a larger class of mechanisms that work just as well? For instance, [START_REF] Lee-Thorp | FNet: Mixing tokens with Fourier transforms[END_REF] propose to replace the attention layer by a non-parametrized Fourier layer. This layer computes the 2D Fourier transform of the sequence of embeddings. The performance trails not much behind the standard Transformer, and the authors suggest that the "token-mixing" property of the Fourier layer is enough to obtain good performance. A mathematical analysis of their results, and of the comparison with standard attention layers, would be of the utmost interest.

Contents 1 . 1

 11 Mathematics of deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Why study the mathematics of deep learning? . . . . . . . . . . . . . . 1.1.2 What are the main mathematical challenges of deep learning? . . . . . 1.1.3 From shallow neural networks to Transformer . . . . . . . . . . . . . . 1.1.4 What to expect in this manuscript? . . . . . . . . . . . . . . . . . . . . 1.2 From discrete to continuous architectures: neural networks in the large-depth regime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Scaling of residual networks at initialization . . . . . . . . . . . . . . . 1.2.2 Implicit regularization of deep residual networks towards neural ODEs 1.2.3 Generalization bounds for neural ODEs and residual networks . . . . . 1.2.4 Recurrent neural networks as kernel methods . . . . . . . . . . . . . . 1.3 Contributions to finite-depth neural networks . . . . . . . . . . . . . . . . . . 1.3.1 Convergence of shallow neural networks in the two-timescale regime . 1.3.2 Structured context and high-coverage grammar for conversational question answering over knowledge graphs . . . . . . . . . . . . . . . . . . 1.4 Résumé détaillé en français . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1. 2

 2 From discrete to continuous architectures: neural networks in the large-depth regime [Part I of the manuscript]

  Figure 1.1: Plot of a given coordinate of V k , after training, as a function of the layer index k ranging from 1 to the depth L = 1000 for three different choices of β and initializations. The left plot depicts a smooth ODE-like weight structure, contrarily to the other plots.

  Figure 1.2: Simulation in the two-timescale regime (ε = 2 • 10 -5). The target function is in blue and the neural network is in orange. The biases b 1 , . . . , b q of the neurons are indicated with vertical dotted lines (the dots are only present for black and white visibility). In a first short phase, only the weights v 1 , . . . , v q of the neurons evolve to match as well as possible the target function (second plot). Then, in a longer phase, the neuron closest to each target discontinuity moves towards it (third plot). Recovery is achieved.

  Figure 1.3: Simulation outside the two-timescale regime (ε = 1). The target function is in blue and the neural network is in orange. The biases b 1 , . . . , b q of the neurons are indicated with vertical dotted lines (the dots are only present for black and white visibility). The dynamics create a zone with no neuron, hindering recovery.

Figure 1

 1 Figure1.5: Data pipeline in our model. The question is first processed by named entity linking in order to find entities in the graph that are contained in the question. In a second step, we explore the graph locally around the found entities to construct a context tree, which forms the input to the Transformer model. It outputs a logical form which is evaluated to produce a candidate answer.

Figure 2 . 1 :Figure 2 . 2 :

 2122 Figure 2.1: Evolution of h Lh 0 / h 0 as a function of L for different values of β and an i.i.d. U(-3/d, 3/d) initialization of model res-3, with d = 40. The input is a random Gaussian observation x in dimension n in = 64. The experiment is repeated with 50 independent randomizations.

Figure 2 . 3 :

 23 Figure 2.3: Evolution of p 0p L / p L as a function of L for different values of β and an i.i.d. U(-3/d, 3/d) initialization of model res-3, with d = 40. The input is a random Gaussian observation x in dimension n in = 64. The experiment is repeated with 50 independent randomizations.

Figure 2 . 4 :

 24 Figure 2.4: Evolution of h Lh 0 / h 0 as a function of L for different values of β and a smooth initialization of model res-3, with d = 40. The input is a random Gaussian observation x in dimension n in = 64. The experiment is repeated with 50 independent randomizations.

Figure 2 . 5 :

 25 Figure 2.5: Evolution of p 0p L / p L as a function of L for different values of β and a smooth initialization of model res-3, with d = 40. The input is a random Gaussian observation x in dimension n in = 64. The experiment is repeated with 50 independent randomizations.

Figure 2 . 6 :

 26 Figure 2.6: Examples of realizations of a fractional Brownian motion B H for different Hurst indexes H. Note that the smaller the value of H, the more irregular the trajectory is.

Figure 2 . 7 :

 27 Figure 2.7: Magnitude of the outputs and of the gradients as a function of the regularity of the weights (Hurst index H) and of the scaling factor β. The orange zone corresponds to the explosion regime, i.e., h Lh 0 h 0 and p 0p L p L . The blue zone corresponds to the identity regime, i.e., h Lh 0 h 0 and p 0p L p L . Finally, the black zone is an intermediate regime, where h Lh 0 ≈ h 0 and p 0p L ≈ p L .

  Figure 2.8: Plot of a given coordinate of θ k , after training, as a function of the layer index k ranging from 1 to the depth L = 1000 for three different choices of β and initializations.

Figure 2 . 9 :

 29 Figure 2.9: Accuracy after training as a function of the regularity of the weights at initialization and scaling. For each point of the heatmap, the model was trained on a grid of learning rates, and the best performance is shown.

Lemma 2 .

 2 16. Let W ∈ R d×d be a matrix whose entries are centered i.i.d. random variables, with finite variance, and let σ be an activation function such that, for all

Figure 2 .Table

 2 8 is obtained by plotting a random coordinate of θ k , after training on MNIST. with Hurst index from 0.05 to 0.97 learning rate grid 10 -4 , 10 -3 , 10 -2 , 10 -1 , 1

Contents 3 . 1 78 3. 2

 31782 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.3 Definitions and notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.4 Large-depth limit of residual networks . . . . . . . . . . . . . . . . . . . . . . 82 3.4.1 Clipped gradient flow and finite training time . . . . . . . . . . . . . . 82 3.4.2 Convergence in the long-time limit for wide networks . . . . . . . . . . 84 3.4.3 Generalizations to other architectures and initialization . . . . . . . . 85 3.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.5.1 Synthetic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.5.2 Real-world data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3.A Some results for general residual networks . . . . . . . . . . . . . . . . . . . 89 3.B Proofs of the results of the main part of the chapter . . . . . . . . . . . . . . 108 3.C Some technical lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 3.D Counter-example for the ReLU case. . . . . . . . . . . . . . . . . . . . . . . . 117 3.E Experimental details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Figure 3

 3 Figure 3.1: Left: 1/L convergence of the maximum distance between two successive weight matrices max 1≤k≤L,t∈[0,T ] ( Z L k (t) -Z L k+1 (t) F ). Right: uniform convergence of Z L to its largedepth limit Z. Here, for a matrix-valued function f , f denotes (

Figure 3 . 2 :

 32 Figure 3.2: Left: Randomly-chosen entry of the weight matrices across layers (x-axis) for various training times t (lighter color indicates higher training time). Right: Loss against training time.

Figure 3

 3 Figure 3.3: Random entries of the convolutions across layers (x-axis) after training. Left: Weighttied initialization leads to smooth weights. Right: i.i.d. initialization leads to non-smooth weights.

  is continuous in s because g and Θ are continuous. Thus the ODE (3.33) has a unique solution on [0, 1] by the Picard-Lindelöf theorem (see Lemma 3.19). Denote by C the uniform Lipschitz constant of g(•, θ) for θ ≤ M . Since g(0, •) ≡ 0 and g(•, Θ(s)) is C-Lipschitz, one has dU ds (s) = g(U (s), Θ(s)) ≤ C U (s) .

  4.1 (finite training time and clipped gradient flow) and Section 3.4.2 (arbitrary training time and standard gradient flow). To this end, we consider a training interval I = [0, T ] ⊆ R + , for T ≤ ∞, and the gradient flow formulation (3.5), which is equivalent to the standard gradient flow (3.4) if π equals the identity. Note that we do not need to assume in the following proof that π is bounded (but only Lipschitz continuous). Therefore, the proof also holds in the case where π equals the identity.Theorem 3.15. Consider the residual network (3.12) initialized as explained in Appendix 3.A and trained with the gradient flow (3.5) on I = [0, T ] ⊆ R + , for some T ∈ (0, ∞]. Assume that there exists a unique solution to the gradient flow, such that (A L ) L∈N * and (B L ) L∈N * each satisfies the assumptions of Corollary 3.11, and (Z L k ) L∈N * ,1≤k≤L satisfies the assumptions of Proposition 3.10. Then the following four statements hold as L tends to infinity:(i) There exist functions A : I → R q×d and B : I → R d ×q such that A L (t) and B L (t) converge uniformly over t ∈ I to A(t) and B(t).

Lemma 3. 19 (

 19 Picard-Lindelöf theorem). Let I = [0, T ] ⊂ R + be an interval, for some T ∈ (0, ∞]. Consider the initial value problem U (s) = U 0 + s 0 g(U (r), r)dr, s ∈ I, (3.54)

  lim x,y→∞ e x,y = lim x→∞ e x,∞ = lim y→∞ e ∞,y = e ∞ .

  (a) Generalization gap as a function of the maximum Lipschitz constant of the weights. Each dot corresponds to a network trained with a varying number of epochs (between 1 and 30).

  (b) Generalization gap as a function of the penalization factor λ. The experiment is repeated 20 times for each value of λ. Each time, the network is trained for 50 epochs. The initial and final matrices are random.

Figure 4 .

 4 Figure 4.1: Link between the generalization gap and the Lipschitz constant of the weights

For

  two integers a and b, denote respectively a//b and a%b the quotient and the remainder of the Euclidean division of a by b. Then, for W ∈ R L×d×d , let φ(W) : [0, 1] → R d 2 the piecewise affine function defined as follows: φ(W) is affine on every interval k L , k+1 L for k ∈ {0, . . . , L -1}; for k ∈ {1, . . . , L} and i ∈ {1, . . . , d 2 },

  Figure 5.1: Approximation of the RNN ODE by the step-N Taylor expansion

Figure 5 . 2 :

 52 Figure 5.2: Adversarial accuracy as a function of the adversarial perturbation ε

Figure 5

 5 Figure 5.3: Evolution of the Frobenius norm of the weights and of the RKHS norm during training

Figure 5 . 4 :

 54 Figure 5.4: Example of a path X ∈ BV c ([0, 1], R) (left) and its corresponding paths X[j] , plotted against time, for different values of j ∈ {1, . . . , T } (right)

  1-L f ( h1:e , he+1:e+d ) I d×d 0 d×1 , where I d×d denotes the identity matrix and 0 •×• the matrix full of zeros. The function H is then solution of assumption (A 1 ), the tensor field F satisfies the assumptions of the Picard-Lindelöf theorem (Theorem 5.15) so that H is well-defined. The projection of this equation on the last d coordinates gives d He+1:e+d t = dX t , He+1:e+d 0 = X 0 , and therefore He+1:e+d t = X t . The projection on the first e coordinates gives d H1:e

Figure 5 .

 5 3 is obtained by selecting randomly one run among the 20 of Figure 5.2.

Contents 6 . 1

 61 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 6.2 Setting and main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 6.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.4 A non-rigorous introduction to the two-timescale limit . . . . . . . . . . . . . 182 6.4.1 Introduction to the two-timescale limit . . . . . . . . . . . . . . . . . . 182 6.4.2 Sketch of the dynamics of the two-timescale limit . . . . . . . . . . . . 182 6.5 Convergence of the gradient flow . . . . . . . . . . . . . . . . . . . . . . . . . 184 6.5.1 In the two-timescale limit . . . . . . . . . . . . . . . . . . . . . . . . . 184 6.5.2 From the two-timescale limit to the two-timescale regime . . . . . . . . 186 6.6 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 6.A Additional notations and technical lemmas . . . . . . . . . . . . . . . . . . . 188 6.B Proofs of the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 6.C Experimental details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Figure 6

 6 Figure 6.1: Notations of the chapter. The target f * is in blue and the neural network f (•; a, u) in orange.

Figure 6 . 2 :

 62 Figure 6.2: Sketch of the dynamics of the neurons in the two-timescale limit with a Heaviside non-linearity. Only the neurons next to a discontinuity of the target move.

  Figure 6.3: Comparison between the SGD (6.9) with η = 4 • 10 -3 in the two-timescale regime (ε = 2 • 10 -5) and the gradient flow in the two-timescale limit (6.5) with η = 0. In the left-hand plot, to align the SGD and the two-timescale limit, we take τ = εhp. In the right-hand plot, the target function is in blue, the gradient flow in the two-timescale limit is in green, and the SGD is in orange.

Figure 6 . 4 :

 64 Figure 6.4: Simulation in the two-timescale regime (ε = 2 • 10 -5). The target function is in blue and the SGD (6.9) is in orange with η = 4 • 10 -3 , h = 10 -5 . The positions u 1 , . . . , u m of the neurons are indicated with vertical dotted lines. In a first short phase, only the weights a 1 , . . . , a m of the neurons evolve to match as good as possible the target function (second plot). Then, in a longer phase, the neuron closest to each target discontinuity moves towards it (third plot). Recovery is achieved.

  Figure 6.5: Simulation outside the two-timescale regime (ε = 1). The target function is in blue and the SGD (6.9) is in orange with η = 4 • 10 -3 , h = 10 -5 . The positions u 1 , . . . , u m of the neurons are indicated with vertical dotted lines. The dynamics create a zone with no neuron, hindering recovery.

Lemma 6. 19 .

 19 Let m be a positive integer and u 1 , . . . , u m be i.i.d. uniform random variables in [

2 Figure 6

 26 Figure 6.6: Dynamics of the neurons next to a discontinuity v i . In the first phase, u L i and u R i move towards v i , until the closest neuron (in this case u L i ) reaches the interval of size η centered in v i . In the second phase, u L i remains in an interval of size 2η around v i , and none of the neurons move significantly.

  0) D and T ∆g = D /10 by definition of T and ∆g. As a consequence, u LL i and u L i do not collide during the first phase, and we have

  6.25) since m ≥ 0, D ≤ 1 and ∆f ≤ 2M . Since each neuron moves by at most D/8 in the time interval [0, T 0 ] and since ∆(u(0)) ≥ D, we deduce that ∆(u(T 0 ))

Table 1 .

 1 2: Chapters of this manuscript organized by nature of the results.

		Theory	Applications
		The classical waltz	The modern tango
	Statistics	Chap. 4 (residual networks) Chap. 5 (recurrent networks)	Chap. 7
	Optimization	Chap. 3 (residual networks) Chap. 6 (shallow networks)	Chap. 3 (residual nets) (Transformer)
	Other properties	Chap. 2 (residual networks)

  de ce chapitre est de déterminer sous quelles conditions ce réseau converge vers une équation différentielle ordinaire (EDO) ou stochastique (EDS) dans la limite en grande profondeur L 1, en fonction du schéma d'initialisation et du paramètre d'échelle β > 0. Plusieurs schémas d'initialisation sont étudiés, principalement l'initialisation i.i.d. et l'initialisation régulière. L'initialisation i.i.d. correspond à la pratique usuelle d'initialiser les poids comme des variables aléatoires i.i.d., par exemple uniformes ou Gaussiennes. L'initialisation régulière est moins courante, et correspond à prendre les V k et W k comme des discrétisations de fonctions régulières (potentiellement aléatoires)

  De plus, l'erreur d'entraînement de l'EDO limite est égale à zéro. Cette analyse représente une première étape dans la compréhension de la régularisation implicite du flot de gradient pour les réseaux résiduels, c'est-à-dire la caractérisation des propriétés du réseau entraîné parmi tous les minimiseurs du risque empirique.

1.4.3 Bornes de généralisation pour EDO neuronales et réseaux de neurones résiduels [Chapitre 4 du manuscrit]

  une donnée séquentielle. Nous faisons de plus l'hypothèse que la donnée (x k ) 1 k L est la discrétisation d'un processus en temps continu X : [0, 1] → R d . Notre but dans ce chapitre est de montrer que le réseau de neurones récurrent résiduel s'écrit en fait comme une méthode à noyaux. Nous reconnaissons dans l'équation précédente la discrétisation d'Euler d'une EDO, et prouvons ainsi que le réseau approche à une distance O(1/L) son équivalent en profondeur continue dH ds

  2.B Technical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 2.C Concentration of sub-Gaussian random matrices . . . . . . . . . . . . . . . . 71 2.D A version of the Picard-Lindelöf theorem . . . . . . . . . . . . . . . . . . . . 73 2.E Detailed experimental setting . . . . . . . . . . . . . . . . . . . . . . . . . . . Residual neural networks (ResNets), introduced by He et al. (2016a) in the field of computer vision, were the first deep neural network models successfully trained with several thousand layers. Since then, extensive empirical evidence has shown that increasing the depth leads to significant improvements in performance, while raising new challenges in terms of training (e.g.,

	74
	2.1 Introduction
	2.1.1 Deep residual neural networks

Table 2 .

 2 1: Examples of ResNet architectures considered in the chapter. In the first two cases, the activation function σ is such that, for all x ∈ R, a|x| |σ(x)| b|x|, 1 /

	√	2

  the lower bound (i) indicates an explosion with high probability of the norm of the last hidden state. On the other hand, when Lα 2 L ≈ 1, the bounds (i) and (ii) show that h L randomly varies around h 0 with fluctuation sizes bounded from below and above. Proposition 2.2. Consider a ResNet(2.4) such that Assumptions (A 1 ) and (A 2 ) are satisfied.

	2 L Proposition 2.3. Consider a ResNet (2.4) such that Assumptions (A 1 ) and (A 2 ) are satisfied. 1 and Lα 2 L ≈ 1. When Lα 2 L L 1, then, for any δ ∈ (0, 1), with probability at least 1 -δ, h L -h 0 2 h 0 2 2Lα 2 L δ . Proof. See Section 2.A.2. 1, If Lα 2 (i) Assume that d 64 and α 2 L 2 ( √ Cs 4 +4 √ C+16s 4 )d . Then, for any δ ∈ (0, 1), with probability at least 1

  are centered i.i.d. random variables, independent of d and L, with unit variance.

	Proof. See Section 2.A.5.
	The next two propositions are the counterparts of Proposition 2.2 and Proposition 2.3 for the
	gradient dynamics.
	Proposition 2.6. Consider a ResNet (2.4) such that Assumptions (A 1 )-(A 4 ) are satisfied. If
	Lα 2 L

Table 2

 2 .3.

	Name	Value
	d	40
	n in	64
	n out	1
	L	1000
	β	0.2 to 1.3
	weight distribution	fractional Brownian motion
		with Hurst index from 0.05 to 0.97
	data distribution	standard Gaussian

Table 2 .

 2

3: Hyperparameters of Figure 2.7

  : i.i.d. initialization leads to non-smooth weights.

	Act. function Init. scheme Train Acc. Test Acc. Smooth trained weights
	Identity GELU ReLU	Weight-tied 56.5 ± 0.1 59.8 ± 0.7 i.i.d. 56.1 ± 0.3 59.6 ± 0.7 Weight-tied 80.5 ± 0.7 79.9 ± 0.2 i.i.d. 89.8 ± 0.5 85.7 ± 0.1 Weight-tied 97.4 ± 0.6 88.1 ± 0.1 i.i.d.	× × ×

  34) is a Lipschitz continuous function with some Lipschitz constant C. To see this, just note that U itself is Lipschitz continuous in s, since g is bounded, and therefore the function (3.34) is a composition of Lipschitz continuous functions. In particular, dU ds is almost everywhere differentiable, and its derivative d 2 U ds 2 (s) is bounded in the supremum norm by C. As a consequence, for k ∈ {0, . . . , L -1}, the Taylor expansion of U on [ k L , k+1 L ] takes the form

  is locally Lipschitz continuous. Applying the first part of the proof to the specific case of x i , we know that H

	P	φ(L) i	φ(L) i	and P i φ(L)	uniformly bounded, and that H i φ(L)	(s, t) and

  .47) By Lemma 3.23, with probability at least 1exp -qm 16 , one has W F ≤ √ 2qm. Together with the previous inequalities, this implies

Table 4 .

 4 and B ∈ R 10×d . Table4.1 gives the value of the hyperparameters.

	Name Value
	d	30
	L	1000
	σ	ReLU

1: Values of the model hyperparameters.

  Theorem 6.2. Let ξ, δ > 0, and f * a piecewise constant function from F n,∆v,∆f,M . Assume that the neural network has m neurons with

	m	6 ∆v	4 + log n + log	1 δ	.	(6.2)

  ∆g , by (6.16). Let us now upper-bound each of the last two terms by D /10 to conclude. By definition of D, η

  .2.

	Name	Value
	m	20
	ε η P h	2 • 10 -5 4 • 10 -3 1.8 • 10 8 10 -5
	Additive noise Uniform on [-1, 1]

Table 6 .

 6 1: Parameters of Figures 6.3 and 6.4.

Table 7 .

 7 The total loss is obtained by averaging over all training samples and over all steps. 2: Some characteristics of the benchmark datasets.

		CSQA		ConvQuestions
	Average length of a dia-log	8 turns		5 turns
	Possible change of topic inside a conversation	Yes		No
	Answer type	Entities, boolean, quantity	Entities (usually a single one), boolean, date, quantity, string
	Entity annotations in the dataset	Yes, with coreference resolution	Only the seed entity (topic of the dialog) and the answer en-tities
	Coreferences in ques-tions	Yes, to the previous turn	Yes, to any preceding turn
	Methods	D2A D2A+ES S2A+MAML MaSP OAT
	Question type	# Ex.		F1
	Simple (Direct)	82k 91.41	83.00	92.66	85.18	82.69
	Simple (Coreferenced)	55k 69.83	64.62	71.18	76.47 79.23
	Simple (Ellipsis)	10k 81.98	83.94	82.21	83.73 84.44
	Logical	22k 43.62	72.93	44.34	69.04 81.57
	Quantitative	9k 50.25	63.95	50.30	73.75 74.83
	Comparative	15k 44.20	55.05	48.13	68.90 70.76
	Question type	# Ex.		Accuracy
	Verification (Boolean)	27k 45.05	45.80	50.16	60.63 66.39
	Quantitative (Count)	23k 40.94	41.35	46.43	43.39 71.79
	Comparative (Count)	15k 17.78	20.93	18.91	22.26 36.00
	Total Average	260k 64.47	64.75	66.54	70.56 75.57
	Table

Table 7 .

 7 4: Coverage per question type for CSQA. Table 7.5: Silver LF depth distribution for both datasets.

	Question type D2A D2A+ES Ours
	Comparative	28.6	45		84.9
	Logical	48.2	92		100.0
	Quantitative	58.1	62		91.1
	Simple	94.4	96		99.7
	Verification	77.9	85		91.4
	Overall	74.3	80		96.2
	Depth	1	2	3	4+
	CSQA (D2A)	0.0 47.0 30.9 22.0
	CSQA (Ours)	5.5 67.9	7.4 19.2
	ConvQuestions 53.9 43.7	2.4	0.0

  Question type # Ex. D2A D2A+ES S2A+MAML MaSP OAT (Ours)

	Clarification	12k 18.31	36.66	19.12	80.79	99.63
	Overall	206k 62.88	72.02	N/R	79.26	81.49
	Table

  Take the example given in the main part of the chapter: "Which musical instrument is played by the maximum number of persons?". The corresponding LF is:

	7.B.1 Meta-operators
	argmax(
	cardinality(
	follow_property(
	for_each(
	members(musical instrument )),

Table 7 .

 7 7.C.1 Comparison with D2A grammar 9: Examples of questions that are difficult to model with the D2A grammar. Examples are mostly chosen from ConvQuestions, as their questions look more realistic than CSQA.

	Intent	Question example	Missing in D2A
	Textual form reason-ing	What was Elvis Presley given name?	string type
	Numerical reasoning	What actor plays the younger child?	get_value, argmin	for_each,
	Numerical reasoning	How old is the younger child?	min
	Selection of the mem-bers of a class	Which television programs have been dubbed by at least 20 people ?	members
	Temporal reasoning	What is the number of seasons until 2018?	for_each, lesser_than, arg get_value,
	Ordinal reasoning	What was the first episode date?	get_first

Table 7 .

 7 13: QA performance on CSQA in oracle mode.

	Question type		CARTON Ours
	Simple (Direct)		85.92	96.95
	Simple (Coreferenced)	87.09	94.77
	Simple (Ellipsis)		85.07	96.66
	Logical		80.80	95.54
	Quantitative		80.62	76.44
	Comparative		62.00	76.66
	Verification (Boolean)	77.82	67.02
	Quantitative (Count)	57.04	75.89
	Comparative (Count)	38.31	35.10
	Total Average		77.89	85.85
	Metric	CSQA ConvQuestions
	Token type	99.88		94.77
	Grammar token	98.86		82.12
	Entity ID	92.47		50.79
	Property ID	99.45		30.26
	Class ID	94.64		N/A
	Numerical value	99.91		N/A
	Avg. token	97.70		61.53

Table 7 .

 7 14: LF token accuracy metrics, on the evaluation splits. For ConvQuestions, Class ID, numerical value and their relative operators are not used (see 7.B.2).

http://yann.lecun.com/exdb/mnist

https://www.cs.toronto.edu/~kriz/cifar.html

< ∞.

https://amritasaha1812.github.io/CSQA

https://convex.mpi-inf.mpg.de

https://www.wikidata.org

https://github.com/guoday/Dialog-to-Action/blob/bb2cbb9de474c0633bac6d01c10eca24c79b951f/BFS/parser.py

Remerciements

This work was supported by the Paris Ile-de-France Region via the DIM Math Innov program. We also acknowledge support from Google via a Google PhD Fellowship and from MINES Paris -PSL.

Proof. See Section 5.B.2.

We introduce in the proposition the time-augmented process X with an additional coordinate corresponding to time. Note that this coordinate is rescaled by 1-L 2 to ensure that the total variation of X is less than 1, since L < 1. Equation (5.4) can be better understood by the following equivalent integral equation:

where the integral should be understood as Riemann-Stieljes integral (Friz and Victoir, 2010, Section I.2). Thus, the output of the RNN can be approximated by the solution of the CDE (5.4), and, according to Proposition 5.1, the approximation error is O( 1 /T ).

Example 5.3. Consider f RNN as in (5.2). If σ is Lipschitz continuous with constant K σ , then, for any

where • op denotes the operator norm-see Appendix 5.A.3. Similarly, f (h 1 , x 1 )f (h 1 , x 2 ) ≤ K σ V op x 1x 2 . Thus, assumption (A 1 ) is satisfied. The tensor field F RNN of Proposition 5.2 corresponding to this network is defined for any h ∈ R ē by

, where W = U V ∈ R e×ē .

(5.5)

The signature

An essential ingredient towards our construction is the signature of a continuous-time process, which we briefly present here. We refer to [START_REF] Chevyrev | A primer on the signature method in machine learning[END_REF] for a gentle introduction and to [START_REF] Lyons | Differential Equations Driven by Rough Paths[END_REF]; [START_REF] Levin | Learning from the past, predicting the statistics for the future, learning an evolving system[END_REF] for details.

Tensor Hilbert spaces. We denote by (R d ) ⊗k the kth tensor power of R d with itself, which is a Hilbert space of dimension d k . The key space to define the signature and, in turn, our RKHS, consists in infinite square-summable sequences of tensors of increasing order:

where the norm is associated to the entrywise scalar product

T is a Hilbert space. We refer to Appendix 5.A.4 for more precise explanations.

Note that for a fixed j, the pairs ( X(i)

[j] , y (i) j ) are i.i.d. Under the assumptions of the theorem, there exists a ball B ⊂ H such that for any 1 ≤ ≤ p, θ ∈ Θ, ξ α ,θ ∈ B . We denote by B p the sum of p such spaces, that is,

Clearly, Ξ θ ∈ B p , and it follows that

We have once again reached a familiar situation, which can be dealt with by an easy extension of Bartlett and Mendelson (2002, Theorem 12). For any

Then with probability at least 1δ,

which concludes the proof.

5.C Differentiation with higher-order tensors

5.C.1 Definition

We define the generalization of matrix product between square tensors of order k and .

Definition 5.26. Let a ∈ (R e ) ⊗k , b ∈ (R e ) ⊗ , p ∈ {1, . . . , k}, q ∈ {1, . . . , }. Then the tensor dot product along (p, q), denoted by a p,q b ∈ (R e ) ⊗(k+ -2) , is defined by

a (i 1 ,...,i p-1 ,j,ip,...,i k-1 ) b (j 1 ,...,j q-1 ,j,jq,...,j -1 ) .

This operation just consists in computing a ⊗ b, and then summing the pth coordinate of a with the qth coordinate of b. The operator is not associative. To simplify notation, we take the convention that it is evaluated from left to right, that is, we write a b c for (a b) c. Definition 5.27. Let a ∈ (R e ) ⊗k . For a given permutation π of {1, . . . , k}, we denote by π(a) the permuted tensor in (R e ) ⊗k such that π(a) (i 1 ,...,i k ) = a (i Π(1) ,...,i Π(k) ) .

Example 5.28. If A is a matrix, then A T = π(A), with π defined by π(1) = 2, π(2) = 1. and SGD randomness. This confirms that, in our setting, the SGD is able to recover the target function in the two-timescale regime (ε 1), but fails outside the two-timescale regime (ε = 1). The transition between the two regimes seems to occur for ε ≈ 0.1.

Part II

Contributions to finite-depth neural networks

Structured Input computation

Hierarchical structure For each query, we construct the input as a JSON-like structure, consisting of lists of objects with their fields (represented in the left part of Figure 7.1). We chose this representation as it allows incorporating general structured information into the model. A field can be a string, a KG integer ID, a numerical value, or a list thereof.

To construct the input, we start from the last utterances in the dialog: the current query, previous query and previous answer. We first perform NEL to retrieve a list of entities E (and numerical values) matching the utterances. The KG is then queried to retrieve additional contextual information: the classes of the entities, and all outgoing and incoming properties from these entities E. This gives a list of properties P. For each property p ∈ P, we fill several fields: its ID, its name, and an Entity IDs field, which corresponds to all entities e ∈ E such that at least one graph operator gives a non-empty result when applied to e and p. For instance, in Wikidata, the property birth name (P1477) is filled for Marie Curie but not for Pierre Curie, so the Entity IDs field of the birth name property only contains Marie Curie.

Let us introduce some formal notations, useful to explain the computation of the input's embeddings (Section 7.4.3). The input is a tree where the root corresponds to the whole input, and each leaf contains the primitive values. For a non-leaf node x, we denote by c(x) its children. For instance, in Figure 7.1, the node Property 2 has three children (leaves) whose values are mother, P25 and Q37463. A node x has also a type, and T (x) denotes the types of all nodes on the path from the root to x. For instance, for the mother node, T (x) is equal to (root, property, name). In our setup, the depth of the input is at most 2. did not pick up the user intent. Or the form of the LF could be right, but (at least) one of the tokens is wrong. Table 7.7 details the error statistics. The most frequent errors concern entity disambiguation. There are two types of errors: either the correct entity was not part of the model input, due to insufficient recall of the NEL system. Or the model picked the wrong entity from the input due to insufficient precision. It is known that the noise from NEL strongly affects model performance [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF]. We tried an oracle experiment with perfect recall NEL (see Appendix 7.D.3), which corroborates this observation, in particular for Simple questions. As we focused on modeling complex questions, improving NEL was not our main focus, but would an interesting direction for future work, in particular via multi-task approaches [START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF]. The model learns the grammar rules. In all inspected cases, the predicted LF is a valid LF according to the grammar, i.e. it could be evaluated successfully. This shows that grammar-guided decoding is not needed to achieve high performance.

Error category

Conclusion

For the problem of weakly-supervised conversational KG-QA, we proposed Object-Aware Transformer, a model capable of processing structured input in a JSON-like format. This allows to flexibly provide the model with structured KG contextual information. We also introduced a KG played by )))

Assume that the KG contains exactly two musical instruments, piano and violin, or in other words, members(musical instrument ) equals {piano, violin }.

for_each creates a dictionary of entities. Each (key, value) pair corresponds to one entity in the argument of for_each, where the key is the entity itself and the value is a singleton set containing the entity. Here for_each({piano, violin }) gives the following dictionary: { piano: {piano}, violin: {violin} }

We then apply the same computation to each of the dictionary values, while keeping the keys untouched. In our example, we apply the expression cardinality( follow_backward(., played by ) ), which gives the result {piano: 20392, violin: 7918}.

Finally, an aggregation operator is computed over the values, and the result is a subset of the keys. In the example, argmax returns the set of keys associated with the maximum values, here {piano}. In other cases, we want to return all the keys associated to a non-empty value, arg allows doing so.

7.B.2 Silver LF generation

Wikidata version For CSQA, we used the preprocessed version of Wikidata made available by the authors, which contains 21.2M triplets over 12.8M entities and 567 distinct properties. For ConvQuestions, we used a more recent version of Wikidata, containing 1.1B triplets over 91.8M entities and 7869 distinct properties.

Named Entity Linking For ConvQuestions, we use gold entity annotations and Google Cloud NLP entity linking service. For CSQA, we use gold entity annotations.

To resolve the coreferences, in ConvQuestions, we use entity annotations from previous utterances during the silver LF generation step. In CSQA, since coreferences are already resolved by the gold annotations, we just use annotations from the current utterance.

Simplifying the BFS We observed that reaching a depth of 4+ is needed for some queries (see Table 7.5 of the main part of the chapter), but is impractical by exhaustive BFS, as the size of the space of LFs grows very quickly with their depth. To improve the efficiency, we used the following ideas:

• Stopping criteria to abort the exploration: timeout t max and maximum depth d max .

• Type checking: by leveraging the operators' signatures (presented in Table 7.1 of the main part), we only construct legal LFs.

• Putting constraints on the form of the LF: we manually forbid certain combinations of operators, e.g. follow_backward after follow_property.

• Restriction of the list of operators: for ConvQuestions, we use the graph operators, the numerical operators, is_in, and get_first. The removal of some set operators and of meta-operators strongly reduces the complexity of the BFS. For CSQA, all operators are needed, but we add more constraints in order to keep the BFS simple enough.

We choose d max = 3 for ConvQuestions and d max = 7 for CSQA, and t max = 1200 seconds. All LFs found by BFS are evaluated over the KG, which gives candidate answers. We keep the LFs whose candidate answers have the highest F1 score w.r.t. the gold answer. The minimal F1 score for keeping a LF is 0.3.

Scores for LF ranking

The BFS often returns several LFs (with the top F1 score, as explained above), among which some are spurious: they do not correspond to the semantic meaning of the question, but their evaluation over the KG yields the correct result by chance. As we keep only one for training, we need a way to rank the candidate LFs. We use the following heuristic scores to do so:

• Complexity: the score is 1 -(d-1) /(dmax-1) where d is the depth of the LF and d max is defined above.

• Property lexical matching: for each property appearing in the LF, we compute the Jaccard index of the words appearing in its name and of the words of the question.

• Annotation coverage: among the entities retrieved by NEL, we compute the percentage of entities which appear in the LF.

As these three scores are between 0 and 1, we average them and keep the LF with the highest total score. We found that this simple method is a good way to reduce spurious LFs, which are often either too complex or not matching lexically the question.

7.B.3 Random examples generation

To generate the random examples for data augmentation for ConvQuestions training, we first sample uniformly 80k entities from the graph. Then, for each entity, we generate a conversation for each triplet that links it to other entities. The question text is made by stitching the entity name and the property name. For instance, the triplet (Marie Curie, native language, Polish) generates the dialog: Q: Marie Curie native language? A: Polish. We also generate variants where the property name is replaced by aliases, which are alternative names in Wikidata, e.g. mother tongue for native language. When the question or answer has more than 256 characters, we eliminate it.

7.B.4 Modeling

Wikidata version As in 7.B.2.

Named Entity Linking NEL is performed again, this time to create the structured context in input to the model. Due to the randomization step described in Section 7.4.2, missing entities in the input cannot be retrieved by the model, so we want to have a high NEL recall. The trade-off is that the NEL precision is low, meaning that we have many spurious entities in the input, which the model has to learn to ignore.

For CSQA, we use simple string matching between the utterances and the names of Wikidata entities. Note that in our model, as well as in all CSQA baselines (in particular Guo et al., 2018;[START_REF] Shen | Multi-task learning for conversational question answering over a large-scale knowledge base[END_REF], the previous gold answer is given as input to the model in an oracle-like setup.

For ConvQuestions, we use the gold seed entity and the Google Cloud NLP entity linking service.

7.D.2 Performance over turns

Tables 7.11 and 7.12 show the evolution of the performance over turns for both datasets. For CSQA, the performance drops after the first two turns, then remains constant. For ConvQuestions, the performance decreases throughout the turns. There is a sharp decrease after the first turn, probably because it is simpler as there is no coreference or ellipsis. The different behavior between the datasets may be due to the realism of ConvQuestions.

7.D.3 Oracle setup and comparison with CARTON

CARTON [START_REF] Plepi | Context transformer with stacked pointer networks for conversational question answering over knowledge graphs[END_REF] gives the entities annotated in the dataset as part of the model input (entities appearing in the previous turn and in the current question), contrarily to the models in Table 7.3 which all use an entity linker. For a fair comparison, we tested our model in an oracle setup, where we also give the gold annotations as input. As shown in Table 7.13, the Total Average score of our model increases by 10% w.r.t. the baseline approach. The improvement is particularly important for the most simple question types (Simple and Logical Questions). In this setup, our performance is 8% higher than CARTON, and we obtain a better score for 7 out of 10 question types.

7.D.4 Further error analysis

An alternative approach for error analysis is to assess the performance of the decoding classifiers (see Section 7.4.5)