Eugenio Clerico
email: eugenio.clerico@gmail.com

Benjamin Guedj
email: b.guedj@ucl.ac.uk

A note on regularised NTK dynamics with an application to PAC-Bayesian training

We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.

Introduction

The analysis of infinitely wide neural networks can be traced back to [START_REF] Neal | Bayesian learning for neural networks[END_REF], who considered this limit for a shallow (1-hidden-layer) network and showed that, before the training, it behaves as a Gaussian process when its parameters are initialised as independent (suitably scaled) normal distributions. A similar behaviour was later established for deep architectures, also allowing for the presence of skip-connections, convolutional layers, etc. [START_REF] Lee | Deep neural networks as Gaussian processes[END_REF]2019;Arora et al., 2019a;[START_REF] Novak | Bayesian deep convolutional networks with many channels are gaussian processes[END_REF][START_REF] Garriga-Alonso | Deep convolutional networks as shallow gaussian processes[END_REF][START_REF] Yang | Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation[END_REF][START_REF] Hayou | Stable ResNet. AISTATS[END_REF]. [START_REF] Lee | Wide neural networks of any depth evolve as linear models under gradient descent[END_REF]2020) among others brought empirical evidence that wide (but finite-size) architectures are still well approximated by the Gaussian limit, while finite size corrections were derived in [START_REF] Antognini | Finite size corrections for neural network Gaussian processes[END_REF] and [START_REF] Basteri | Quantitative Gaussian approximation of randomly initialized deep neural networks[END_REF].

Although the previous results hold only at the initialisation (as the Gaussian process approximation is only valid before the training), [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF] established that the evolution of an infinitely wide network can still be tracked analytically during the training, under the so-called neural tangent kernel (NTK) regime. In a nutshell, they showed that the usual gradient flow on the parameters space induces the network's output to follow a kernel gradient flow in functional space, governed by the NTK. A main finding of [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF] is that although for general finite-sized networks the NTK is time-dependent and random at the initialisation, in the infinite-width limit it becomes a deterministic object that can be exactly computed, and it stays fixed throughout the training. Later, [START_REF] Lee | Wide neural networks of any depth evolve as linear models under gradient descent[END_REF] provided a new proof of the convergence to the NTK regime, while [START_REF] Yang | Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation[END_REF] established similar results for more general architectures, such as convolutional networks. [START_REF] Chizat | On lazy training in differentiable programming[END_REF] extended the idea of linearised dynamics to more general models, introducing the concept of lazy training and finding sufficient conditions for a network to reach such regime. They also pointed out that this linearised behaviour may be detrimental to learning, as also highlighted by [START_REF] Yang | Feature learning in infinite-width neural networks[END_REF] who showed that the NTK dynamics prevents a network hidden layers from effectively learning features. However, the NTK has been a fruitful tool to analyse convergence (Allen-Zhu et al., 2019b;[START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF] and generalisation (Allen-Zhu et al., 2019a;Arora et al., 2019b;[START_REF] Cao | Generalization error bounds of gradient descent for learning over-parameterized deep relu networks[END_REF] for over-parameterised settings under (stochastic) gradient descent.

The standard derivation of the NTK dynamics [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF][START_REF] Lee | Wide neural networks of any depth evolve as linear models under gradient descent[END_REF] requires the network to be trained by gradient descent to optimise an objective that depends on the parameters only through the network's output. This setting does not allow for the presence of regularising terms that directly involve the parameters. Yet, in practice, a network reaches the NTK regime when its training dynamics can be linearised around the initialisation. This happens if the network parameters stay close enough to their initial value throughout the training, the defining property of the lazy training regime. It is then natural to expect that a regularising term that enforces the parameters to stay close to their initialisation will still favour linearised dynamics, and so bring a training evolution that still can be expressed in terms of a fixed and deterministic NTK. This is the focus of the present paper, where we discuss the evolution of a network in the NTK regime trained with an ℓ 2 -regularisation that constrains the parameters to stay close to their initial values. Regularisers of this kind typically appear in PAC-Bayes-inspired training objectives, where the mean vector of normally distributed stochastic parameters is trained via (stochastic) gradient descent on a generalisation bound (an approach initiated by the seminal work of [START_REF] Langford | Not) bounding the true error[END_REF] and further explored by [START_REF] Alquier | On the properties of variational approximations of Gibbs posteriors[END_REF][START_REF] Dziugaite | Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data[END_REF][START_REF] Neyshabur | A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks[END_REF][START_REF] Letarte | Dichotomize and generalize: PAC-Bayesian binary activated deep neural networks[END_REF][START_REF] Nagarajan | Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience[END_REF][START_REF] Zhou | Non-vacuous generalization bounds at the ImageNet scale: a PAC-Bayesian compression approach[END_REF][START_REF] Nozawa | PAC-Bayesian contrastive unsupervised representation learning[END_REF][START_REF] Biggs | Differentiable PAC-Bayes objectives with partially aggregated neural networks[END_REF]2022;[START_REF] Dziugaite | On the role of data in PAC-Bayes bounds[END_REF]Pérez-Ortiz et al., 2021a;[START_REF] Guedj | A primer on PAC-Bayesian learning[END_REF]Pérez-Ortiz et al., 2021;[START_REF] Chérief-Abdellatif | On PAC-Bayesian reconstruction guarantees for VAEs[END_REF][START_REF] Lotfi | PAC-Bayes compression bounds so tight that they can explain generalization[END_REF][START_REF] Tinsi | Risk bounds for aggregated shallow neural networks using gaussian priors[END_REF][START_REF] Clerico | Conditionally Gaussian PAC-Bayes[END_REF]2023a;[START_REF] Viallard | Learning via Wasserstein-based high probability generalisation bounds[END_REF]. As these training objectives yield generalisation guarantees, we conjecture that the exact dynamics that we derive could be a starting point to obtain generalisation bounds for more general kernel gradient descent algorithms.

As final remarks, we note that Chen et al. ( 2020) considers a NTK regime that allows for regularisation, but they only consider the mean-field setting of two-layer neural networks (introduced by [START_REF] Chizat | On the global convergence of gradient descent for over-parameterized models using optimal transport[END_REF][START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF], and further explored by [START_REF] Mei | Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit[END_REF][START_REF] Wei | Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel[END_REF][START_REF] Fang | Convex formulation of overparameterized deep neural networks[END_REF], where the network is not initialised with a scaling yielding a Gaussian process. Finally, we mention that also [START_REF] Huang | Demystify optimization and generalization of over-parameterized PAC-Bayesian learning[END_REF] attempted the analysis of PAC-Bayesian dynamics via NTK. However, their approach approach differs from ours, it does not highlight the effect of the regularisation term, and the whole analysis deals with a simple shallow stochastic architecture where only one layer is trained.

Outline. We present our framework and notation in Section 2 and then treat the unregularised NTK dynamics in Section 3 as a starter. We then move on to the more interesting case of regularised dynamics in Section 4, first for the simple case of ℓ 2 -regularisation and then for a more general regularising term. We instantiate our analysis to the example of least square regression in Section 5 and illustrate the merits of our work with an application to PAC-Bayes training of neural networks in Section 6. The paper ends with concluding remarks in Section 7 and we defer technical proofs to Appendix A.

Setting and notation

We consider a fully-connected feed-forward neural network of depth L, which we denote as F : X → R q , where X ⊂ R p is a compact set. We denote as n l the width of the l-th hidden layer of the network, as n 0 = p the input dimension and n L = q the output dimension. We consider the network

F (x) = U L (x) ; U l+1 i (x) = 1 √ n l n l j=1 W l+1 ij φ(U l j (x)) ; U 1 i (x) = 1 √ n 0 n0 j=1 W 1 ij x j ;
where φ is the network's activation, acting component-wise. The network's prediction in the label space Y is ŷ(x) = f (F (x)), for some f : R q → Y. We denote as W the parameter space where the weights lie, and as W the parameters of the network. We consider the infinite-width limit, where all the hidden widths n l (l = 1, . . . , L -1) are taken to infinity1 .

Data consists in pairs instance-label z = (x, y) ∈ Z = X × Y, with x ∈ X and y ∈ Y. We consider a non-negative loss function ℓ : W × Z → [0, ∞). For a dataset s ∈ Z m , we define the empirical loss L s as the average of ℓ on s, namely

L s (W ) = 1 m z∈s ℓ(W, z) .
As we will often encounter empirical averages, we define the following handy notation

g(Z) s = g(X, Y ) s = 1 m (x,y)∈s g(x, y) , so that L s (W ) = ℓ(W, Z) s .
We assume that ℓ depends on W only through the network's output F , i.e., there exists a function l such that we can rewrite

ℓ(W, z) = l(F (x), y) .
The network training follows the gradient of a learning objective C s , namely

∂ t W (t) = -∇C s (W (t)) ,
where ∇ denotes the gradient with respect to the parameters. We assume that C s can be split into two terms, the empirical loss, and a regularisation term that depends directly on the parameters (without passing through the network's output F ) and does not depend on s:

C s = L s + λR , (1)
for some λ ≥ 0. We let ρ : R + → R + be a strictly increasing differentiable function such that ρ(0) = 0, and we consider the case of a regulariser in the form

R(W ) = ρ 1 2 W -W (0) 2 F , (2)
with W (0) denoting the value of the parameters at the initialisation, and with • 2 F denoting the square of the Frobenius norm on W , namely W 2 F = l,i,j (W l ij ) 2 . For conciseness, we write ∆g(t) for g(t)g(0), where g is any time-dependent term. Moreover, we introduce the following notation

ψ l;l ′ k;ij (x; t) = ∂U l k (x; t) ∂W l ′ ij ; Θ l kk ′ (x, x ′ ; t) = l l ′ =1 ψ l;l ′ k (x; t) • ψ l;l ′ k ′ (x ′ ; t) ; Ξ l k (x; t) = l l ′ =1 ψ l;l ′ k (x; t) • ∆W l ′ (t) , (3)
where ' • ' denotes the component-wise inner product between matrices (or vectors) of the same size. Θ L is the so-called neural tangent kernel (NTK) of the network.

The unregularised dynamics

Without regularisation (namely when we set ρ ≡ 0) the standard NTK dynamics holds [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF][START_REF] Lee | Wide neural networks of any depth evolve as linear models under gradient descent[END_REF]. We briefly cover this case, where the width of the network is taken to infinity.

At the initialisation, the network's output behaves as a centred Gaussian process, whose covariance kernel can be computed recursively. More precisely, we have that all the components of the output are i.i.d. Gaussian processes defined by

F k ∼ GP(0, Σ L ) ,
where

Σ 1 (x, x ′ ) = x • x ′ n 0 ; Σ l+1 (x, x ′ ) = E (ζ,ζ ′ )∼N (0,Σ l (x,x ′ )) [φ(ζ)φ(ζ ′ )] . (4)
Moreover, the neural tangent kernel Θ L , defined in (3), tends to a diagonal deterministic limit (with respect to the initialisation randomness), and stays constant during the training. In particular, we have

Θ L kk ′ (x, x ′ ; t) = ΘL (x, x ′ )δ kk ′ .
Moreover, ΘL can be computed recursively as follows:

Θ1 = Σ 1 ; Θl+1 (x, x ′ ) = Σ l+1 (x, x ′ ) + E (ζ,ζ ′ )∼N (0,Σ l (x,x ′ )) [ φ(ζ) φ(ζ ′ )] Θl (x, x ′ ) , ( 5
)
where φ is the derivative of φ.

During the training, the network's output obeys the dynamics

∂ t F k (x; t) = - 1 m (x,ȳ)∈s ΘL (x, x) ∂ l(F (x; t), ȳ) ∂F k = -ΘL (x, X) ∂ l(F (X), Y ) ∂F k s . ( 6
)
4 The regularised dynamics

In this section, we discuss the impact of the regularisation term on the NTK dynamics. We first particularise this to the specific case of ℓ 2 -regularisation, then move to the treatment of more general regularisers.

Simple ℓ 2 -regularisation

We first consider the case ρ ≡ id in (2), so that R(W ) = 1 2 ∆W 2 F and

C s (W ) = L s (W ) + λ 2 ∆W 2 F . ( 7
)
By just applying the chain rule to ∂ t W (t) = -∇C s (W (t)), we find that

∂ t W l ij (t) = -∇C s (W (t)) = - nL k=1 ψ L;l k;ij (X; t) ∂ l(F (X; t), Y ) ∂F k s -λ∆W l ij (t) . ( 8
)
This translates into the following functional evolution of the network's output

∂ t F k (x; t) = - q k ′ =1 Θ L kk ′ (x, X; t) ∂ l(F (X; t), Y ) ∂F k ′ s -λΞ L k (x; t) .
Our goal is to prove that in the infinite width limit the next two properties hold:

1. The NTK is constant at its initial value, which coincides with the standard deterministic NTK in (5), and more generally for all layers

Θ l kk ′ (x, x ′ ; t) = δ kk ′ Θl (x, x ′ ) ;
2. The term Ξ L is exactly ∆F , and more generally

Ξ l (x; t) = ∆U l (x; t) = U l (x; t) -U l (x; 0) .
We note that these two properties are actually rather intuitive: the first one tells us that the Jacobian of the output stays fixed during the training, as if the dynamics where linear; the second one that we can indeed linearise U around the initialisation. We recall that the standard NTK regime (with no regularisation) holds when the parameters do not move too much from their initial values, so that the dynamics can be linearised.

The addition of a regularising term does actually enforce the parameters to stay close to their initial value. Hence, adding regularisation does not hinder the linearisation of the learning dynamics.

From the two properties above, we conclude that the NTK evolution of F is given by the following.

Theorem 1. In the infinite-width limit, taken recursively layer by layer, the network's output evolves as

∂ t F k (x; t) = - 1 m (x,ȳ)∈s ΘL (x, x) ∂ l(F (x; t), ȳ) ∂F k -λ(F k (x; t) -F k (x; 0)) ,
where ΘL is defined in (5).

Note the term -λ(F k (x; t)-F k (x; 0)), which constrains the network's output to stay close to its initialisation, is not present in the standard NTK dynamics (6).

Evolution of the training objective.

We can see how the training objective C s evolves during the training. First,

∂ t L s (W (t)) = -∇ F l(F (X; t); Y ) • ∂ t F (X; t) s = Θ(X, X ′ )∇ F ℓ(F (X; t); Y ) • ∇ F ℓ(F (X ′ ; t); Y ′ ) s⊗s -λ ∇ F ℓ(F (X; t); Y ) • ∆F (X; t) s ,
where the notation g(Z, Z ′ ) s⊗s denotes 1 m 2 z∈s z ′ ∈s g(z, z ′ ). On the other hand, for the regularising term we have that

∂ t R(W (t)) = -∇ F l(F (X; t), Y ) • ∆F (X; t) s -2λR(W (t)) . ( 9
)
Thus, overall we get that

∂ t C s (W (t)) = -Θ(X, X ′ )∇ F ℓ(F (X; t); Y ) • ∇ F ℓ(F (X ′ ; t); Y ′ ) s⊗s -2λ ∇ F ℓ(F (X; t); Y ) • ∆F (X; t) s -2λ 2 R(W (t)) .
As a side remark, we note that if l is convex in F , then we always have that

∇ F l(F (x; t), y) • ∆F (x; t) ≥ ∆ l(F (x; t), y) = l(F (x; t), y) -l(F (x; 0), y) .
In particular we get that

∂ t R(W (t)) ≤ ∆ℓ(F (x; t), Y ) s -2λR(W (t)) = -∆C s (W (t)) -λR(W (t)) . Since t → ∆C s (W (t)) is non-decreasing, we obtain that R(W (t)) ≤ 1 λ C s (W (0)) -C s (W (t)) (1 -e -λt ) ,
from which it follows that R(W (t)) can be controlled by the variation in L s (W (t)) as

λR(W (t)) ≤ 1 -e -λt 2 -e -λt L s (W (0)) -L s (W (t)) .

General regulariser

We consider the case of a more general regularising term, which still leads to tractable training dynamics. Let ρ : R + → R + be a differentiable strictly increasing function. Define

D(t) = 1 2 ∆W (t) 2 F . and let R(W (t)) = ρ(D(t)) .
Theorem 2. Consider a function ρ : [0, ∞) → [0, ∞) as above and assume that l is locally Lipschitz. Assume that the dynamics is given by

∂ t W (t) = -∇C s (W (t)) = -∇L s (W (t)) -λR(W (t)) .
Then, in the infinite-width limit (taken recursively layer by layer) we have

∂ t F k (x; t) = -Θ(x; X) ∂ l(F (X; t), Y ) ∂F k s -λρ ′ (D(t))∆F k (x; t) ; ∂ t D(t) = - q k=1 ∆F k (X; t) ∂ l(F (X; t), Y ) ∂F k s -2λρ ′ (D(t))D(t) .
Proof. The proof is based on the following result, which we establish in Appendix A.1.

Proposition 1. Fix a time horizon T > 0 and a depth L. Assume that for t ∈ [0, T ] and for all l ∈ [1 : L]

∂ t W l ij (t) = - nL k=1 ψ L;l k;ij (X; t)V k (Z; t) s -λr(D(t); t)∆W l ij (t) ,
for some mappings V : Z × R → R nL and r : [0, ∞) 2 → R. Then we have that U L obeys the dynamics

∂ t U L k (x; t) = - nL k ′ =1 Θ L kk ′ (x, X; t)V k ′ (F (X; t), Y ) s -λr(D(t); t)Ξ L k (x; t) . Moreover, if D(t) = O(1) for all t ∈ [0, T ], T 0 V (Z; t) s dt = O(1) and T 0 |r(D(t); t)|dt = O(1) ,
if φ is γ φ -Lipschitz and β φ -smooth, then (in the infinite-width limit taken starting from the layer 1 and then going with growing index), for all l ∈ [1 : L], Θ l is constant during the training, and Ξ l = ∆U l .

To prove Theorem 2, we need to check that the assumptions of the above proposition hold when setting r(D; t) = ρ ′ (D(t)) and V (z; t) = ∇ F ℓ(F (x; t), y). First, notice that C s (t) ≤ C s (0) = L s (0) for all t ≥ 0, as we are following the gradient flow. Now, with arbitrarily high probability (on the initialisation), we can find a finite upperbound J for L s (0), which holds when taking the infinite-width limit (as the network output becomes Gaussian). In particular, we have that since J is independent of the width and we can write J = O(1) (where the O notation is referred to the infinite width limit). Now, we have that

R(t) ≤ J/λ = O(1) .
Since ρ is invertible, in particular for all t ≥ 0 we have that

D(t) ≤ ρ -1 (J) = O(1) .
We prove in Lemma 1 (Appendix A.1) that ∆F (t) = O(1), and so we know that with arbitrarily high probability we can find a radius J ′ such that F (x; t) ≤ J ′ for all t > 0. In particular, the regularity of l implies that V (z; t) = ∇ F l(F (x; t), y) is uniformly bounded for all t > 0. So,

T 0 V (Z; t) s dt = O(1)
. Finally, we have that the r(D; t) in the previous statement is ρ ′ (D(t)). Since D(t) is bounded throughout the training and ρ is locally Lipschitz we can bound the integral over r and apply Proposition 1.

Finally, the dynamics for D(t) follows from the chain rule.

Clearly Theorem 1 is just a particular instance of Theorem 2, as we only need to set ρ as the identity.

The example of least square regression

As a simple application of what we established, we study the evolution of a network under least square regression, namely when we have l(ŷ, y) = 1 2 (ŷ, y) 2 . The dynamics of the training is given by

∂ t F (x; t) = -Θ(x, X)(F (X; t) -Y ) s -λ(F (x; t) -F (x; 0)) .
This is a linear ODE and can be solved exactly. For convenience we introduce the following notation. We let Θ denote the NTK Gram matrix whose entries are Θ(x, x ′ )/m, with x and x ′ ranging among the instances of the training sample s. Similarly F (t) and Y are the vectors made of the network's output and labels, for the datapoints in s. We thus have

∂ t F (t) = -Θ( F (t) -Y ) -λ( F (t) -F (0)) , which brings
F (t) = F (0) + (Id -e -tV λ )V -1 λ Θ( Y -F (0)) ,
where V λ = λId + Θ, which is always invertible for λ > 0.

Note that asymptotically for large t we have that

F (t) → F (∞) = Id -V -1 λ Θ F (0) + V -1 λ Θ Y ,
which is exactly what one would obtain optimising C s in (1). Clearly, for small values of λ the labels are almost perfectly approximated, as we have

F (∞) = Y + λ Θ -1 (Id + λ Θ -1 ) -1 ( F (0) -Y ) = Y + O(λ) .
On the other hand, if λ is very large then F (∞) ≃ F (0), as

F (∞) = F (0) + Θ λ Id + Θ λ -1 (Y -F (0)) = F (0) + O(1/λ) .
Once the evolution of F on the training datapoints has been computed, one can directly evaluate the value of F (x; t) for any input that is not in the training sample. Defining

τ (x; t) = Θ(x, X)(F (X; t) -Y ) s ,
we have

F (x; t) = F (x; 0) + (1 -e -λt ) t 0 τ (x; t ′ )e λt ′ dt ′ .

An application to PAC-Bayesian training

A motivation to study dynamics in the form of (8) comes from the PAC-Bayesian literature. We consider a dataset s made of i.i.d. draws from a distribution µ on Z. We are seeking for parameters W with a small population loss

L Z (W ) = E µ [ℓ(W )] .
As µ is unknown, we rely on the empirical loss L s as a proxy for L Z .

The PAC-Bayesian bounds (introduced in the seminal works of Shawe-Taylor & Williamson, 1997;[START_REF] Mcallester | PAC-Bayesian model averaging[END_REF][START_REF] Seeger | PAC-Bayesian Generalization Error Bounds for Gaussian Process Classification[END_REF][START_REF] Maurer | A note on the PAC Bayesian theorem[END_REF][START_REF] Catoni | Statistical Learning Theory and Stochastic Optimization[END_REF][START_REF] Catoni | PAC-Bayesian supervised classification: The thermodynamics of statistical learning[END_REF] -we refer to the recent surveys from [START_REF] Guedj | A primer on PAC-Bayesian learning[END_REF][START_REF] Alquier | User-friendly introduction to PAC-Bayes bounds[END_REF][START_REF] Hellström | Generalization bounds: Perspectives from information theory and PAC-Bayes[END_REF] for a thorough introduction to PAC-Bayes) are generalisation guarantees that upperbound in high probability the population loss of stochastic architectures, in our case networks whose parameters W are random variables (this means that every time that the network sees an input x, it draws W from some distribution Q, and then evaluate F (x) for this particular realisation of the parameters). The PAC-Bayesian bounds hold in expectation under the parameters law Q (commonly referred to as posterior). Here is a concrete example of this kind of guarantees. Fix a probability measure P and η > 0, for a bounded loss ℓ ∈ [0, 1] we have (see, e.g., Alquier, 2021)

E W ∼Q [L µ (W )] ≤ 1-δ E W ∼Q [L s (W )] + 1 √ 8m η + KL(Q|P ) + log(1/δ) η , ( 10
)
where the inequality holds uniformly for every probability measure Q, in high probability (at least 1δ) with respect to the draw of s. The probability measure P (typically called prior) in ( 10) is arbitrary, as long as it is chosen independently of the particular dataset s used for the training.

Several studies (see Section 1 -this line of work started with [START_REF] Langford | Not) bounding the true error[END_REF], was reignited by Dziugaite & Roy, 2017 and then followed by a significant body of work by many authors) have proposed to train stochastic neural networks by optimising a PAC-Bayesian bound. A possible approach consists in considering the case when all the parameters of the network are independent Gaussian variables with unit variance (namely W l ij ∼ N (m l ij , 1)). The training then usually amounts to tune the means m. Typically, the initial values of the means are randomly initialised (we denote them as m(0)), and P can be chosen as the distribution of the networks parameters at initialisation [START_REF] Dziugaite | Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data[END_REF], namely a multivariate normal with the identity as covariance matrix and m(0) as mean vector. In this way, one gets that

KL(Q|P ) = 1 2 m(t) -m(0) 2 F .
Defining Ls (m) = E W ∼N (m,Id) [L s (W )], we see that using the bound (10) as training objective is equivalent to optimise

C s (m) = Ls (m) + 1 2η √ 8m m -m(0) 2 F , ( 11
)
which is exactly in the form of ( 7) with λ = 1/(η √ 8m). Note that many other typical PAC-Bayesian bounds (potentially non-linear in the KL term) can also fit in our framework when considering more general regularising terms, as in Section 4.2. Clerico et al. (2023b) has recently shown that, when considering the infinite-width limit for a single-hiddenlayer stochastic network, a close form for Ls (m) can be computed explicitly, and one can actually see the stochastic network as a deterministic one (with a different activation function), where the means m are the trainable parameters. We summarise and rephrase the results of Clerico et al. (2023b), and show explicitly how exact continuous dynamics can be established via our results, in the infinite-width limit.

Training of wide and shallow stochastic networks

We focus on a binary classification problem (i.e., Y = {±1}), where we assume that all the inputs x are normalised so that x = √ n 0 (i.e., X = S n0-1 ( √ n 0 ), the sphere of radius √ n 0 in R n0 ). We consider a stochastic network with a single hidden layer and one-dimensional output, and Lipschitz and smooth activation φ,

F (x) = 1 √ n n j=1 W 2 j φ(U 1 j (x)) ; U 1 i (x) = 1 √ n 0 n0 j=1 W 1 ij x j .
The prediction of the network is the sign of the output. The stochastic parameters can be rewritten as

W l = ζ l + m l ,
where ζ l is a matrix of the same dimension of W l , whose components are all independent standard normals (resampled every time that a new input is fed to the network) and m l is a matrix of trainable parameters (the means of W l ). We assume that the components of m l are all randomly initialised as independent draws from a standard normal distribution. In this setting, Clerico et al. (2023b) showed that in the infinite-width

limit n → ∞ F (x) ∼ N (M 2 (x), Q 2 (x)) , 2 (12)
where M2 and Q 2 are the output mean and variance in the limit.

When the training objective is in the form (11), the network is constraint to remain close to its initialisation. Interestingly, in this lazy training regime Q 2 stays constant to its initial value, which is deterministic (with respect to the initialisation randomness) and independent of x when X is a sphere. We can thus define σ > 0 such that Q 2 (x; t) = σ 2 , for all x ∈ X and t. We refer to the Appendix A.2 for details.

Following the derivation of Clerico et al. (2023b) we note that M 2 can actually be seen as the output of a neural network with parameters m, whose activation function is ψ, defined as a Gaussian convolution of φ,

ψ(u) = E ζ∼N (0,1) [φ(ζ + u)] .
Concretely, this means that

M 2 (x) = 1 √ n m 2 ψ(M 1 (x)) ; M 1 (x) = 1 √ n 0 m 1 x .
Now, for a loss function l(F, z), we can define the expected loss

l(M, z) = E ζ∼N (0,1) [ l(σζ + M, z)] .
In this way, we have that the expected empirical loss Ls (m) appearing in ( 11) is given by

Ls (m) = 1 m z∈s l(M 2 (x), y) .
Hence, optimising the PAC-Bayes bound (10) induces the dynamics

∂ t M 2 (x; t) = -Θ(x; X) ∂ l(M 2 (X; t), Y ) ∂M 2 s - 1 η √ 8m ∆M 2 (x; t) , ( 13
)
where Θ(x,

x ′ ) = ∇ m M 2 (x; 0) • ∇ m M 2 (x ′ ; 0) is given by Θ(x, x ′ ) = Σ(x, x ′ ) + x • x ′ n 0 E[ ψ(ζ) ψ(ζ ′ )] ; Σ(x, x ′ ) = E[ψ(ζ)ψ(ζ ′ )] , with (ζ, ζ ′ ) ∼ N 0, 1 n0 n0 x•x ′ x•x ′ n0
and ψ denoting the derivative of ψ.

Misclassification loss.

A common choice is to set l(F, z) = 1 if sign F (x) = y, and 0 otherwise, that is the so-called misclassification loss. In such a case we can easily derive that

l(M, z) = P ζ∼N (0,1) ζ > yM (x) σ = 1 2 1 -erf yM (x) σ √ 2 .
It follows that ( 13) now reads

∂ t M 2 (x; t) = Y Θ(x; X) e -M 2 (X;t)/(2σ 2 ) σ √ 2π s - 1 η √ 8m ∆M 2 (x; t) .
This does not have a simple close-form solution, and can only be solved using numerical integrators.

Quadratic loss. In order to obtain simpler dynamics we consider the loss l(F, z) = (1 -yF (x)) 2 . Note that this quadratic loss is unbounded, and so a generalisation bound such as ( 10) is not guaranteed to hold. However, the quadratic loss is always greater than the misclassification loss, so the RHS of ( 10) is still a valid generalisation upperbound for the misclassification population loss. This choice of l yields the dynamics

∂ t M 2 (x; t) = -2 Θ(x; X)(M 2 (x; t) -Y ) s - 1 η √ 8m ∆M 2 (x; t) .
Proceeding as in Section 5, and again using a tilde to denote vectors and matrices indexed on the training sample s, we get that

M 2 (t) = M 2 (0) + Id -e -2tV λ/2 V -1 λ/2 Θ( Y -M 2 (0)) ,
where V λ/2 = λId/2 + Θ and λ = 1/(η √ 8m). Asymptotically, for large t, M 2 will approach

M 2 (∞) = M 2 (0) + V -1 λ/2 Θ( Y -M 2 (0)) .
For large t, the empirical loss approaches

L∞ = 1 m M 2 (0) -Y 2 (λ/2) 2 V -2 λ/2
, where for a positive definite matrix A we define v 2 A = v ⊤ Av. We note that the eigenvalues of (λ/2) 2 V -2 λ/2 are in the form

α i = λ 2 /4 (λ/2 + θ i ) 2 ,
where the θ i 's are the eigenvalues of Θ. In practice, we can expect the largest eigenvalues of Θ to be of order 1 (i.e., max i θ i ∼ 1) when the sample size m grows to infinity [START_REF] Murray | Characterizing the spectrum of the NTK via a power series expansion[END_REF]. Since λ ∼ 1/ √ m, we get that the network will be able to reach a small empirical loss (of order

λ 2 ∼ 1/m) if M 2 (0) -Y lies in eigenspaces of Θ where the eigenvalues θ i ∼ 1 ≫ 1/ √ m.
On the other hand, for large t the regularising term R will approach R ∞ , which from (9) must satisfy

2 m ∆ M (∞) • ( M (∞) -Y ) = -2λR ∞ .
From this we can derive that

R ∞ = 1 2m M 2 (0) -Y 2 V -2 λ/2 Θ .
Here, the eigenvalues of V -2 λ/2 Θ are of the form

β i = θ i (λ/2 + θ i ) 2 .
If we are in the regime where the datapoints are such that M 2 (0) -Y again lies where θ i ∼ 1, then we can expect R ∞ ∼ 1. This means that if we are able to learn well the labels while optimising the PAC-Bayesian bound, we are ensured that the KL term is of order 1, and so the objective (11) will be of order 1/ √ m, resulting in a non-vacuous bound. We argue that this is what happens when data comes from a reasonable underlying distribution, matching the implicit regularisation induced by the NTK.

We finally note that the β i 's are small also when θ i ≪ 1/ √ m. However this is due to the fact that these directions are not promoted by the NTK dynamics and so if M 2 (0) -Y completely lies in eigenspaces with very small eigenvalues of Θ, then the network will essentially stay fixed to its initial configuration, keeping a small penalty, but also not improving its performance.

Conclusion

We established explicit dynamics for infinitely wide fully connected networks trained to optimise a regularised objective, where the regularisation pushes the parameters to stay close to their initialisation. Under this regime we show that the model undergoes linearised dynamics during the training, that appears to be a regularised version of the standard NTK evolution.

Our analysis follows similar ideas to the NTK convergence proof of [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF] and presents the first regularised NTK analysis that can also be applied to PAC-Bayesian training. We conjecture that stronger follow-up results could be derived, for instance following the approach of [START_REF] Lee | Wide neural networks of any depth evolve as linear models under gradient descent[END_REF] to show that the convergence holds when the infinite width limit is taken for all the hidden layers simultaneously and to study discretised dynamics.

We also anticipate that further analytical and empirical studies of the induced PAC-Bayesian dynamics might be of interest to shed some light on generalisation-driven training of neural networks.

A Appendix

A.1 Proof of Proposition 1

Lemma 1. Assume that, when the infinite-width limit is taken recursively layer by layer, D(t) = O(1). Assume that φ is γ φ -Lipschitz, then we have that for each layer l ∆U l (x; t) = O(1) .

Proof. We have that

∆U 1 (x; t) ≤ 1 √ n 0 ∆W 0 (t) x = O(1) .
Then, recall that at initialisation the components of U l (x, 0) are independent normal distributions, with mean 0 and variance Σ l (x, x), as defined in (4). So, we have that all the φ(U l i (x; 0)) 2 's are independent and equally distributed, with finite variance (thanks to the Lipschitzness of φ). By the standard CLT we get that 1

n l φ(U l (x; 0)) 2 = 1 n l n l i=1 φ(U l i (x; 0)) 2 ∼ 1 n l n l i=1 E[φ(U l i (x; 0)) 2 ] = O(1) ,
and so φ(U l (x; 0)) = O( √ n l ). Now, using the Lipschitzness of φ we get

∆U l+1 (x; t) ≤ 1 √ n l W l+1 (0) γ φ ∆U l (x; t) + ∆W l+1 (t) φ(U l (x; 0)) + γ φ ∆U l (x; t) ∆W l+1 (t) = O 1 + 1 + n l+1 n l ∆U l (x; t) ,
where we used that

W l+1 (0) = O( √ n l + √ n l+1
), a classical result in random matrix theory [START_REF] Vershynin | Introduction to the non-asymptotic analysis of random matrices[END_REF]. Assuming that the limit is taken layer after layer, we have that n l+1 /n l → 0 and so ∆U l+1 (x; t) = O(1) by induction.

For the rest of this section, we define D l (t) as

D l (t) = 1 2 l l ′ =1 W l ′ (t) -W l ′ (0) 2 F .
Now, we restate and prove Proposition 1.

Proposition 1. Fix a time horizon T > 0 and a depth L. Assume that for t ∈ [0, T ] and for all l ∈ [1 : L]

∂ t W l ij (t) = - nL k=1 ψ L;l k;ij (X; t)V k (Z; t) s -λr(D L (t); t)∆W l ij (t) ,
for any mappings V : Z × R → R nL and r : [0, ∞) 2 → R. Then we have that U L obeys the dynamics

∂ t U L k (x; t) = - nL k ′ =1 Θ L kk ′ (x, X; t)V k ′ (F (X; t), Y ) s -λr(D L (t); t)Ξ L k (x; t) . Moreover, if D L (t) = O(1) for all t ∈ [0, T ], T 0 V (Z; t) s dt = O(1) and T 0 |r(D L (t); t)|dt = O(1) ,
if φ is γ φ -Lipschitz and β φ -smooth, then (in the infinite-width limit taken starting from the layer 1 and then going with growing index), for all l ∈ [1 : L], Θ l is constant during the training, and Ξ l = ∆U l .

Proof. The first statement follows directly from the chain rule. For the second statement, we proceed by induction, taking inspiration in the original NTK proof of [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF]. For L = 1 the model is linear and the statement holds. Now, assume that the statement holds for networks of depth L, we want to show that it is true also for architectures of depth L + 1. We hence consider a network of depth L + 1 following the dynamics

∂ t W l ij (t) = - nL+1 k=1 ψ L+1;l k;ij (X; t)V k (Z; t) s -λr(D L+1 (t); t)∆W l ij (t)
and satisfying

T 0 V (Z; t) s dt = O(1), T 0 r(D L+1 (t); t)dt = O(1), D L+1 (t) = O(1), for all t ∈ [0, T ]. We note that for l ∈ [1 : L] ψ L+1;l k;ij (x; t) = ∂U L+1 k (x; t) ∂W l ij = nL k ′ =1 ∂U L+1 k (x; t) ∂U L k ′ ψ L;l k ′ ;ij (x; t) = 1 √ n L nL k ′ =1 W L+1 kk ′ (t) φ(U L k ′ (x; t))ψ L;l k ′ ;ij (x; t) .
We define

V k ′ (z; t) = 1 √ n L nL+1 k=1 W L+1 kk ′ (t) φ(U L k ′ (x; t))V k (z; t)
and r(D; t) = r( ∆W L+1 (t) 2 F /2 + D; t) , so that we can rewrite the dynamics for l ∈ [1 : L] as

∂ t W l ij (t) = - nL k ′ =1 ψ L;l k ′ ;ij (X; t) V k ′ (Z; t) s -λ r t (D L (t))∆W l ij (t) .
We have that

T 0 V (Z; t) s dt ≤ γ φ √ n L T 0 W L+1 (t) V (Z; t) s dt , which is of order O(1) if W L+1 (t) / √ n L is
. This is indeed the case, as we know that ∆W L+1 (t) ≤ ∆W L+1 (t) F = O(1), and W L+1 (0) = O( √ n L+1 + √ n L ) (this is a classical result on random matrix theory; see for instance [START_REF] Vershynin | Introduction to the non-asymptotic analysis of random matrices[END_REF]. Moreover, for each t we have that r(D L (t); t) = r(D L+1 (t); t), so that in particular We can hence apply the inductive hypothesis to the sub-network made of the first L layers, and obtain that, for l ∈ [1 : L], Θ l stays constant during the training and Ξ l = ∆U l . We also recall that at initialisation the kernels Θ l are diagonal, and so we have that for all t ∈ [0 : T ]

Θ l kk ′ (x, x ′ ; t) = δ kk ′ Θl (x, x ′ ) .
Now, in order to conclude we need to check that the claim holds also for the last layer. We have

Θ L+1 kk ′ (x, x ′ ; t) = L+1 l=1 n l k=1 ψ L+1;l k (x; t) • ψ L+1;l k ′ (x ′ ; t) = ψ L+1;L+1 k (x; t) • ψ L+1;L+1 k ′ (x ′ ; t) + nL j,j ′ =1 ∂U L+1 k (x; t) ∂U L j ∂U L+1 k ′ (x ′ ; t) ∂U L j ′ Θ L jj ′ (x, x ′ ; t) = ψ L+1;L+1 k (x; t) • ψ L+1;L+1 k ′ (x ′ ; t) + nL j=1 ∂U L+1 k (x; t) ∂U L j ∂U L+1 k ′ (x ′ ; t) ∂U L j ΘL (x, x ′ ) .
Let us denote as u k (x; t) the vector with components u k;j (x; t) = ∂U L+1 k (x;t) ∂U L j . We easily see that

u k (x; t) ≤ γ φ √ n L W L+1 k• (t) ≤ γ φ √ n L W L+1 k• (0) + γ φ √ n L ∆W L+1 (t) = O(1) ,
where we used that W L+1 k•

(0) = √ n L and that the norm of the row of a matrix is always bounded by the Frobenius norm of the matrix. On the other hand, we have that

∆u k (x; t) ≤ γ φ √ n L ∆W L+1 (t) + β φ √ n L W L+1 k• (0) ∞ ∆U L (t) ,
where we used that φ is γ φ -Lipschitz and β φ -smooth. We know that ∆W [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF]. On the other hand, ∆U L (t) = O(1) by Lemma 1. We hence easily conclude that

L+1 (t) = O(1) as D L+1 = O(1). Moreover W L+1 k• (0) ∞ behaves as the maximum of n L independent standard normals, that is W L+1 k• (0) ∞ = O( √ log n L )
∆   nL j=1 ∂U L+1 k (x; t) ∂U L j ∂U L+1 k ′ (x ′ ; t) ∂U L j ΘL (x, x ′ )   = O( log(n L )/n L ) .
Now to show that the variation of the NTK vanishes during the training we only need to control ∆(ψ

L+1;L+1 k (x; t) • ψ L+1;L+1 k ′ (x ′ ; t)). First, notice that ψ L+1;L+1 k (x; 0) ≤ ΘL+1 (x, x ′ ) = O(1) . Moreover, we have that ψ L+1;L+1 k (x; t) = δ ik √ nL φ(U L j (x; t)) and so ∆ψ L+1;L+1 k (x; t) ≤ γ φ √ n L ∆U L (x; t) = O(1/ √ n L ) .
We thus deduce that ∆(ψ L+1;L+1 k

(x; t) • ψ L+1;L+1 k ′ (x ′ ; t)) = O(1/ √ n L )
and so ∆Θ L+1 kk ′ (x, x ′ ; t) = O( log(n L )/n L ) , which shows that in the infinite-width limit the NTK stays constant during the training. Now we are left with showing that Ξ L+1 = ∆U L+1 . Recalling the notation u

k;k ′ (x; t) = ∂U L+1 k (x;t) ∂U L k ′ , we can write Ξ L+1 k (x; t) = L+1 l=1 ψ L+1;l k • ∆W l (t) = ψ L+1;L+1 k (x; t) • ∆W L+1 (t) + u k (x; t) • Ξ L (x; t) .

Using the induction hypothesis Ξ

L = ∆U L , we get that Ξ L+1 k (x; t) = ψ L+1;L+1 k (x; t) • ∆W L+1 (t) + u k (x; t) • ∆U L (x; t) . Using that ∂ t U L+1 k = ψ L+1;L+1 k • ∂ t W L+1 + u k • ∂ t U L , we can write Ξ L+1 k (x; t) -∆U L+1 k (x; t) = t 0 ψ L+1;L+1 k (x; t) -ψ L+1;L+1 k (x; t ′ ) • ∂ t W L+1 (t ′ )dt ′ + t 0 (u k (x; t) -u k (x; t ′ )) • ∂ t U L (x; t ′ )dt ′ .
In particular,

|Ξ L+1 k (x; t) -∆U L+1 k (x; t)| ≤ 2 sup t ′ ∈[0,t] ∆ψ L+1;L+1 k (x; t ′ ) t 0 ∂ t W L+1 (t ′ ) F dt ′ + 2 sup t ′ ∈[0,t] ∆u k (x; t ′ ) t 0 ∂ t U L (x; t ′ ) dt ′ .
From what we have shown already, we know that sup

t ′ ∈[0,t] ∆ψ L+1;L+1 k (x; t ′ ) = O(1/ √ n L ) and sup t ′ ∈[0,t] ∆u k (x; t ′ ) = O( log(n L )/n L ) ,
so we are left with checking that the last two integrals are of order O(1) in order to conclude. First, we have

∂ t W L+1 ij (t) = - 1 √ n L φ(U L j (X; t))V i (X; t) s -λr(D L+1 (t); t)∆W L+1 ij (t) .
Since ∆φ(U L (x; t)) = O(1), we have φ(U L (x; t)) = O( √ n L ), and we can define

K ′ = 1 √ n L sup x ′ ∈s φ(U L (x ′ ; t)) = O(1) .
We have thus obtained

∂ t W L+1 (t) ≤ K ′ V (Z; t) s + λ|r(D L+1 (t); t)| ∆W L+1 (t) . So, t 0 ∂ t W L+1 (t ′ ) dt ′ ≤ K ′ t 0 V (Z; t ′ ) s dt ′ + sup t ′ ∈[0,t] ∆W L+1 (t ′ ) λ t 0 |r(D L+1 (t ′ ); t ′ )|dt ′ = O(1) ,
since both integrals in the RHS can be controlled by hypothesis.

Now we just need to control

t 0 ∂ t U L (x; t ′ ) dt ′ . Defining K(x) = sup x ′ ∈s | ΘL (x, x ′ )|, we easily get that ∂ t U L (x; t) ≤ K(x) V (Z; t) s + λ| r(D L (t); t)| ∆U L (x; t) .
We have established that With this last step we have shown that in the infinite width limit Ξ L+1 (x; t) = ∆U L+1 (x; t) , which concludes the proof.

A.2 Variance of the wide stochastic network

From Clerico et al. (2023b) (eq. 5 therein, applied to a one-dimensional output) we know that the output's variance Q 2 of a shallow wide stochastic network is given by

Q 2 (x; t) = 1 n n j=1
(1 + (m 2 j (t)) 2 )ξ(M 1 j (x; t)) - We now consider an initialisation where each component of m 1 and m 2 is sampled independently from N (0, 1). Then, since for any input x we have x = √ n 0 , each component of M 1 j (0) is distributed (with respect to the initialisation's randomness) as a standard normal distribution. Since m 2 is independent of M 1 (0) we can easily derive that, in the limit n → ∞,

Q 2 (x; 0) = 2E ζ∼N (0,1) [ξ(ζ)] -E ζ∼N (0,1) [ψ(ζ) 2 ] = σ 2 ,
which is a deterministic value.

We now show here that Q 2 keeps constant during the training. We have that ∆M 1 (x; t) ≤ 1 √ n 0 ∆m 1 (t) x = ∆m 1 (t) = O(1) . Now, let u j (t) = 1 + m 2 j (t) 2 . We have that ∆   n j=1

(1 + (m 2 j (t)) 2 )ξ(M 1 j (x; t))   = u(0) • ∆ξ(M 1 (x; t)) + ∆u(t) • ξ(M 1 (x; 0)) + ∆u(t) • ∆ξ(M 1 (x; t)) .

Let us start by the first term. We have that ∆ξ(M 1 j (x; t)) = E ζ∼N (0,1) (2φ(M

T 0 |

 0 r(D L (t); t)|dt = T 0 |r(D L+1 (t); t)|dt = O(1) .

  ) 2 ψ(M 1 j (x; t)) 2 ,where we define ξ(u) = E ζ∼N (0,1) [φ(ζ + u) 2 ] and we recall that ψ(u) = E ζ∼N (0,1)[φ(ζ + u)].

  1 j (x; 0) + ζ) + ∆φ(M 1 j (x; t) + ζ))∆φ(M 1 j (x; t) + ζ) ,and so (recalling that we are assuming that φ is C φ Lipschitz)namely the output's variance is constant to the deterministic value σ 2 throughout the training, as n → ∞.

	and
	n
	∆m 2 j (t) 2 ∆ξ(M 1 j (t))
	j=1
	≤ 2C φ E ζ∼N (0,1) φ(ζ + M 1 (x; 0)) 2 1/2 ∆m 2 (t) 2 8 ∆M 1 (x; t) 4 + C 2 φ ∆m 2 (t) 2 8 ∆M 1 (x; t) 2 8 = O( √	n) .
	With analogous reasoning, we can obtain that
						∆	  1 n	j=1 n	(m 2 j (t)) 2 ψ(M 1 j (x; t)) 2	  = O(1/	√ n) ,
	and so conclude that					∆Q 2 (x; t) = O(1/	√ n) ,
	|u(0) • ∆ξ(M 1 (x; t))|
		n
	≤ 2	j=1	E ζ∼N (0,1) u j (0)φ(M 1 j (x; 0) + ζ)∆φ(M 1
									n		1/2
									u j (0)φ(M 1 j (x; 0) + ζ) 2		+ C 2
								j=1
	For large n we have that
			1 n	n j=1	u j (0)φ(M 1
	Proceeding similarly we get that
						n			1/2
	|∆u(t) • ξ(M 1 (x; 0))| ≤ 2		j=1	m 2 j (0) 2 ξ(M 1 (x; 0)) 	∆m 2 (t) + ∆m 2 (t) 2 4 ξ(M 1 (x; 0)
				≤ 2	 	j=1 n	 j (0) 2 ξ(M 1 (x; 0)) m 2 	1/2	∆m 2 (t) + ∆m 2 (t) 2 2 ξ(M 1 (x; 0) = O( √	n) .
	Finally, we have that
								n	n
	∆u(t) • ∆ξ(M 1 (x; t)) = 2	j=1	m 2 j (0)∆m 2 j (t)∆ξ(M 1 j (t)) +	j=1	(∆m 2 j (t)) 2 ∆ξ(M 1 j (t)) .
	We have that
	n
	m 2 j (0)∆m 2 j (t)∆ξ(M 1 j (t))
	j=1
									n	 1/2
			≤ 2C φ E ζ∼N (0,1)		j=1	m 2 (0) 2 φ(ζ + M 1 j (x; 0)) 2		2 8 = O( √	n)

j (x; t) + ζ) + n j=1 E u j (0)∆φ(M 1 j (x; t) + ζ) 2 ≤ 2C φ ∆M 1 (x; t) E ζ∼N (0,1) φ ∆M 1 (x; t) 2 u(0) . j (x; 0) + ζ) 2 → 2E ζ ′ ∼N (0,1) [φ(ζ ′ + ζ) 2 ] = O(1)

(in probability with respect to the random initialisation) and u(0) → √ 2n. Since ∆M 1 (x; t) = O(1), we have that

u(0) • ∆ξ(M 1 (x; t)) = O( √ n) .

∆m 2 (t) 4 ∆M 1 (x; t) 4 + C 2 φ m 2 (0) ∆m 2 (t) 4 ∆M 1 (x; t)

Following the approach of[START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF], we will consider the case where this limit is taken recursively, layer by layer (that is we also have n l+1 /n l → 0).

The limit is in distribution with respect to the intrinsic stochasticity of the ζ's, and in probability with respect to the random initialisation.

Acknowledgements E.C. acknowledges partial support from the U.K. Engineering and Physical Sciences Research Council (EP-SRC) through the grant EP/R513295/1 (DTP scheme), from the Magdalen College Oxford travel grant, and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 950180). B.G. acknowledges partial support from the U.K. Engineering and Physical Sciences Research Council (EP-SRC) under grant number EP/R013616/1, and partial support from the French National Agency for Research, grants ANR-18-CE40-0016-01 and ANR-18-CE23-0015-02.


David Vallenet

Alexandra Calteau

Mathieu Dubois

Mylène Beuvin

Laura Burlot

Xavier Bussell

Stéphanie Fouteau

Guillaume Gautreau

Aurélie Lajus

Jordan Langlois

David Roche

Zoé Rouy

Rémi Planel

Johan Rollin

Claudine Médigue

MicroScope: an integrated platform for the annotation and exploration of microbial gene functions through genomic and metabolic comparative analysis

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Genome Annotation

MicroScope

Visualization System

Pangenomes Genomic Regions

Platform Architecture

• MicroScope is part of National infrastructures:

• France Génomique (www.france-genomique.com)

• IFB (www.france-bioinformatique.fr) • Computational results

• Genomic Object

• Primary databanks

Analyses & Curations

Relational Database (PkGDB)

Users & Interactions

: users from Europe, North America, South America, Africa, Asia, Oceania.

XX -XX : shared genome -annotators

Gene editor

Current annotation

Primary annotation

Method results

Genomic map of the annotated genome

••

  Search within single/multiple tools • Search in single/High throughput automatic annotation of microbial genomes • Expression and evolutionary studies by NGS data integration • Function and biological process predictions • Data exploration tools (keyword search, comparative genomics and metabolism) • Collaborative curation environment for expert annotation • User support & MicroScope Open Days • Sessions: • General training on comparative genomics, metabolism exploration and expert annotation (4.5 days) • Advanced course (2 days) Professional Trainings • Audience: researchers, engineers, technicians, students, industrials • Location: at Evry University (2 sessions per year) and external training session in France and abroad • Since 2008 > 500 persons participated in our trainings • Check our website! • About 11 000 genomes • 3 000 human-expertized annotations a month • 140 collaborative publications • More

Manage & Process Your Data Workflow Engine (JBPM) User Interface (MaGe) with a new genome selector

	Syntactic	Functional	Relational
	annotation	annotation	annotation
	• CDSs, RNAs	• Biological	• Metabolic
	• Repeats	function of genes	pathways
	• Codon usage	• Domains and	• Phylogenetic
	• Genomic islands	motifs	profiles
	• …	• Subcellular localization	• Synteny conservation

• Collaborations with industrial & biotech companies • ANR projects • European infrastructure (IBISBA, Elixir Microb. Biotech.)