Marco Dozzi

Rim Touibi

Pierre-A Vuillermot

Global variational solutions to a class of fractional spde's on unbounded domains

Global variational solutions to a class of fractional spde's on unbounded domains

Marco Dozzi, Rim Touibi, Pierre-A Vuillermot

1 Outline and statement of the main result Many applications ranging from the worlds of engineering and …nance to the natural sciences call for a mathematical modelling in terms of stochastic differential equations. In particular, there have been many works devoted to the analysis of the ultimate behavior of solutions to stochastic partial di¤erential equations of parabolic type which speci…cally occur in population dynamics, population genetics, nerve pulse propagation and related topics, to name only a few (see, e.g., [START_REF] Vuillermot | On the time evolution of Bernstein processes associated with a class of parabolic equations[END_REF] for a brief account of some of those works and the references therein). Moreover, there have also been several more recent articles dealing with the analysis of solutions to various types of semilinear parabolic stochastic partial di¤erential equations driven either by a Brownian noise, or by a fractional noise with Hurst parameter H 2 1 2 ; 1 (see, e.g., [START_REF] Alvarez | Blowup estimates for a family of semilinear SPDE's with time-dependent coe¢ cients[END_REF], [START_REF] Balan | Stochastic heat equation with multiplicative fractional-colored noise[END_REF], [START_REF] Dozzi | Exponential functionals of Brownian motion and explosion times of a system of semilinear SPDEs[END_REF]- [START_REF] Dozzi | Finite-time blowup and existence of global positive solutions of a semi-linear SPDE[END_REF], and the plethora of references therein, particularly [START_REF] Lv | Impacts of noise on a class of partial di¤ erential equations[END_REF]). While these works have been primarily centered around questions of global existence, uniqueness and blowup in …nite time, there have also been investigations essentially motivated by issues in …nancial mathematics devoted to the analysis of problems that involve a mixture of a Brownian noise with a fractional noise, within the realm of both ordinary and partial stochastic di¤erential equations (see, e.g., [START_REF] Guerra | Stochastic di¤ erential equations driven by fractional Brownian motion and standard Brownian motion[END_REF], [START_REF] Mishura | Existence and uniqueness of the solution of stochastic di¤ erential equation involving Wiener process and fractional Brownian motion with Hurst index H > 1 2[END_REF]- [START_REF] Mishura | Existence and uniqueness of mild solution to stochastic heat equation with white and fractional noises[END_REF] and the references therein).

It is our purpose here to contribute further to the analysis of some of the above questions by proving the existence, and in some cases the uniqueness, of global variational solutions to Neumann initial-boundary value problems associated with a class of non-autonomous stochastic parabolic partial di¤erential equations de…ned on certain regions of Euclidean space. In what follows we assume that all the functional spaces are real, use the standard notations for the usual spaces of Lebesgue integrable functions and their norms, and begin by de…ning the Wiener process that will generate the fractional noise we need. Let D R d be an unbounded open domain where d 2 N + , and let us consider a linear, self-adjoint, positive, non-degenerate trace-class operator C in L 2 (D) whose eigenfunctions and eigenvalues we write (e i ) i2N + and ( i ) i2N + , respectively. Let H = (H i ) i2N + be a sequence of Hurst parameters satisfying H i 2 1 2 ; 1 for every i, and for t 2 R + 0 let B Hi (t) i2N + be a sequence of onedimensional, independent fractional Brownian motions de…ned on the complete probability space ( ; F; P) and starting at the origin. Assuming that (e i ) i2N + constitutes an orthonormal basis of L 2 (D), we then de…ne the L 2 (D)-valued fractional Wiener process W H (:; t) by

This series converges strongly in L 2 (D) P-a.s. by virtue of the basic properties of B Hi (t), the boundedness of the sequence H and the fact that C is traceclass. From this we conclude that (1) de…nes a centered Gaussian process whose covariance is entirely determined by C, that is,

E W H (:; s) ; v 2 W H (:; t) ; v 2 = 1 2 +1 X i=1 s 2Hi + t 2Hi js tj 2Hi (Cv; e i ) 2 (v; e i ) 2
for all s; t 2 R + 0 and all v; v 2 L 2 (D), where (:; :) 2 denotes the standard inner product in this space and E the expectation functional on ( ; F; P) (see, e.g., [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF] for a discussion of various basic properties of fractional Brownian motion). For T 2 R + arbitrary, we then consider the class of stochastic initial-boundary value problems formally given by

du(x; t) = (div (k(x; t)r x u(x; t)) + g(u(x; t)))dt + h(u(x; t))W H (x; dt); (x; t) 2 D (0; T ] ; u(x; 0) = '(x); x 2 D;
(2) @u(x; t) @n(k) = 0; (x; t) 2 @D (0; T ]

where @D = D n D stands for the boundary of D and the last line for the conormal derivative of u relative to the matrix-valued function k. Moreover, g and h are real-valued while ' is a random initial condition. Regarding these functions we will need the following hypotheses:

(K) The function k : D (0; T ] 7 ! R d 2 is Lebesgue-mesurable and we have k i;j (:) = k j;i (:) for all i; j 2 f1; :::; dg. Moreover, there exist constants k; k 2 R + such that the inequalities

k jyj 2 (k(x; t)y; y) k jyj 2 (3)
hold uniformly in (x; t) 2 D (0; T ] for all y 2 R d , where (:; :) and j:j denote the standard Euclidean inner product and the related norm in R d , respectively.

(L) The functions g; h : R 7 ! R are Lipschitz continuous. Moreover, the derivative h 0 exists, is Hölder continuous with exponent 2 (0; 1] and bounded. In addition we impose that

H 2 1 + 1 ; 1 (4)
where H := inf i2N + H i .

(I) The initial condition ' is an L 2 (D)-valued random variable.

Finally, whereas the above properties of the operator C are su¢ cient to de-…ne W H , they are not quite strong enough to allow us to prove the result we are looking for. Recalling that C is necessarily an integral transform whose generating kernel we denote by , we still impose the following spectral condition:

(C) We have

x 7 ! Z D dy j (x; y)j 2 2 L 1 (D) (5)
and

+1 X i=1 p i ke i k 1 < +1: (6)
As to the consistency of this hypothesis, we simply remark that we have indeed e i 2 L 1 (D) as a consequence of (5), which follows easily from the eigenvalue equation Ce i = i e i , Schwarz inequality and the fact that ke i k 2 = 1.

We will also need some functional spaces in order to de…ne the notion of variational solution we are interested in. To this end we …x once and for all an 2 1 H; +1 where H satis…es (4), and introduce the Banach space

B ;2 [0; T ] ; L 2 (D) of all Lebesgue measurable mappings u : [0; T ] 7 ! L 2 (D)
endowed with the norm kuk 2 ;2;T := sup

t2[0;T ] ku(t)k 2 ! 2 + Z T 0 dt Z t 0 d ku(t) u( )k 2 (t ) +1 ! 2 < +1. (7)
Furthermore, let H 1 (D (0; T )) be the isotropic Sobolev space consisting of all real-valued functions v 2 L 2 (D (0; T )) that possess distributional derivatives v xi ; v 2 L 2 (D (0; T )), endowed with the inner product

(v 1 ; v 2 ) 1;2;T : = Z D (0;T ) dxd v 1 (x; )v 2 (x; ) + d X i=1 Z D (0;T ) dxd v 1;xi (x; )v 2;xi (x; ) + Z D (0;T ) dxd v 1; (x; )v 2; (x; ) (8)
and the corresponding norm

kvk 1;2;T = (v; v) 1 2
1;2;T .

We note that for any function v 2 H 1 (D (0; T )) which does not depend on time we have v 2 H 1 (D), the usual Sobolev space on D whose norm we denote by k:k 1;2 . Moreover, let H 1 ((0; T )) be the Sobolev space of functions de…ned on the time interval (0; T ). From these de…nitions it follows immediately that if v 2 H 1 (D) and 2 H 1 ((0; T )) then v 2 H 1 (D (0; T )). This allows us to consider the vector space of all …nite linear combinations of such tensor products as an inner product space with respect to (8), and we write H (D (0; T )) for its completion in H 1 (D (0; T )) with respect to [START_REF] Guerra | Stochastic di¤ erential equations driven by fractional Brownian motion and standard Brownian motion[END_REF]. About this Hilbert space we shall prove in Section 2 the existence of the trace embedding

H (D (0; T )) ! L 2 (D f = tg) (10)
valid for each t 2 [0; T ] and every d 2 N + , which is important to formulate the following notion of variational solution:

De…nition 1. We say the H 1 (D)-valued random …eld (u V;' (:; t)) t2[0;T ] de…ned on ( ; F; P) is a global variational solution to (2) if the following two conditions hold:

(1) We have u V;' 2 L 2 0; T ; H 1 (D) \ B ;2 [0; T ] ; L 2 (D) P-a.s., which means that the relations

Z T 0 d ku V;' (:; )k 2 1;2 = Z T 0 d ku V;' (:; )k 2 2 + kru V;' (:; )k 2 2 < +1 (11)
and ku V;' k ;2;T < +1 hold P-a.s. In the sequel we shall sometimes write ku V;' k L 2 (0;T ;H 1 (D)) for norm [START_REF] Lv | Impacts of noise on a class of partial di¤ erential equations[END_REF].

(

+ Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x; d ) (12)
holds P-a.s. for each v 2 H (D (0; T )) and every t 2 [0; T ], where x 7 ! v(x; t) 2 L 2 (D) stands for the trace of v in the sense of [START_REF] Hu | A multiparameter Garsia-Rodemich-Rumsey inequality and some applications[END_REF], and where the stochastic integral with respect to (1) is de…ned as

Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x; d ) : = +1 X i=1 p i Z t 0 (v(:; ); h(u V;' (:; ))e i ) 2 B Hi (d ): (13)
In order to make sense out of each term in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] and prove the existence of such a solution we will still need the following geometric hypothesis on the domain D:

(D) There exists a sequence (D N ) N 2N + of open sets such that for every N we have D N D N +1 D and the compactness of the embedding

H 1 (D N ) ,! L 2 (D N ) : (14)
Moreover, let B H 1 (D) be any bounded set. Then for each > 0 there exists

N 2 N + such that Z DnD N dx ju(x)j 2 < ( 15
)
for every u 2 B.

Remark. It is essential to observe here that any domain satisfying (D) is necessarily of …nite Lebesgue measure as a consequence of the theory of tesselations developed in Chapter 6 of [START_REF] Adams | Sobolev Spaces[END_REF], and more importantly that (D) implies the compactness of the embedding H 1 (D) ,! L 2 (D) as it amounts to a geometric condition implying that D becomes rapidly narrow at in…nity, as the two examples discussed at the end of this section will show. It is also easily veri…ed that each term in ( 12) is well de…ned and …nite P-a.s. as a consequence of all of the above hypotheses. In particular, we may conclude that ( 13) is an in…nite sum of one-dimensional, pathwise, generalized Stieltjes integrals which de…nes a real-valued random variable as a consequence of Hypothesis (C) and of the fact that h is Lipschitz continuous. We shall dwell a bit more on this and on related properties of the stochastic integral in the Appendix. We note that Problem (2) was thoroughly analyzed in [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF] in case D is a bounded domain satisfying the cone condition and with a single Hurst parameter in [START_REF] Adams | Sobolev Spaces[END_REF].

Under these conditions our main result is the following:

Theorem. Assume that Hypotheses (K), (L), (I), (C) and (D) hold. Then Problem (2) possesses a global variational solution u V;' . Moreover, if h is an a¢ ne function then u V;' is the unique solution to [START_REF] Adams | Some imbedding theorems for Sobolev spaces[END_REF].

In order to prove this result we shall organize the remaining part of this article in the following way: in Section 2 we …rst state the existence of u V;' when the test functions in ( 12) are independent of time, that is, with v 2 H 1 (D).

We then extend the statement to the case of an approximating family of test functions in R d+1 consisting of …nite linear combinations of the tensor products we alluded to above, and eventually to all test functions v 2 H (D (0; T )) by a suitable density argument and by invoking the trace embedding of the form [START_REF] Hu | A multiparameter Garsia-Rodemich-Rumsey inequality and some applications[END_REF] which we prove by elementary means. In Section 2 we also illustrate our results with two examples, while we prove some crucial estimates for the stochastic integral in the Appendix by means of a necessary modi…cation of the theory set forth in [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF] due to the fact that we are dealing there with an in…nite sequence of Hurst parameters satisfying (4). The method rests in an essential way on a particular case of an inequality proved by Garsia, Rodemich and Rumsey in [START_REF] Garsia | A real variable lemma and the continuity of paths of some Gaussian processes[END_REF], and on Minkowski's integral inequality (see, e.g., Appendix A in [START_REF] Stein | Singular Integrals and Di¤ erentiability Properties of Functions[END_REF]).

Proof of the main result and two examples

We begin with the following result: Proposition 1. Assume that Hypotheses (K), (L), (I), (C) and (D) hold. Then there exists u

V;' 2 L 2 0; T ; H 1 (D) \ B ;2 [0; T ] ; L 2 (D) such that the integral relation (12) holds for every v 2 H 1 (D), that is, (v; u V;' (:; t)) 2 = (v; ') 2 d X i;j=1 Z t 0 d v xi ; k i;j (:; )u V;';xj (:; ) 2 + Z t 0 d (v; g(u V;' (:; ))) 2 + Z t 0 Z D dxv(x)h(u V;' (x; ))W H (x; d ) (16)
P-a.s. for every t 2 [0; T ].

Proof. Since the theory developed in Chapter 6 of [START_REF] Adams | Sobolev Spaces[END_REF] implies the compactness of the embedding H 1 (D) ,! L 2 (D), the result follows from a direct adaptation of all the arguments put forward in the …rst part of [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF].

The next intermediary result is: Lemma 1. Assume that Hypotheses (K), (L), (I), (C) and (D) hold and let u V;' be the random …eld of Proposition

+ Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x; d ) (17)
P-a.s. for every t 2 [0; T ].

Proof. By linearity it is su¢ cient to prove that (17) holds for v = v . The easiest way out is to start with the …rst integral on the right-hand side. Using [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF] we obtain

Z t 0 d (v (:; ); u V;' (:; )) 2 = Z t 0 d 0 ( ) (v; u V;' (:; )) 2 = (v(:; t); ') 2 (v(:; 0); ') 2 d X i;j=1 Z t 0 d 0 ( ) Z 0 d v xi ; k i;j (:; )u V;';xj (:; ) 2 + Z t 0 d 0 ( ) Z 0 d (v; g(u V;' (:; ))) 2 + Z t 0 d 0 ( ) Z 0 Z D dxv(x)h(u V;' (x; ))W H (x; d ): (18)
We then integrate by parts the last three terms with respect to and reintroduce v whenever possible to obtain respectively

d X i;j=1 Z t 0 d 0 ( ) Z 0 d v xi ; k i;
and

Z t 0 d 0 ( ) Z 0 d (v; g(u V;' (:; ))) 2 = (t) Z t 0 d (v; g(u V;' (:; ))) 2 Z t 0 d (v(:; ); g(u V;' (:; ))) 2 (20)
for the deterministic integrals, while we get

Z t 0 d 0 ( ) Z 0 Z D dxv(x)h(u V;' (x; ))W H (x; d ) = (t) Z t 0 Z D dxv(x)h(u V;' (x; ))W H (x; d ) Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x; d ) (21)
for the stochastic integral. The substitution of ( 19)-( 21) into (18) then leads to the desired result, after having lumped together the three terms containing the factor (t) and used there (16) once again.

We infer from the preceding considerations that for any v 2 H (D (0; T )) there exist functions vn satisfying [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF] for every n 2 N + such that kv vn k 1;2;T ! 0

as n ! +1, where k:k 1;2;T is given by [START_REF] Guerra | Stochastic di¤ erential equations driven by fractional Brownian motion and standard Brownian motion[END_REF]. The …rst consequence of this is that we can already approximate the three deterministic integrals in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF], deferring to separate propositions the analysis of the remaining terms:

ku V;' k L 2 (0;T ;H 1 (D)) Z D (0;T ) dxd jv (x; ) vn; (x; )j 2 ! 1 2 ku V;' k L 2 (0;T ;H 1 (D)) kv vn k 1;2;T ! 0
as n ! +1 P-a.s., by using successively Schwarz inequalities in L 2 (D) and on the time interval (0; T ) along with [START_REF] Lv | Impacts of noise on a class of partial di¤ erential equations[END_REF] and ( 22). The proof of (24) follows from similar arguments. Thus, noting that the matrix elements k i;j are uniformly bounded for all i; j 2 f1; :::; dg as a consequence of (3), we eventually get

c 1 + c 2 ku V;' k L 2 (0;T ;H 1 (D)) lim n!+1
kv vn k 1;2;T = 0 P.-a.s. with c 1;2 > 0, which follows from the fact that g is Lipschitz continuous and jDj < +1 where jDj stands for the Lebesgue measure of D.

In the sequel we will also need approximation properties for the remaining terms in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] that are similar to those of the above proposition. However, the proof of those properties and eventually of the main theorem will require two more preparatory results. The …rst one is: Lemma 2. Let H (D (0; T )) be the Hilbert space de…ned in Section 1. Then there exists the continuous trace embedding

H (D (0; T )) ! L 2 (D f = tg) (26)
valid for each t 2 [0; T ] and every d 2 N + . Moreover, there also exists the continuous embedding

H (D (0; T )) ! B ;2 [0; T ] ; L 2 (D) (27)
where B ;2 [0; T ] ; L 2 (D) is endowed with norm [START_REF] Dozzi | Finite-time blowup and existence of global positive solutions of a semi-linear SPDE[END_REF].

Proof. It is su¢ cient to prove the result for H 1 (D (0; T )). Let v 2 H 1 (D (0; T )) and let us write momentarily k:k 1;2;H 1 ((0;T )) for the norm in H 1 ((0; T )). From ( 9) and Fubini's theorem we infer that x 7 ! v(x; t) 2 L 2 (D) for almost every t, and more importantly that t 7 ! v(x; t) 2 H 1 ((0; T )) for almost every x 2 D. Therefore, writing C ([0; T ]) for the space of all continuous functions de…ned on [0; T ] endowed with the uniform norm k:k 1;T we have t 7 ! v(x; t) 2 C ([0; T ]) by virtue of the embedding H 1 ((0; T )) ,! C ([0; T ]). Consequently we obtain jv(x; t)j kv(x; :)k 1;T c kv(x; :)k 1;2;H 1 ((0;T )) for each t 2 [0; T ] and almost every x 2 D for some c > 0, and thereby the desired result

Z D dx jv(x; t)j 2 c 2 Z D dx kv(x; :)k 2 1;2;H 1 ((0;T ))
c 2 kvk 2 1;2;T according to [START_REF] Guerra | Stochastic di¤ erential equations driven by fractional Brownian motion and standard Brownian motion[END_REF]. As for the proof of the second embedding, we note that for every v 2 H 1 (D (0; T )) and t we have

kv(:; t) v(:; )k 2 (t ) 1 2 Z D (0;T ) dxd jv (x; )j 2 ! 1 2 (t ) 1
2 kvk 1;2;T by ( 9), and therefore kvk 2 ;2;T = sup

t2[0;T ] kv(:; t)k 2 ! 2 + Z T 0 dt Z t 0 d kv(:; t) v(:; )k 2 (t ) +1 ! 2 c kvk 2 1;2;T + Z T 0 dt Z t 0 d (t ) 1 2 2 kvk 2 1;2;T c kvk 2
1;2;T by virtue of (26) and the fact that < 1 2 , changing the value of c whenever necessary.

The desired approximation property for the stochastic integral in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] will now rest on (27) and on the following estimate, which also shows that ( 13) is Hölder continuous with respect to the time variable: Lemma 3. Let us consider the stochastic integral as de…ned in [START_REF] Mishura | Existence and uniqueness of the solution of stochastic di¤ erential equation involving Wiener process and fractional Brownian motion with Hurst index H > 1 2[END_REF]. Then there exists a P-a.s. …nite, positive random variable r H such that the estimate

Z t t Z D dxv(x; )h(u V;' (x; ))W H (x; d )
r H 1 + ku V;' k ;2;T kvk ;2;T jt tj 1 2

(28)

holds P-a.s. for every v 2 B ;2 [0; T ] ; L 2 (D) and all t; t 2 [0; T ].

We defer the proof of this lemma to the Appendix, as it requires technical tools regarding generalized Stieltjes integrals that do not directly pertain to the main core of this article.

The expected approximation property of the stochastic integral is then the following: Proposition 3. Let v 2 H (D (0; T )) and (v n ) n2N + be as in Proposition 2. Then we have

lim n!+1 Z t 0 Z D dxv n (x; )h(u V;' (x; ))W H (x; d ) = Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x;
+ Z t 0 d (v n (:; ); g(u V;' (:; ))) 2 + Z t 0 Z D dxv n (x; )h(u V;' (x; ))W H (x; d ) (29)
P-a.s. for each t 2 [0; T ] and every n, and we already know that each integral on the right-hand side of this expression converges to the desired integral in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] according to Propositions 2 and 3. Let us now investigate the term on the left-hand side. From Schwarz inequality in L 2 (D) and ( 7) we have (v(:; t) vn (:; t); u V;' (:; t)) 2 ku V;' k ;2;T sup t2[0;T ] kv(:; t) vn (:; t)k 2 ku V;' k ;2;T kv vn k ;2;T c ku V;' k ;2;T kv vn k 1;2;T ! 0 P-a.s. as n ! +1 for some c > 0 by virtue of ( 22) and ( 27). Therefore we get lim n!+1 (v n (:; t); u V;' (:; t)) 2 = (v(:; t); u V;' (:; t)) 2 P-a.s. uniformly in t 2 [0; T ], where v(:; t) stands for the trace of v in the sense of (26). Passing to the limit in (29) we then obtain for the remaining term

lim n!+1 (v n (:; 0); ') 2 = (v(:; t); u V;' (:; t)) 2 Z t 0 d (v (:; ); u V;' (:; )) 2 + d X i;j=1 Z t 0 d v xi (:; ); k i;j (:; )u V;';xj (:; ) 2 Z t 0 d (v(:; ); g(u V;' (:; ))) 2 Z t 0 Z D dxv(x; )h(u V;' (x; ))W H (x; d ) (30)
P-a.s. for every t 2 [0; T ], hence

lim n!+1
(v n (:; 0); ') 2 = (v(:; 0); u V;' (:; 0)) 2 by choosing t = 0. But from ( 16) at t = 0 we have (v; u V;' (:; 0) ') 2 = 0 for every v 2 H 1 (D) and thereby for every smooth and compactly supported v 2 C 1 c (D), the latter space being dense in L 2 (D). Therefore u V;' (:; 0) ' is orthogonal to L 2 (D), hence u V;' (:; 0) = ' so that lim n!+1 (v n (:; 0); ') 2 = (v(:; 0); ') 2 :

The substitution of the preceding expression into (30) then proves [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] for every v 2 H (D (0; T )). Finally, the proof that u V;' is the unique variational solution to (2) satisfying ( 16) when h is an a¢ ne function is identical to that carried out in [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF]. Therefore, let ũV;' be another variational solution to (2) satisfying [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF]. We then have ũV;' = u V;' P-a.s. as a solution to [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF], hence also as a solution to [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF].

Remarks. (1) The preceding considerations show that there are actually two distinct types of equivalent variational solutions to [START_REF] Adams | Some imbedding theorems for Sobolev spaces[END_REF], to wit, one that sat-is…es ( 16) which involves test functions independent of time, and one satisfying [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF]. As was done in [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF] where the class of problems given by ( 2) was de…ned and analysed on a bounded domain D satisfying the cone condition, we may refer to them as variational solutions of type I and II, respectively. Moreover, for the kind of equations considered in this article and to the best of our knowledge, the problem of uniqueness of the solution in case of a general nonlinearity h in the noise term remains open.

(2) If D is a bounded domain that satis…es the cone condition, the natural space of test functions for variational solutions of type II is H 1 (D (0; T )), as was shown in [START_REF] Nualart | Variational solutions for partial di¤ erential equations driven by a fractional noise[END_REF] by …rst using [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF] to prove [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF] for polynomial test functions, then for smooth test functions by invoking the C 1 -version of Weierstrass'approximation theorem, and eventually for functions in H 1 (D (0; T )) by a density argument. It is not possible to argue in that way when D is unbounded and merely satis…es Hypothesis (D). Thus, it is still true that the space C 1 D [0; T ] of all uniformly continuous and uniformly bounded functions on D (0; T ) is dense in H 1 (D (0; T )) if we assume in addition that D and therefore D (0; T ) satisfy the so-called segment condition (see, e.g. [START_REF] Adams | Sobolev Spaces[END_REF]). However, we can no longer approximate functions in C 1 D [0; T ] by polynomials in a suitable topology to make such approximations worthwile in the context of this article (an approximation of the functions in C 1 D [0; T ] by polynomials is still possible if one endows C 1 (D (0; T )) with the structure of a Fréchet space, see, e.g., Section 3 of Chapter 3 in [START_REF] Kirillov | Theorems and Problems in Functional Analysis[END_REF], but this did not turn out to be strong enough in our case). We therefore bypassed the di¢ culty by constructing the space H (D (0; T )) in Section 1, but the determination of the relative size of this space within H 1 (D (0; T )) remains an open problem at the present time.

We complete this section by illustrating the above results by means of two examples. We begin with: Proposition 4. Let us consider the two-dimensional domain

D = (x 1 ; x 2 ) 2 R 2 : x 1 > 0; 0 < x 2 < b(x 1 ) (31)
where the positive boundary curve b 2 C 1 R + 0 is decreasing, has a bounded derivative and satis…es

lim s!+1 b (s + ) b (s) = 0 (32)
for every > 0. Then the conditions of Hypothesis (D) in Section 1 hold.

Proof. For every N 2 N + let us de…ne the bounded open set

D N := (x 1 ; x 2 ) 2 R 2 : 0 < x 1 < N \ D.
Then we have D N D N +1 D and D N satis…es the cone condition, so that compact embedding [START_REF] Mishura | Mixed stochastic di¤ erential equations with long-range dependence: existence, uniqueness and convergence of solutions[END_REF] holds by the Rellich-Kondrachov theorem. Furthermore, the properties of the ‡ow associated with D in the sense of Example 6.46 and Theorem 6.47 in [START_REF] Adams | Sobolev Spaces[END_REF] by means of the function b satisfying (32) imply that for every u 2 H 1 (D), the inequality Z

DnD N dx ju(x)j 2 c N kuk 2 1;2
is valid with lim N !+1 c N = 0. Consequently, for each > 0 there exists N 2 N + such that (15) holds for each u belonging to a bounded set of H 1 (D).

We conclude that for a domain of the form (31) the embedding H 1 (D) ,! L 2 (D) is compact, so that our main theorem holds true in this case. Typical examples of boundary functions satisfying the above conditions include b(x) = exp x 1+ where > 0, whereas the mere exponential b(x) = exp [ x] does not belong to that class of curves. In fact, since b is decreasing it follows easily from (32) that

lim s!+1 exp [s] b(s) = 0:
Generally speaking, it is in fact the …niteness of the Lebesgue measure of D, along with the rapid decrease of the measure of the part of D situated outside the disk of radius R centered at the origin as R ! +1, which makes all this possible.

Our second example refers to the horn-shaped region discussed in [START_REF] Adams | Some imbedding theorems for Sobolev spaces[END_REF], for which our main theorem is also valid: Proposition 5. Let us consider the three-dimensional domain

D = (x 1 ; x 2 ; x 3 ) 2 R 3 : x 3 > 0; 0 < x 2 1 + x 2 2 < b 2 (x 3 )
where the boundary curve b satis…es the same hypotheses as in Proposition 4 .

Then the conditions of Hypothesis (D) in Section 1 hold.

Proof. The argument is similar to that given in the proof of Proposition 4 if we de…ne D N := (x 1 ; x 2 ; x 3 ) 2 R 3 : 0 < x 3 < N \ D for every N 2 N + . Thus the compactness of embedding [START_REF] Mishura | Mixed stochastic di¤ erential equations with long-range dependence: existence, uniqueness and convergence of solutions[END_REF] along with [START_REF] Mishura | Existence and uniqueness of mild solution to stochastic heat equation with white and fractional noises[END_REF] hold.

Appendix: some remarks on generalized Stieltjes integrals and proof of Lemma 3

The following considerations constitute a necessary modi…cation of the theory developed in [START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF], needed to take into account the fact that we are dealing here with an in…nite sequence of Hurst parameters. For every i 2 N + let us introduce the random variable Hi = sin ( ) sup

0 t<t T B Hi (t) B Hi (t ) (t t) 1 + (1 )
Z t t d B Hi (t) B Hi ( ) ( t) 2
(33) where is the …xed parameter chosen in Section 1, for which we have the inequality

Hi sup 0 t<t T B Hi (t ) B Hi (t) (t t) 1 + (1 )
Z t t d B Hi ( ) B Hi (t) ( t) 2 ! :
(34) For reasons that will soon be apparent, we …rst need to get appropriate estimates for the moments of (33) that are uniform in i 2 N + , and to achieve this suitable upper bounds for the increments of B Hi (t). Indeed the main result that will lead to the proof of Lemma 3 is: Proposition A. 1. Let B Hi (t) i2N + be the one-dimensional, independent fractional Brownian motions introduced in Section 1, where the sequence H = (H i ) i2N + satis…es (4). Then for each i 2 N + we have

sup i2N + E Hi p < +1 (35)
for every p 2 [1; +1).

The proof of this proposition rests on the following particular case of the Garsia-Rodemich-Rumsey inequality, obtained by rescaling the basic estimate stated in Lemma 1.1 of [START_REF] Garsia | A real variable lemma and the continuity of paths of some Gaussian processes[END_REF] to establish its validity on [0; T ] rather than just on [0; 1], and by applying the particular choice of the functions involved made at the very beginning of [START_REF] Hu | A multiparameter Garsia-Rodemich-Rumsey inequality and some applications[END_REF]:

Lemma A. 1. Let f : [0; T ] 7 ! R be continuous, let q 2 [1; +1) and 2 1 q ; +1 . If the integral between parentheses in (36) is …nite, then the inequality

jf (t ) f (t)j c ;q;T Z T 0 Z T 0 d d jf ( ) f ( )j q j j q+1 ! 1 q jt tj 1 q (36)
holds for all t; t 2 [0; T ], where c ;q;T = c T + 1 q

1 q 1 ( 37
)
and where c T > 0 depends only on T .

The upper bounds for the increments of B Hi (t) we need turn out to be provided by (36) for particular values of the parameters. In all that follows we write for a …xed auxiliary quantity which we can eventually express in terms of the constants , and H ¯:

Lemma A.

for every p 2 [1; +1).

Proof. Let us …rst de…ne the random variable

~ i; ;T := Z T 0 Z T 0 d d B Hi ( ) B Hi ( ) 2 j j 2H i
and prove that sup

i2N + E ~ i; ;T p < +1 (40)
for every p 2 [1; +1). For any r 2 2 ; +1 we have p := r 2 2 [1; +1) so that on the one hand we obtain

E ~ i; ;T p = E 0 @ Z T 0 Z T 0 d d B Hi ( ) B Hi ( ) 2 j j 2H i 1 A r 2 0 @ Z T 0 Z T 0 d d E B Hi ( ) B Hi ( ) r j j rHi ! 2 r 1 A r 2 (41)
from Minkowski's integral inequality. On the other hand, from the basic properties of the one-dimensional fractional Brownian motion we have

E B Hi ( ) B Hi ( ) r c r j j rHi
for some constant c r depending only on r, so that the substitution of this expression into (41) annihilates the dependence in H i , thus leading to

E ~ i; ;T p c r T 2p < +1
uniformly in i, which is (40). Let us now choose q = 2 and i = H i 2 for every i 2 N + in Lemma A. 1. Then clearly q 2 [1; +1) and i 2 2 ; +1 by virtue of (4), and since (40) implies in particular that ~ i; ;T < +1 P-a.s. we may thus apply (36) to obtain B Hi (t ) B Hi (t)

c i ;q;T 0 @ Z T 0 Z T 0 d d B Hi ( ) B Hi ( ) 2 j j 2H i 1 A 2 jt tj Hi c i ;q;T 0 @ Z T 0 Z T 0 d d B Hi ( ) B Hi ( ) 2 j j 2H i 1 A 2 jt tj H = c i ;q;T ~ 2 i; ;T jt tj H P-a.
s. for all t; t 2 [0; T ], where c i ;q;T di¤ers from c i ;q;T by a trivial factor depending only on T and H. In order to get (38) it is thus su¢ cient to take

and prove that sup

i2N + c i ;q;T < +1 (43)
along with (39). Ignoring the trivial dependence in T and H in c i ;q;T we …rst infer from (37) that the simple estimate

i + 2 i 2 1 = H i (H i ) 1 1 + 1 1 < +1
holds uniformly in i thanks to (4) and our choice of , which gives (43). Finally, let us partition the probability space as

= n ! 2 : ~ i; ;T (!) 1 o [ n ! 2 : ~ i; ;T (!) > 1 o
and split the expectation functional accordingly. From (42) it is then plain that

sup i2N + E j i; ;T j p = sup i2N + E ~ i; ;T p 2 1 + sup i2N + E ~ i; ;T p < +1
according to (40) since 2 1, which is the desired result.

The proof of (35) then follows from (39) and from the substitution of (38) into (34) provided we impose a further restriction on the parameter to make the singularities integrable:

Proof of Proposition A. 1. We …rst notice that

H 1 + < 1 + 1
as a consequence of the conditions we imposed on these parameters in Section 1. Then, …xing 2 (0; H 1 + ) we may substitute (38) into (34) to obtain Remark. To follow up on our remark preceding the statement of Lemma A. 2., we see a posteriori that we could have chosen for instance = 1 2 (H 1 + ) throughout.

We are now ready for the following:

Proof of Lemma 3. We begin by estimating each integral on the righthand side of [START_REF] Mishura | Existence and uniqueness of the solution of stochastic di¤ erential equation involving Wiener process and fractional Brownian motion with Hurst index H > 1 2[END_REF]. For every i 2 N + let us set f i ( ) := (v(:; ); h(u V;' (:; ))e i ) 2 :

Then, from the basic estimate (4.11)

W

  H (:; t) := +1 X i=1 p i e i (:) B Hi (t):

ZD 2 ! 1 2cd

 21 (:; ) vn;xi (:; ); k i;j (:; )u V;';xj (:; )2 c ku V;' k L 2 (0;T ;H 1 (D)) d X j=1 (0;T )dxd v xj (x; ) vn;xj (x; ) ku V;' k L 2 (0;T ;H 1 (D)) kv vn k 1;2;T ! 0 as n ! +1 P-a.s., where c > 0 is a constant depending only on k, k and d. In a similar way for (25) we have lim (v(:; ) vn (:; ); g(u V;' (:; ))) 2

  .s. after an explicit integration, where the prefactor is uniform in i 2 N + . Therefore, (35) indeed follows from (39).

  [START_REF] Adams | Sobolev Spaces[END_REF]. Then for all …nite linear combinations v of functions of the form v where v 2 H 1 (D) and 2 H 1 ((0; T )) we have

		Z t
	(v(:; t); u V;' (:; t)) 2 = (v(:; 0); ') 2 + d X Z t	0	d (v (:; ); u V;' (:; )) 2
	i;j=1

0 d vxi (:; ); k i;j (:; )u V;';xj (:; ) 2 + Z t 0 d (v(:; ); g(u V;' (:; ))) 2

  Proposition 2. Let v 2 H (D (0; T )) and let (v n ) n2N + be as in (22).

				lim n!+1	d X i;j=1	Z t 0	d vn;xi (:; ); k i;j (:; )u V;';xj (:; ) 2
		=	d X i;j=1	Z t 0	d v xi (:; ); k i;j (:; )u V;';xj (:; ) 2	(24)
	and
	Z t					Z t
	lim n!+1	0	d (v n (:; ); g(u V;' (:; ))) 2 =	0	d (v(:; ); g(u V;' (:; ))) 2	(25)
	P-a.s. for every t 2 [0; T ].
	Proof. Regarding (23) we have
		Z t
	Then we have
		Z t				Z t
	lim n!+1	0		d (v n; (:; ); u V;' (:; )) 2 =	0	d (v (:; ); u V;' (:; )) 2 ;	(23)

0 d (v (:; ) vn; (:; ); u V;' (:; )) 2

  Proof of the main result. Let v 2 H (D (0; T )) and let (v n ) n2N + be as in(22). According to Lemma 1 we have

		Z t
	(v n (:; t); u V;' (:; t)) 2 = (v n (:; 0); ') 2 + d Z t X i;j=1 0 d vn;xi (:; ); k i;j (:; )u V;';xj (:; ) 2 d (v n; (:; ); u V;' (:; )) 2 0
	Proof. From Relation (28) and embedding (27) we have
	Z t	Z
	0
	The preceding considerations now lead to the following:

d ) P-a.s. for every t 2 [0; T ].

D

dx (v(x; ) vn (x; )) h(u V;' (x; ))W H (x; d )

cr H 1 + ku V;' k ;2;T kv vn k 1;2;T P-a.s. for some c > 0, hence the desired result from (22).

  in Proposition 4.1 of[START_REF] Nualart | Di¤ erential equations driven by fractional Brownian motion[END_REF] regarding generalized Stieltjes integrals we infer that the inequality P-a.s., where Hi is given by (33). Furthermore, from Schwarz inequality and the basic hypotheses of Section 1 we havejf i ( )j c ke i k 1 1 + ku V;' (:; )k 2 kv (:; )k 2 c ke i k 1 1 + ku V;' k ;2;T kvk ;2;T(45)where the second inequality follows from[START_REF] Dozzi | Finite-time blowup and existence of global positive solutions of a semi-linear SPDE[END_REF], and similarlyjf i ( ) f i ( )j c ke i k 1 ku V;' (:; ) u V;' (:; )k 2 kvk ;2;T +c ke i k 1 1 + ku V;' k ;2;T kv (:; ) v (:; )k 2(46)for all ; 2 [0; T ], for some constant c > 0. Consequently, on the one hand the substitution of (45) into the …rst integral on the right-hand side of (44) gives On the other hand, the substitution of (46) into the second integral on the right-hand side of (44), Schwarz inequality relative to the measure d and (7) lead to Z t N + since 1 > 1 2 according to our original choice of this parameter. In order to prove (28) it is therefore su¢ cient to show that according to[START_REF] Mishura | Existence and uniqueness of the solution of stochastic di¤ erential equation involving Wiener process and fractional Brownian motion with Hurst index H > 1 2[END_REF] and (49), with the obvious choice for r H . But (50) follows from Relation (35) of Proposition A.1. with p = 1 since spectral condition (6) holds.

	Z t t	d	(	jf i ( )j t )		c ke 1 2 :
										(48)
	Therefore, with (47) and (48) into (44) we obtain
	Z t								1 2	(49)
	for every i 2 +1 X i=1	p	i ke i k 1	Hi < +1	(50)
	P-a.s., for then
					Z t	Z
	Z t t	f i ( )B Hi (d ) t c +1 X i=1	p	Hi i ke i k 1 Z t t	d	Hi	( !	jf i ( )j t ) 1 + ku V;' k ;2;T kvk ;2;T jt + t d ( ) jf i ( ) f i ( )j +1 Z tj ! 1 2	(44)

holds i k 1 1 + ku V;' k ;2;T kvk ;2;T jt tj

1 (47) by direct integration. t d Z t d jf i ( ) f i ( )j ( ) +1

c ke i k 1 1 + ku V;' k ;2;T kvk ;2;T jt tj

t f i ( )B Hi (d ) c ke i k 1

Hi 1 + ku V;' k ;2;T kvk ;2;T jt tj D dxv(x; )h(u V;' (x; ))W H (x; d )

Acknowledgements The research of M. D. was supported in part by the FCT of the Portuguese government under grant FCT UID/MAT/00208/2013. He would like to thank Professors A. B. Cruzeiro, P.-A. Vuillermot and J. C. Zambrini for their very kind invitation to the GFMUL, where this work was completed. R. T. would like to extend her thanks and express her appreciation and gratitude to the entire sta¤ of the IECL for their warm hospitality and strong support during the completion of her Ph.D. dissertation there. The research of P.-A. V. was supported in part by the FCT of the Portuguese government under grant PDTC/MAT-STA/0975/2014.


Arthur Jacot
email: arthur.jacot@netopera.net

Franck Gabriel
email: franckrgabriel@gmail.com

Clément Hongler
email: clement.hongler@epfl.ch

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function f θ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinitewidth limit, the network function f θ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

Introduction

Artificial neural networks (ANNs) have achieved impressive results in numerous areas of machine learning. While it has long been known that ANNs can approximate any function with sufficiently many hidden neurons [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF][START_REF] Leshno | Multilayer feedforward networks with a non-polynomial activation function can approximate any function[END_REF], it is not known what the optimization of ANNs converges to. Indeed the loss surface of neural networks optimization problems is highly non-convex: the high number of saddle points slows down the convergence [START_REF] Dauphin | Identifying and attacking the saddle point problem in high-dimensional non-convex optimization[END_REF]. A number of results [START_REF] Choromanska | The Loss Surfaces of Multilayer Networks[END_REF][START_REF] Pascanu | On the saddle point problem for nonconvex optimization[END_REF][START_REF] Pennington | Geometry of neural network loss surfaces via random matrix theory[END_REF] suggest that for wide enough networks, there are very few "bad" local minima, i.e. local minima with much higher cost than the global minimum.

Another mysterious feature of ANNs is their good generalization properties in spite of their usual over-parametrization [START_REF] Sagun | Empirical analysis of the hessian of over-parametrized neural networks[END_REF]. It seems paradoxical that a reasonably large neural network can fit random Preprint. Work in progress. arXiv:1806.07572v1 [cs.LG] 20 Jun 2018 labels, while still obtaining good test accuracy when trained on real data [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF]. It can be noted that in this case, kernel methods have the same properties [START_REF] Belkin | To understand deep learning we need to understand kernel learning[END_REF].

In the infinite-width limit, ANNs have a Gaussian distribution described by a kernel [START_REF] Neal | Bayesian Learning for Neural Networks[END_REF][START_REF] Lee | Deep neural networks as gaussian processes[END_REF]. These kernels are used in Bayesian inference or Support Vector Machines, yielding results comparable to ANNs trained with gradient descent [START_REF] Lee | Deep neural networks as gaussian processes[END_REF][START_REF] Cho | Kernel methods for deep learning[END_REF]. We will see that in the same limit, the behavior of ANNs during training is described by a related kernel, which we call the neural tangent network (NTK).

Contribution

We study the network function f θ of an ANN, which maps an input vector to an output vector, where θ is the vector of the parameters of the ANN. In the limit as the widths of the inner layers tend to infinity, the network function at initialization, f θ converges to a Gaussian distribution [START_REF] Neal | Bayesian Learning for Neural Networks[END_REF][START_REF] Lee | Deep neural networks as gaussian processes[END_REF].

In this paper, we investigate fully connected networks in this infinite-width limit, and describe the dynamics of the network function f θ during training:

• During gradient descent, we show that the dynamics of f θ follows that of the so-called kernel gradient descent in function space with respect to a limiting kernel, which only depends on the depth of the network, the choice of nonlinearity and the initialization variance.

• The convergence properties of ANNs during training can then be related to the positivedefiniteness of the infinite-width limit NTK. In the case when the dataset is supported on a sphere, we prove this positive-definiteness using recent results on dual activation functions [START_REF] Daniely | Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity[END_REF].

• For a least-squares regression loss, the network function f θ follows a linear differential equation in the infinite-width limit, and the eigenfunctions of the Jacobian are the kernel principal components of the input data. This shows a direct connection to kernel methods and motivates the use of early stopping to reduce overfitting in the training of ANNs.

• Finally we investigate these theoretical results numerically for an artificial dataset (of points on the unit circle) and for the MNIST dataset. In particular we observe that the behavior of wide ANNs is close to the theoretical limit.

Neural networks

In this article, we consider fully-connected ANNs with layers numbered from 0 (input) to L (output), each containing n 0 , . . . , n L neurons, and with a Lipschitz, twice differentiable nonlinearity function σ : R → R, with bounded second derivative 1 .

This paper focuses on the ANN realization function F (L) : R P → F, mapping parameters θ to functions f θ in a space F. The dimension of the parameter space is P = L-1 =0 (n + 1)n +1 : the parameters consist of the connection matrices W ( ) ∈ R n ×n +1 and bias vectors b ( ) ∈ R n +1 for = 0, ..., L -1. In our setup, the parameters are initialized as iid Gaussians N (0, 1).

For a fixed distribution p in on the input space R n0 , the function space F is defined as {f : R n0 → R n L }. On this space, we consider the seminorm || • || p in , defined in terms of the bilinear form f, g p in = E x∼p in f (x) T g(x) .

In this paper, we assume that the input distribution p in is the empirical distribution on a finite dataset x 1 , ..., x N , i.e the sum of Dirac measures 1 N N i=0 δ xi . We define the network function by f θ (x) := α(L) (x; θ), where the functions α( ) (•; θ) : R n0 → R n (called preactivations) and α ( ) (•; θ) : R n0 → R n (called activations) are defined from the 0-th to the L-th layer by:

α (0) (x; θ) = x α( +1) (x; θ) = 1 √ n W ( ) α ( ) (x; θ) + βb ( ) α( )(x; θ) = σ(α ( ) (x; θ)),
where the nonlinearity σ is applied entrywise. The scalar β > 0 is a parameter which allows us to tune the influence of the bias on the training.

Remark 1. Our definition of the realization function F (L) slightly differs from the classical one. Usually, the factors 1 √ n and the parameter β are absent and the parameters are initialized using the so-called Xavier initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF], taking W

( ) ij ∼ N (0, 1 n ) and b ( ) j ∼ N (0, 1) (or sometimes b ( ) j = 0) to compensate.
While the set of representable functions F (L) (R P ) is the same for both parametrizations (with or without the factors 1 √ n and β), the derivatives of the realization function with respect to the connections ∂ W ( ) ij F (L) and bias ∂ b ( ) j F (L) are scaled by 1 √ n and β respectively in comparison to the classical parametrization.

The factors 1

√ n are key to obtaining a consistent asymptotic behavior of neural networks as the widths of the hidden layers n 1 , ..., n L-1 grow to infinity. However a side-effect of these factors is that they reduce greatly the influence of the connection weights during training when n is large: the factor β is introduced to balance the influence of the bias and connection weights. Taking β ∼ 1 √ n makes the behavior comparable to a classical ANN parametrization of width n. Also, as a result of the presence of the factors, one may take a larger learning rate than in the case of classical parametrizations. The choices of β and of the learning rate in numerical experiments are further discussed in Section 6.

Kernel gradient

The training of an ANN consists in optimizing f θ in the function space F with respect to a functional cost C : F → R, such as a regression or cross-entropy cost. Even for a convex functional cost C, the composite cost C • F (L) : R P → R is in general highly non-convex [START_REF] Choromanska | The Loss Surfaces of Multilayer Networks[END_REF]. We will show that during training, the network function f θ follows a descent along the kernel gradient with respect to the Neural Tangent Kernel (NTK) which we introduce in Section 4. This makes it possible to study the training of ANNs in the function space F, on which the cost C is convex.

A multi-dimensional kernel K is a function R n0 × R n0 → R n L ×n L , which maps any pair (x, x ) to an n L × n L -matrix such that K(x, x ) = K(x , x) T (equivalently K is a symmetric tensor in F ⊗ F).
Such a kernel defines a bilinear map on F, taking the expectation over independent x, x ∼ p in :

f, g K := E x,x ∼p in f (x) T K(x, x )g(x )
.

The kernel K is positive definite with respect to || • || p in if ||f || p in > 0 =⇒ ||f || K > 0.
We denote by F * the dual of F with respect to p in , i.e. the set of linear forms µ : F → R of the form µ = d, • p in for some d ∈ F. Two elements of F define the same linear form if and only if they are equal on the data. The constructions in the paper do not depend on the element d ∈ F chosen in order to represent µ as d, • p in . Using the fact that the partial application of the kernel

K i,• (x, •) is a function in F, we can define a map Φ K : F * → F mapping a dual element µ = d, • p in to the function f µ = Φ K (µ) with values: f µ,i (x) = µK i,• (x, •) = d, K i,• (x, •) p in .
For our setup, which is that of a finite dataset x 1 , . . . , x n ∈ R n0 , the cost functional C only depends on the values of f ∈ F at the data points. As a result, the (functional) derivative of the cost C can be viewed as an element of F * , which we write ∂ in f C. We denote by d| f ∈ F, a corresponding dual element, such that

∂ in f C = d| f , • p in . The kernel gradient ∇ K C| f0 ∈ F is defined as Φ K ∂ in f C| f0 . In contrast to ∂ in f C
which is only defined on the dataset, the kernel gradient generalizes to values x outside the dataset thanks to the kernel K:

∇ K C| f0 (x) = 1 N N j=1 K(x, x j )d| f (x j ).
A time-dependent function f (t) follows the kernel gradient descent with respect to K if it satisfies the differential equation

∂ t f (t) = -∇ K C| f (t) .
During kernel gradient descent, the cost C(f (t)) evolves as

∂ t C| f (t) = -d| f (t) , ∇ K C| f (t) p in = -d| f (t) 2
K . Convergence to a critical point of C is hence guaranteed if the kernel K is positive definite with respect to || • || p in : the cost is then strictly decreasing except at points such that ||d| f (t) || p in = 0. If the cost is convex and bounded from below, the function f (t) therefore converges to a global minimum as t → ∞.

Random functions approximation

As a starting point to understand the convergence of ANN gradient descent to kernel gradient descent in the infinite-width limit, we introduce a simple model, inspired by the approach of [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF].

A kernel K can be approximated by a choice of P random functions f (p) sampled independently from any distribution on F whose (non-centered) covariance is given by the kernel K:

E[f (p) k (x)f (p) k (x )] = K kk (x, x ).
These functions define a random linear parametrization F lin : R P → F

θ → f lin θ = 1 √ P P p=1 θ p f (p) .
The partial derivatives of the parametrization are given by

∂ θp F lin (θ) = 1 √ P f (p) .
Optimizing the cost C • F lin through gradient descent, the parameters follow the ODE:

∂ t θ p (t) = -∂ θp (C • F lin )(θ(t)) = - 1 √ P ∂ in f C| f lin θ(t) f (p) = - 1 √ P d| f lin θ(t) , f (p) p in .
As a result the function f lin θ(t) evolves according to

∂ t f lin θ(t) = 1 √ P P p=1 ∂ t θ p (t)f (p) = - 1 P P p=1 d| f lin θ(t) , f (p) p in f (p) ,
where the right-hand side is equal to the kernel gradient -∇ K C with respect to the tangent kernel

K = P p=1 ∂ θp F lin (θ) ⊗ ∂ θp F lin (θ) = 1 P P p=1 f (p) ⊗ f (p) .
This is a random n L -dimensional kernel with values Kii (x, x ) = 1

P P p=1 f (p) i (x)f (p)
i (x ). Performing gradient descent on the cost C • F lin is therefore equivalent to performing kernel gradient descent with the tangent kernel K in the function space. In the limit as P → ∞, by the law of large numbers, the (random) tangent kernel K tends to the fixed kernel K, which makes this method an approximation of kernel gradient descent with respect to the limiting kernel K.

Neural tangent kernel

For ANNs trained using gradient descent on the composition C • F (L) , the situation is very similar to that studied in the Section 3.1. During training, the network function f θ evolves along the (negative) kernel gradient

∂ t f θ(t) = -∇ Θ (L) C| f θ(t)
with respect to the neural tangent kernel (NTK)

Θ (L) (θ) = P p=1 ∂ θp F (L) (θ) ⊗ ∂ θp F (L) (θ).
However, in contrast to F lin , the realization function F (L) of ANNs is not linear. As a consequence, the derivatives ∂ θp F (L) (θ) and the neural tangent kernel depend on the parameters θ. The NTK is therefore random at initialization and varies during training, which makes the analysis of the convergence of f θ more delicate.

In the next subsections, we show that, in the infinite-width limit, the NTK becomes deterministic at initialization and stays constant during training. Since f θ at initialization is Gaussian in the limit, the asymptotic behavior of f θ during training can be explicited in the function space F.

Initialization

As observed in [START_REF] Neal | Bayesian Learning for Neural Networks[END_REF][START_REF] Lee | Deep neural networks as gaussian processes[END_REF], the output functions f θ,i for i = 1, ..., n L tend to iid Gaussian functions in the infinite-width limit (a proof in our setup is given in the appendix): Proposition 1. For a network of depth L at initialization, with a Lipschitz nonlinearity σ, and in the limit as n 1 , ..., n L-1 → ∞, the output functions f θ,k , for k = 1, ..., n L , tend (in law) to iid centered Gaussian functions of covariance Σ (L) , where Σ (L) is defined recursively by:

Σ (1) (x, x ) = 1 n 0 x T x + β 2 Σ (L+1) (x, x ) = E f [σ(f (x))σ(f (x ))] + β 2 ,
taking the expectation with respect to a centered Gaussian function f of covariance Σ (L) . Remark 2. Strictly speaking, the existence of a suitable Gaussian measure with covariance Σ (L) is not needed: we only deal with the values of f at x, x (the joint measure on f (x), f (x ) is simply a Gaussian vector in 2D). For the same reasons, in the proof of Proposition 1 and Theorem 1, we will freely speak of Gaussian processes without discussing their existence.

The first key result of our paper (proven in the appendix) is the following: in the same limit, the Neural Tangent Kernel (NTK) converges in probability to an explicit deterministic limit. Theorem 1. For a network of depth L at initialization, with a Lipschitz nonlinearity σ, and in the limit as the layers width n 1 , ..., n L-1 → ∞, the NTK Θ (L) converges in probability to a deterministic limiting kernel:

Θ (L) → Θ (L) ∞ ⊗ Id n L . The scalar kernel Θ (L) ∞ : R n0 × R n0 → R is defined recursively by Θ (1) ∞ (x, x ) = Σ (1) (x, x ) Θ (L+1) ∞ (x, x ) = Σ(L) (x, x )Θ (L) ∞ (x, x ) + Σ (L+1) (x, x ), where for L ≥ 1 Σ(L) (x, x ) = E f ∼N (0,Σ (L) ) [ σ (f (x)) σ (f (x ))] ,
taking the expectation with respect to a centered Gaussian function f of covariance Σ (L) , and where σ denotes the derivative of σ. Remark 3. By Rademacher's theorem, σ is defined everywhere, except perhaps on a set of zero Lebesgue measure.

Note that the limiting Θ (L)

∞ only depends on the choice of σ, the depth of the network and the variance of the parameters at initialization (which is equal to 1 in our setting).

Training

Our second key result is that the NTK stays asymptotically constant during training. This applies for a slightly more general definition of training: the parameters are updated according to a training direction d t ∈ F:

∂ t θ p (t) = ∂ θp F (L) (θ(t)), d t p in .
In the case of gradient descent, d t = -d| f θ(t) (see Section 3), but the direction may depend on another network, as is the case for e.g. Generative Adversarial Networks [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]. We only assume that the integral T 0 d t p in dt stays stochastically bounded as the width tends to infinity, which is verified for e.g. least-squares regression, see Section 5. Theorem 2. Assume that σ is a Lipschitz, twice differentiable nonlinearity function, with bounded second derivative. For any T such that the integral

T 0 d t p in dt stays stochastically bounded, as n 1 , ..., n L-1 → ∞, we have, uniformly for t ∈ [0, T ], Θ (L) (t) → Θ (L) ∞ ⊗ Id n L .
As a consequence, in this limit, the dynamics of f θ is described by the differential equation

∂ t f θ(t) = Φ Θ (L) ∞ ⊗Idn L d t , • p in .
Remark 4. As the proof of the theorem (in the appendix) shows, the variation during training of the individual activations in the hidden layers shrinks as their width grows. However their collective variation is significant, which allows the parameters of the lower layers to learn: in the formula of the limiting NTK Θ (L+1) ∞ (x, x ) in Theorem 1, the second summand Σ (L+1) represents the learning due to the last layer, while the first summand represents the learning performed by the lower layers.

As discussed in Section 3, the convergence of kernel gradient descent to a critical point of the cost C is guaranteed for positive definite kernels. The limiting NTK is positive definite if the span of the derivatives ∂ θp F (L) , p = 1, ..., P becomes dense in F w.r.t. the p in -norm as the width grows to infinity. It seems natural to postulate that the span of the preactivations of the last layer (which themselves appear in ∂ θp F (L) , corresponding to the connection weights of the last layer) becomes dense in F, for a large family of measures p in and nonlinearities (see e.g. [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF][START_REF] Leshno | Multilayer feedforward networks with a non-polynomial activation function can approximate any function[END_REF] for classical theorems about ANNs and approximation). In the case when the dataset is supported on a sphere, the positive-definiteness of the limiting NTK can be shown using Gaussian integration techniques and existing positive-definiteness criteria, as given by the following proposition, proven in Appendix A.4: Proposition 2. For a non-polynomial Lipschitz nonlinearity σ, for any input dimension n 0 , the restriction of the limiting NTK Θ (L)

∞ to the unit sphere S n0-1 = {x ∈ R n0 : x T x = 1} is positive- definite if L ≥ 2.

Least-squares regression

Given a goal function f * and input distribution p in , the least-squares regression cost is

C(f ) = 1 2 ||f -f * || 2 p in = 1 2 E x∼p in f (x) -f * (x) 2 .
Theorems 1 and 2 apply to an ANN trained on such a cost. Indeed the norm of the training direction d(f ) p in = f * -f p in is strictly decreasing during training, bounding the integral. We are therefore interested in the behavior of a function f t during kernel gradient descent with a kernel K (we are of course especially interested in the case

K = Θ (L) ∞ ⊗ Id n L ): ∂ t f t = Φ K f * -f, • p in .
The solution of this differential equation can be expressed in terms of the map Π :

f → Φ K f, • p in : f t = f * + e -tΠ (f 0 -f * ) where e -tΠ = ∞ k=0 (-t) k k! Π k is the exponential of -tΠ.
If Π can be diagonalized by eigenfunctions f (i) with eigenvalues λ i , the exponential e -tΠ has the same eigenfunctions with eigenvalues e -tλi . For a finite dataset x 1 , ..., x N of size N , the map Π takes the form

Π(f ) k (x) = 1 N N i=1 n L k =1 f k (x i )K kk (x i , x).
The map Π has at most N n L positive eigenfunctions, and they are the kernel principal components f (1) , ..., f (N n L ) of the data with respect to to the kernel K [START_REF] Schölkopf | Nonlinear component analysis as a kernel eigenvalue problem[END_REF][START_REF] Shawe-Taylor | Kernel Methods for Pattern Analysis[END_REF]. The corresponding eigenvalues λ i correspond to the variance captured by the components.

Decomposing the difference

(f * -f 0 ) = ∆ 0 f + ∆ 1 f + ... + ∆ N n L f
along the eigenspaces of Π, the trajectory of the function f t reads

f t = f * + ∆ 0 f + N n L i=1 e -tλi ∆ i f ,
where ∆ 0 f is in the kernel (null-space) of Π and ∆ i f ∝ f (i) . The above decomposition can be seen as a motivation for the use of early stopping. The convergence is indeed faster along the eigenspaces corresponding to larger eigenvalues λ i . Early stopping hence focuses the convergence on the most relevant kernel principal components, while avoiding to fit the ones in eigenspaces with lower eigenvalues (such directions are typically the 'noisier' ones: for instance, in the case of the RBF kernel, lower eigenvalues correspond to high frequency functions).

Note that by the linearity of the map e -tΦ , if f 0 is initialized with a Gaussian distribution (as is the case for ANNs in the infinite-width limit), then f t is Gaussian for all times t. Assuming that the kernel is positive definite on the data (implying that K is invertible), as t → ∞ limit, we get that

f ∞ = f * + ∆ 0 f = f 0 -i ∆ i f takes the form f ∞,k (x) = κ T x,k K-1 y * + f 0 (x) -κ T x,k K-1 y 0 ,
with the N n l -vectors κ x,k , y * and y 0 given by κ

x,k = (K kk (x, x i )) i,k y * = (f * k (x i )) i,k y 0 = (f 0,k (x i )) i,k
and where the Gram matrix K is given by (K kk (x i , x i )) (i,k),(i ,k ) . The first term is the kernel ridge regression [START_REF] Shawe-Taylor | Kernel Methods for Pattern Analysis[END_REF] without regularization (λ = 0) and the second term is a centered Gaussian of variance 0 on the points of the dataset. Remark 5. From Proposition 2 and the discussion above, we obtain in particular that in a leastsquares regression setup, when the dataset p in is supported on the unit sphere S n0-1 = {x ∈ R n0 :

x T x = 1}, the infinite-width limit of the ANN training converges, as t → ∞, to a global minimum of the cost.

Numerical experiments

In the following numerical experiments, fully connected ANNs of various widths are compared to the theoretical infinite-width limit. We choose the size of the hidden layers to all be equal to the same value n := n 1 = ... = n L-1 and we take the ReLU nonlinearity σ(x) = max(0, x).

In the first two experiments, we consider the case n 0 = 2. Moreover, the input elements are taken on the unit circle. This can be motivated by the structure of high-dimensional data, where the centered data points often have roughly the same norm 2 .

In all experiments, we took n L = 1 (note that by our results, a network with n L outputs behaves asymptotically like n L networks with scalar outputs which evolve independently). Finally, the value of the parameter β is chosen as 0.1, see Remark 1.

Convergence of the NTK

The first experiment illustrates the convergence of the NTK Θ (L) of a network of depth L = 4 for two different widths n = 500, 10000. The function Θ (4) (x 0 , x) is plotted for a fixed x 0 = (1, 0) and x = (cos(γ), sin(γ)) on the unit circle. To observe the distribution of the NTK, 10 independent initializations are performed for both widths. The kernels are plotted at initialization t = 0 and then after 200 steps of gradient descent with learning rate 1.0 (i.e. at t = 200). The cost is a regression cost approximating the function f * (x) = x 1 x 2 on random N (0, 1) inputs.

For the wider network, the NTK shows less variance and is smoother. It is interesting to note that the expectation of the NTK is very close for both networks widths. During training, we observe that the NTK tends to "inflate". As expected, this is much more apparent on the smaller network (n = 500) than for the wider network (n = 10000) where the NTK stays almost fixed.

Kernel regression

For a regression cost, the infinite-width limit network function f θ(t) has a Gaussian distribution for all times t (see Section 5). We compare this theoretical limit to the distribution of the network function at convergence f θ(∞) for finite-width networks. We optimized a least-squares regression cost on 4 points of the unit circle and plotted the function f θ(t) on the unit circle for a large time t = 1000 (1000 steps with learning rate 1.0) for two different widths n = 50, 1000 and for 10 random initializations each.

We also approximated the kernels Θ

∞ and Σ (4) using a large-width network (n = 10000) and used them to calculate and plot the 10th, 50th and 90-th percentiles of the t → ∞ limiting Gaussian distribution.

The distributions of the network functions are very similar for both widths: their mean and variance appear to be close to those of the limiting distribution t → ∞. The NTK gives a good indication of what the neural network will converge to even for relatively small widths (n = 50).

Convergence along a principal component

We now illustrate our result on the MNIST dataset of handwritten digits made up of grayscale images of dimension 28 × 28, yielding a dimension of n 0 = 784.

We computed the first 3 principal components of a batch of N = 512 digits with respect to the NTK of a high-width network n = 10000 (giving an approximation of the limiting kernel) using a power iteration method. The respective eigenvalues are λ 1 = 0.0457, λ 2 = 0.00108 and λ 3 = 0.00078. The PCA is not centered, the first component is therefore almost equal to the constant function, which 8 Figure 3 explains the large gap between the first and second eigenvalues 3 . The next two components are much more interesting as can be seen in Figure 3a, where the batch is plotted with x and y coordinates corresponding to the 2nd and 3rd components.

2 1 0 1 2 f (2) (x)
We have seen in Section 5 how the convergence of kernel gradient descent follows the kernel principal components. If the difference at initialization f 0 -f * is equal (or proportional) to one of the principal components f (i) , then the function will converge in a straight line to f * at an exponential rate e -λit .

We tested whether ANNs of various widths n = 100, 1000, 10000 behave in a similar manner. We set the goal of the regression cost to f * = f θ(0) + 0.5f (2) and let the network converge. At each time step t, we decomposed the difference f θ(t) -f * into a component g t proportional to f (2) and another one h t orthogonal to f (2) . In the infinite-width limit, the first component decays exponentially fast ||g t || p in = 0.5e -λ2t while the second is null (h t = 0), as the function converges along a straight line.

As expected, we see in Figure 3b that the wider the network, the less it deviates from the straight line (for each width n we performed two independent trials). As the width grows, the trajectory along the 2nd principal component (shown in Figure 3c) converges to the theoretical limit shown in blue.

A surprising observation is that smaller networks appear to converge faster than wider ones. This may be explained by the inflation of the NTK observed in our first experiment. Indeed, multiplying the NTK by a factor a is equivalent to multiplying the learning rate by the same factor. However, note that since the NTK of large-width network is more stable during training, larger learning rates can in principle be taken. One must hence be careful when comparing the convergence speed in terms of the number of steps (rather than in terms of the time t): both the inflation effect and the learning rate must be taken into account.

Conclusion

This paper introduces a new tool to study ANNs, the Neural Tangent Kernel (NTK), which describes the local dynamics of an ANN during gradient descent. This leads to a new connection between ANN training and kernel methods: in the infinite-width limit, an ANN can be described in the function space directly by the limit of the NTK, an explicit constant kernel Θ (L)

∞ , which only depends on its depth, nonlinearity and parameter initialization variance. More precisely, in this limit, ANN gradient descent is shown to be equivalent to a kernel gradient descent with respect to Θ (L) ∞ . The limit of the NTK is hence a powerful tool to understand the generalization properties of ANNs, and it allows one to study the influence of the depth and nonlinearity on the learning abilities of the network. The analysis of training using NTK allows one to relate convergence of ANN training with the positive-definiteness of the limiting NTK and leads to a characterization of the directions favored by early stopping methods.

A Appendix

This appendix is dedicated to proving the key results of this paper, namely Proposition 1 and Theorems 1 and 2, which describe the asymptotics of neural networks at initialization and during training, as well as Proposition 2, which guarantees the positive-definiteness of the limiting NTK on the sphere.

We study the limit of the NTK as n 1 , ..., n L-1 → ∞ sequentially, i.e. we first take n 1 → ∞, then n 2 → ∞, etc. This leads to much simpler proofs, but our results could in principle be strengthened to the more general setting when min(n 1 , ..., n L-1 ) → ∞.

A natural choice of convergence to study the NTK is with respect to the operator norm on kernels:

K op = max f p in ≤1 f K = max f p in ≤1 E x,x [f (x) T K(x, x )f (x )],
where the expectation is taken over two independent x, x ∼ p in . This norm depends on the input distribution p in . In our setting, p in is taken to be the empirical measure of a finite dataset of distinct samples x 1 , ..., x N . As a result, the operator norm of K is equal to the leading eigenvalue of the N n L × N n L Gram matrix (K kk (x i , x j )) k,k <n L ,i,j<N . In our setting, convergence in operator norm is hence equivalent to pointwise convergence of K on the dataset.

A.1 Asymptotics at Initialization

It has already been observed [START_REF] Neal | Bayesian Learning for Neural Networks[END_REF][START_REF] Lee | Deep neural networks as gaussian processes[END_REF] that the output functions f θ,i for i = 1, ..., n L tend to iid Gaussian functions in the infinite-width limit. Proposition 1. For a network of depth L at initialization, with a Lipschitz nonlinearity σ, and in the limit as n 1 , ..., n L-1 → ∞ sequentially, the output functions f θ,k , for k = 1, ..., n L , tend (in law) to iid centered Gaussian functions of covariance Σ (L) , where Σ (L) is defined recursively by:

Σ (1) (x, x ) = 1 n 0 x T x + β 2 Σ (L+1) (x, x ) = E f [σ(f (x))σ(f (x ))] + β 2 ,
taking the expectation with respect to a centered Gaussian function f of covariance Σ (L) .

Proof. We prove the result by induction. When L = 1, there are no hidden layers and f θ is a random affine function of the form:

f θ (x) = 1 √ n 0 W (0) x + βb (0) .
All output functions f θ,k are hence independent and have covariance Σ (1) as needed.

The key to the induction step is to consider an (L + 1)-network as the following composition: an L-network R n0 → R n L mapping the input to the pre-activations α(L) i , followed by an elementwise application of the nonlinearity σ and then a random affine map R n L → R n L+1 . The induction hypothesis gives that in the limit as sequentially n 1 , ..., n L-1 → ∞ the preactivations α(L) i tend to iid Gaussian functions with covariance Σ (L) . The outputs

f θ,i = 1 √ n L W (L) i α (L) + βb (L) i
conditioned on the values of α (L) are iid centered Gaussians with covariance

Σ(L+1) (x, x ) = 1 n L α (L) (x; θ) T α (L) (x ; θ) + β 2 .
By the law of large numbers, as n L → ∞, this covariance tends in probability to the expectation

Σ(L+1) (x, x ) → Σ (L+1) (x, x ) = E f ∼N (0,Σ (L) ) [σ(f (x))σ(f (x ))] + β 2 .
In particular the covariance is deterministic and hence independent of α (L) . As a consequence, the conditioned and unconditioned distributions of f θ,i are equal in the limit: they are iid centered Gaussian of covariance Σ (L+1) .

In the infinite-width limit, the neural tangent kernel, which is random at initialization, converges in probability to a deterministic limit. Theorem 1. For a network of depth L at initialization, with a Lipschitz nonlinearity σ, and in the limit as the layers width n 1 , ..., n L-1 → ∞ sequentially, the NTK Θ (L) converges in probability to a deterministic limiting kernel:

Θ (L) → Θ (L) ∞ ⊗ Id n L . The scalar kernel Θ (L) ∞ : R n0 × R n0 → R is defined recursively by Θ (1) ∞ (x, x ) = Σ (1) (x, x ) Θ (L+1) ∞ (x, x ) = Σ(L) (x, x )Θ (L) ∞ (x, x ) + Σ (L+1) (x, x ), where for L ≥ 1 Σ(L) (x, x ) = E f ∼N (0,Σ (L) ) [ σ (f (x)) σ (f (x ))] ,
taking the expectation with respect to a centered Gaussian function f of covariance Σ (L) , and where σ denotes the derivative of σ.

Proof. The proof is again by induction. When L = 1, there is no hidden layer and therefore no limit to be taken. The neural tangent kernel is a sum over the entries of W (0) and those of b (0) :

Θ kk (x, x ) = 1 n 0 n0 i=1 n1 j=1 x i x i δ jk δ jk + β 2 n1 j=1 δ jk δ jk = 1 n 0 x T x δ kk + β 2 δ kk = Σ (1) (x, x )δ kk .
Here again, the key to prove the induction step is the observation that a network of depth L + 1 is an L-network mapping the inputs x to the preactivations of the L-th layer α(L) (x) followed by a nonlinearity and a random affine function. For a network of depth L + 1, let us therefore split the parameters into the parameters θ of the first L layers and those of the last layer (W (L) , b (L) ).

By Proposition 1 and the induction hypothesis, as n 1 , ..., n L-1 → ∞ the pre-activations α(L) i are iid centered Gaussian with covariance Σ (L) and the neural tangent kernel Θ (L) ii (x, x ) of the smaller network converges to a deterministic limit:

∂ θ α(L) i (x; θ) T ∂ θ α(L) i (x ; θ) → Θ (L) ∞ (x, x )δ ii .
We can split the neural tangent network into a sum over the parameters θ of the first L layers and the remaining parameters W (L) and b (L) .

For the first sum let us observe that by the chain rule:

∂ θp f θ,k (x) = 1 √ n L n L i=1 ∂ θp α(L) i (x; θ) σ(α (L) i (x; θ))W (L) ik .
By the induction hypothesis, the contribution of the parameters θ to the neural tangent kernel Θ (L+1) kk (x, x ) therefore converges as n 1 , ..., n L-1 → ∞:

1 n L n L i,i =1 Θ (L) ii (x, x ) σ α(L) i (x; θ) σ α(L) i (x ; θ) W (L) ik W (L) i k → 1 n L n L i=1 Θ (L) ∞ (x, x ) σ α(L) i (x; θ) σ α(L) i (x ; θ) W (L) ik W (L) ik
By the law of large numbers, as n L → ∞, this tends to its expectation which is equal to

Θ (L) ∞ (x, x ) Σ(L) (x, x )δ kk .
It is then easy to see that the second part of the neural tangent kernel, the sum over W (L) and b (L) converges to Σ (L+1) δ kk as n 1 , ..., n L → ∞.

A.2 Asymptotics during Training

Given a training direction t → d t ∈ F, a neural network is trained in the following manner: the parameters θ p are initialized as iid N (0, 1) and follow the differential equation:

∂ t θ p (t) = ∂ θp F (L) , d t p in .
In this context, in the infinite-width limit, the NTK stays constant during training: Theorem 2. Assume that σ is a Lipschitz, twice differentiable nonlinearity function, with bounded second derivative. For any T such that the integral T 0 d t p in dt stays stochastically bounded, as n 1 , ..., n L-1 → ∞ sequentially, we have, uniformly for t ∈ [0, T ],

Θ (L) (t) → Θ (L)
∞ ⊗ Id n L . As a consequence, in this limit, the dynamics of f θ is described by the differential equation

∂ t f θ(t) = Φ Θ (L) ∞ ⊗Idn L d t , • p in .
Proof. As in the previous theorem, the proof is by induction on the depth of the network. When L = 1, the neural tangent kernel does not depend on the parameters, it is therefore constant during training.

For the induction step, we again split an L + 1 network into a network of depth L with parameters θ and top layer connection weights W (L) and bias b (L) . The smaller network follows the training direction

d t = σ α(L) (t) 1 √ n L W (L) (t) T d t for i = 1, . . . , n L , where the function α(L) i (t) is defined as α(L) i (•; θ(t))
. We now want to apply the induction hypothesis to the smaller network. For this, we need to show that T 0 d t p in dt is stochastically bounded as n 1 , . . . , n L → ∞. Since σ is a c-Lipschitz function, we have that

d t p in ≤ c 1 √ n L W (L) (t) op d t p in .
To apply the induction hypothesis, we now need to bound

1 √ n L W (L) (t) op .
For this, we use the following lemma, which is proven in Appendix A.3 below: Lemma 1. With the setting of Theorem 2, for a network of depth L + 1, for any = 1, . . . , L, we have the convergence in probability:

lim n L →∞ • • • lim n1→∞ sup t∈[0,T ] 1 √ n W ( ) (t) -W ( ) (0) op = 0
From this lemma, to bound

1 √ n L W (L) (t) op , it is hence enough to bound 1 √ n L W (L) (0) op .
From the law of large numbers, we obtain that the norm of each of the n L+1 rows of W (L) (0) is bounded, and hence that

1 √ n L W (L) (0) op is bounded (keep in mind that n L+1 is fixed, while n 1 , . . . , n L grow).
From the above considerations, we can apply the induction hypothesis to the smaller network, yielding, in the limit as n 1 , . . . , n L → ∞ (sequentially), that the dynamics is governed by the constant kernel

Θ (L) ∞ : ∂ t α(L) i (t) = 1 √ n L Φ Θ (L) ∞ σ α(L) i (t) W (L) i (t) T d t , • p in .
At the same time, the parameters of the last layer evolve according to

∂ t W (L) ij (t) = 1 √ n L α (L) i (t), d t,j p in .
We want to give an upper bound on the variation of the weights columns W (L) i (t) and of the activations α(L) i (t) during training in terms of L 2 -norm and p in -norm respectively. Applying the Cauchy-Schwarz inequality for each j, summing and using

∂ t || • || ≤ ||∂ t • ||), we have ∂ t W (L) i (t) -W (L) i (0) 2 ≤ 1 √ n L ||α (L) i (t)|| p in ||d t || p in . Now, observing that the operator norm of Φ Θ (L) ∞ is equal to ||Θ (L)
∞ || op , defined in the introduction of Appendix A, and using the Cauchy-Schwarz inequality, we get

∂ t α(L) i (t) - α(L) i (0) p in ≤ 1 √ n L Θ (L) ∞ op σ α(L) i (t) ∞ W (L) i (t) 2 d t p in ,
where the sup norm

• ∞ is defined by f ∞ = sup x |f (x)|.
To bound both quantities simultaneously, study the derivative of the quantity

A(t) = ||α (L) i (0)|| p in + c α(L) i (t) - α(L) i (0) p in + ||W (L) i (0)|| 2 + W (L) i (t) -W (L) i (0) 2 .
We have

∂ t A(t) ≤ 1 √ n L c 2 Θ (L) ∞ op W (L) i (t) 2 + ||α (L) i (t)|| p in ||d t || p in ≤ max{c 2 Θ (L) ∞ op , 1} √ n L d t p in A(t),
where, in the first inequality, we have used that | σ| ≤ c and, in the second inequality, that the sum

W (L) i (t) 2 + ||α (L) i (t)|| p in is bounded by A(t)
. Applying Grönwall's Lemma, we now get

A(t) ≤ A(0) exp max{c 2 Θ (L) ∞ op , 1} √ n L t 0 d s p in ds .

Note that Θ (L)

∞ op is constant during training. Clearly the value inside of the exponential converges to zero in probability as n L → ∞ given that the integral We can now use these bounds to control the variation of the NTK and to prove the theorem. To understand how the NTK evolves, we study the evolution of the derivatives with respect to the parameters. The derivatives with respect to the bias parameters of the top layer ∂ b (L) j f θ,j are always equal to δ jj . The derivatives with respect to the connection weights of the top layer are given by

∂ W (L) ij f θ,j (x) = 1 √ n L α (L) i (x; θ)δ jj .
The pre-activations α(L) i evolve at a rate of 1 √ n L and so do the activations α

(L) i . The summands ∂ W (L) ij f θ,j (x) ⊗ ∂ W (L) ij f θ,j (x )
∂ θk f θ,j (x) = 1 √ n L n L i=1 ∂ θk α(L) i (x; θ) σ α(L) i (x; θ) W (L) ij .
Their contribution to the NTK Θ

(L+1) jj (x, x ) is 1 n L n L i,i =1 Θ (L) ii (x, x ) σ α(L) i (x; θ) σ α(L) i (x ; θ) W (L) ij W (L) i j .
By the induction hypothesis, the NTK of the smaller network Θ (L) tends to Θ (L)

∞ δ ii as n 1 , ..., n L-1 → ∞. The contribution therefore becomes

1 n L n L i=1 Θ (L) ∞ (x, x ) σ α(L) i (x; θ) σ α(L) i (x ; θ) W (L) ij W (L) ij .
The connection weights W (L) ij vary at rate 1 √ n L , inducing a change of the same rate to the whole sum.

We simply have to prove that the values σ(α (L) i (x; θ)) also change at rate 1 √ n L . Since the second derivative of σ is bounded, we have that

∂ t σ α(L) i (x; θ(t)) = O ∂ t α(L) i (x; θ(t)) .
Since ∂ t α(L) i (x; θ(t)) goes to zero at a rate 1 √ n L by the bound on A(t) above, this concludes the proof.

It is somewhat counterintuitive that the variation of the activations of the hidden layers α ( ) i during training goes to zero as the width becomes large 4 . It is generally assumed that the purpose of the activations of the hidden layers is to learn "good" representations of the data during training. However note that even though the variation of each individual activation shrinks, the number of neurons grows, resulting in a significant collective effect. This explains why the training of the parameters of each layer has an influence on the network function f θ even though it has asymptotically no influence on the individual activations of the layers for < < L.

A.3 A Priori Control during Training

The goal of this section is to prove Lemma 1, which is a key ingredient in the proof of Theorem 2. Let us first recall it: Lemma 1. With the setting of Theorem 2, for a network of depth L + 1, for any = 1, . . . , L, we have the convergence in probability:

lim n L →∞ • • • lim n1→∞ sup t∈[0,T ] 1 √ n W ( ) (t) -W ( ) (0) op = 0
Proof. We prove the lemma for all = 1, . . . , L simultaneously, by expressing the variation of the weights 1 √ n W ( ) and activations 1

√ n α( ) in terms of 'back-propagated' training directions d (1) , . . . , d (L) associated with the lower layers and the NTKs of the corresponding subnetworks:

1. At all times, the evolution of the preactivations and weights is given by:

∂ t α( ) = Φ Θ ( ) < d ( ) t , • > p in ∂ t W ( ) = 1 √ n < α ( ) , d ( +1) t > p in ,
where the layer-wise training directions d (1) , . . . , d (L) are defined recursively by

d ( ) t =    d t if = L + 1 σ α( ) 1 √ n W ( ) T d ( +1) t if ≤ L,
and where the sub-network NTKs Θ ( ) satisfy

Θ (1) = 1 √ n 0 α (0) T 1 √ n 0 α (0) ⊗ Id n + β 2 ⊗ Id n Θ ( +1) = 1 √ n W ( ) σ(α ( ) )Θ ( ) σ(α ( ) ) 1 √ n W ( ) + 1 √ n α ( ) T 1 √ n α ( ) ⊗ Id n + β 2 ⊗ Id n . 2. Set w (k) (t) := 1 √ n k W (k) (t) op and a (k) (t) := 1 √ n k α (k) (t) p in .
The identities of the previous step yield the following recursive bounds:

d ( ) t p in ≤ cw ( ) (t) d ( +1) t p in ,
where c is the Lipschitz constant of σ. These bounds lead to

d ( ) t p in ≤ c L+1- L k= w (k) (t) d t p in .
For the subnetworks NTKs we have the recursive bounds

Θ (1) op ≤ (a (0) (t)) 2 + β 2 . Θ ( +1) op ≤ c 2 (w ( ) (t)) 2 Θ ( ) op + (a ( ) (t)) 2 + β 2 , which lead to Θ ( +1)
op ≤ P a (1) , . . . , a ( ) , w (1) , . . . , w ( ) , where P is a polynomial which only depends on , c, β and p in .

Set

ã(k) (t) := 1 √ n k α(k) (t) -α(k) (0) p in w(k) (t) := 1 √ n k W (k) (t) -W (k) (0) op and define
A (t) = L k=1 a (k) (0) + cã (k) (t) + w (k) (0) + w(k) (t) . Since a (k) (t) ≤ a (k) (0) + cã (k) (t) and w (k) (t) ≤ w (k) (0) + w(k) (t), controlling A (t)
will enable us to control the a (k) (t) and w (k) (t). Using the formula at the beginning of the first step, we obtain

∂ t ã( ) (t) ≤ 1 √ n Θ ( ) (t) op d ( ) t p in ∂ t w( ) (t) ≤ 1 √ n a ( ) (t) d ( +1) t p in .
This allows one to bound the derivative of A (t) as follows:

∂ t A (t) ≤ L =1 c √ n Θ ( ) (t) op d ( ) t p in + 1 √ n a ( ) (t) d ( +1) t p in .
Using the polynomial bounds on Θ ( ) (t) op and d ( +1) t p in in terms of the a (k) and w (k) for k = 1, . . . obtained in the previous step, we get that

∂ t A (t) ≤ 1 min {n 1 , . . . , n L } Q w (1) (t) , . . . , w (L) (t) , a (1) (t) , . . . , a (L) (t) d t p in ,
where the polynomial Q only depends on L, c, β and p in and has positive coefficients. As a result, we can use a (k) (t) ≤ a (k) (0) + cã (k) (t) and w (k) (t) ≤ w (k) (0) + w(k) (t) to get the polynomial bound

∂ t A (t) ≤ 1 min {n 1 , . . . , n L } Q (A (t)) d t p in .
4. Let us now observe that A (0) is stochastically bounded as we take the sequential limit lim n L →∞ • • • lim n1→∞ as in the statement of the lemma. In this limit, we indeed have that w ( ) and a ( ) are convergent: we have w ( ) → 0, while a ( ) converges by Proposition 1.

The polynomial control we obtained on the derivative of A (t) now allows one to use (a nonlinear form of, see e.g. [START_REF] Dragomir | Some Gronwall Type Inequalities and Applications[END_REF]) Grönwall's Lemma: we obtain that A (t) stays uniformly bounded on [0, τ ] for some τ = τ (n 1 , . . . , n L ) > 0, and that τ → T as min (n 1 , . . . , n L ) → ∞, owing to the This subsection is devoted to the proof of Proposition 2, which we now recall:

Proposition 2. For a non-polynomial Lipschitz nonlinearity σ, for any input dimension n 0 , the restriction of the limiting NTK Θ

(L) ∞ to the unit sphere S n0-1 = {x ∈ R n0 : x T x = 1} is positive- definite if L ≥ 2.
A key ingredient for the proof of Proposition 2 is the following Lemma, which comes from [START_REF] Daniely | Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity[END_REF]. If the expansion of µ in Hermite polynomials (h i ) i≥0 is given by µ = ∞ i=0 a i h i , we have

μ (ρ) = ∞ i=0 a 2 i ρ i .
The other key ingredient for proving Proposition 2 is the following theorem, which is a slight reformulation of Theorem 1(b) in [START_REF] Gneiting | Strictly and non-strictly positive definite functions on spheres[END_REF], which itself is a generalization of a classical result of Schönberg: With Lemma 2 and Theorem 3 above, we are now ready to prove Proposition 2.

Proof of Proposition 2. We first decompose the limiting NTK Θ (L) recursively, relate its positivedefiniteness to that of the activation kernels, then show that the positive-definiteness of the activation kernels at level 2 implies that of the higher levels, and finally show the positive-definiteness at level 2 using Lemma 2 and Theorem 3:

1. Observe that for any L ≥ 1, using the notation of Theorem 1, we have Θ (L+1) = Σ(L) Θ (L) + Σ (L+1) .

Note that the kernel Σ(L) Θ (L) is positive semi-definite, being the product of two positive semi-definite kernels. Hence, if we show that Σ (L+1) is positive-definite, this implies that Θ (L+1) is positive-definite.

c i c j Σ (L+1) (x i , x j ) = E   i c i σ (f (x i )) 2   + β i c i 2 .
Hence the left-hand side only vanishes if c i σ (f (x i )) is almost surely zero. If Σ (L) is positive-definite, the Gaussian (f (x i )) i=1,...d is non-degenerate, so this only occurs when c 1 = • • • = c d = 0 since σ is assumed to be non-constant. This shows that the positivedefiniteness of Σ (L+1) is implied by that of Σ (L) . By induction, if Σ (2) is positive-definite, we obtain that all Σ (L) with L ≥ 2 are positive-definite as well. By the first step this hence implies that Θ (L) is positive-definite as well.

3. By the previous steps, to prove the proposition, it suffices to show the positive-definitess of Σ (2) on the unit sphere S n0-1 . We have

Σ (2) (x, x ) = E (X,

Writing the expansion of µ in Hermite polynomials

(h i ) i≥0 µ = ∞ i=0 a i h i ,
we obtain that μ is given by the power series

μ (ρ) = ∞ i=0 a 2 i ρ i ,
Since σ is non-polynomial, so is µ, and as a result, there is an infinite number of nonzero a i 's in the above sum.

5. Using (1) above, we obtain that

Σ (2) (x, x ) = ν x T x ,
where ν : R → R is defined by

ν (ρ) = β 2 + ∞ i=0 a i n 0 β 2 + ρ n 0 β 2 + 1 i ,
where the a i 's are the coefficients of the Hermite expansion of µ. Now, observe that by the previous step, the power series expansion of ν contains both an infinite number of nonzero even terms and an infinite number of nonzero odd terms. This enables one to apply Theorem 3 to obtain that Σ (2) is indeed positive-definite, thereby concluding the proof.

Remark 6. Using similar techniques to the one applied in the proof above, one can show a converse to Proposition 2: if the nonlinearity σ is a polynomial, the corresponding NTK Θ (2) is not positivedefinite S n0-1 for certain input dimensions n 0 .

20 Figure 1 :

 201 Figure1: Convergence of the NTK to a fixed limit for two widths n and two times t.

Figure 2 :

 2 Figure 2: Networks function f θ near convergence for two widths n and 10th, 50th and 90th percentiles of the asymptotic Gaussian distribution.

  The 2nd and 3rd principal components of MNIST.

  Deviation of the network function f θ from the straight line. Convergence of f θ along the 2nd principal component.

t 0 d 1 √

 01 t p in ds stays stochastically bounded. The variations of the activations αc -1 (A(t)-A(0)) and A(t)-A(0) respectively, which converge to zero at rate O n L .

  of the NTK hence vary at rate of n -3/2 L which induces a variation of the NTK of rate 1 √ n L . Finally let us study the derivatives with respect to the parameters of the lower layers

Lemma 2 (

 2 Lemma 12(a) in suppl. mat. of[START_REF] Daniely | Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity[END_REF]). Let μ : [-1, 1] → R denote the dual of a Lipschitz function µ : R → R, defined by μ (ρ) = E (X,Y ) [µ (X) µ (Y )] where (X, Y ) is a centered Gaussian vector of covariance Σ, with

Theorem 3 .

 3 For a function f : [-1, 1] → R with f (ρ) = ∞ n=0 b n ρ n , the kernel K (n0) f : S n0-1 × S n0-1 → R defined by K (n0) f (x, x ) = f x T xis positive-definite for any n 0 ≥ 1 if and only if the coefficients b n are strictly positive for infinitely many even and infinitely many odd integers n.

2 . 2 .

 22 By definition, with the notation of Proposition 1 we haveΣ (L+1) (x, x ) = E f ∼N (0,Σ (L) ) [σ (f (x)) σ (f (x ))] + βThis gives, for any collection of coefficients c 1 , . . . , c d ∈ R and any pairwise distinctx 1 , . . . , x d ∈ R n0 , that d i,j=1

2 where Σ = 1 n0 + β 2 1 n0 x T x + β 2 1 n0 x T x + β 2 1 n0 + β 2 . 1 + β 2 , ( 1 )

 211112121 Y )∼N (0, Σ) [σ (X) σ (Y )] + β A change of variables then yields E (X,Y )∼N (0, Σ) [σ (X) σ (Y )] + β 2 = μ n 0 β 2 + x T x n 0 β 2 +where μ : [-1, 1] → R is the dual in the sense of Lemma 2 of the function µ : R → R defined by µ (x) = σ x 1 n0 + β 2 .

  is bounded, the differential bound on A (t) gives that the derivative ∂ t A (t) converges uniformly to 0 on [0, τ ] for any τ < T , and hence A (t) → A (0). This concludes the proof of the lemma.

		√	1 min{1,...,n L }	in front of the polynomial. Since
	A (t) A.4 Positive-Definiteness of Θ	(L) ∞

While these smoothness assumptions greatly simplify the proofs of our results, they do not seem to be strictly needed for the results to hold true.

The classical example is for data following a Gaussian distribution N (0, Idn 0 ): as the dimension n0 grows, all data points have approximately the same norm √ n0.

Note that if we choose β = 1.0 instead of our recommended 0.1, the gap between the first and the second principal component is about ten times bigger, which makes training more difficult.

As a consequence, the pre-activations stay Gaussian during training as well, with the same covariance Σ ( ) .

Acknowledgements

The authors thank K. Kytölä for many interesting discussions. The second author is supported by the ERC CG CRITICAL. The last author acknowledges support from the ERC SG Constamis, the NCCR SwissMAP, the Blavatnik family foundation and the Latsis foundation.