Lénaïc Chizat
email: lenaic.chizat@u-psud.fr

Edouard Oyallon
email: edouard.oyallon@centralesupelec.fr

Francis Bach
email: francis.bach@inria.fr

On Lazy Training in Differentiable Programming

In a series of recent theoretical works, it was shown that strongly overparameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to overparameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.

Introduction

Differentiable programming is becoming an important paradigm in signal processing and machine learning that consists in building parameterized models, sometimes with a complex architecture and a large number of parameters, and adjusting these parameters in order to minimize a loss function using gradient-based optimization methods. The resulting problem is in general highly non-convex. It has been observed empirically that, for fixed loss and model class, changes in the parameterization, optimization procedure, or initialization could lead to a selection of models with very different properties [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF]. This paper is about one such implicit bias phenomenon, that we call lazy training, which corresponds to the model behaving like its linearization around the initialization. This work is motivated by a series of recent articles [START_REF] Du | Gradient descent provably optimizes over-parameterized neural networks[END_REF][START_REF] Li | Learning overparameterized neural networks via stochastic gradient descent on structured data[END_REF][START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF][START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Zou | Stochastic gradient descent optimizes over-parameterized deep ReLU networks[END_REF] where it is shown that overparameterized neural networks could converge linearly to zero training loss with their parameters hardly varying. With a slightly different approach, it was shown in [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] that infinitely wide neural networks behave like the linearization of the neural network around its initialization. In the present work, we argue that this behavior is not specific to neural networks, and is not so much due to over-parameterization than to an implicit choice of scaling. By introducing an explicit scale factor, we show that essentially any parametric model can be trained in this lazy regime if its output is close to zero at initialization. This shows that guaranteed fast training is indeed often possible, but 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. at the cost of recovering a linear method 1 . Our experiments on two-layer neural networks and deep convolutional neural networks (CNNs) suggest that this behavior is undesirable in practice.

Presentation of lazy training

We consider a parameter space2 R p , a Hilbert space F, a smooth model h : R p → F (such as a neural network) and a smooth loss R : F → R + . We aim to minimize, with gradient-based methods, the objective function F : R p → R + defined as

F (w) := R(h(w)).
With an initialization w 0 ∈ R p , we define the linearized model h(w) = h(w 0 ) + Dh(w 0 )(w -w 0 ) around w 0 , and the corresponding objective F : R p → R + as F (w) := R( h(w)).

It is a general fact that the optimization path of F and F starting from w 0 are close at the beginning of training. We call lazy training the less expected situation where these two paths remain close until the algorithm is stopped.

Showing that a certain non-convex optimization is in the lazy regime opens the way for surprisingly precise results, because linear models are rather well understood. For instance, when R is strongly convex, gradient descent on F with an appropriate step-size converges linearly to a global minimizer [START_REF] Bottou | Optimization methods for large-scale machine learning[END_REF]. For two-layer neural networks, we show in Appendix A.2 that the linearized model is a random feature model [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] which lends itself nicely to statistical analysis [START_REF] Carratino | Learning with SGD and random features[END_REF]. Yet, while advantageous from a theoretical perspective, it is not clear a priori whether this lazy regime is desirable in practice.

This phenomenon is illustrated in Figure 1 where lazy training for a two-layer neural network with rectified linear units (ReLU) is achieved by increasing the variance τ 2 at initialization (see next section). While in panel (a) the ground truth features are identified, this is not the case for lazy training on panel (b) that manages to interpolate the observations with just a small displacement in parameter space (in both cases, near zero training loss was achieved). As seen on panel (c), this behavior hinders good generalization in the teacher-student setting [START_REF] Saad | On-line learning in soft committee machines[END_REF]. The plateau reached for large τ corresponds exactly to the performance of the linearized model, see Section 3.1 for details.

When does lazy training occur?

A general criterion. Let us start with a formal computation. We assume that w 0 is not a minimizer so that F (w 0 ) > 0, and not a critical point so that ∇F (w 0 ) = 0. Consider a gradient descent step w 1 := w 0 -η∇F (w 0 ), with a small stepsize η > 0. On the one hand, the relative change of the objective is

∆(F ) := |F (w1)-F (w0)| F (w0) ≈ η ∇F (w0) 2 F (w0)
. On the other hand, the relative change of the differential of h measured in operator norm is ∆(Dh) := Dh(w1)-Dh(w0) Dh(w0)

≤ η ∇F (w0) • D 2 h(w0) Dh(w0)
. Lazy training refers to the case where the differential of h does not sensibly change while the loss enjoys a significant decrease, i.e., ∆(F ) ∆(Dh). Using the above estimates, this is guaranteed when

∇F (w 0 ) F (w 0 ) D 2 h(w 0 ) Dh(w 0 )
.

For the square loss R(y) = 1 2 y -y 2 for some y ∈ F, this leads to the simpler criterion

κ h (w 0 ) := h(w 0 ) -y D 2 h(w 0 ) Dh(w 0 ) 2 1, (1)
using the approximation

∇F (w 0 ) = Dh(w 0 ) (h(w 0 ) -y ) ≈ Dh(w 0 ) • h(w 0 ) -y .
This quantity κ h (w 0 ) could be called the inverse relative scale of the model h at w 0 . We prove in Theorem 2.3 that it indeed controls how much the training dynamics differs from the linearized training dynamics when R is the square loss 3 . For now, let us explore situations in which lazy training can be shown to occur, by investigating the behavior of κ h (w 0 ).

Rescaled models. Considering a scaling factor α > 0, it holds

κ αh (w 0 ) = 1 α αh(w 0 ) -y D 2 h(w 0 ) Dh(w 0 ) 2 .
Thus, κ αh (w 0 ) simply decreases as α -1 when α grows and αh(w 0 ) -y is bounded, leading to lazy training for large α. Training dynamics for such rescaled models are studied in depth in Section 2. For neural networks, there are various ways to ensure h(w 0 ) = 0, see Section 3.

Homogeneous models.

If h is q-positively homogeneous 4 then multiplying the initialization by λ is equivalent to multiplying the scale factor α by λ q . In equation,

κ h (λw 0 ) = 1 λ q λ q h(w 0 ) -y D 2 h(w 0 ) Dh(w 0 ) 2 .
This formula applies for instance to q-layer neural networks consisting of a cascade of homogenous non-linearities and linear, but not affine, operators.

: (R d ) m → F of the form h m (w) = α(m) m i=1 φ(θ i ),
where α(m) > 0 is a normalization, w = (θ 1 , . . . , θ m ) and φ : R d → F is a smooth function. This setting covers the case of two-layer neural networks (see Appendix A.2). When initializing with independent and identically distributed variables (θ i ) m i=1 satisfying Eφ(θ i ) = 0, and under the assumption that Dφ is not identically 0 on the support of the initialization, we prove in Appendix A.2 that for large m it holds

E[κ hm (w 0 )] m -1 2 + (mα(m)) -1 .
As a consequence, as long as mα(m) → ∞ when m → ∞, such models are bound to reach the lazy regime. In this case, the norm of the initial output becomes negligible in front of the scale as m grows due to the statistical cancellations that follow from the assumption Eφ(θ i ) = 0. In contrast, the critical scaling α(m) = 1/m, allows to converge as m → ∞ to a non degenerate dynamic described by a partial differential equation and referred to as the mean-field limit [START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF][START_REF] Chizat | On the global convergence of gradient descent for overparameterized models using optimal transport[END_REF][START_REF] Rotskoff | Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error[END_REF][START_REF] Sirignano | Mean field analysis of neural networks: A central limit theorem[END_REF].

Content and contributions

The goal of this paper is twofold: (i) understanding in a general optimization setting when lazy training occurs, and (ii) investigating the practical usefulness of models in the lazy regime. It is organized as follows:

• in Section 2, we study the gradient flows for rescaled models αh and prove in various situations that for large α, they are close to gradient flows of the linearized model. When the loss is strongly convex, we also prove that lazy gradient flows converge linearly, either to a global minimizer for over-parameterized models, or to a local minimizer for underparameterized models.

• in Section 3, we use numerical experiments on synthetic cases to illustrate how lazy training differs from other regimes of training (see also Figure 1). Most importantly, we show empirically that CNNs used in practice could be far from the lazy regime, with their performance not exceeding that of some classical linear methods as they become lazy.

Our focus is on general principles and qualitative description.

Updates of the paper.

F α (w) := 1 α 2 R(αh(w)) (2)
are close, when the scaling factor α is large, to those of the scaled objective for the linearized model

Fα (w) := 1 α 2 R(α h(w)), (3)
where h(w) := h(w 0 ) + Dh(w 0 )(w -w 0 ) and w 0 ∈ R p is a fixed initialization. Multiplying the objective by 1/α 2 does not change the minimizers, and corresponds to the proper time parameterization of the dynamics for large α. Our basic assumptions are the following:

Assumption 2.1. The parametric model h : R p → F is differentiable with a locally Lipschitz differential5 Dh. Moreover, R is differentiable with a Lipschitz gradient.

This setting is mostly motivated by supervised learning problems, where one considers a probability distribution ρ ∈ P(R d × R k ) and defines F as the space L 2 (ρ x ; R k ) of square-integrable functions with respect to ρ x , the marginal of ρ on R d . The risk R is then built from a smooth loss function

: (R k ) 2 → R + as R(g) = E (X,Y )∼ρ (g(X), Y
). This corresponds to empirical risk minimization when ρ is a finite discrete measure, and to population risk minimization otherwise (in which case only stochastic gradients are available to algorithms). Finally, one defines h(w) = f (w, •) where f : R p × R d → R k is a parametric model, such as a neural network, which outputs in R k depend on parameters in R p and input data in R d .

Gradient flows. In the rest of this section, we study the gradient flow of the objective function F α which is an approximation of (accelerated) gradient descent [START_REF] Gautschi | Numerical analysis[END_REF][START_REF] Scieur | Aspremont. Integration methods and optimization algorithms[END_REF] and stochastic gradient descent [19, Thm. 2.1] with small enough step sizes. With an initialization w 0 ∈ R p , the gradient flow of F α is the path (w α (t)) t≥0 in the space of parameters R p that satisfies w α (0) = w 0 and solves the ordinary differential equation

w α (t) = -∇F α (w α (t)) = - 1 α Dh(w α (t)) ∇R(αh(w α (t))), (4)
where Dh denotes the adjoint of the differential Dh. We will study this dynamic for itself, and will also compare it to the gradient flow ( wα (t)) t≥0 of Fα that satisfies wα (0) = w 0 and solves

w α (t) = -∇ Fα ( wα (t)) = - 1 α Dh(w 0 ) ∇R(α h( wα (t))). (5)
Note that when h(w 0 ) = 0, the renormalized dynamic w 0 + α( wα (t) -w 0 ) does not depend on α, as it simply follows the gradient flow of w → R(Dh(w 0 )(w -w 0 )) starting from w 0 .

Bounds with a finite time horizon

We start with a general result that confirms that when h(w 0 ) = 0, taking large α leads to lazy training. We do not assume convexity of R. Theorem 2.2 (General lazy training). Assume that h(w 0 ) = 0. Given a fixed time horizon

T > 0, it holds sup t∈[0,T ] w α (t) -w 0 = O(1/α), sup t∈[0,T ] w α (t) -wα (t) = O(1/α 2 ) and sup t∈[0,T ] αh(w α (t)) -α h( wα (t)) = O(1/α).
For supervised machine learning problems, the bound on w α (t) -wα (t) implies that αh(w α (T )) also generalizes like α h( wα (T )) outside of the training set for large α, see Appendix A.3. Note that the generalization behavior of linear models has been widely studied, and is particularly well understood for random feature models [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF], which are recovered when linearizing two layer neural networks, see Appendix A.2. It is possible to track the constants in Theorem 2.2 but they would depend exponentially on the time horizon T . This exponential dependence can however be discarded for the specific case of the square loss, where we recover the scale criterion informally derived in Section 1.2. Theorem 2.3 (Square loss, quantitative). Consider the square loss R(y) = 1 2 y -y 2 for some y ∈ F and assume that for some (potentially small) r > 0, h is Lip(h)-Lipschitz and Dh is Lip(Dh)-Lipschitz on the ball of radius r around w 0 . Then for an iteration number K > 0 and corresponding time

T := K/Lip(h) 2 , it holds αh(w α (T )) -α h( wα (T )) αh(w 0 ) -y ≤ K 2 α Lip(Dh) Lip(h) 2 αh(w 0 ) -y as long as α ≥ K αh(w 0 ) -y /(rLip(h)).
We can make the following observations:

• For the sake of interpretability, we have introduced a quantity K, analogous to an iteration number, that accounts for the fact that the gradient flow needs to be integrated with a stepsize of order 1/Lip(∇F α ) = 1/Lip(h) 2 . For instance, with this step-size, gradient descent at iteration K approximates the gradient flow at time T = K/Lip(h) 2 , see, e.g., [START_REF] Gautschi | Numerical analysis[END_REF][START_REF] Scieur | Aspremont. Integration methods and optimization algorithms[END_REF]. • Laziness only depends on the local properties of h around w 0 . These properties may vary a lot over the parameter space, as is the case for homogeneous functions seen in Section 1.2.

For completeness, similar bounds on w α (T ) -w 0 and w α (T ) -wα (T ) are also provided in Appendix B.2. The drawback of the bounds in this section is the increasing dependency in time, which is removed in the next section. Yet, the relevance of Theorem 2.2 remains because it does not depend on the conditioning of the problem. Although the bound grows as K 2 , it gives an informative estimate for large or ill-conditioned problems, where training is typically stopped much before convergence.

Uniform bounds and convergence in the lazy regime

This section is devoted to uniform bounds in time and convergence results under the assumption that R is strongly convex. In this setting, the function Fα is strictly convex on the affine hyperspace w 0 + ker Dh(w 0 ) ⊥ which contains the linearized gradient flow ( wα (t)) t≥0 , so the latter converges linearly to the unique global minimizer of Fα . In particular, if h(w 0 ) = 0 then this global minimizer does not depend on α and sup t≥0 wα (t) -w 0 = O(1/α). We will see in this part how these properties reflect on the gradient flow w α (t) for large α.

Over-parameterized case.

αh(w α (t)) -y * ≤ √ κ αh(w 0 ) -y * exp(-mσ 2 min t/4). If moreover h(w 0 ) = 0, it holds as α → ∞, sup t≥0 w α (t) -w 0 = O(1/α), sup t≥0 αh(w α (t)) -α h( wα (t)) = O(1/α) and sup t≥0 w α (t) -wα (t) = O(log α/α 2 ).
The proof of this result relies on the fact that αh(w α (t)) follows the gradient flow of R in a timedependent and non degenerate metric: the pushforward metric [START_REF] Lee | Smooth manifolds. In Introduction to Smooth Manifolds[END_REF] induced by h on F. For the first part, we do not claim improvements over [START_REF] Du | Gradient descent provably optimizes over-parameterized neural networks[END_REF][START_REF] Li | Learning overparameterized neural networks via stochastic gradient descent on structured data[END_REF][START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF][START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Zou | Stochastic gradient descent optimizes over-parameterized deep ReLU networks[END_REF], where a lot of effort is also put in dealing with the non-smoothness of h, which we do not study here. As for the uniform in time comparison with the tangent gradient flow, it is new and follows mostly from Lemma B.2 in Appendix B where the constants are given and depend polynomially on the characteristics of the problem.

Under-parameterized case. We now remove the over-parameterization assumption and show again linear convergence for large values of α. This covers in particular the case of population loss minimization, where F is infinite-dimensional. For this setting, we limit ourselves to a qualitative statement 6 . Theorem 2.5 (Under-parameterized lazy training). Assume that F is separable, R is strongly convex, h(w 0 ) = 0 and rank Dh(w) is constant on a neighborhood of w 0 . Then there exists α 0 > 0 such that for all α ≥ α 0 the gradient flow (4) converges at a geometric rate (asymptotically independent of α) to a local minimum of F α .

Thanks to lower-semicontinuity of the rank function, the assumption that the rank is locally constant holds generically, in the sense that it is satisfied on an open dense subset of R p . In this underparameterized case, the limit lim t→∞ w α (t) is for α large enough a strict local minimizer, but in general not a global minimizer of F α because the image of Dh(w 0 ) does not a priori contain the global minimizer of R. Thus it cannot be excluded that there exists parameters w farther from w 0 with a smaller loss. This fact is clearly observed experimentally in Section 3, Figure 2-(b). Finally, a comparison with the linearized gradient flow as in Theorem 2.4 could be shown along the same lines, but would be technically slightly more involved because differential geometry comes into play.

Relationship to the global convergence result in [START_REF] Chizat | On the global convergence of gradient descent for overparameterized models using optimal transport[END_REF]. A consequence of Theorem 2.5 is that in the lazy regime, the gradient flow of the population risk for a two-layer neural network might get stuck in a local minimum. In contrast, it is shown in [START_REF] Chizat | On the global convergence of gradient descent for overparameterized models using optimal transport[END_REF] that such gradient flows converge to global optimality in the infinite over-parameterization limit p → ∞ if initialized with enough diversity in the weights. This is not a contradiction since Theorem 2.5 assumes a finite number p of parameters. In the lazy regime, the population loss might also converge to its minimum when p increases: this is guaranteed if the tangent kernel Dh(w 0 )Dh(w 0 ) [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] converges (after normalization) to a universal kernel as p → ∞. However, this convergence might be unreasonably slow in high-dimension, as Figure 1-(c) suggests. As a side note, we stress that the global convergence result in [START_REF] Chizat | On the global convergence of gradient descent for overparameterized models using optimal transport[END_REF] is not limited to lazy dynamics but also covers non-linear dynamics, such as seen on Figure 1 where neurons move.

Numerical Experiments

We realized two sets of experiments, the first with two-layer neural networks conducted on synthetic data and the second with convolutional neural networks (CNNs) conducted on the CIFAR-10 dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. The code to reproduce these experiments is available online7 .

Two-layer neural networks in the teacher-student setting

We consider the following two-layer student neural network h m (w) = f m (w, •) with f m (w, x) = m j=1 a j max(b j • x, 0) where a j ∈ R and b j ∈ R d for j = 1, . . . , m. It is trained to minimize the square loss with respect to the output of a two-layer teacher neural network with same architecture and m 0 = 3 hidden neurons, with random weights normalized so that a j b j = 1 for j ∈ {1, 2, 3}. To obtain a 2-D representation, we plot |a j (t)|b j (t) throughout training (lines) and at convergence (dots) for j ∈ {1, . . . , m}. The blue or red colors stand for the signs of a j (t) and the unit circle is displayed to help visualizing the change of scale. On panel (c), we set n = 1000, m = 50 with symmetrized initialization and report the average and standard deviation of the test loss over 10 experiments. To ensure that the bad performances corresponding to large τ are not due to a lack of regularization, we display also the best test error throughout training (for kernel methods, early stopping is a form of regularization [START_REF] Yao | On early stopping in gradient descent learning[END_REF]).

Increasing number of parameters. Figure 2-(a) shows the evolution of the test error when increasing m as discussed in Section 1.2, without symmetrized initialization. We report the results for two choices of scaling functions α(m), averaged over 5 experiments with d = 100. The scaling 1/ √ m leads to lazy training, with a poor generalization as m increases, in contrast to the scaling 1/m for which the test error remains relatively close to 0 for large m (more experiments with this scaling can be found in [START_REF] Chizat | On the global convergence of gradient descent for overparameterized models using optimal transport[END_REF][START_REF] Rotskoff | Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error[END_REF][START_REF] Mei | A mean field view of the landscape of two-layer neural networks[END_REF]).

Under-parameterized case. Finally, Figure 2-(b) illustrates the under-parameterized case, with d = 100, m = 50 with symmetrized initialization. We used SGD with batch-size 200 to minimize the population square loss, and displayed average and standard deviation of the final population loss (estimated with 2000 samples) over 5 experiments. As shown in Theorem 2.5, SGD converges to a a priori local minimum in the lazy regime (i.e., here for large τ ). In contrast, it behaves well when τ is small, as in Figure 1. There is also an intermediate regime (hatched area) where convergence is very slow and the loss was still decreasing when the algorithm was stopped.

Deep CNNs experiments

We now study whether lazy training is relevant to understand the good performances of convolutional neural networks (CNNs).

Interpolating from standard to lazy training. We first study the effect of increasing the scale factor α on a standard pipeline for image classification on the CIFAR10 dataset. We consider the VGG-11 model [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], which is a widely used model on CIFAR10. We trained it via mini-batch SGD with a momentum parameter of 0.9. For the sake of interpretability, no extra regularization (e.g., BatchNorm) is incorporated, since a simple framework that outperforms linear methods baselines with some margin is sufficient to our purpose (see Figure 3(b)). An initial learning rate η 0 is linearly decayed at each epoch, following η t = η0 1+βt . The biases are initialized with 0 and all other weights are initialized with normal Xavier initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. In order to set the initial output to 0 we use the centered model h, which consists in replacing the VGG model h by h(w) := h(w) -h(w 0 ). Notice that this does not modify the differential at initialization. The model h is trained for the square loss multiplied by 1/α 2 (as in Section 2), with standard data-augmentation, batch-size of 128 [START_REF] Zagoruyko | Wide residual networks[END_REF] and η 0 = 1 which gives the best test accuracies over the grid 10 k , k ∈ {-3, 3}, for all α. The total number of epochs is 70, adjusted so that the performance reaches a plateau for α = 1. Figure 3(a) reports the accuracy after training αh for increasing values of α ∈ 10 k for k = {0, 1, 2, 3, 4, 5, 6, 7} (α = 1 being the standard setting). For α < 1, the training loss diverges with η 0 = 1. We also report the stability of activations, which is the share of neurons over ReLU layers that, after training, are activated for the same inputs than at initialization, see Appendix C. Values close to 100% are strong indicators of an effective linearization.

We observe a significant drop in performance as α grows, and then the accuracy reaches a plateau, suggesting that the CNN progressively reaches the lazy regime. This demonstrates that the linearized model (large α) is not sufficient to explain the good performance of the model for α = 1. For large α, we obtain a low limit training accuracy and do not observe overfitting, a surprising fact since this amounts to solving an over-parameterized linear system. This behavior is due to a poorly conditioned linearized model, see Appendix C.

Performance of linearized CNNs. In this second set of experiments, we investigate whether variations of the models trained above in a lazy regime could increase the performance and, in particular, could outperform other linear methods which also do not involve learning a representation [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF][START_REF] Oyallon | Deep roto-translation scattering for object classification[END_REF]. To this end, we train widened CNNs in the lazy regime, as widening is a well-known strategy to boost performances of a given architecture [START_REF] Zagoruyko | Wide residual networks[END_REF]. We multiply the number of channels of each layer by 8 for the VGG model and 7 for the ResNet model [START_REF] He | Deep residual learning for image recognition[END_REF] (these values are limited by hardware constraints). We choose α = 10 7 to train the linearized models, a batch-size of 8 and, after cross-validation, η 0 = 0.01, 1.0 for respectively the standard and linearized model. We also multiply the initial weights by respectively 1.2 and 1.3 for the ResNet-18 and VGG-11, as we found that it slightly boosts the training accuracies. Each model is trained with the cross-entropy loss divided by α 2 until the test accuracy stabilizes or increases, and we check that the average stability of activations (see Appendix C) was 100%.

As seen on Figure 3(b), widening the VGG model slightly improves the performances of the linearized model compared to the previous experiment but there is still a substantial gap of performances from other non-learned representations [START_REF] Recht | Do ImageNet classifiers generalize to ImageNet[END_REF][START_REF] Oyallon | Deep roto-translation scattering for object classification[END_REF] methods, not to mention the even wider gap with their non-lazy counterparts. This behavior is also observed on the state-of-the-art ResNet architecture. Note that [START_REF] Arora | On exact computation with an infinitely wide neural net[END_REF] reports a test accuracy of 77.4% without data augmentation for a linearized CNN with a specially designed architecture which in particular solves the issue of ill-conditioning. Whether Remark on wide NNs. It was proved [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] that neural networks with standard initialization (random independent weights with zero mean and variance O(1/n ) at layer , where n is the size of the previous layer), are bound to reach the lazy regime as the sizes of all layers grow unbounded. Moreover, for very large neural networks of more than 2 layers, this choice of initialization is essentially mandatory to avoid exploding or vanishing initial gradients [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF][START_REF] Hanin | How to start training: The effect of initialization and architecture[END_REF] if the weights are independent with zero mean. Thus we stress that we do not claim that wide neural networks do not show a lazy behavior, but rather that those which exhibit good performances are far from this asymptotic behavior.

Discussion

Lazy training is an implicit bias phenomenon in differentiable programming, that refers to the situation when a non-linear parametric model behaves like a linear one. This arises when the scale of the model becomes large which, we have shown, happens implicitly under some choices of hyper-parameters governing normalization, initialization and number of iterations. While the lazy training regime provides some of the first optimization-related theoretical insights for deep neural networks [START_REF] Du | Gradient descent finds global minima of deep neural networks[END_REF][START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Zou | Stochastic gradient descent optimizes over-parameterized deep ReLU networks[END_REF][START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF], we believe that it does not explain yet the many successes of neural networks that have been observed in various challenging, high-dimensional tasks in machine learning. This is corroborated by numerical experiments where it is seen that the performance of networks trained in the lazy regime degrades and in particular does not exceed that of some classical linear methods. Instead, the intriguing phenomenon that still defies theoretical understanding is the one illustrated on Figure 1(c) for small τ and on Figure 3(a) for α = 1: neural networks trained with gradient-based methods (and neurons that move) have the ability to perform high-dimensional feature selection through highly non-linear dynamics.

Justification for asymptotics. In this paragraph, we justify the formula for the asymptotic upper bound on κ hm (w 0 ) given for such models in Section 1.2. Using the assumption that Eφ(θ i ) = 0 and the fact that the parameters are independents, one has E h(w 0 ) 2 = mα(m) 2 E φ(θ) 2 . For the differential, from the law of large numbers, we have the estimate

1 mα(m) 2 Dh(w 0 ) Dh(w 0 ) = 1 m m i=1 Dφ(θ i )Dφ(θ i ) -→ m→∞ E [Dφ(θ)Dφ(θ) ] .
It follows that E Dh(w 0 ) 2 = E Dh(w 0 )Dh(w 0 ) ∼ mα(m) 2 E[Dφ(θ)Dφ(θ) ] because we have assumed that Dφ is not identically 0 on the support of θ. One also has

D 2 h(w 0 ) = sup u∈R d×m u ≤1 α(m) m i=1 u i D 2 φ(θ i )u i ≤ α(m) sup θi D 2 φ(θ i ) ≤ α(m)Lip(Dφ).
From the definition of κ hm (w 0 ) and the upper bound

h m (w 0 )-y ≤ h(w 0 ) + y we conclude that E[κ hm (w 0 )] m -1 2 + (mα(m)) -1 .
Limit kernels and random feature. In this section, we show that the tangent kernel is a random feature kernel for neural networks with a single hidden layer. For simplicity, we consider the scaling α(m) = 1/ √ m as in [START_REF] Du | Gradient descent provably optimizes over-parameterized neural networks[END_REF] which leads to a non-degenerated limit of the kernel 9 as m → ∞. The associated tangent kernel in Eq. ( 7) is the sum of two kernels

K m (x, x ) = K (a) m (x, x )+K (b) m (x, x ),
one for each layer, where

K (a) m (x, x ) = 1 m m j=1 (x • x )b 2 j σ (a j • x)σ (a j • x ) and K (b) m (x, x ) = 1 m m j=1 σ(a j • x)σ(a j • x ).
If we assume that the initial weights a j (resp. b j ) are independent samples of a distribution on R d (resp. a distribution on R), these are random feature kernels [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] that converge as m → ∞ to the kernels

K (a) (x, x ) = E (a,b) (x • x )b 2 σ (a • x)σ (a • x ) and K (b) (x, x ) = E a [σ(a • x)σ(a • x )] .
The second component K (b) , corresponding to the differential with respect to the output layer, is the one traditionally used to make the link between these networks and random features [START_REF] Rahimi | Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning[END_REF]. When σ(s) = max{s, 0} is the rectified linear unit activation and the distribution of the weights a j is rotation invariant in R d , one has the following explicit formulae [START_REF] Cho | Kernel methods for deep learning[END_REF]:

K (a) (x, x ) = (x • x )E(b 2 ) 2π (π -ϕ), K (b) (x, x ) = x x E( a 2 ) 2πd ((π -ϕ) cos ϕ + sin ϕ) (8)
where ϕ ∈ [0, π] is the angle between the two vectors x and x . See Figure 4 for an illustration of this kernel and the convergence of its random approximations. The link with (independent) random sampling is lost for deeper neural networks, but it is shown in [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] that tangent kernels still converge when the size of networks increase, for certain architectures.

A.3 Generalization for the lazy model

As noted in the main text, in supervised machine learning, F is often a Hilbert space of functions on R d and the model h is often of the form h(w) = f (w, •) where f : R p × R d → R k . A natural question that arises in this context and that is not directly answered by the theorems of Section 2, is whether the trained lazy model and the trained tangent model also generalize the same way, i.e. whether at training time T , it holds f (w(T ), x) ≈ f ( w(T ), x) for points x ∈ R d that are not in the training set, where f (w, x) = f (w 0 , x) + D w f (w 0 , x)(w -w 0 ). We will see here that it is actually a simple consequence of the bounds.

Figure 4: Random realizations of the kernels K m and the limit kernel K of Eq. ( 8). We display the value of K(x, x ) as a function of ϕ = angle(x, x ) with x fixed, on a section of the sphere in R 10 . Parameters are normal random variables of variance 1, so E(b 2 ) = 1 and E( a 2 ) = d.

Proposition A.1 (Generalizing like the tangent model). Assume that for some C > 0 it holds w α (T ) -w(T ) ≤ C log(α)/α 2 . Assume moreover that there exists a set X ⊂ R d such that

M 1 := sup x∈X D w f (w 0 , x) < ∞ and M 2 := sup x∈X Lip(w → D w f (w, x)) < ∞. Then it holds sup x∈X αf (w α (T ), x) -α f ( wα (T ), x) ≤ C log α α M 1 + 1 2 C • M 2 • log(α) -→ α→∞ 0.
Proof. Let us call A the quantity to be upper bounded, and start with the decomposition

A ≤ sup x∈X αf (w α (T ), x) -α f (w α (T ), x) + sup x∈X α f (w α (T ), x) -α f ( wα (T ), x) = A 1 + A 2
By Taylor's theorem applied at each point x ∈ X, one has

A 1 ≤ α 2 M 2 w α (T ) -wα (T ) 2 ≤ C 2 • M 2 log(α) 2 2α .
It also holds

A 2 = α sup x∈X D w f (w 0 , x)(w α (T ) -wα (T )) ≤ M 1 C log(α)
α and the conclusion follows.

B Proofs of the theoretical results

In all the forthcoming proofs, we use the notations y(t) = αh(w α (t)) and ȳ(t) = α h( wα (t)) for the dynamics in F (they also depend on α although this is not reflected in the notation). We also write Σ(w) := Dh(w)Dh(w) for the so-called tangent kernel [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF], which is a quadratic form on F. By using the chain rule, we find that the trajectories in F solve the differential equation

d dt y(t) = -Σ(w α (t))∇R(y(t)), d dt ȳ(t) = -Σ(w(0))∇R(ȳ(t)).
with y(0) = ȳ(0) = αh(w 0 ). Remark that the first differential equation is coupled with w α (t).

B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative)

For this first proof, we only track the dependency in α, and we use C to denote a quantity independent of α, that may vary from line to line. For T > 0, it holds

T 0 w α (t) dt = T 0 ∇F α (w α (t)) dt ≤ √ T T 0 ∇F α (w α (t)) 2 dt 1 2
.

It follows, by using the fact that d dt F α (w α (t)) = -∇F α (w α (t)) 2 , that sup t∈[0,T ] w α (t)w(0) ≤ (T • F α (w α (t)))

1 2 1 α . In particular, we deduce that sup t∈[0,T ] y(t) -y(0) ≤ C and sup t∈[0,T ] ∇R(y(t)) ≤ C.

Let us now consider the evolution of ∆(t) := y(t) -ȳ(t) . It satisfies ∆(0) = 0 and ∆ (t) ≤ Σ(w α (t))∇R(y(t)) -Σ(w(0))∇R(ȳ(t))

≤ (Σ(w α (t)) -Σ(w(0)))∇R(y(t)) + Σ(w(0))(∇R(y(t)) -∇R(ȳ(t))

≤ C 1 /α + C 2 ∆(t)
The ordinary differential equation u (t) = C 1 /α + C 2 u(t) with initial condition u(0) = 0 admits the unique solution u(t) = C1 αC2 (exp(C 2 t) -1). Since ∆(t) is a sub-solution of this system, it follows that ∆(t) ≤ C1 αC2 (exp(C 2 t) -1) ≤ C/α (notice the exponential dependence in the final time and some other characteristics of the problem). Finally, consider the quantity δ(t) = w α (t) -wα (t) . It holds

δ (t) ≤ α -1 Dh(w α (t)) ∇R(f (t)) -Dh(w 0 ) ∇R(ȳ(t)) ≤ α -1 Dh(w α (t)) -Dh(w 0 ) ∇R(y(t)) + α -1 Dh(w 0 ) ∇R(y) -∇R(ȳ(t)) ≤ Cα -2
We thus conclude, since δ(0) = 0, that sup t∈[0,T ] δ(t) ≤ α -2 .

B.2 Proof of Theorem 2.3 (finite horizon, square loss)

Step 1. With the square loss, the objective is still potentially non-convex, but we have the property

d dt y(t) -y * 2 = -Σ(w(t))(y(t) -y ), y(t) -y ≤ 0.
The proof scheme is otherwise similar as above, but we carry all constants. Let us denote T exit = inf{t > 0 ; w α (t) -w 0 > r}. For t ≤ T exit it holds w α (t) = ∇F α (w α (t)) ≤ α -1 y(t) -y Dh(w α (t)) ≤ α -1 y(0) -y Lip(h)

It follows that w α (t) -w(0) ≤ tα -1 y(0) -y Lip(h) (this bound is tighter for small times, compared to the bound in √ t used in the previous proof). Since we have assumed that α ≥ k y(0) -y /(rLip(h)), it holds w α (t) -w 0 ≤ (t/K) • rLip(h) 2 = r so T exit > T .

Step 2. Now we consider ∆(t) = y(t) -ȳ(t) . It holds

1 2 d dt ∆(t) 2 = y (t) -ȳ (t), y(t) -ȳ(t) ≤ -Σ(w α (t))∇R(y(t)) -Σ(w(0))∇R(ȳ(t)), y(t) -ȳ(t) ≤ -(Σ(w α (t)) -Σ(w(0)))∇R(y(t)), y(t) -ȳ(t)
where we have used the fact that Σ(w(0))(∇R(y(t)) -∇R(ȳ(t)), y(t) -ȳ(t) ≥ 0, which is specific to the square loss. Taking the norms and dividing both sides by ∆(t), it follows

∆ (t) ≤ Lip(Σ) • w α (t) -w(0) y(0) -y ≤ 2Lip(h) 2 Lip(Dh)tα -1 y(0) -y 2
where we have used Lip(Σ) ≤ 2Lip(h)Lip(Dh). Since ∆(0) = 0, it follows

∆(t) ≤ t 2 α Lip(h) 2 Lip(Dh) y(0) -y 2 .
The bound in the statement then follows by writing this upper bound at time T = K/Lip(h) 2 .

Step 3. Finally, consider δ(t) = w α (t) -wα (t) . The bound that we will obtain is not reported in the main text due to space constraints, but proved here for the sake of completeness. As in the previous proof, it holds αδ (t) ≤ Dh(w α (t)) -Dh(w 0 ) ∇R(y(t)) + Dh(w 0 ) ∇R(y)-∇R(ȳ(t)) = A(t)+B(t).

• •

Figure 5: There is a small neighborhood W 0 ⊂ R p of the initialization w 0 , which image by h is a differentiable manifold in F. In the lazy regime, the optimization paths (both in W and in F) for the non-linear model h (dashed gray paths) are close to those of the linearized model h (dashed black paths) until convergence or stopping time (Section 2). This figure illustrates the under-parameterized case where p < dim(F).

B.5 Proof of Theorem 2.5 (under-parameterized case)

The setting of this theorem is depicted on Figure 5. By the rank theorem (a result of differential geometry, see [START_REF] Lee | Smooth manifolds. In Introduction to Smooth Manifolds[END_REF]Thm. 4.12] or [START_REF] Abraham | Manifolds, Tensor Analysis, and Applications[END_REF] for a statement in separable Hilbert spaces), there exists open sets W 0 , W0 ⊂ R p and F 0 , F0 ⊂ F and diffeomorphisms ϕ : W 0 → W0 and ψ :

F 0 → F0 such that ϕ(w 0 ) = 0, ψ(h(w 0 )) = 0 and ψ • h • ϕ -1 = π r
, where π r is the map that writes, in suitable bases, (x 1 , . . . , x p ) → (x 1 , . . . , x r , 0, . . . ). Up to restricting these domains, we may assume that F0 is convex. We also denote by Π r the r-dimensional hyperplan in F that is spanned by the first r vectors of the basis. The situation is is summarized in the following commutative diagram:

W 0 F 0 W0 F0 ϕ h π r ψ
In the rest of the proof, we denote by C > 0 any quantity that depends on m, M and Lipschitz smoothness constants of h, ψ, ϕ, ψ -1 , ϕ -1 , but not on α. Although we do not do so, this could be translated into explicit constants that depends on the smoothness of h and R, on the strong convexity constant of R and on the smallest positive singular value of Dh(w 0 ) using quantitative versions of the rank theorem [5, Thm. 2.7].

Step 1. Our proof is along the same lines as that of Theorem 2.4, but performed in Π r which can be thought of as a straighten up version of h(W 0 ). Consider the function G α defined for g ∈ F0 as G α (g) = R(αψ -1 (g))/α 2 . The gradient and Hessian of G α satisfy, for

v 1 , v 2 ∈ R p , ∇G α (g) = 1 α (Dψ(g) -1 ) ∇R(αψ -1 (g)), D 2 G α (g)(v 1 , v 2 ) = v 1 (Dψ(g) -1 ) ∇ 2 R(αψ -1 (g))Dψ(g) -1 v 2 + 1 α D 2 ψ(g) -1 (v 1 , v 2 ) ∇R(αψ -1 (g)).
The second order derivative of G α is the sum of a first term with eigenvalues in an interval [C -1 , C], and a second term that goes to 0 as α increases. It follows that G α is smooth and strongly convex for α large enough. Note that if R or ψ -1 are not twice continuously differentiable, then the Hessian computations should be understood in the distributional sense (this is sufficient because the functions involved are Lipschitz smooth). Also, let g * be a minimizer of the lower-semicontinuous closure of G α on the closure of F0 . By strong convexity of R and our assumptions, it holds

g * 2 ≤ 2 m (G α (0) -G α (g * )) ≤ 2R(0) α 2 m ,
so g * is in the interior of F0 for α large enough and is then the unique minimizer of G α .

Step 2. Now consider T := inf{t ≥ 0 ; w α (t) / ∈ W 0 }. For t ∈ [0, T ), the trajectory w α (t) of the gradient flow (4) has "mirror" trajectories in the four spaces in the diagram above. Let us look more particularly at g(t) := π r • ϕ(w α (t)) = ψ • h(w α (t)) for t < T . In the following computation, we write Dϕ for the value of the differential at the corresponding point of the dynamic Dϕ(w α (t)) (and similarly for other differentials). By noticing that Dh = Dψ -1 Dπ r Dϕ, we have

g (t) = - 1 α DψDhDh ∇R(αψ -1 (g(t)) = - 1 α Dπ r DϕDϕ Dπ r (Dψ -1 ) ∇R(αψ -1 (g(t)).
so g(t) remains in Π r . Also, the first r × r block of Dπ r DϕDϕ Dπ r is positive definite on Π r , with a positive lower bound (up to taking W 0 and F 0 smaller if necessary). Thus by Lemma B.1, there are constants

C 1 , C 2 > 0 independent of α such that, for t ∈ [0, T ), g(t) -g * ≤ C 1 g(0) - g * exp (-C 2 t) .
Step 3. Now we want to show that T = +∞ for α large enough. It holds Thus, by choosing α large enough, one has w α (t) ∈ W 0 for all t ≥ 0, so T = ∞ and the theorem follows.

w (t) = - 1
C Experimental details and additional results

C.1 Many neurons dynamics visualized

The setting of Figure 6 is the same as for panels (a)-(b) in Figure 1 except that m = 200, n = 200: it allows to visualize behavior of the training dynamics for a larger number of neurons. Symmetrized initialization to set f (w 0 , •) = 0 was used on panel (c) but not on panel (b), where we see that the neurons need to move slightly more in order to compensate for the non-zero initialization. As on Figure 1, we observe a good behavior in the non-lazy regime for small τ .

C.2 Stability of activations

We define here the "stability of activations" mentioned in Section 3.2. We consider a ReLU layer of size n in a neural network and the test input data (x i ) N i=1 (the test images of CIFAR10 in our case). We call z ij (T ) ∈ R the value of the pre-activation (i.e. the value that goes through the ReLU function as an input) of index j on the data sample i, obtained with the parameters of the network at epoch T . The "stability of activations" for this layer is defined as s := Q n ×N where L is the number of ReLU layers, Q is the number of indices (i, j) that satisfy sign(z ij (T last )) = sign(z ij (T init )) for i ∈ {1, . . . , B} and j ∈ {1, . . . , n }, where T init refers to initialization and T last to the end of training. The quantity that we report on Figure 3(a) is the average of s over all ReLU layers of the VGG-11 network, for various values of α.

C.3 Spectrum of the tangent kernel

In the setting of Figure 3(a), we want to understand why the linearized model (that is, trained for large α) could not reach low training accuracies in spite of being highly over-parameterized. Figure 7(a) shows the train and test losses after 70 epochs where we see that the training loss is far from 0 for all α ≥ 10. We report on Figure 7(b) the normalized and sorted eigenvalues σ 2 i of the tangent kernel Dh(w 0 )Dh(w 0 ) (notice the log-log scale) evaluated for two distinct input data sets (x i ) n i=1 of size n = 500: (i) images randomly sampled from the training set of CIFAR10 and (ii) images with uniform random pixel values. Since there are 10 output channels, the corresponding space F has 10 × 500 dimensions. We observe that there is a gap of 1 order of magnitude between the 0.2% largest eigenvalues and the remaining ones-which causes the ill conditionning-and then a decrease of order approximately O(1/i). We observe a similar pattern with the CIFAR10 inputs and completely random inputs, which suggests that this conditioning is intrinsic to the linearized VGG model. Note that modifying the neural network architecture to improve this conditioning, or using optimization methods that are better adapted to ill-conditionned models, is beyond the scope of the present paper.

Figure 1 :

 1 Figure 1: Training a two-layer ReLU neural network initialized with normal random weights of variance τ 2 : lazy training occurs when τ is large. (a)-(b) Trajectory of weights during gradient descent in 2-D (color shows sign of output layer). (c) Generalization in 100-D: it worsens as τ increases. The ground truth is generated with 3 neurons (arrows in (a)-(b)). Details in Section 3.

Figure 2 :

 2 Figure 2: (a) Test loss at convergence for gradient descent, when α depends on m as α = 1/m or α = 1/ √ m, the latter leading to lazy training for large m (not symmetrized). (b) Population loss at convergence versus τ for SGD with a random N(0, τ 2 ) initialization (symmetrized). In the hatched area the loss was still slowly decreasing.

  For the student network, we use random Gaussian weights, except when symmetrized initialization is mentioned, in which case we use random Gaussian weights for j ≤ m/2 and set for j > m/2, b j = b j-m/2 and a j = -a j-m/2 . This amounts to training a model of the form h(w a , w b ) = h m/2 (w a ) -h m/2 (w b ) with w a (0) = w b (0) and guaranties zero output at initialization. The training data are n input points uniformly sampled on the unit sphere in R d and we minimize the empirical risk, except for Figure 2b(b) where we directly minimize the population risk with Stochastic Gradient Descent (SGD). Cover illustration. Let us detail the setting of Figure 1 in Section 1. Panels (a)-(b) show gradient descent dynamics with n = 15, m = 20 with symmetrized initialization (illustrations with more neurons can be found in Appendix C).

Figure 3 :

 3 Figure 3: (a) Accuracies on CIFAR10 as a function of the scaling α. The stability of activations suggest a linearized regime when high. (b) Accuracies on CIFAR10 obtained for α = 1 (standard, non-linear) and α = 10 7 (linearized) compared to those reported for some linear methods without data augmentation: random features and prior features based on the scattering transform.

  α Dh ∇R(αh(w α (t)) = Dϕ Dπ r ∇G α (g(t))and, by Lipschitz-smoothness of G α (Step 1), ∇G α (g(t)) ≤ C α g(t) -g * hence w α (t) -w 0 ≤

Figure 6 :

 6 Figure6: Training a two-layer ReLU neural network initialized with normal random weights of variance τ 2 , as in Figure1, but with more neurons. In this 2-homogeneous setting, changing τ 2 is equivalent to changing α by the same amount so lazy training occurs for large τ .

Figure 7 :

 7 Figure 7: (a) End-of training train and test loss. (b) Spectrum of the tangent kernel Dh(w 0 )Dh(w 0 ) for the VGG11 model on two data sets.

  This article is an expanded version of "A Note on Lazy Training in Supervised Differential Programming" that appeared online in December 2018. Compared to the first version, it has been complemented with finite horizon bounds in Section 2.2 and numerical experiments on CNNs in Section 3.2 while the rest of the material has just been slightly reorganized.

	2 Analysis of Lazy Training Dynamics
	2.1 Theoretical setting
	Our goal in this section is to show that lazy training dynamics for the scaled objective

  The following proposition shows global convergence of lazy training under the condition that Dh(w 0 ) is surjective. As rank Dh(w 0 ) gives the number of effective parameters or degrees of freedom of the model around w 0 , this over-parameterization assumption guarantees that any model around h(w 0 ) can be fitted. Of course, this can only happen if F is finite-dimensional. Theorem 2.4 (Over-parameterized lazy training). Consider a M -smooth and m-strongly convex loss R with minimizer y and condition number κ := M/m. Assume that σ min , the smallest singular value of Dh(w 0 ) is positive and that the initialization satisfies h(w 0 ) ≤ C 0 := σ 3 min /(32κ 3/2 Dh(w 0 ) Lip(Dh)) where Lip(Dh) is the Lipschitz constant of Dh. If α > y * /C 0 , then for t ≥ 0, it holds

Here we mean a prediction function linearly parameterized by a potentially infinite-dimensional vector.

Our arguments could be generalized to the case where the parameter space is a Riemannian manifold.

Note that lazy training could occur even when κ h (w0) is large, i.e. Eq. (1) only gives a sufficient condition.

That is, for q ≥ 1, it holds h(λw) = λ q h(w) for all λ > 0 and w ∈ R p .

Dh(w) is a continuous linear map from R p to F. The Lipschitz constant of Dh : w → Dh(w) is defined with respect to the operator norm. When F has a finite dimension, Dh(w) can be identified with the Jacobian matrix of h at w.

In contrast to the finite horizon bound of Theorem 2.3, quantitative statements would here involve the smallest positive singular value of Dh(w0), which is anyways hard to control.

https://github.com/edouardoyallon/lazy-training-CNN

We have omitted the bias/intercept, which is recovered by fixing the last coordinate of x to 1.

Since the definition of gradients depends on the choice of a metric, this scaling is not of intrinsic importance. Rather, it reflects that we work with the Euclidean metric on R p . The choice of scaling however becomes important when dealing with training (see also discussion in Section 1.2).

Acknowledgments

We acknowledge supports from grants from Région Ile-de-France and the European Research Council (grant SEQUOIA 724063). Edouard Oyallon was supported by a GPU donation from NVIDIA. We thank Alberto Bietti for interesting discussions and Brett Bernstein for noticing an error in a previous version of this paper.

Supplementary material

Supplementary material for the paper: "On Lazy Training in Differentiable Programming" authored by Lénaïc Chizat , Edouard Oyallon and Francis Bach (NeurIPS 2019). This supplementary material is organized as follows:

• Appendix A: Remarks on the linearized model • Appendix B: Proofs of the theoretical results • Appendix C: Experimental details and additional results

A The linearized model in supervised machine learning A.1 Differentiable models and their linearization

In this section, we give some details on the interpretation of the linearized model in the case of supervised machine learning. In this setting, a differentiable model is a typically a function f : R p × R d → R k where R p is the parameter space, R d is the input space and R k the output space. One defines a Hilbert space F of functions from R d to R k , typically L 2 (ρ x , R k ) where ρ x is the distribution of input samples. The function h : R p → F considered in the article is then the function which to a vector of parameters associates a predictor h : w → f (w, •).

In first order approximation around the initial parameters w 0 ∈ R p , the parametric model f (w, x) reduces to the following linearized or tangent model :

where D w f is the differential of f in the variable w. The corresponding hypothesis class is affine in the space of predictors. It should be stressed that when f is a neural network, f is generally not a linear neural network because it is not linear in x ∈ R d , but in the features D w f (w 0 , x) ∈ R p×k which generally depend non-linearly on x. For large neural networks, the dimension of the features might be much larger than d, which makes f similar to a non-parametric method. Finally, if f is already a linear model, then f and f are identical.

Kernel method with an offset. In the case of the square loss, training the affine model ( 6) is equivalent to training a linear model in the variables (x, ỹ) := (D w f (w 0 , x), y -f (w 0 , x)).

When k = 1, this is equivalent to a kernel method with the tangent kernel [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] defined as K :

This kernel is different from the one traditionally associated to neural networks [START_REF] Rahimi | Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning[END_REF][START_REF] Daniely | Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity[END_REF] which involve the derivative with respect to the output layer only. Also, the output data is shifted by the initialization of the model h(w 0 ) = f (w 0 , •). This term inherits from the randomness due to the initialization: it is for instance shown in [START_REF] Lee | Deep neural networks as Gaussian processes[END_REF][START_REF] De | Gaussian process behaviour in wide deep neural networks[END_REF] that the distribution of h(w 0 ) converges to a Gaussian process for certain over-parameterized neural networks initialized with random normal weights.

A.2 Two-layer neural networks

Lazy training has some interesting consequences when looking more particularly at two-layer neural networks. These are functions of the form

where m ∈ N is the size of the hidden layer and σ : R → R is an activation function and the parameters 8 are (θ j ) m j=1 where θ j = (a j , b j ) ∈ R d+1 , so here the number of parameters is p = m(d + 1). We have also introduced a scaling α(m) > 0 as in Section 1.2.

Let us bound these two quantities separately. On the one hand, it holds for t ∈ [0, T ],

On the other hand, it holds for t ∈ [0, T ],

By integrating these two bounds and summing, we get

After rearranging the terms, we obtain

Note that this bound is arranged so that both sides of the inequality are dimensionless, in the sense that they would not change under a simple rescaling of either the norm on F or on R p . The left-hand side should be understood as the relative difference between the non-linear and the linearized dynamics, while the right-hand side involves the scale of Section 1.2.

B.3 Proof of Theorem 2.4 (over-parameterized case)

Consider the radius r 0 := σ min /(2Lip(Dh)). By smoothness of h, it holds Σ(w) σ 2 min Id/4 as long as w -w 0 < r 0 . Thus Lemma B.1 below guarantees that y(t) converges linearly, up to time T := inf{t ≥ 0 ; w α (t) -w 0 > r 0 }. It only remains to find conditions on α so that T = +∞. The variation of the parameters w α (t) can be bounded for 0 ≤ t ≤ T as

By Lemma B.1, it follows that for 0 ≤ t ≤ T ,

This quantity is smaller than r 0 , and thus T = ∞, if y(0) -y * ≤ 2αC 0 . This is in particular guaranteed by the conditions on h(w 0 ) and α in the theorem.

When h(w 0 ) = 0, the previous bound also implies the "laziness" property sup t≥0 w α (t) -w 0 = O(1/α) since in that case y(0) does not depend on α. For the comparison with the tangent gradient flow, the first bound is obtained by applying the stability Lemma B.2, and noticing that the quantity denoted by K in that lemma is in O(1/α) thanks to the previous bound on w α (t) -w 0 . For the last bound, we compute the integral over [0, +∞) of the bound

It is easy to see from the derivations above that the integral of the first term is in O(1/α). For the second term, we define t 0 := 4 log α/(µσ 2 min ) and on [0, t 0 ] we use the smoothness bound ∇R(y(t)) -∇R(ȳ(t)) ≤ M y(t) -ȳ(t) which integral over [0, t 0 ] is in O(log α/α), while on [t 0 , +∞) we use the crude bound

∇R(y(t)) -∇R(ȳ(t)) ≤ ∇R(y(t)) + ∇R(ȳ(t))

which integral over [t 0 , +∞) is in O(1/α) thanks to the definition of t 0 and the exponential decrease of ∇R along both trajectories. This is sufficient to conclude. As a side note, we remark that the assumption that Dh is globally Lipschitz could be avoided by considering the more technical definition Lip(Dh) := inf L > 0 ; Dh is L-Lipschitz on a ball centered at w 0 of radius σ min 2L > 0, because then the path w α (t) never escapes the ball of radius σmin 2L around w 0 for α > y * /C 0 . Lemma B.1 (Strongly-convex gradient flow in a time-dependent metric). Let F : F → R be a m-strongly-convex function with M -Lipschitz continuous gradient and with global minimizer y * and let Σ(t) : F → F be a time dependent continuous self-adjoint linear operator with eigenvalues lower bounded by λ > 0 for 0 ≤ t ≤ T . Then solutions on [0, T ] to the differential equation

B.4 Stability Lemma

The following stability lemma is at the basis of the equivalence between lazy training and linearized model training in Theorem 2.4. We limit ourselves to a rough estimate sufficient for our purposes. Lemma B.2. Let R : F → R + be a m-strongly convex function and let Σ(t) be a time dependent positive definite operator on F such that Σ(t) λId for t ≥ 0. Consider the paths y(t) and ȳ(t) on F that solve for t ≥ 0, y (t) = -Σ(t)∇R(y(t)) and ȳ (t) = -Σ(0)∇R(ȳ(t)).

Defining K := sup t≥0 (Σ(t) -Σ(0))∇R(y(t)) , it holds for t ≥ 0, Since the function z → R(Σ 1/2 0 z) is λm-strongly convex, one has that A(t) ≤ -2λmh(t). Using the quantity K introduced in the statement, one has also B(t) ≤ K z(t) -z(t) / √ λ = K 2h(t)/λ. Summing these two terms yields the bound h (t) ≤ K 2h(t)/λ -2λmh(t).

The right-hand side is a concave function of h(t) which is nonnegative for h(t) ∈ [0, K 2 /(2λ 3 m 2 )] and negative for higher values of h(t). Since h(0) = 0 it follows that for all t ≥ 0, one has h(t) ≤ K 2 /(2λ 3 µ 2 ) and the result follows since y(t) -ȳ(t) ≤ Σ(0) 1/2 2h(t).


Sebastian Geisler
email: sebastian.geisler@rub.de

Sarah Beumann
email: beumann@uni-wuppertal.de

"Sometimes it goes wrong!" -Teachers' beliefs concerning experiments in mathematics

Keywords: Experiments, teachers' beliefs, grounded theory

In didactic literature there are many practical materials to use experiments in mathematics classroom. Even though, experiments seem to be beneficial in mathematics education, little is known about what teachers think concerning this method. In this explorative study we want to examine the teachers' beliefs about experiments and which relations they see between experiments and mathematics. We asked teachers from various school types in Germany, with an open-ended questionnaire. Our results indicate that most teachers see experiments either as controlled and planful procedure or as "trial and error". Furthermore, teachers state mainly two relations between experiments and mathematics.

Introduction

Despite the fact, that teachers' beliefs have been of increasing interest in educational research, the lack of a common shared definition or understanding of beliefs is obvious [START_REF] Pajares | Teachers' Beliefs and Educational Research: Cleaning Up a Messy Construct[END_REF][START_REF] Philipp | Mathematics teachers' beliefs and affect[END_REF]. Most authors locate beliefs within the affective domain but attest beliefs to be more cognitive and stable than attitudes and emotions [START_REF] Mcleod | Research on Affect in Mathematics Education: A Reconceptualization[END_REF][START_REF] Philipp | Mathematics teachers' beliefs and affect[END_REF][START_REF] Hannula | Exploring new dimensions of mathematics-related affect: Embodied and social theories[END_REF]. We refer to beliefs as "Psychologically held understandings, premises, or propositions about the world that are thought to be true" (Philipp, 2007, p. 259). It has been proven that beliefs influence teachers' activity [START_REF] Grigutsch | Mathematische Weltbilder von Schülern. Struktur, Entwicklung[END_REF]. Thus, beliefs are the basis for later actions and are part of the teacher's professional knowledge [START_REF] Kuntze | Situation-specific and generalized components of professional knowledge of mathematics teachers -Research on a video-based in-service teacher learning program[END_REF].

However, during the last years the focus of interest has changed from rather global beliefs to domain specific beliefs or beliefs concerning single methods (e.g. [START_REF] Eichler | Domain Specificity of Mathematics Teachers' Beliefs and oals[END_REF]. One method, that has not been taken into account so far, are experiments or hands-on laboratory activities. Since experiments play an important role in both scientific mathematics and mathematics education, we want to shed light on teachers' beliefs concerning the nature of experiments and the relations between experiments and mathematics.

Experiments

Experiments play a crucial role in the epistemology of nature science. The experimental testing of hypotheses, which have been derived from theories, can be seen as the core of scientific work in subjects like physics and chemistry. It is, therefore, not surprising that experiments have an outstanding position in science education as well. For example, in physics lessons experiments and laboratory work fill about 30% of the time spent in class [START_REF] Tesch | Experimentieren im Physikunterricht -Ergebnisse einer Videostudie[END_REF]. Before illustrating the role of experiments in mathematics we will give a brief overview of the discussion about the question what exactly an experiment is.

Characteristics of Experiments

It is rather surprising that there is no common shared definition of experiments. [START_REF] Borba | Humans with Media and the Reorganization of Mathematical Thinking[END_REF] summarized definitions from several dictionaries concluding that "in an experiment, conditions or factors are manipulated and facts are observed in order to prove or disprove a given hypothesis" (p. 66). They added that besides the testing of hypotheses, experiments can be used to "discover something unknown […] or to provide examples of a known truth" (p.66). [START_REF] Ganter | Experimentell zum Funktionalen Denken: Eine empirische Untersuchung zur Wirkung von Schülerexperimenten als Ausgangspunkt mathematischer Begriffsbildung -Teilprojekt 7[END_REF] stress that experiments form a cyclical process, starting with a concrete hypothesis, the planning of an experiment and followed by the performing of the experiment itself and analysing the resultswhich themselves can lead to a following cycle. It should not remain unmentioned that in every day senseand even in some dictionariesevery hands-on activity or laboratory work is considered to be an experiment, even if the process is not controlled and follows rather the principles of "trial and error" [START_REF] Borba | Humans with Media and the Reorganization of Mathematical Thinking[END_REF].

Experiments and (the Learning of) Mathematics

Even if it is not often mentioned or recognized: experiments and empirical observations play an important role in mathematics and the learning of the same. Polya (1945) described that:

"mathematics has two faces; it is the rigorous science of Euclid but it is also something else. Mathematics presented in the Euclidean way appears as a systematic, deductive science; but mathematics in the making appears as an experimental, inductive science. Both aspects are as old as the science of mathematics itself" (p. vii).

Experiments are part of the proving process in the sense that they help to identify structures and they give ideas how to proof theorems (Kortenkamp, 2014). A (young) subdomain of mathematics that uses (mainly) computer-supported experiments and simulations for this purpose is called experimental mathematics [START_REF] Borwein | Experimentation in Mathematics[END_REF]. Besides the role experiments play in the proving process, mathematics and experiments meet each other in the experimental processes as it is performed in nature science, too. Quantitative experiments cannot be analyzed without mathematics the data derived from an experiment undergoes a mathematical modeling process. [START_REF] Henning | M nster: erlag f r wissenschaftliche Texte und Medien[END_REF] postulates that one of the most important functions of mathematics lies in supporting the understanding of our world.

The role of experiments in the mathematics classroom is closely connected to the relations between experiments and mathematics in scientific settings. In the existing literature we identified three learning situations related to experiments in mathematics:

 Proving processes: Regardless that proving is less present in school than in scientific mathematics, experimental work plays the same role in the proving process in school.

Experiments can help to find a rule that could be proved and the ideas for the proof itself (Kortenkamp, 2014). "Experimental proofs" are part of the proving process of secondary students at school [START_REF] Brunner | Mathematisches Argumentieren, Begründen und Beweisen. Grundlagen, Befunde und Konzepte[END_REF].  Modeling: Modeling is a key competence in mathematics lessons [START_REF] Blum | Applied mathematical problem solving, modelling, applications and links to other subjects -State, trends and issues in mathematics instruction[END_REF]. But since modeling activities can be performed with "real" data derived from experiments as well as with data found in textbooks, the question arises whether the implementation of experiments can enrich the modeling process. Following [START_REF] Carreira | Mathematical modelling with hands-on experimental tasks: On the student's sense of credibility[END_REF][START_REF] Carreira | Students' Modelling Routes in the Context of Object Manipulation and Experimentation in Mathematics[END_REF], experiments can enhance the credibility of modeling tasks and foster students' understanding of the real situation that is modeled and the involved mathematics. The use of "real" data from experiments can also enrich the validation of models and opens teachers' and students' view for the question what characterizes a good model and how one should act upon measurement errors which are inherent of experiments [START_REF] Carrejo | What is mathematical modelling? Exploring prospective teachers' use of experiments to connect mathematics to the study of motion[END_REF].  Learning of new concepts: Furthermore, experiments can support students' learning of new concepts. [START_REF] Ganter | Experimentell zum Funktionalen Denken: Eine empirische Untersuchung zur Wirkung von Schülerexperimenten als Ausgangspunkt mathematischer Begriffsbildung -Teilprojekt 7[END_REF] showed that experiments can create a fruitful start for the learning of functional relations and, in particular, the manipulation of a variable during an experiment can help students to distinguish between dependent and independent variable.

Furthermore, experiments can increase students' interest in mathematics and their motivation [START_REF] Beumann | Versuch´s doch mal -Eine empirische Untersuchung zur Förderung von Motivation und Interesse durch mathematische Schülerexperimente[END_REF]. As conclusion one might say that experiments can enrich mathematics lessons in various ways and learning situations. However, experiments are neither explicitly mentioned in the German mathematics curriculum, nor do the play a crucial role in teacher education. Thus, teachers' beliefs concerning experiments and the teachers' use of experiments in the classroom is widely unexplored. This contribution is a first step to investigate teachers' beliefs concerning experiments and their connection to mathematics.

Beliefs and Experiments

Some recent studies addressed the beliefs of teachers in nature sciences concerning the role of experiments in the classroom and the goals obtained by using experiments. Science teachers' reasons to use experiments in their teaching practice include that experiments involve hands-onactivities, increase students' motivation and their conceptual and procedural understanding of nature sciences [START_REF] Lavonen | Effect of a long-term in-service training program on teachers' beliefs about the role of experiments in physics education[END_REF]. In the special case of mathematics, teachers value the use of hands-on activities in classroom and believe that mathematics lessons should be student-centered [START_REF] Wang | United States teachers' views of effective mathematics teaching and learning[END_REF] experiments fall within both: hands-on activities and student-centered methods.

However, to the best of our knowledge there is currently no study (neither in science education nor mathematics education) focusing on teachers' beliefs concerning the nature of experiments and how experiments are connected to mathematics and mathematics learning. For a profitably use of experiments in class, teachers need to have appropriate beliefs about this method itself as well as about the relation to mathematics.

Research Questions

Since on the one hand experiments can be beneficial for the learning of mathematics, and on the other hand teachers' beliefs affect their classroom practices [START_REF] Grigutsch | Mathematische Weltbilder von Schülern. Struktur, Entwicklung[END_REF], we want to explore beliefs that teachers hold concerning experiments. This leads to the following questions:

1) Which beliefs are held by mathematics teachers concerning the notion experiment?

2) Which relations between experiments and mathematics do mathematics teachers see?

Methods

Since the answers in liker-scale surveys are not easy to interpret when dealing with beliefs (Phillips, 2007) we decided to use open-ended questions within a paper and pencil questionnaire. The questions where formulated as follows: 1) What do you understand under the notion "Experiment"? Give a brief explanation and typical characteristics. 2) In which way do experiments and mathematics fit together? Which relations between mathematics and experiments do you see?

The questionnaire was handed to 50 in-service teachers who worked at different types of schools in North Rhine Westphalia, Germany (primary school, lower and upper secondary schools). All of them were mathematics teacher and 24 taught another STEM-subject (like chemistry or physics) as well. All teachers participated voluntarily. We choose that variety of participants in order to gain a more global (yet not necessarily generalizable) picture of the teachers' beliefs. However, not all teachers answered every question in the questionnaire.

The data analysis of this study is based on the "Grounded Theory" by [START_REF] Glaser | Grounded theory: Strategien qualitativer Forschung[END_REF].

We constantly compared the data with each other by using open codes. Thus, we identified various beliefs concerning experiments. These beliefs have been discussed and grouped in categories. Given the limited space, we only give an overview of the main categories in this contribution. Regarding the nature of experiments, we distinguish beliefs concerning the procedure of experimenting and the aims which are connected to experiments. Most teachers stated that they see experiments as either a controlled and planful action to check a given hypothesis or as uncontrolled trying without guiding hypothesis and plan. However, some teachers wrote that they see every hands-on activity as an experiment as far as it involves autonomous working. Besides these characteristics of the conduction of an experiment, most teachers stated that experiments aim at a certain goal: 16 teachers believe that experiments should lead to new insights and new knowledgeat least this knowledge should be subjectively new for the person that obtains the experiments. Other teachers focus more on the aim to better understand previously gained knowledge (Table 1).

Results

Category

Concerning the relation between mathematics and experiments we could identify two main beliefs (Table 2): 13 teachers stated that experiments are part of the process to generate new mathematical knowledge. Experiments are the inductive starting point of this process. 9 teachers see mathematics as part of the experimental process in other sciencesin this view mathematics is used to describe and model experiments. Two teachers stated no direct connection between mathematics and experiments but listed characteristics that both have in common. 3 teachers wrote that they see no relation between mathematics and experiments. However, despite the fact that we explicitly asked for a connection between experiments and mathematics, many teachers only wrote about the connection of experiments and classroom practices.

Category Description Example Quantity

Experiments as a method in the mathematical process

Experiments are used in the mathematical process to generate new hypothesis that can be proven later.

"Mathematical statements are the result of a long process, obtained with the help of experiments"

13 Mathematics as toolbox in the experimental process

Mathematics is used to describe and evaluate experiments in other sciences and everyday life.

"Experiments and their results can usually be described with mathematics." 9

Common characteristics of Mathematics and Experiments

Mathematics is not part of the experimental process or vice versa but both have common characteristics.

"Mathematics as well as experiments is based on plans to solve certain problems, the systematic conduction of the plan and the reflection upon the results."

Table 2: Beliefs concerning the relation between mathematics and experiments

Like beliefs concerning the nature of mathematics, beliefs about experiments are not disjoint in the sense that teachers can either believe that experiments are planful and controlled or trial and error. The different beliefs can be held in various combinations. We illustrate this fact by describing the beliefs of three teachers more precisely: Mr. A teaches mathematics and biology at an upper secondary school and wrote that he used experiments in his mathematics lessons before. For him a "clear mandatory setting and procedure" characterizes experiments. But he is aware of the fact that in every day life "experimenting" can be viewed as less planful and less controlled: "Colloquial, experimenting can be aimless without mandatory setting and determined procedure." He sees the connection between mathematics and experiments in modeling activities: "Insights from experiments can be mathematically modeled. The mathematical models themselves can be used for further predictions concerning the phenomenon. These predictions can be checked by new experiments. Mathematics and experiments can support each other in this sense."

Mr. B teaches mathematics and physics at an upper secondary school and used experiments in his mathematics lessons before. For him, experiments follow a very clear and determined procedure: "Experiments consist of the observation of results, the investigation of dependences from single parameter, a defined procedure, and the reflection on the objects and used methods." In contrast to Mr. A he sees no direct connection between mathematics and experiments but he stresses that the experimental process and the process of creating new mathematical knowledge have some characteristics in common: "Similar to experiments, an appropriate question and the development of ideas, hypothesis and conceptions belong to mathematics. On this basis theories can be developed and later be proven."

Mrs. C works at an upper secondary school, too, but teaches no nature science and has not used experiments in her lessons so far. Mrs. C's beliefs concerning the nature of experiments are less sophisticated. For her experimenting means to "try things out." However, in her opinion experiments are an integral part of mathematical discoveries: "Mathematics is based on experimental work, like searching and finding schemas -that's how discoveries are made!"

The exemplary comparison of Mr. A and Mr. B reveals, that the teachers' beliefs concerning the nature of experiments as well as their background (both are teaching a second STEM-subject and used experiments in their lessons before) do not directly determine which connections to mathematics they see. Both teachers stress that experiments follow a defined procedure. However, Mr. A sees a clear connection between experiments and mathematics, while Mr. B does not see this connection. This result was observed amongst many teachers in our sample. Furthermore, we found no effect of STEM-teaching on the teachers' beliefs concerning the nature or experiments.

Discussion and Outlook

The teachers' beliefs concerning the nature of experiments mainly reflect the discussion on a scientific definition of experiments [START_REF] Borba | Humans with Media and the Reorganization of Mathematical Thinking[END_REF] with the two (extreme) poles "experiments as trial and error" and "experiments as controlled procedure". Given the fact, that modeling plays an important role in modern mathematics education, it is rather surprising that not more teachers saw this connection between mathematics and experiments. Especially since many teachers in our sample taught a nature science as well and, therefore, should be familiar with experiments. In our understanding, every analysis of a quantitative experiment in nature sciences needs mathematical modeling to some degree. However, 25% of the teachers reported that they see experiments as part of the process to gain new mathematical knowledge which is in line with [START_REF] Brunner | Mathematisches Argumentieren, Begründen und Beweisen. Grundlagen, Befunde und Konzepte[END_REF] model of proving processes in school. Furthermore, (computer-supported) mathematical experiments are of growing importance in scientific mathematics which should be reflected in mathematics education as well, according to Kortenkamp et al. (2014).

One limitation for our analysis is that we have no information about the teachers' beliefs concerning the nature of mathematics (mathematical worldview). These beliefs might affect which connections teachers see between experiments and mathematics.

With this pilot study we made a first step to understand teachers' beliefs concerning experiments in mathematics. Our ongoing research will now focus on two facets: 1) The reasons that lie behind teachers' beliefs concerning experiments (e.g. teachers' background and connections between teachers' beliefs concerning the nature of mathematics and their beliefs concerning experiments) and 2) the question whether (and how) mathematics teachers implement experiments in their mathematics lessons. In order to gain deeper insights into the connection of the teachers' beliefs and their use of experiments in mathematics lessons, semi-structured interviews as well as classroom observations will be used.

Table 1 : Beliefs concerning the nature of experiments

 1

			Description	Example		Quantity
	Procedure (characteristics of the experimental process)
	Experiments	as	Experiments are characterized by planful	"An experiment is an	17
	planful	and	and controlled actions to verify a given	action, that is conducted
	controlled procedure	hypothesis	under fixed circumstances
				[…] to check a hypothesis
				or generate one."
	Experiments as "trial	Experiments are not planful or controlled.	"Try things out -involves	22
	and error"		They do not need a hypothesis and are	that sometimes it goes
			characterized by trying out and	wrong."
			attempting different things.
	Experiments	as	Every kind of hands-on activity that	"To solve a task with	10
	autonomous hands-	provides autonomous working is viewed	hands-on activities and
	on activity		as experiment -regardless if it is planful	practical solutions."
			or just trying.
	Aim (aims connected to experiments)
	Experiments aiming	The aim of experiments is to (better)	"Better	understand	13
	on	(better)	understand previously known things.	relations."
	understanding
	Experiments aiming	The aim of experiments is to provide new	"Experiments	are	a
	at (new) knowledge	(at least from the viewpoint of the	method to gain new
			experimenting person) insights and	insights."
			knowledge.