Frédéric

Tomás

Jean-Bastien Julien

Matteo Ronan

Philippe

Jérémie, Romaric, Emilie, Daniil Odal- Ric Olivier

Piero Andrea Matteo

Lillo

My gratitude goes to my advisors, Alessandro Lazaric and Michal Valko. Thanking them for teaching me how to be a good scientist would be telling only half of the story, as showing me how to be professional has been priceless for my growth. The hardest part of sequential learning is

Abstract

The main advantage of non-parametric models is that the accuracy of the model (degrees of freedom) adapts to the number of samples. The main drawback is the so-called "curse of kernelization": to learn the model we must first compute a similarity matrix among all samples, which requires quadratic space and time and is unfeasible for large datasets. Nonetheless the underlying effective dimension (effective d.o.f.) of the dataset is often much smaller than its size, and we can replace the dataset with a subset (dictionary) of highly informative samples. Unfortunately, fast data-oblivious selection methods (e.g., uniform sampling) almost always discard useful information, while data-adaptive methods that provably construct an accurate dictionary, such as ridge leverage score (RLS) sampling, have a quadratic time/space cost. In this thesis we introduce a new single-pass streaming RLS sampling approach that sequentially construct the dictionary, where each step compares a new sample only with the current intermediate dictionary and not all past samples. We prove that the size of all intermediate dictionaries scales only with the effective dimension of the dataset, and therefore guarantee a per-step time and space complexity independent from the number of samples. This reduces the overall time required to construct provably accurate dictionaries from quadratic to near-linear, or even logarithmic when parallelized. Finally, for many non-parametric learning problems (e.g., K-PCA, graph SSL, online kernel learning) we we show that we can can use the generated dictionaries to compute approximate solutions in near-linear that are both provably accurate and empirically competitive.

Résumé

L'avantage principal des méthodes d'apprentissage non-paramétriques réside dans le fait que la nombre de degrés de libertés du modèle appris s'adapte automatiquement au nombre d'échantillons. Ces méthodes sont cependant limitées par le "fléau de la kernelisation": apprendre le modèle requière dans un premier temps de construire une matrice de similitude entre tous les échantillons. La complexité est alors quadratique en temps et espace, ce qui s'avère rapidement trop coûteux pour les jeux de données de grande dimension. Cependant, la dimension "effective" d'un jeu de donnée est bien souvent beaucoup plus petite que le nombre d'échantillons lui-même. Il est alors possible de substituer le jeu de donnée réel par un jeu de données de taille réduite (appelé "dictionnaire") composé exclusivement d'échantillons informatifs. Malheureusement, les méthodes avec garanties théoriques utilisant des dictionnaires comme "Ridge Leverage Score" (RLS) ont aussi une complexité quadratique. Dans cette thèse nous présentons une nouvelle méthode d'échantillonage RLS qui met à jour le dictionnaire séquentiellement en ne comparant chaque nouvel échantillon qu'avec le dictionnaire actuel, et non avec l'ensemble des échantillons passés. Nous montrons que la taille de tous les dictionnaires ainsi construits est de l'ordre de la dimension effective du jeu de données final, guarantissant ainsi une complexité en temps et espace à chaque étape indépendante du nombre total d'échantillons. Cette méthode présente l'avantage de pouvoir être parallélisée. Enfin, nous montrons que de nombreux problèmes d'apprentissage non-paramétriques peuvent être résolus de manière approchée grâce à notre méthode.

Introduction

This thesis focuses on developing efficient algorithms for constrained environments. Here by efficient we mean that the algorithm should be able to scale to large problem instances, and the constraints on the environment can be computational, such as limited memory to store the problem and time to find a solution, or statistical, such as limited amount of labeled data to learn on. Consider an algorithm that can find good solutions for small instances but is too expensive to run on large instances. The final goal of this thesis is to develop new algorithms that, using only a fraction of the resources, can compute an approximate solution that is provably close to the one of the original algorithm.

Why Efficient Machine Learning?

The recent widespread adoption of machine learning in many fields is leading to the emergence of new challenges. In some applications, we are now collecting data at a faster rate than ever. It is not uncommon for a dataset to contain million of samples, and even more massive dataset are constructed starting from social networks and web crawls composed of billions or even trillions of edges. Faced with the challenge of scalability, the most commonly used solution is to simply increase the engineering effort. For a long time, it was enough to add more memory as datasets increased. Similarly, if the runtime of the algorithms grew too large, one could wait for Moore's law to bring it back down to feasibility.

The core motivation of this thesis is that, when the size of a problem surpasses a certain threshold, simple hardware or software improvements cannot counterbalance an intrinsically inefficient algorithm. To cope with the growing size of data in our times, the only way forward is the development of more efficient algorithms.

Since the computational problem often lies in the size of the dataset, it seems reasonable Chapter 1. Introduction to solve the problem by throwing away a part of our samples and reduce the size of the dataset. Unfortunately, throwing away information can negatively affect the performance of our learning algorithms. In this thesis we present two new algorithms that can find a small dictionary of highly informative samples in a single pass over the data. We also prove that the samples contained in these dictionaries can be used in place of the whole dataset to compute approximate solutions to many learning problems, and provably find a solution that is close to the one computed using all the data available. In particular, we show that this is the case when our goal is to learn a regression function in a linear space, complete the labeling of a graph, or compete against an adversary in an online optimization game.

1.2 Sequential dictionary learning in a streaming setting Dictionary learning. Given a large collection of objects, that we will call atoms, in a dictionary learning problem our goal is to approximate the whole collection using only a small subset of atoms. Simple practical examples of this problem is learning to represent colors, where we only need to combine red, blue and green to cover the whole space, or learning a language, where we can approximately express a word as a combination of other words (e.g., a star is a far sun). We consider the case where the atoms that we have to choose are vectors belonging to a linear space, which can be infinite-dimensional, i.e., a Reproducing Kernel Hilbert Space (RKHS) H. Given a RKHS and a dataset of samples arbitrarily drawn from this space, we are interested in approximating the spectral information contained in this dataset, using only a small subset of the whole dataset. This can be done by looking at the covariance matrix of these samples, that contains information on the geometry of the data (eigenvectors), as well as which directions are more important and which less (eigenvalues). To be accurate, our method will have to adapt to the difficulty of the data. If all samples are very unique and lie in all different directions, we want to keep them all, if they are very redundant and they are all grouped in a few directions, we want to keep only a few representatives. Unfortunately, deciding the uniqueness of a sample naively is too expensive, since we need to compare each sample against every other sample in the dataset, resulting in a quadratic space and time complexity. Nonetheless, if we already had a small dictionary of representative samples we could efficiently compute the uniqueness of each sample in the dataset simply by comparing it to the dictionary. This is a classical chicken-and-egg problem: we need accurate estimation of the sample's importance to construct a good dictionary, but we need a good dictionary to compute accurate estimations. To break the cycle, we propose to take a sequential sampling approach. We begin by processing a small subset of the dataset, making it possible to extract an accurate dictionary out of it efficiently. When we process the next portion of the dataset, we only compare the new data to previously selected dictionary, and update our dictionary if necessary. If the previous dictionary is accurate, we will again choose the right samples and construct an accurate dictionary, repeating this process until the whole dataset is examined. This will allow us to break the quadratic barrier of previous dictionary learning algorithm, since we do not compare each object to each other, and translate this improvements to downstream problems that also could not be solved in linear time. Moreover it will allow 1.3 Our contributions us to apply our algorithms to more stringent computational settings, in particular online and streaming settings.

Streaming algorithms An important metric to measure algorithmic efficiency is how many passes over the dataset the algorithm performs during its execution. The main difference is between algorithms that can operate in a single pass over the input vs. algorithm that require multiple passes. The former can be applied not only in cases where all the data is ready beforehand, but also to more difficult setting where the data arrives sequentially over a stream, and the algorithm can only access past data if it explicitly stored it. Being able to operate in a streaming setting has a number of practical efficiency consequences, in order to avoid bottlenecks when dealing with with data storage and centralization. Moreover, streaming algorithms are naturally applicable to sequential and online learning problems, because they provide the option of continuing to update the current solution when more data arrives rather than restarting from scratch.

Our contributions

Dictionary learning for covariance matrices We propose two new sequential sampling algorithms, KORS and SQUEAK, that efficiently construct a core dictionary capable of approximating the covariance matrix of the dataset up to a small constant additive and multiplicative error. This small constant error is achieved with high probability without any assumption on the geometry of the data. To guarantee this, our methods automatically adapt the size of the dictionary until it captures all relevant directions in the data, where a direction is relevant if its associated eigenvalue is larger than our chosen error threshold. The total number of these relevant directions, called the effective dimension of the dataset, can be potentially much smaller than the ambient dimension of the linear space. If the atoms are very spread out these two dimensions coincide, but if the majority of the atoms follow some structure and lie on a small subspace, the size of our dictionary becomes independent on the number of samples and their dimensionality. For the problem of spectral approximation in a RKHS, we introduce the first dictionary learning streaming algorithm that operates in a single-pass over the dataset, with a time complexity linear in the number of samples, and a space complexity independent from the number of samples that scales only with the effective dimension of the dataset. Previous results had either a quadratic time complexity, or a space complexity that scaled with the coherence of the dataset, a quantity always larger than the effective dimension. In addition, we present a modification to our algorithm that can be run in a distributed computation environment. This allows us to introduce the first dictionary learning algorithm that, with enough machines to parallelize the dictionary construction, can achieve a runtime logarithmic in the number of samples.

Learning with Kernels Reconstructing the covariance matrix of the samples allow us to provide reconstruction guarantees for low-rank kernel matrix approximation. Similarly, many existing error analysis for fundamental problems such as kernel ridge regression (KRR) or kernel principal component analysis (PCA) can be extended to work with our dictionary learning methods. Therefore, the approximate solutions computed computed using the dictionary are provably close to the exact ones, and overall both dictionary and approximate solution can be computed in space constant and time linear in the number of samples. This greatly outperforms computing the exact solution, that requires at least Chapter 1. Introduction quadratic time and space, and is also an improvement over all other existing dictionary learning algorithms with comparable accuracy guarantees.

Learning on Graph

We show that our general dictionary learning approach can be successfully applied to the task of graph spectral sparsification, recovering a well known algorithm by [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF]. In doing so, we rectify a flaw in the original analysis of the accuracy and computational complexity of the algorithm. In addition, similarly to how dictionaries are used for kernel learning, we argue that graph sparsifiers can be used to accelerate graph learning algorithms without compromising learning performance. As an example, we show that a semi-supervised learning (SSL) solution computed on the sparsified graph will have almost the same performance as a solution computed on the whole graph, but can be computed in a time that is linear in the number of edges of the graph, and in a space linear in the number of nodes. This is also experimentally validated on graphs with tens of thousands of nodes.

Online learning We present Sketched-KONS and PROS-N-KONS, two approximate second-order algorithms for online kernel learning (OKL). They both achieve the optimal logarithmic regret of exact second-order methods, while reducing their quadratic complexity. Sketched-KONS is the first algorithm to show that it is possible to approximate the second-order updates of an exact second-order OKL method, without losing the logarithmic regret rate. Computationally, Sketched-KONS offers a favorable regret-performance trade-off. For a given factor β ≤ 1, we can increase the regret by a linear 1/β factor while obtaining a quadratic β 2 improvement in runtime compared to exact methods. PROS-N-KONS improves on Sketched-KONS, and is the first second-order OKL method that guarantees both logarithmic regret and a sub-quadratic runtime, taking only a nearconstant per-step time and space cost that scales only with the effective dimension of the problem.

Thesis structure

Chapter 2. We formally introduce the problem of dictionary learning for covariance matrices, and provide two definitions that quantify when a dictionary accurately reconstructs the spectrum of a covariance matrix, either in an Euclidean space (Section 2.1) or in a general RKHS (Section 2.2). We also survey existing batch sampling algorithms, and show how using the ridge leverage score (RLS) of a sample to quantify its uniqueness provably constructs small and accurate dictionaries. Unfortunately computing ridge leverage scores is too computationally expensive. While there exists efficient methods to approximate them when we restrict our problems to Euclidean spaces, prior to the results presented in this manuscript no method could approximate leverage scores in a general RKHS taking less than quadratic time.

Chapter 3. We continue our survey of sampling methods in a general RKHS, presenting two batch algorithms, the first an new result of this thesis and the second a recent result from [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF], that can approximate RLS efficiently but require multiple passes over the data (Section 3.1). Unfortunately multiple passes are extremely expensive or impossible (i.e., streaming setting) on large dataset. We address this problem with two new sequential RLS sampling algorithms, that represent the main contribution of the thesis. Both of these algorithms process the dataset incrementally, alternating between using an intermediate dictionary to estimate 1.5 Notation 13 the RLSs of new samples, and using the estimate to choose which informative new sample should be inserted in the dictionary and which samples became redundant and must be removed. We first introduce (Section 3.2) the simpler of the two, KORS, that takes an insertion-only sampling approach, where a sample already included in the dictionary is never removed. Afterwards (Section 3.3), we show with SQUEAK how adding the possibility of removing already sampled atoms from the dictionary allows us to parallelize the sampling process and distribute it across multiple machines. Rigorously proving that KORS and SQUEAK succeed with high probability (Section 3.4) requires a careful stochastic dominance argument that is of independent interest for adaptive importance sampling algorithms. This conclude the first part of the thesis dedicated to introducing improved dictionary learning methods. We continue in the second part showing how they can be applied to solve important batch and online machine learning problems.

Chapter 4 We show how our fast approximation scheme can be applied to many batch learning problems that are based on kernels. In particular, using KORS and SQUEAK, we derive linear time algorithms that can compute low-rank approximations of kernel matrices (Section 4.1), and approximate solutions for kernel ridge regression and kernel principal component analysis problems (Section 4.2). Similarly, we show how a specialized variant of SQUEAK can be used to compute spectral graph sparsifiers (Section 4.3), and how spectral graph sparsifiers can be applied to machine learning problems to derive efficient algorithms (Section 4.4). In particular, we focus on the problem of graph semi-supervised learning, proving that computing an approximate solution on a sparse approximation of the graph constructed using SQUEAK performs almost as well as the exact solution computed on the whole graph.

Chapter 5 We introduce the problem of online kernel learning (Section 5.1) and survey existing first-order and second-order gradient methods proposed to solve them. All of these methods suffer from the curse of kernelization, and require at least O(t) per-step time to guarantee low regret, resulting in a quadratic O(T 2 ) runtime. For the first time, we break this quadratic barrier, while provably maintaining logarithmic regret, using two novel approximate second-order OKL algorithms. Sketched-KONS (Section 5.2) takes the approach of approximating gradient updates, while PROS-N-KONS (Section 5.3) takes the approach of approximating the RKHS where the gradient descent takes place.

Notation

We now outline the general notation used in this thesis. We will also add more specific notation when the corresponding concept is introduced. Any deviation from this notation is explicitly stated in the text. We use curly bracket notation {a i } n i=1 to indicate sets of arbitrary elements, possibly with duplicates, and the union operator to represent adding (combining) two sets {a i } n i=1 ∪{b i } n i=1 , or a set and a single element {a i } n i=1 ∪ b. Finally, we use round bracket notation (•, •, •) to indicate tuples, and the set of integers between 1 and n is denoted by [n] := {1, . . . , n}. We use upper-case bold letters A for matrices and maps, lower-case bold letters a for vectors and points in an RKHS, lower-case letters a for scalars. We denote by [A] i,j and [a] i the (i, j) element of a matrix and i-th element of a vector respectively, with [A] i, * the Chapter 1. Introduction i-th columns of a matrix, and with [A] * ,j its j-th row. We denote by I n ∈ R n×n the identity matrix of dimension n and by Diag({a i } n i=1 ) ∈ R n×n the diagonal matrix with the elements of the set {a i } n i=1 on the diagonal. We use e n,i ∈ R n to denote the indicator vector of dimension n for element i. When the dimension of I and e i is clear from the context, we omit the n. We use A B to indicate the Löwner ordering [START_REF] Horn | Matrix analysis[END_REF], i.e., that A -B is a positive semi-definite (PSD) operator. We use I{A} for the {0, 1} indicator function of event A, and #{{A i }} = n i=1 I{A i } for the count of how many events are realized in a set of events.

Norms and decompositions. Throughout the thesis norms of vector and matrices are -2 and induced -2 norm (operator norm), unless when explicitly indicated. For a vector a, a = √ a T a, and for a matrix A = sup{ Aa : a = 1}. Given a matrix A, we indicate with A = VΣU T its singular value decomposition (SVD), where the matrix Σ = Diag({σ i } Rank(A) i=1

) contains the singular value of A, with σ n ≥ σ n-1 ≥ • • • ≥ σ 1 > 0. Moreover, we indicate with σ max = σ n the largest singular value, and with σ min the smallest. Similarly, given a symmetric matrix A, we indicate with A = UΛU T its eigenvalue decomposition (eigendecomposition), where the matrix Λ = Diag({λ i } Rank(A) i=1

) contains the singular value of A, with λ n ≥ λ n-1 ≥ • • • ≥ λ 1 . Moreover, we indicate with λ max = λ n the largest eigenvalue. and with λ min the smallest (possibly negative). Note that the induced -2 norm of a matrix is equivalent to its largest singular value σ max , which is equal to the eigenvalue with largest magnitude in self-adjoin operators (symmetric matrices).

Dictionary Learning for Covariance Matrices

In this chapter we formally introduce the problem of dictionary learning, first for finite dimensional spaces R d and then for general Reproducing Kernel Hilbert Spaces (RKHS) H. In particular, we introduce a metric to quantify the accuracy of a dictionary, and survey existing batch sampling methods that can construct small and accurate dictionaries. Unfortunately, these batch sampling methods are too computationally expensive and require an oracle to execute. How to implement such an oracle in practice will be the focus of Chapter 3.

Nyström in an Euclidean space: leverage score sampling

Consider a collection D = {x i } n i=1 of n vectors x i selected arbitrarily1 from R d , possibly with duplicates, and let X ∈ R d×n be the matrix with x i as its i-th column. Throughout this section we will focus on the case n d where we have many more samples than dimensions. Associated with X we have the covariance matrix

XX T = n i=1 x i x T i .
In machine learning the covariance matrix is a recurrent element, which can be used among other things to extract principal components, fit (non-)parametric linear models and perform density estimation [START_REF] Hastie | The elements of statistical learning[END_REF]. The space and time complexity of all of these methods scale proportionally to both the dimension of the covariance matrix d, as well as the number n of rank-1 matrices x i x T i contained in it. When n is very large, it is computationally very hard to process XX T , so we need to reduce this dependency. Given our dataset D, in this section we are interested in finding a dictionary I that satisfies two goals: A dictionary is a generic collection of weighted atoms I := {(weight, atom)} = {(s i , a i )}, where the atoms a i are arbitrary elements of R d , and s i are non-negative weights2 . The size of the dictionary |I| is the number of distinct atoms with a non-zero weight. We say that a dictionary I can be used to exactly represent dataset D when it is possible to exactly reconstruct the covariance matrix XX T = s i a i a T i as a weighted sum of the elements in the dictionary (see Fig. 2.1). If the dictionary correctly reconstructs XX T but its size does not scale with n (i.e., it is small ), we can use it in place of XX T to answer our machine learning questions (e.g., fit a linear model).

Clearly the dictionary necessary to represent XX T is not unique, and many alternatives are possible, that differ in the choice of atoms and weights. We will now consider some options:

Without learning. The simplest choice for a dictionary that needs to represent D is D itself, that is we do not change neither atoms or weights and set I = {(1, x i )} n i=1 . This perfectly reconstructs D since n i=1 1x i x T i = XX T , but does not save any space. Learn atoms. 3 As soon as n > d, I = D is not the most succinct (in number of atoms) representation possible. Consider the thin Singular Value Decomposition of X = VΣU T and let r be the rank of X 1. V ∈ R d×r contains the orthonormal left singular vectors v i ∈ R d as columns, such that V T V = I r and VV T is the projection on Im(X). 2. Σ : R r×r is the diagonal matrix where [Σ] i,i = σ i is the i-th singular value, and σ r ≥ σ r-1 ≥ • • • ≥ σ 1 > 0. 3. U : R n×r contains the orthonormal right singular vectors u i ∈ R n as columns, such that Im(U) = Im(X T ).

Then we can rewrite

XX T = VΣU T UΣ T V T VΣΣ T V T = r i=1 σ 2 i v i v T i
and use I = {1, σ i v i } as a dictionary for D, replacing the atoms x i with σ i v i . Clearly the number of atoms in the SVD-based dictionary is smaller than the one in the original dataset, but unfortunately when n is very large computing the SVD is prohibitively expensive.

Learn weights but not atoms. This is the family of dictionaries I = {(s i , x i )} n i=1 . When s i = 1 for all i (i.e., I = D) we saw that this is not the most succinct representation possible, and there must be some redundancies that we can eliminate. A simple example of redundancy can be seen when x i and x i are equal. In this case, we can simply remove x i from the dictionary setting s i = 0 and compensate by setting s i = 2. This leaves n i=1 s i x i x T i = n i=1 x i x T i = XX T unchanged. In other words we can leave the atoms x i unchanged, but adapt the weights s i to the data in order to reduce the number of non-zero weights and the size of the dictionary. While general dictionary learning consider the overall problem of jointly learning the weights and atoms, in the remainder of this thesis we will focus only on this last approach of selecting and reweighting a subset of D itself, sometime called sparse coding or sparse approximation. Given the selected subset, we can define the selection matrix S associated with a dictionary as S = Diag({ √ s i } n i=1 ) ∈ R n×n . Then XS is a R d×n matrix where the atoms contained in I are properly reweighted so that

XSS T X T = n i=1 s i x i x T i .
Unfortunately (even when n > d) it is not always possible to find a subset of input samples x i that is strictly smaller than D and exactly reconstructs D. If we want to reduce the size of the dictionary we must incur in some approximation error. We will consider two metrics to quantify our approximation error. We present the first here, and postpone the second to Section 2.2.1.

Subspace approximation

As we mentioned before, the covariance matrix XX T plays a central role in many linear problems, such as linear regression or PCA, that try to find the vector or vectors in a linear space that best fit the data collected. More formally, the solution to all of these problems must lie in Im(X) = Im(XX T ), and is computed using the spectral information contained in XX T , in the form of eigenvectors and eigenvalues. Therefore if the span and the spectrum of XSS T X is close to that of XX T , we can use only the data contained in the dictionary to compute an accurate approximate solution. Chapters 4 and 5 will quantify more rigorously how an accurate dictionary translates into a good approximate solution, while we will now give our definition of accurate dictionaries.

We begin by introducing the orthogonal projection matrix on Im(X) as a proxy for the spectrum of X, Definition 2.1 The orthogonal projection Π on Im(X) is defined as

Π = (XX T ) +/2 XX T (XX T ) +/2 = n i=1 π i π T i , 2.1
with π i = (XX T ) +/2 x i . The approximate orthogonal projection Π for a dictionary I and its selection matrix S is defined as

Π = (XX T ) +/2 XSS T X T (XX T ) +/2 = n i=1 s i π i π T i .

2.2

Note that Π = I d when X is full rank, and that we can also express Π = r i=1 v i v T i , i.e., the SVD dictionary can represent Π using only r atoms, while the {π i } n i=1 dictionary

Chapter 2. Dictionary Learning for Covariance Matrices requires n atoms. Finally, note that the approximate orthogonal projection Π is not the orthogonal projection on Im(XS) Π = Π I := (XSS T X T ) +/2 XSS T X T (XSS T X T ) +/2 .

We chose this definition for Π because we are not only interested in reconstructing the subspace spanned by XX T , but we also want to reconstruct its defining spectral properties. To show that Π is the right tool for the job, we use Definition 2.2 and Proposition 2.1. Definition 2.2 A dictionary I = {(s i , x i )} n i=1 and its associated selection matrix S ∈ R n×n are ε-accurate w.r.t. a matrix X if

Π -Π = (XX T ) +/2 X(I n -SS T )X T (XX T ) +/2 ≤ ε 2.3
Proposition 2.1 - [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF].

(XX T ) +/2 (XX T -XSS T X T )(XX T ) +/2 ≤ ε (1 -ε)XX T XSS T X T (1 + ε)XX T 2.4
Due to the definition of Löwner PSD ordering, if the dictionary I is ε-accurate Proposition 2.1 guarantees that, for any a ∈ R d , a T XSS T X T a is at most a (1 ± ε) constant multiplicative error away from a T XX T a. Since a T XX T a = 0 only if a ∈ Ker(XX T ), this implies that Ker(XX T ) = Ker(XSS T X), Im(XX T ) = Im(XSS T X), and the dictionary perfectly reconstructs the projection Π Π = (XX T ) +/2 XX T (XX T ) +/2 = (XSS T X T ) +/2 XSS T X T (XSS T X T ) +/2 = Π I .

We can also show that the spectrum of XX T is preserved. In particular, ε-accuracy and Proposition 2.1 implies that all eigenvalues of XX T are approximated by XSS T X T up to the same constant factor. Therefore, ε-accuracy not only preserves the subspace that contains X, but also approximates well the importance of each direction in that subspace. As we will see this translates in stronger guarantees for complex problems such as PCA and regression. In addition, since XSS T X T has to approximate at least r eigenvalues and eigenvectors, it is easy to see that an ε-accurate dictionary must contain at least r atoms (e.g., to approximate a diagonal matrix such as I r , we need at least r atoms).

Now that we defined an accuracy measure, there are many possible ways to try to find a dictionary and selection matrix S that satisfy it. Gradient descent. We can rewrite Eq. 2.3 as a function of the weights s i as

min {s i } n i=1 n i=1 (1 -s i )π i π T i
A simple approach would be to use gradient descent mixed with an -0 or -1 penalty on the weights s i to minimize this objective. Unfortunately computing the vectors π i requires O(nd 2 ) time and is not efficient. Moreover computing a gradient w.r.t. n weights s i , and repeating this operation at each step again makes the algorithm's complexity scale poorly with n.

Greedy choice with barrier function. Instead of performing expensive full gradient

Algorithm 1 Nyström method Input: Budget q, set of probabilities {p i } n i=1 such that for all i 0 < p i ≤ 1 Output: I 1: Initialize I = ∅ 2: for i = {1, . . . , n} do

3:

Draw the Binomial r.v. q i ∼ B(p i , q)

4:

If q t = 0, add 1

p i q i
q , x i to I 5: end for updates, several papers [START_REF] Batson | Twice-ramanujan sparsifiers[END_REF][START_REF] Silva | Sparse sums of positive semidefinite matrices[END_REF] propose to use simpler greedy methods, providing increasingly stronger guarantees for these approaches. Given a partial dictionary, their strategy is to sequentially compute which would be the vector π i that would further the most the approximation goal while keeping the existing dictionary fixed. [START_REF] Batson | Twice-ramanujan sparsifiers[END_REF] show that using appropriately constructed barrier functions to quantify the usefulness of each π i vector, it is possible to prove that this sequential approach generates an ε-accurate and in many cases optimally small dictionary. Unfortunately each step of this strategy requires solving a linear system in X and it is not computationally efficient, unless X satisfies stringent structural assumptions, e.g., when XX T is the Laplacian of a graph [START_REF] Lee | Constructing Linear-Sized Spectral Sparsification in Almost-Linear Time[END_REF]. Random sampling. A traditional [START_REF] Williams | Using the Nystrom method to speed up kernel machines[END_REF] and common heuristic to construct a dictionary is to build an unbiased estimator of Π choosing the weights with uniform sampling. In particular, we sample a small subset of q atoms uniformly at random (i.e., with probability p = 1/n), and set their weight to s i = 1/(pq) = n/q. This strategy is simple, requires no knowledge of π i , and almost no computation. Unfortunately, it often fails to construct an ε-accurate dictionary [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF]. Nonetheless if we replace the data oblivious uniform distribution with a more accurate data adaptive distribution, we can improve our accuracy. In the next section we will focus on this family of sampling methods, and discuss how to find a good data adaptive distribution.

Nyström sampling

In Section 2.1 we restricted our attention to dictionaries that select and reweight a subset of the input matrix. The Nyström method is a general name for all methods that choose this subset according to a random process. The blueprint for batch Nyström sampling is reported in Algorithm 1. The algorithm iterates over each atom x i and draws a Binomial random variable q i ∼ B(p i , q) containing q independent copies of a Bernoulli Ber(p i ). While we refer to q i as the number of copies of atom x i , Algorithm 1 does not explicitly add duplicate copies of x i to the dictionary I, but a single copy with weight 1

p i q i q .
Notice that traditionally Nyström sampling draws q( n i=1 p i ) samples without [START_REF] Williams | Using the Nystrom method to speed up kernel machines[END_REF] or with [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] replacement from the multinomial distribution M({p i / n i=1 p i }). Algorithm 1 draws q samples from each of the n Binomials B(p i , q), which can be seen as a "Bernoulli with replacement". On a statistical level this sampling scheme differs from Multinomial sampling in two aspects. The first difference is that the size of the dictionary |I| = q = n i=1 q i is not fixed in advance to q( n i=1 p i ) but random and equal to the number of non-zero weights contained in the dictionary |I| = n i=1 I{ 1

p i q i q = 0} = n i=1 I{q i = 0}
. Nonetheless, we can bound w.h.p.

20

Chapter 2. Dictionary Learning for Covariance Matrices the final size of the dictionary to show that it is at most 3q( n i=1 p i ).

Proposition 2.2 Given probabilities {p i } and budget q, let I be the dictionary generated by Algorithm 1. Then

P |I| ≥ 3q n i=1 p i ≤ exp -q n i=1 p i .
Proof sketch of Proposition 2.2. While the true size of the dictionary is the number of non-zero weights, it is easier to analyze the upper bound |I| ≤ n i=1 q i , as if we assumed that all the copies included in the dictionaries belonged to different atoms, or we explicitly stored each copy of an atom separately. Then we can use a Hoeffding-like bound, proved in Proposition 2.13, to bound the sum of independent, non-identically distributed Binomials

n i=1 q i .
The second difference is that, although in expectation both the number of drawn samples for each atom (E[q i ] = qp i ) and in total (E[q] = q n i=1 p i ) match between the two approaches, our sequential Binomial sampling is not equivalent (same distribution) as a series of q( n i=1 p i ) Multinomial draws with replacement4 . On an algorithmic level, unlike traditional Nyström, we do not need to explicitly compute the normalization constant n i=1 p i to perform the sampling procedure. This will become useful in Section 2.2.2, where the p i will be computed starting from data. Other algorithm that use this Binomial sampling approach are present in the literature [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF], as well as variants that draw q i proportionally to an overestimated Bernoulli Ber(p i q) rather than a Binomial B(p i , q) (Cohen et al., 2015a;[START_REF] Cohen | Ridge Leverage Scores for Low-Rank Approximation[END_REF][START_REF] Cohen | Online Row Sampling[END_REF][START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]. It is easy to see that P(Ber(p i q) = 0) = 1p i q ≤ (1p i ) q = P(B(p i , q) = 0), and drawing from the Binomial will produce smaller dictionaries. From the definition of sampling probabilities p i and weights 1 p i q i q we can see that the dictionary I is an unbiased estimator of D since

E[XSS T X T ] = n i=1 1 p i E[q i ] q x i x T i = n i=1 U 1 p i q p i q x i x T i = XX T .
This holds for any valid set of probabilities p i and any q but it is easy to see that some choices are better than others. For example selecting extremely small p i for all i and q = 1 will almost certainly result in an empty (and very inaccurate) dictionary. From an accuracy perspective, we want to know when XSS T X T , and in turn Π, are close to their mean XX T and Π, or in other words when XSS T X T concentrates. From a computational improvement perspective, we want to know when the generated dictionary I is small. Two factors impact these two quantities.

The probabilities

The probabilities p i used to sample the Binomials tells us which atoms to choose and how many. More in general, the probability p i should tell us how unique or redundant each sample is, and the sum of these probabilities p i encodes how many atoms overall are necessary to represent the dataset. The concentration of XSS T X T around its mean will be guided by how closely the distribution p i captures the true underlying redundancies. Remember from the previous section that we can find a small dictionary that accurately reconstruct the dataset because there are redundancies between samples. Consider the example dataset D = {x, x, x , x , x } with two identical copies of x, three identical copies of x , and x T x = 0 orthogonal. The dictionary that perfectly reconstruct this dataset is {(2, x), (3, x )} with minimal size 2. How would we set our probabilities in Algorithm 1 to generate this dictionary? To obtain 2 and 3 as our weights, we set p 1 = p 2 = 1/2 and p 3 = p 4 = p 5 = 1/3, with p i = 2 total number of needed atoms. This example also highlights why we need randomization: selecting greedily the two most unique atoms associated with the highest probabilities (x 1 and x 2 ) will not generate a good dictionary.

The number of copies

The budget q tells us the extra budget we have to pay to compensate the intrinsic randomness induced by the random sampling, and any mistake in choosing the probability p i (mismatch between chosen p i and true redundancies). Take again the example D = {x, x, x , x , x } with p 1 = p 2 = 1/2 and p 3 = p 4 = p 5 = 1/3. If we set q = 1, simply due to the way q 1 and q 2 are randomly selected, we have a 1/4 possibility to generate a bad dictionary that does not contain x. Increasing q reduces the probability of this event happening. Similarly, if we underestimate the importance of a sample (e.g., set p 1 = p 2 = 1/4), increasing q will bring the final dictionary close to the mean no matter what kind of error we make. While increasing q helps with accuracy, we saw that our dictionary size increases as q, so raising it excessively might result in a very large dictionary that will not improve our computational complexity.

Uniform sampling

To see an example of how the probabilities and budget q impact accuracy and dictionary size, we begin by considering uniform probabilities p i = 1/n. We will refer to Algorithm 1 using uniform probabilities as Uniform. Historically the first Nyström methods introduced in machine learning [START_REF] Williams | Using the Nystrom method to speed up kernel machines[END_REF] used uniform sampling, and it remains a popular choice for its conceptual simplicity and the fact that computing this distribution takes no computation at all. More recently, Bach, (2013) provided sufficient conditions under which uniform sampling generates an ε-accurate dictionary, using the dataset's coherence to quantify the number of samples necessary.

Definition 2.3 - [START_REF] Drineas | Fast approximation of matrix coherence and statistical leverage[END_REF]. Given a matrix X, the coherence µ of X is

µ = max i=1,...,n π i π T i = max i=1,...,n π i 2 2.5
Proposition 2.3 - (Bach, 2013, App. B1, Lem. 2). Consider an arbitrary matrix X ∈ R d×n with x i as its i-th column and rank r. Let Π and Π be defined according to

22

Chapter 2. Dictionary Learning for Covariance Matrices Definition 2.1. Let I be the dictionary generated by Algorithm 1. If

p i = 1/n, q ≥ nµ 4 log(2d/δ) ε 2 ,
with n i=1 p i = 1, then for any 0 < ε < 1 and w.p.1δ Accuracy: I is ε-accurate.

Space: the size of the dictionary is bounded by |I| ≤ 3q.

Proposition 2.3 tells us that to construct an ε-accurate dictionary, we should sample roughly O(nµ) atoms, which can correspond to almost the whole dataset when µ is close to 1. Unfortunately, estimating the coherence µ is a hard problem in itself [START_REF] Drineas | Fast approximation of matrix coherence and statistical leverage[END_REF], therefore running uniform sampling with guarantees is not easy. As a solution, it is common in the Nyström literature to make an assumption of low coherence. Since the coherence µ is directly related to how much samples in the dataset are orthogonal with each other, Proposition 2.3 also gives us intuition on how uniform sampling can fail. For example, consider the case where a single sample x i is completely orthogonal to the others (µ = 1), and we try to use uniform probabilities in Algorithm 1 but set q to a constant (w.l.o.g. q = 1) to have a dictionary size that does not scale with n. Then the probability of not including the orthogonal sample is going to be roughly (1 -1/n) ∼ 1. And if we do not include the sample, it is easy to see that Π -Π ∼ 1 simply by testing these two matrices along the sample direction, where x T i Πx i = x T i x i and x T i Πx i = 0 since x i is orthogonal to all other samples. Note that this problem will not happen with high probability only when a single sample is orthogonal. As an example, imagine that we have n 1/2 vectors identical and orthogonal to the rest, we are still going to miss them with probability roughly (1 -1/n 1/2 ). This also is in line with our intuition of q as an extra budget to compensate inaccuracies in choosing p i . When the coherence is small, and no sample is orthogonal to the others, all samples look the same and have the same importance of roughly 1/n, making uniform sampling very accurate. But if the coherence is high, uniform probabilities are very likely to not correspond at all to the true underlying importance of each samples (they are chosen before even looking at the samples), and to concentrate we need a large number of copies q that potentially scales with n.

To see how to remove this dependency on the coherence, let us first give a sketch of Proposition 2.3's proof. The proof is based on the following general concentration for matrices, Proposition 2.4 - (Tropp, 2012, Theorem 1.6). Consider a sequence [X j ] of independent random symmetric matrices with dimension t. Assume that E[X j ] = 0, X j ≤ R, and let Y = j X j . Furthermore, assume that there exists σ > 0 that bounds the matrix variance statistic of the sum

E[Y 2 ] 2 ≤ σ 2 . Then P( Y ≥ ε) ≤ t exp - ε 2 /2 σ 2 + Rε/3 .
Proof sketch of Proposition 2.3. We provide a sketch of the proof here and include the full result in Section 2.3. Intuitively, the core of the proof consist in finding appropriate values for R and σ 2 to apply Proposition 2.4.

In our case we use uniform sampling with probabilities p i = 1/n and the matrix we are

interested in is Y = Π = n i=1 π i π T i . We can show that R := 1 q max i π i 2 p i = n max i π i 2 q = nµ q , σ 2 := 1 q max i π i 2 p i Π = n q max i π i 2 Π ≤ nµ q Π ≤ nµ q ,
satisfy the necessary conditions, and then apply the proposition.

Oracle leverage score sampling

From the proof sketch of Proposition 2.3 we saw that the norms of the individual atoms π i 2 = (XX T ) +/2 x i x T i (XX T ) +/2 plays a large role in bounding the range and variance of the random process induced by Algorithm 1. These quantities are well known in Statistics under the name of leverage scores.

Definition 2.4 - [START_REF] Drineas | Fast approximation of matrix coherence and statistical leverage[END_REF]. Given a matrix X, the leverage score (LS) of column x i ∈ X is

τ i = π i π T i = (XX T ) +/2 x i x T i (XX T ) +/2 = x T i (XX T ) + x i . 2.6
Furthermore, the sum of the leverage scores

n i=1 τ i = Tr(Π) = Tr XX T (XX T ) + = Rank(XX T ) = r,
is equal to the rank of XX T .

The leverage score τ i intuitively captures how orthogonal x i is w.r.t. the rest of the samples X, and serves to rigorously measure what we so far called redundancy w.r.t. to preserving the spectrum of XX T . LSs have the following properties:

• LSs are always smaller than 1. The maximum LS for sample x i is achieved when x i is orthogonal to all other samples: then

x T i (XX T ) + x i = x T i (x i x T i ) + x i = 1.
• LSs are always larger than 0. The minimum LS is achieved when x i is identical to all other samples (all samples are equal): then

x T i (XX T ) + x i = x T i (nx i x T i ) + x i = 1/n.
• The sum of the leverage scores captures the overall orthogonal directions contained in the data, making it equal to the rank of X. • The coherence of the matrix X is equal to the maximum LS (max i τ i ). Therefore, the low coherence assumption commonly used when applying uniform sampling is saying that no leverage score should be higher than the other, or alternatively that the leverage scores should be uniform. • Computing the LSs is computationally expensive, requiring nd 2 operations to construct the XX T matrix. For simplicity we assume that an efficient oracle can compute them for us. In practice there are many efficient algorithms (Cohen et al., 2015a;[START_REF] Drineas | Fast approximation of matrix coherence and statistical leverage[END_REF] Proposition 2.5 - (Alaoui and Mahoney, 2015, App. A, Lem. 1). Consider an arbitrary matrix A ∈ R d×n with a i as its i-th column and assume max i=1,...,n a i 2 ≤ 1. Let I be the dictionary randomly generated by Algorithm 1 using probabilities p i ≥ a i 2 . For any t > 0 we have

P( AA T -ASS T A T ≥ ε ≤ 2d exp - qε 2 /2 ( AA T + ε/3) 2.7
Proof. The proof remains substantially the same as (Alaoui and Mahoney, 2015, Thm.2) up to two differences. (1) We consider the spectral norm • rather than the largest eigenvalue since in general AA T -ASS T A T is not guaranteed to be PSD, and (2) we use our "Bernoulli with replacement" rather than Multinomial with replacement. As a sketch, the proof uses the same approach as Proposition 2.3, applying Proposition 2.4 to Π, but this time using p i ≥ a i 2 instead of p i = 1/n. This lets us develop a tighter bound

R := 1 q max i a i 2 p i = 1 q max i p i p i = 1 q , σ 2 := AA T q max i a i 2 p i ≤ AA T q ,
Intuitively, what this proposition tells us is that to preserve in spectral norm the sum of rank-1 matrices AA T = n i=1 a i a T i , we want to sample more often matrices with larger norm. Under the hood of the proof, this is because in the random sum ASS T A T = n i=1 s i a i a T i rank-1 matrices with larger norm cause larger variance, and sampling them more often help us to reduce the variance and concentrate around the expectation.

Noting that the LS are the norm of the columns of Π, we can instantly apply Proposition 2.5 to obtain

Corollary 2.6 -Proposition 2.5, Proposition 2.13. Consider an arbitrary matrix X ∈ R d×n with x i as its i-th column and rank r. Let Π and Π be defined according to Definition 2.1, and the LS τ i according to Definition 2.4. Let I be the dictionary generated by Algorithm 1 run with parameters

p i = τ i , q ≥ 4 log(4r/δ) ε 2 ,
and n i=1 p i = r. Then for any 0 < ε < 1 and w.p.1δ Accuracy: I is ε-accurate.

Space: the size of the dictionary is bounded by |I| ≤ 3qr Proof of Corollary 2.6. The proof follows directly from Proposition 2.5. To replace the dependency on d with the dependency on r, it is sufficient to decompose Π before applying the concentration using the SVD decomposition

Π -Π = (XX T ) +/2 X(I n -SS T )X T (XX T ) +/2 = (ΣΣ T ) +/2 ΣU T (I n -SS T )UΣ T (ΣΣ T ) +/2 ,
where ΣΣ T ∈ R r×r . This leaves unchanged the norms a i 2 and all other quantities.

Algorithm 2 Nyström method in H Input: Budget q, set of probabilities {p i } n i=1 such that for all i 0 < p i ≤ 1 Output: I 1: Initialize I = ∅ 2: for i = {1, . . . , n} do

3:

Draw the binomial r.v. q i ∼ B(p i , q)

4:

If q t = 0, add 1

p i q i
q , φ i = ϕ(x i ) to I 5: end for Note that this result was first derived by [START_REF] Drineas | Relative-error CUR matrix decompositions[END_REF]. With this corollary we have answered the question on which and how many atoms we need to sample to guarantee an accurate reconstruction of . We have finally satisfied the two goals of this section. For example, setting q = O(log(r)) gives us w.h.p. an ε-accurate dictionary that can reconstruct Π and XX T , with a total size smaller than O(r log(r)) that only scales with the actual rank of the problem, and independent from n. With this dictionary, we can solve downstream tasks in a time independent from n, making it feasible to scale to problems on very large datasets. Compared to uniform sampling, we do not need to assume a low coherence or select a large q to guarantee accuracy, because we will be actively looking for those orthogonal samples that made uniform sampling fail. In other words, our p i now capture much more closely our true underlying importance, and we can use a small logarithmic q because we only need to compensate the intrinsic randomness of the sampling process and not wrong probabilities. Moreover, the size of the dictionary automatically scales as qr, adapting to the rank and difficulty of the problem.

Nonetheless the result is still not fully satisfactory. In particular, we have no way to automatically estimate r, so we typically resort to setting q = O(log(d)). In addition, two obvious shortcomings emerge from looking at the LS sampling approach Fast-decaying eigenvalues. Sometimes the rank of the matrix r is large and close to d, but only due to small numerical fluctuations. Or alternatively the smaller eigenvalues of XX T are negligible compared to the larger and we would like to ignore them.

Large n, larger d. In many cases of interest d itself is also very large, and potentially larger than n. This is commonly the case when we project our point in some high dimensional space using tile-coding, wavelets, one-hot encodings or other combinatorial features. While these higher dimensional spaces have room for richer dictionaries, using LS to measure importance becomes vacuous, since it becomes possible for X to have n orthogonal vectors with τ i = 1 and Algorithm 1 would simply keep all elements.

In both of these cases, not only it is impossible to exactly reconstruct XX T using a dictionary smaller than n ≤ d, even ε approximation becomes impossible without storing all n input samples in the dictionary. In the next section, we will see how to further relax the approximation condition to tackle this setting, and how to generalize the LS to Regularized Leverage Scores.

Nyström in a RKHS: ridge leverage score sampling

To move from R d to a Reproducing Kernel Hilbert Space H we must introduce some additional notation. First we do not restrict our samples x i to come from R d anymore, but make no assumption on the input space X . Then, given our dataset D of n arbitrarily chosen points5 x i ∈ X we use a feature map φ(•) : X → H to map the samples into the RKHS as φ i = ϕ(x i ). In the remainder of the thesis, the points φ i as will play the roles of points in a RKHS, samples from a dataset, columns of a map, or atoms in a dictionary, and we will refer to them accordingly depending on the context. To simplify the notation, we indicate with φ T i φ j the inner product φ i , φ j H in H. Also for simplicity and without loss of generality we assume that for all ϕ(x i ) the feature map ϕ induces a unit norm φ T i φ i = φ i 2 = 1. This is commonly satisfied in practice, for example by using Gaussian or cosine similarities for our H, or by normalizing the inputs x i in the R d case. Alternatively, all our results can be extended to non-unit norm, with extra multiplicative terms of the form (max i φ i 2 )/(min i φ i 2 ) appearing in space and time complexities. When evaluating computational costs, we also assume that computing the inner product φ T i φ j takes a time independent from n (i.e., constant). We also replace the matrix X with the map Φ : R n → H composed by stacking together all points φ i . To simplify again the notation, we indicate with Φ T : H → R n its adjoint map. It is easy to see that the self-adjoint operator ΦΦ T : H → H that replaces the covariance matrix XX T remains PSD and it can again be expressed as ΦΦ T = n i=1 φ i φ T i . The thin SVD of Φ = VΣU T must also be appropriately redefined. Let r be the rank of Φ, we have that 1. The projection matrix becomes a projection operator on Im(Φ)

Π = (ΦΦ T ) +/2 ΦΦ T (ΦΦ T ) +/2 = VV T
based on the map V : R r → H, where v i are the orthonormal eigenfunctions that span Im(Φ), such that V T V = I r . Note that Π also acts as the identity for all vectors in Φ (Φ = ΠΦ) 2. Σ : R r×r remains the diagonal matrix where [Σ] i,i = σ i is the i-th singular value, and σ r ≥ σ r-1 ≥ • • • ≥ σ 1 > 0 3. U : R n×r remain the orthonormal vectors that span Im(Φ T ) Compared to the simpler leverage score sampling case in R d , we have two new difficulties. First, we cannot store or manipulate directly the vectors φ i , but only access them through inner products. Therefore it is not clear how to compute the LS φ T (ΦΦ T ) + φ. Second, when we choose our RKHS we prefer those with large dimension, because we want a rich space that can contain good solutions for our downstream task problems. As a consequence, it is very easy to end up constructing an H where Rank(Φ) = n. For example, using the Gaussian similarity the only way for the rank of the matrix to be smaller than the number of points is for some of the points to be exactly identical to each other. When the rank of the matrix is n, even if we could compute φ T (ΦΦ T ) + φ we would end up with all LS equal to 1, and the sampling would be meaningless. Nonetheless, in many cases the eigenvalues of ΦΦ decay very fast, or in other words only a few of the many (infinite) directions in H capture most of the information contained in Φ.

Regularized subspace approximation

We can encode this intuition using a regularized projection, and a new (ε, γ)-accuracy definition.

Definition 2.5 The γ-regularized projection Γ on Im(Φ) is defined as

Γ = (ΦΦ T + γΠ) -1/2 ΦΦ T (ΦΦ T + γΠ) -1/2 = n i=1 ψ i ψ T i , 2.8
with ψ i = (ΦΦ T + γΠ) -1/2 φ i . The approximate γ-regularized projection Γ for a dictionary I and its selection matrix S is defined as

Γ = (ΦΦ T + γΠ) -1/2 ΦSS T Φ T (ΦΦ T + γΠ) -1/2 = n i=1 s i ψ i ψ T i . 2.9 Definition 2.6 A dictionary I = {(s i , φ i )} n i=1 and its associated selection matrix S ∈ R n×n are (ε, γ)-accurate w.r.t. a dataset Φ if Γ -Γ = (ΦΦ T + γΠ) -1/2 Φ(I n -SS T )Φ T (ΦΦ T + γΠ) -1/2 ≤ ε 2.10
Associated with Definition 2.6 we have another reconstruction result Lemma 2.7

(ΦΦ T + γΠ) -1/2 (ΦΦ T -ΦSS T Φ T )(ΦΦ T + γΠ) -1/2 ≤ ε (1 -ε)ΦΦ T -εγΠ ΦSS T Φ T (1 + ε)ΦΦ T + εγΠ 2.11
Looking at the equation, we see that an (ε, γ)-accurate dictionary gives us the same (1 ± ε) multiplicative error guarantees of an ε-accurate dictionary, but also adds a εγ additive error. This means for example that only eigenvalues larger than εγ will be accurately reconstructed by the dictionary, and that the equivalence Ker(ΦΦ T ) = Ker(ΦSS T Φ) does not hold. Nonetheless, in many interesting problems (e.g., PCA, regularized regression) we were not interested in approximating the directions associated with smaller eigenvalues anyway, and as we will see in Chapter 4 and Chapter 5 this extra small error does influence too much the quality of the solutions computed on the dictionary. In exchange for this new mixed multiplicative-additive error, we can use Γ to define a generalization of the LS to this new d n setting

Definition 2.7 - [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF]. Given a map Φ : R n → H, the γ-ridge leverage score (RLS)

of column i ∈ [n] is τ i (γ) = φ T i (ΦΦ T + γΠ) -1 φ i = n j=1 σ 2 j σ 2 j + γ [U] i,j .

2.12

Furthermore, the effective dimension d eff (γ) of the operator is defined as

d eff (γ) = n i=1 τ i (γ) = Tr ΦΦ T (ΦΦ T + γΠ) -1 = n i=1 σ 2 i σ 2 i + γ .

2.13

Ridge leverage scores

The RLSs τ i (γ) generalize the way LSs captures the overall orthogonality of a sample in the dataset, and LSs can be seen as a special case τ i (0). In particular, the directions Chapter 2. Dictionary Learning for Covariance Matrices associated with smaller eigenvalues are non-linearly penalized by the regularization γ, and being orthogonal along those direction will not make the sample important. Since the value of γ will be often clear from the context, we will refer to both LSs and RLSs as τ i and only include the full notation τ i (γ) when important.

We can now look at two opposite cases to get an idea of how RLSs behave. When all samples are identical, the RLS of φ i is at its minimum

0 ≤ φ T i (ΦΦ T + γΠ) -1 φ i = φ T i (nφφ T + γΠ) -1 φ i = φ T i φ i nφ T i φ i + γ = 1 n + γ ,
even lower than the corresponding lowest LS. When φ i is orthogonal to all other samples its RLS is at its maximum

φ T i (ΦΦ T + γΠ) -1 φ i = φ T i (φ i φ T i + γΠ) -1 φ i = φ T i φ i φ T i φ i + γ = 1 1 + γ ≤ 1.
But when we have many near-orthogonal samples the difference between LSs and RLSs emerge. While all the LSs would be very close to 1 (uniform sampling), looking at Eq. 2.12 we see that as γ grows, the RLSs are non-uniformly shrunk. In particular samples φ i correlated with the larger eigenvalues of ΦΦ remain close to 1, while the ones correlated with smaller eigenvalues are shrunk (nonuniform sampling). For example a sample orthogonal to the others and fully aligned with the smallest non-zero eigenvalue λ min would be shrunk by roughly a λ min /γ factor.

In the linear case where ϕ(•) : X → R d (φ i = x i ), the RLS can be explicitly computed as [START_REF] Cohen | Ridge Leverage Scores for Low-Rank Approximation[END_REF][START_REF] Cohen | Online Row Sampling[END_REF]. In the general H setting, to compute RLS in practice we can use the following equality Lemma 2.8 - (Calandriello et al., 2017a). The RLSs can be reformulated as

x T i (XX T + γI d ) -1 x i (
φ T i (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i φ i -φ T i Φ(Φ T Φ + γI n ) -1 Φ T φ i 2.14
While evaluating Eq. 2.14 is now possible, since Φ T Φ ∈ R n×n and Φ T φ i ∈ R n are real matrices and vectors, it is still to computationally expensive. Operations involving Φ T Φ comprise the dominant part of this cost: it takes O(n 2 ) space and operations to construct and store it, O(n 3 ) operations to invert (Φ T Φ + γI n ) -1 , and O(n 2 ) operations to compute each φ T i Φ(Φ T Φ + γI n ) -1 Φ T φ i using the precomputed inverse. For now, we will assume that an efficient oracle can compute τ i (γ) for us, and we postpone the implementation of such an oracle to Chapter 3.

Ridge leverage scores are also known under a number of different names in Statistics, often appearing together with the Gaussian distribution. Consider the problem of Gaussian process regression [START_REF] Rasmussen | Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)[END_REF], where our function is drawn from a f ∼ GP (0, ϕ(x) T ϕ(x )) with mean function 0 and covariance function induced by ϕ(•). The target variables are generated as y i = f (x i ) + i , with i drawn i.i.d. from N (0, 1). Then, after drawing n samples the posterior variance of a new point x * w.r.t. the sample is

σ n (x * ) = φ T * φ * -φ * Φ(Φ T Φ + I n ) -1 Φ T φ * .
This is almost equivalent to computing the RLS of φ * w.r.t. Φ using the reformulation from Lemma 2.8. The only difference is that φ * is not included in the inverse, or in other words that we compute the RLS w.r.t. Φ rather than w.r.t. [Φ, φ * ] as we did so far. Taking this interpretation, [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design[END_REF] shows that for Gaussian processes, points that maximize RLSs (large variance) also approximately maximize the information gain I({y i }|f ) [START_REF] Cover | Elements of information theory[END_REF].

Effective dimension

We saw that, compared to LSs, the RLSs shrink depending on the overall spectrum of Φ. This is also reflected in the sum of the RLSs: although the LSs sum to the ambient dimensionality (and rank) of Φ in H, which might be large, the RLSs sum to its effective dimensionality, which is potentially much smaller. Intuitively d eff (γ) scales with the number of relevant directions ΦΦ T , where a direction is relevant if it is associated with an eigenvalue larger than the regularization γ. We can also see d eff (γ) as the rank of ΦΦ T after thresholding all eigenvalues smaller than γ, and it is easy to see that d eff (γ) < r for any γ > 0, and d eff (0) = r. The effective dimension is also monotone in γ, which means we can always make it smaller by increasing the regularization. While in general d eff (γ) can greatly vary depending on the specific ΦΦ T , if we know something on the structure of the operator (e.g., because of the way φ i are generated, or some property of ϕ) we can still find bounds on it. Note that the partial derivative of d eff (γ) w.r.t. the values λ i is

∂ ∂λ i n j=1 λ j λ j + γ = ∂ ∂λ i n j=1 1 - γ λ j + γ = γ (λ i + γ) 2 ,
with the same partial derivative for all λ i and the maximum increase in λ i = 0. Therefore, when the trace Tr(ΦΦ T ) = n i=1 λ i is bounded (e.g., Tr(ΦΦ T ) = n as in our case) the maximum d eff (γ) is achieved when all λ i are equal, which corresponds to the case where all samples are orthogonal to each other. Conversely, the minimum is achieved when all samples are equal and only a single λ i is non-zero, resulting in

d eff (γ) = nφ T i φ i nφ T i φ i +γ ∼ 1.
In general, knowing that the operator has a bounded trace Tr(ΦΦ T ) = n gives us a bound d eff (γ) ≤ n/γ which scales as O(n) when γ is a constant, but a smaller O( √ n) when γ = √ n. Stronger bounds are possible when we can control each individual eigenvalue.

For example if the eigenvalues decay polynomially

λ i = i -1 then d eff (γ) = n i=1 i -1 i -1 +γ = n i=1
1 1+γi ≤ log(n + min{1, γ -1 })/γ, while clearly the rank is much larger and equal to n. Together with the RLSs, the effective dimension appears in many statistical problems. If we assume that the points are drawn according to a distribution µ(x) rather than arbitrarily, then the effective dimension is the empirical equivalent of the capacity of the RKHS H under µ [START_REF] Caponnetto | Fast Rates for Regularized Least-squares Algorithm[END_REF]. One of the contributions of this thesis, reported in Section 3.2, will be showing that the log-determinant log(Det(Φ T Φ/γ + I n )) of the covariance matrix can be upper bounded using its effective dimension. This log-determinant appears, among other topics, as the sampling function in determinantal point processes [START_REF] Kulesza | Determinantal point processes for machine learning[END_REF], as an upper bound for the total information gain I({y i }|f ) in Gaussian process regression [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design[END_REF], and as an upper bound for the regret of online learning algorithms [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF].

Oracle ridge leverage score sampling

Using RLSs as probabilities in Algorithm 2, we can construct an (ε, γ)-accurate dictionary w.h.p.. We refer to this algorithm as Oracle-RLS.

Corollary 2.9 -Lemma 2.10, Proposition 2.5, Proposition 2.13. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I be the Chapter 2. Dictionary Learning for Covariance Matrices dictionary generated by Algorithm 2 run with parameters

p i = τ i (γ), q ≥ 4 log(4r/δ) ε 2 ,
and n i=1 p i = d eff (γ), i.e., Oracle-RLS. Then for any 0 < ε < 1 and w.p.1δ Accuracy: I is (ε, γ)-accurate.

Space: the size of the dictionary is bounded by |I| ≤ 3qd eff (γ) Differently from Corollary 2.6 in the previous section, Corollary 2.9 cannot always guarantee a constant (in n) error and at the same time guarantee a constant (again in n) dictionary size. If we set ϕ(•) to be the identity map and X = R d , we can compare better the two settings.

When n d, we know that d eff (γ) ≤ r ≤ d no matter the γ, and therefore we can always find an ε-accurate dictionary that does not grow with n. But in the d n setting (or in a general H) the regularization γ drives the trade-off between dictionary size and accuracy. Going back to the bounded trace Tr(ΦΦ T ) = n, in the worst case (all orthogonal samples and diagonal Φ T Φ) we can only trade-off a n/γ dictionary size for a γ approximation error. Nonetheless, we still achieved the goal that we set at the beginning of the section: when the spectrum of ΦΦ T is not uniform, RLSs sampling will automatically adapt to the difficulty of the problem and construct O(d eff (γ)) sized dictionaries that are not only much smaller than d or n, but also of the numerically high rank r.

Finally, note that for simplicity we chose our bounds to scale with q ≥ O(log(r)), in order to continue using the simple inequalities of Proposition 2.4. Using more refined concentrations based on the stable rank rather than the rank it is possible to obtain tighter bounds that scale with q ≥ O(log(d eff (γ))). Unfortunately, estimating both n and d eff (γ) is computationally expensive, and in practice q is often set proportionally to log(n), since scaling logarithmically w.r.t. to n is quite mild and still acceptable.

Proof of Corollary 2.9. For the proof we cannot directly apply Proposition 2.5, since Γ is an operator and not a matrix and Proposition 2.4 cannot be applied. Nonetheless, we can define a proxy for the regularized projection Γ that is a matrix.

Definition 2.8 The γ-regularized (approximate) projection P ∈ R n×n on Im(Φ T ) is defined as

P = Φ T (ΦΦ T + γΠ) -1 Φ = CC T , P = CSS T C T . 2.15 with C = U(ΣΣ T + γI r ) -1/2 ΣU T , c i = U(ΣΣ T + γI r ) -1/2 ΣU T e i .

2.16

Among other important things, we can use this definition as an alternative formulation for RLSs

τ i = ψ i 2 = ψ T i ψ i = φ T i (ΦΦ T + γΠ) -1 φ i = e T i Φ T (ΦΦ T + γΠ) -1 Φe i = c i 2 = c T i c i = e T i C T Ce i = e T i UΣ T (ΣΣ T + γI r ) -1/2 U T U(ΣΣ T + γI r ) -1/2 ΣU T e i ,
and to reformulate the definition of (ε, γ)-accurate dictionaries Lemma 2.10 - (Calandriello et al., 2017a). Let Γ, Γ be defined according to Definition 2.5, and P, P according to Definition 2.8. Then

Γ -Γ = (ΦΦ T + γΠ) -1/2 Φ(I n -SS T )Φ T (ΦΦ T + γΠ) -1/2 = U(ΣΣ T + γI r ) -1/2 ΣU T (I n -SS T )UΣ T (ΣΣ T + γI r ) -1/2 U T = P -P .
Adding also Proposition 2.2 we conclude the proof.

In the next chapter we address the computational cost of computing RLSs, in order to remove the necessity of an oracle and derive sampling algorithms that can be implemented in practice.

Extended proofs

We report some linear algebra identities that will be useful through the thesis.

Proposition 2.11 - [START_REF] Horn | Matrix analysis[END_REF]. For any matrix A ∈ R n×m and any γ > 0

A(A T A + γI m ) -1 A T = AA T (AA T + γI n ) -1 .
For any symmetric matrix A ∈ R n×n and diagonal matrix B ∈ R n×n such that B has ns zero entries, and s non-zero entries, define C ∈ R n×s as the matrix obtained by removing all zero columns in B. Then

AB(BAB + γI m ) -1 BA = AC(C T AC + γI s ) -1 C T A.
For any appropriately shaped matrix A, B, C, with A and B invertible, the Woodbury matrix identity states

(A + CBC T ) -1 = A -1 -A -1 C C T A -1 C + B -1 -1 C T A -1 .
Proposition 2.12 - [START_REF] Horn | Matrix analysis[END_REF]. For any map A : R n → H, projection operator Π such that A = ΠA, and any γ > 0

A(A T A + γI n ) -1 A T = AA T (AA T + γΠ) -1 .
For any map A : R n → H and diagonal matrix B ∈ R n×n such that B has ns zero entries, and s non-zero entries, define C ∈ R n×s as the matrix obtained by removing all zero columns in B. Then

AB(BA T AB + γI n ) -1 BA T = AC(C T AC + γI s ) -1 C T A.
For the proof of Proposition 2.2 we need a preliminary result Proposition 2.13 - (Calandriello et al., 2017a where we use the fact that 1 + x ≤ e x , z s ∼ Ber(p s ) and t s=1 p s = d. The choice of θ minimizing the previous expression is obtained as

d dθ
e (d(e θ -1)-θgd) = e (d(e θ -1)-θgd) de θgd = 0, and thus θ = log(g). Plugging this in the previous bound, Proof of Proposition 2.13. Let z i,j be the j-th Bernoulli r.v. independently sampled when computing q i , such that q i = q j=1 z i,j . To bound the size |I| ≤ n i=1 q i = n i=1 q j=1 z i,j , we simply apply Proposition 2.13 with g = 3 and d = q n i=1 p i .

Proof of Corollary 2.3. We will use Proposition 2.4 from [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF]. Denote again with z i,j the Bernoulli random variables contained in each Binomial random variable q i with q i = q j=1 z i,j . Then we can rewrite

ASS T A T = 1 q n i=1 q j=1 1 p i z i,j a i a T i = 1 q q j=1 n i=1 1 p i z i,j a i a T
i and it's easy to see that z i,j = 1 with probability p i and 0 otherwise. Let Y = AA T -ASS T A T , then

Y = 1 q q j=1 n i=1 1 - z i,j p i a i a T i
We choose X i,j to be

1 q 1 - z i,j p i a i a T i for every i ∈ [n], j ∈ [q]
. Now we verify the assumptions of the above theorem. The matrices X i,j are all independent due to the independence of all z i,j . We can see

E[X i,j ] = 1 q 1 - E[z i,j ] p i a i a T i = 1 q 1 - ¡ ¡ ¡ p i p i a i a T i = 0 And X i,j ≤ 1 q 1 - z i,j p i a i a T i ≤ 1 q 1 p i a i a T i = 1 q a i 2 p i ≤ 1 q max i a i 2 p i = n max i a i 2 q := R
Now we need to control the spectral norm of the second moment of Y. Again with

E[Y 2 ] = (i,j) (i ,j ) E[X i,j X i ,j ] = n i=1 q j=1 E[X 2 i,j ] + (i,j) (i ,j ) =(i,j) E[X i,j X i ,j ] = n i=1 q j=1 E[X 2 i,j ] + (i,j) (i ,j ) =(i,j) E[X i,j ] E[X i ,j ] = n i=1 q j=1 E[X 2 i,j ]
where the expectation decomposes due to the independence of X i,j and X i,j . Now for all i and j

E[X 2 i,j ] = 1 q 2 E 1 - z i,j p i 2 a i a T i a i a T i = 1 q 2 E 1 -2 z i,j p i + z 2 i,j p 2 i a i a T i a i a T i = 1 q 2 1 -2 E[z i,j ] p i + E[z 2 i,j ] p 2 i a i a T i a i a T i = 1 q 2 1 -2 E[z i,j ] p i + E[z i,j ] p 2 i a i a T i a i a T i = 1 q 2 1 -2 ¡ ¡ ¡ p i p i + p i p 2 i a i a T i a i a T i = 1 q 2 1 p i -1 a i a T i a i a T i
Where we used the fact that E[z i,j ] = p i and that z 2 i,j = z i,j . We can now derive an upper bound

E[Y 2 ] 2 = n i=1 q j=1 E[X 2 i,j ] = 1 q n i=1 1 p i -1 a i 2 a i a T i ≤ 1 q n i=1 a i 2 p i a i a T i ≤ 1 q max i a i 2 p i n i=1 a i a T i ≤ n max i a i 2 q AA T := σ 2
We can now apply the theorem

P ( Y ≥ ε) ≤ d exp    - ε 2 /2 n max i a i 2 q AA T + n max i a i 2 ε 3q   
Whenever ε ≤ 1 and AA T 2 ≥ 1 (as it is in our case) this can be simplified to

P ( Y ≥ ε) ≤ d exp - ε 2 q 4n max i a i 2 AA T 2
Proof of Corollary 2.5. The first part of the proof is identical to the proof of Proposition 2.3. The main change is in the bounding of R and σ 2 .

We begin with R, and using the definition of the probabilities p i ≥ a i 2 we have

X i,j ≤ 1 q 1 - z i,j p i a i a T i ≤ 1 q 1 p i a i a T i = 1 q a i 2 p i ≤ 1 q a i 2 a i 2 ≤ 1 q := R
To bound σ 2 We can now derive an upper bound

E[Y 2 ] 2 = n i=1 q j=1 E[X 2 i,j ] = 1 q n i=1 1 p i -1 a i 2 a i a T i ≤ 1 q n i=1 a i 2 p i a i a T i ≤ 1 q n i=1 a i 2 a i 2 a i a T i = 1 q n i=1 a i a T i = 1 q AA T := σ 2
We can now apply the theorem

P ( Y ≥ ε) ≤ d exp - ε 2 /2 1 q AA T + ε 3q
Whenever ε ≤ 1 and AA T 2 ≥ 1 (as it is in our case) this can be simplified to

P ( Y ≥ ε) ≤ d exp - ε 2 q 4 AA T 2
Proof of Lemma 2.7. We begin by noting that since Im(ΦΦ T + γΠ) = Im(Π) Π = (ΦΦ T + γΠ) 1/2 (ΦΦ T + γΠ) -1/2 , and that

Γ = (ΦΦ T + γΠ) -1/2 (ΦΦ T -ΦSS T Φ T )(ΦΦ T + γΠ) -1/2 = (ΦΦ T + γΠ) -1/2 Φ(I n -SS T )Φ T (ΦΦ T + γΠ) -1/2
is Hermitian all its eigenvalues are real. Consider an arbitrary φ a ∈ H. We have

φ T a (ΦΦ T -ΦSS T Φ T )φ a = φ T a Φ(I n -SS T )Φ T φ a = φ T a ΠΦ(I n -SS T )Φ T Πφ a = φ T a (ΦΦ T + γΠ) 1/2 (ΦΦ T + γΠ) -1/2 Φ(I n -SS T )Φ T (ΦΦ T + γΠ) -1/2 (ΦΦ T + γΠ) 1/2 φ a = φ T a (ΦΦ T + γΠ) 1/2 Γ(ΦΦ T + γΠ) 1/2 φ a ≤ λ max (Γ)φ T a (ΦΦ T + γΠ) 1/2 (ΦΦ T + γΠ) 1/2 φ a ≤ Γ φ T a (ΦΦ T + γΠ)φ a ≤ εφ T a (ΦΦ T + γΠ)φ a , and
φ T a ΦΦ T φ a -φ T a ΦSS T Φ T φ a ≤ εφ T a ΦΦ T φ a + εγφ T a Πφ a .
Similarly

φ T a (ΦSS T Φ T -ΦΦ T )φ a = φ T a (ΦΦ T + γΠ) 1/2 (-Γ)(ΦΦ T + γΠ) 1/2 φ a ≤ λ max (-Γ)φ T a (ΦΦ T + γΠ) 1/2 (ΦΦ T + γΠ) 1/2 φ a ≤ -Γ φ T a (ΦΦ T + γΠ)φ a = Γ φ T a (ΦΦ T + γΠ)φ a ≤ εφ T a (ΦΦ T + γΠ)φ a . and φ T a ΦSS T Φ T φ a -φ T a ΦΦ T φ a ≤ εφ T a ΦΦ T φ a + εγφ T a Πφ a .
Putting them together and using the definition of Positive Semi-Definiteness [START_REF] Horn | Matrix analysis[END_REF] concludes the proof.

Proof of Lemma 2.8. Using Proposition 2.11

φ T i (ΦΦ T + γΠ) -1 φ i = φ T i γ γ Π (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i (γΠ) (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i (ΦΦ T + γΠ -ΦΦ T )(ΦΦ T + γΠ) -1 φ i = 1 γ φ T i (ΦΦ T + γΠ)(ΦΦ T + γΠ) -1 φ i -φ T i ΦΦ T (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i Πφ i -φ T i ΦΦ T (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i φ i -φ T i ΦΦ T (ΦΦ T + γΠ) -1 φ i = 1 γ φ T i φ i -φ T i Φ(Φ T Φ + γI n ) -1 Φ T φ i
Proof of Lemma 2.10. Using the SVD decomposition and the invariance of the operator norm to orthonormal transformations

Γ -Γ = (ΦΦ T + γΠ) -1/2 Φ(I n -SS T )Φ T (ΦΦ T + γΠ) -1/2 = Φ(Φ T Φ + γI n ) -1/2 (I n -SS T )(Φ T Φ + γI n ) -1/2 Φ T = VΣ T (ΣΣ T + γI r ) -1/2 U T (I n -SS T )U(ΣΣ T + γI r ) -1/2 ΣV T
= U(ΣΣ T + γI r ) -1/2 ΣU T (I n -SS T )UΣ T (ΣΣ T + γI r ) -1/2 U T = P -P .

Sequential RLS Sampling

In the previous section we showed that batch LS and RLS sampling can be used to construct small and accurate dictionaries. Unfortunately this requires an oracle that returns the RLSs.

One possibility way to implement RLS sampling without the oracle is to use Definition 2.4 and the equivalence from Lemma 2.8 to compute all τ i exactly. Unfortunately, this algorithm, that we will call Exact-RLS, requires decomposing or inverting an n × n matrix Φ T Φ, which takes O(n 3 ) time and O(n 2 ) space.

In this chapter we survey existing sequential sampling processes that try to reduce this computational cost, and introduce two new algorithms for sequential RLS sampling without (KORS) and with (SQUEAK) the capability to remove samples already included in the dictionary. We show that these approach outperform all previous methods, ultimately reducing the overall computational cost of RLS sampling from O(n 3 ) time, O(n 2 ) space and repeated passes over the dataset to a near-linear O(nd eff (γ) 3 ) time, near-constant O(d eff (γ) 2 ) space and a single pass over the data.

This allows us to run our algorithms not only in the traditional random memory access computational model, but also in the more constrained sequential memory access model. The main difference between data storage solutions in the real world is whether they support random access, i.e., the capacity of reading any entry in the dataset in constant time regardless of their position in the dataset; or only support sequential access, i.e., they can only read the entries of the dataset in order, or pay a much larger time penalty for skipping to another position. Notable examples of the former is the aptly named Random Access Memory (RAM) in modern computers, while an example of the latter are magnetic hard-drives, where due to the spin times of the magnetic disk random access reading throughput is 2 to 3 orders of magnitude slower than sequential access throughput. Set approximate probability p i = τ i 5:

Draw the binomial r.v. q i ∼ B( p i , q)

6:

If q t = 0, add1

p i q i
q , φ i to I 7: end for since as the size of the storage grows indexing logic and circuitry becomes more complex and slow. Restricting ourselves to perfectly sequential access (a single pass over the dataset) places us in a streaming setting. In other words, we assume that we can see each sample only once, and we must either choose to explicitly store it in the dictionary to access it later, or discard it and permanently lose access to it. As single-pass algorithms with a constant memory requirement, KORS and SQUEAK can operate in a streaming setting.

Restrictions similar to the streaming setting also apply to the distributed computing setting, where the data is split across multiple databases or datacenters, and communication costs across centers are much higher. To avoid these costs, algorithms must run in a distributed manner, with all computations restricted to data local to each site. To counterbalance this drawback, they can now exploit the additional computational power of multiple machines, and any parallelism that the distributed algorithm allows. We will show that SQUEAK can operate in a distributed setting, running efficiently in parallel across multiple machine. This will further reduce its runtime to O(d eff (γ) 3 ).

Note that although the sampling schemes reviewed in this section were proposed in chronological order [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF], Calandriello et al., (2017a), Calandriello et al., (2017c) and [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] 1 , we slightly change the order in the presentation to capture all the methods in a general framework.

Multi-Pass RLS sampling

We begin by presenting three multi-pass algorithms that will help us introduce concepts necessary to build our single-pass, time-efficient RLS sampling algorithm.

Two-Step RLS sampling

The first algorithm that we present was introduced by [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF], together with the definition of RLSs and the first results on oracle RLSs sampling. To begin with, [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] note that the proof of Proposition 2.5 still follow through if instead of sampling according to exact RLSs τ i , we sample according to ρ-accurate RLSs.

Algorithm 4 Two-Step RLS sampling (Two-Step-RLS) Input: Map Φ, regularization γ, accuracy ε, budget q 1 , q 2 1: Initialize I = ∅ 2: for i = {1, . . . , n} do 3:

draw q i ∼ B(1/n, q 1 ) and if q i = 0, add n q i q 1

, φ i to I 4: end for 5: compute Γ = ΦS(S T Φ T ΦS + γI n ) -1 S T Φ T 6: Initialize I = ∅ 7: for i = {1, . . . , n} do 8:

compute τ i = φ T i Γ 1/2 (Γ 1/2 ΦΦ T Γ 1/2 + γΠ) -1 Γ 1/2 φ i 9: compute p i = τ i 10:
draw q i ∼ B( p i , q 2 ) and if q i = 0, add 1

p i q i q 2
, φ i to I

11: end for

Definition 3.1 An approximate RLS τ i is called ρ-accurate for ρ ≥ 1 if it satisfies 1 ρ τ i ≤ τ i ≤ τ i .
In other words, to be ρ-accurate, an approximation τ i of a RLS τ i must be smaller than τ i , so as to keep being a probability, while at the same time be larger than τ i /ρ to avoid under-representing the importance of φ i . We have the following guarantees for oracle ρ-approximate RLS sampling, reported in Algorithm 3.

Corollary 3.1 -Lemma 2.10, Proposition 2.5, Proposition 2.13. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I be the dictionary generated by Algorithm 3 run with parameters

p i = τ i , q ≥ 4ρ log(4r/δ) ε 2 ,
where the approximate RLSs τ i are ρ-accurate w.r.t. τ i , then for any 0 < ε < 1 and w.p.1δ

Accuracy: I is (ε, γ)-accurate.
Space: the size of the dictionary is bounded by

|I| ≤ 3q n i=1 τ i ≤ 3qd eff (γ).
This is coherent with our interpretation of the extra budget q: if we make a constant error on the importance τ i of φ i , i.e., running Algorithm 3 with ρ-accurate RLSs estimate, we only need a constant increase in q and in final dictionary size to obtain the desired (ε, γ)-accurate dictionary.

Knowing that ρ-accurate RLSs suffice to sample, [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] propose a simple two-step algorithm Two-Step-RLS, reported in Algorithm 4:

1 Do a first sampling pass with probabilities that are simple to compute (e.g., uniform) to get a mildly accurate dictionary I, and use I to build ρ-accurate approximations τ i . 2 Do a second sampling pass using τ i , and build an (ε, γ)-accurate dictionary I.

Proposition 3.2 - [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF]. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I, { τ i } be the dictionary and approximate RLSs generated by Two-Step-RLS (Algorithm 4) using Eq. 3.1. If

q 1 ≥ n γ 4 log(4r/δ) ε 2 , q 2 ≥ λ max -γ λ min -γ 4 log(4r/δ) ε 2 ,
then for any 0 < ε < 1 and w.

i (γ) ≤ 1 1+γ ≤ 1/γ, setting q 1 ≥ n γ 4 log(4r/δ) ε 2 guarantees that q 1 ≥ n max i τ i (γ) 4 log(4r/δ) ε 2
as requested by Proposition 2.3. We can then use Proposition 2.3 to guarantee that the dictionary built during Step 1 is an (ε, γ)-accurate dictionary with |I| ≤ 3q 1 atoms. In their original work, rather than uniform sampling [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] use the slightly more refined probabilities p i = φ T i φ i / Tr(ΦΦ T ), which provides a slightly tighter dictionary size bound of 4 Tr(ΦΦ) γ log(n/δ)/ε 2 . Nonetheless, in many cases the samples are already in unitary norm φ T i φ i = 1, (e.g., all self normalized kernels such as the Gaussian kernel) which for simplicity we assumed in this thesis, and the adaptive distribution reverts to uniform sampling. Given I, [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] propose the following estimator for τ i , that we report using our notation,

τ i (γ ) = φ T i Γ 1/2 (Γ 1/2 ΦΦ T Γ 1/2 + γ Π) -1 Γ 1/2 φ i 3.1 with Γ = ΦS(S T Φ T ΦS + γI n ) -1 S
T Φ T being the exact regularized projection matrix on the approximate Im(ΦS) (not to be confused with the approximate regularized projection matrix Γ). Note also that γ inside the inverse does not need to be equal to the γ used in Γ.

Regardless of γ , the ρ multiplicative approximation constant for τ i given by Alaoui and Mahoney, (2015, Theorem 4)

is ρ = λmax-γ λ min -γ resulting in λ min -γ λ max -γ τ i (γ ) ≤ τ i ≤ τ i (γ ).
Plugging this into an approximate RLSs sampling in Step 2, invoking Corollary 3.1, and setting q 2 appropriately, we obtain a final (ε, γ)-accurate dictionary I containing only |I| ≤ q 2 d eff (γ ) atoms.

Unfortunately the definition of q 2 include a (λ minγ) -1 factor due to the rough ρ-accuracy of the RLSs τ i computed using Eq. 3.1. This forces us to set γ < λ min , and λ min is typically smaller than 1, resulting in q 1 ≥ n/γ > n/λ min > n and an extremely large first-step dictionary that keeps all the input atoms, resulting in no computational improvement over exact RLSs calculation.

To make comparisons with following algorithms easier, we present a slightly modified version of Alaoui and Mahoney's algorithm that replaces Eq. 3.1 with an improved estimator introduced by Calandriello et al., (2017a). Given a dictionary I, let I I be the R n×n diagonal matrix with [I I ] i,i = 1 if q i = 0 and 0 otherwise.

Lemma 3.3 - (Calandriello et al., 2017a, Lemma 2). Given an (ε, γ)-accurate dictionary I, compute the approximate RLSs τ i as

τ i = (1 -ε)φ T i (ΦSS T Φ T + γΠ) -1 φ i = 1-ε γ (φ T i φ i -φ T i ΦS(S T Φ T ΦS + γI I ) -1 S T Φ T φ i ).

3.2

Then for all i in D, the estimator in Eq. 3. Proof sketch of Lemma 3.3. The full proof is reported in Section 3.5. To bound the accuracy, we know from the dictionary that

ΦSS T Φ T + γΠ (1 + ε)ΦΦ T + εγΠ + γΠ = (1 + ε)(ΦΦ T + γΠ),
and therefore (ΦSS T Φ T + γΠ) -1 1 1+ε (ΦΦ T + γΠ) -1 . Using a similar approach to obtain a lower bound, and adjusting constants, we can prove the ρ = (1+ε) (1-ε) -accuracy. Note that Lemma 2.8 is crucial to the proof. We are first showing that (1ε)φ T i (ΦSS T Φ T + γΠ) -1 φ i is ρ-accurate w.r.t. φ T i (ΦΦ T + γΠ) -1 φ i , and only afterwards we apply the equivalence of Eq. 2.14 to obtain a reformulation of the estimator that can be computed using only inner products. Proving directly the ρ-accuracy of the computable version is significantly more difficult.

Plugging Lemma 3.3 in the first step of Alaoui and Mahoney's two-step algorithm we have Corollary 3.4 -Proposition 2.3, Corollary 3.1, Lemma 3.3. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I, { τ i } be the dictionary and approximate RLSs generated by Two-Step-RLS (Algorithm 4) using Eq. 3.2 in place of Eq. 3.1. If

q 1 ≥ n γ 4 log(4r/δ) ε 2 τ i , q 2 ≥ 1 + ε 1 -ε 4 log(4r/δ) ε 2 ,
then for any 0 < ε < 1 and w. Time: the runtime of the algorithm is O(nq 2 1 ) Passes: the algorithm requires 3 passes over Φ Compared to Proposition 3.2, we have greatly improved the λmax-γ λ min -γ constant in q 2 to 1+ε 1-ε , which in turn leave us free to choose γ more freely without impacting the n/γ factor in q 1 . The total runtime of the algorithm is dominated by the RLSs estimation, that takes O(|I| 3 ) = O(q 3 1 ) to precompute the inverse or a decomposition, and O(nq 2 1 ) time to compute all individual τ i . Since q 1 must scale with n/γ, this results in a O(n 3 /γ 2 ) time and O(n 2 /γ 2 ) space complexity, down from the O(n 3 ) time and O(n 2 ) space of exact RLSs sampling. As [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] note, when γ is large, this is a significantly smaller than the exact approach. As we will see, a typical value for γ is n 1/2 , resulting in a O(n 2 ) time and O(n) space complexity.

To recap, Alaoui and Mahoney's Two-Step-RLS gave us two key ideas to quickly construct a dictionary: (1) sample according to approximate RLSs instead of exact RLSs, and (2) instead of directly sampling proportionally to the final RLSs (second step) build an intermediate approximation that is easier to compute (first step). While Lemma 3.3 is already close to the best we can do for RLSs estimation given an (ε, γ)-accurate dictionary, the idea of sequentially approximating better and better RLSs can be pushed further. In the next sections we present two algorithms that perform multiple approximation steps rather than one.

Many-Step RLS sampling

In the analysis of Two-Step-RLS [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] use guarantees for uniform sampling (Proposition 2.3) to decide how many samples are necessary in the first sampling step. In this section we take the alternative approach of showing that the estimator τ i = 1/(n + γ) can be used as an accurate ρ-accurate approximation, with constant ρ, but only for specific values of γ. It is easy to show that this is not true for all γ: if a sample is orthogonal to all others and γ = 1 we have τ i = 1/2 and τ i = 1/(n + 1), which is an arbitrarily poor approximation. Nonetheless, using the generic upper and lower bounds 1 n+γ ≤ τ i (γ) ≤ 1 1+γ we know that τ i must satisfy

1 ρ τ i (γ) ≤ 1 ρ 1 1 + γ ≤ τ i = 1 n + γ ≤ τ i (γ),
which means that in general 1 n+γ achieve ρ = n+γ 1+γ accuracy. This accuracy depends on n and γ, therefore uniform τ i might be ρ-accurate w.r.t. τ i (γ) with constant ρ for certain values of γ, and not for others. For example, setting γ = n tells us that uniform sampling is 2-accurate, w.r.t. τ i (n). In other words, when γ is small, uniform sampling is only accurate when the RLSs τ i (γ) are uniform due to the low coherence in the data; but using a high enough γ we can shrink all probabilities so much that uniform sampling becomes accurate regardless of the data. Indeed, an additive error of εnΠ on the reconstruction for an operator with norm ΦΦ T ≤ n means that we do not need to reconstruct much at all to meet (ε, n)-accuracy. In particular, Algorithm 5 Many-Step RLS sampling (Many-Step-RLS) Input: Map Φ, regularization γ, accuracy ε, budget q Output: I 1: Initialize p 0,i = τ 0,i = 1/(n + γ) for all i, , γ 0 = n 2: for t = {1, . . . , log 2 (n/γ) + 1 } do 3:

Initialize γ t = γ t-1 /2, I t = ∅ 4:
for i = {1, . . . , n} do 5:

draw q t,i ∼ B( p t-1,i , q) and if q t,i = 0, add

1 p t-1,i q t,i q , φ i to I t 6: end for 7: construct S t from I t 8: for i = {1, . . . , n} do 9: set τ t,i = (1 -ε)φ T i (ΦS t S T t Φ T + (1 + ε)γ t Π) -1 φ i 10: set p t,i = τ t,i 11:
end for 12: end for applying Corollary 3.1 tells us that we can construct an (ε, n)-accurate dictionary having only

24d eff (n) log(n/δ)/ε 2 ≤ 24 log(n/δ)/ε 2 atoms since d eff (γ) ≤ n/γ = 1 when γ = n. 2 .
Despite the low accuracy of a (ε, n)-accurate dictionary, the following lemma, based on Lemma 3.3 shows how we can use this inaccurate dictionary to compute slightly better estimates.

Lemma 3.5 Given an (ε, 2γ)-accurate dictionary I, compute the approximate RLSs τ i as

τ i = (1 -ε)φ T i (ΦSS T Φ T + (1 + ε)γΠ) -1 φ i = 1-ε (1+ε)γ (φ T i φ i -φ T i ΦS(S T Φ T ΦS + (1 + ε)γI I ) -1 S T Φ T φ i ).

3.3

Then for all i in D, the estimator in Eq. 3.3 is ρ-accurate w.r.t. τ i (γ), with accuracy ρ = (1 + 3ε)/(1ε). Moreover, τ i can be computed in O(|I| 3 ) time and O(|I|2 ) space using only φ i and atoms present in I itself.

Therefore, using Lemma 3.5 and the (ε, n)-accurate dictionary, we can compute an (ε, n/2) dictionary with only 12( 1+3ε 1-ε )d eff (n/2) log(n/δ)/ε 2 atoms, that can be used to compute an (ε, n/4) dictionary with only 12( 1+3ε 1-ε )d eff (n/4) log(n/δ)/ε 2 atoms, and so on until after log 2 (n/γ) + 1 steps we have a 12( 1+3ε 1-ε )d eff (γ) log(n/δ)/ε 2 dictionary. The resulting algorithm, Many-Step-RLS is reported in Algorithm 5.

Using a union bound on the steps we obtain the following result.

Theorem 3.6 -Lemma 3.5, Corollary 3.1, Proposition 2.13. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I t be the dictionaries generated by Many-Step-RLS (Algorithm 5). If

q ≥ 1 + 3ε 1 -ε 4 ε 2 log 4r log 2 (n/γ) + 1 δ ,
Chapter 3. Sequential RLS Sampling then for any 0 < ε < 1 and w.p.1δ Accuracy: Passes: the algorithm requires 2 log 2 (n/γ) + 1 passes over Φ or 4n random accesses Using Many-Step-RLS we can compute an (ε, γ)-accurate dictionary with a size that is bounded by 12d eff (γ) log(n log 2 (n/γ) + 1 /δ)/ε 2 . Up to a log( log 2 (n/γ) + 1 ) factor, this is the same result as Oracle-RLS, but with a computational complexity of

Each I t is (ε, n/2 t-
O(nq 2 ) ≤ O(nd eff (γ) 2 ).
Since we are interested in cases where d eff (γ) is small, this is essentially a linear runtime in n, which is an enormous improvement over the O(n 3 ) cost of the exact oracle and finally breaks the quadratic barrier that blocked previous methods. We refer to algorithms that achieve this time complexity as near-linear time algorithms.

On the other hand, when d eff (γ) is large, potentially as large as n, Many-Step-RLS recovers the O(n 3 ) runtime of computing exact RLSs using the definition and sampling. Nonetheless, in the cases where d eff (γ) is large even an oracle algorithm would generate a large dictionary, that would not be very useful to accelerate downstream applications. In this case we can either accept that the problem is intrinsically hard, i.e., there is no free lunch, or start considering increasing γ to trade-off computational cost and accuracy. We postpone a full discussion of Theorem 3.6 to Section 3.3.3, where we will make an in-depth comparison across all near-linear time oracles. We now present an alternative approach that eliminates the extra log(log(n/γ)) factor in the size of the dictionary.

Recursive RLS sampling

Another approach to multiple-step RLS sampling, called Recursive-RLS and reported in Algorithm 6, was recently proposed by [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]. For clarity, we also report an equivalent iterative version. Similarly to Many-Step-RLS, the algorithm's goal is to approximate sufficiently well τ i in order to be able to perform approximate RLSs sampling and use Corollary 3.1. While Many-Step-RLS is a strictly bottom-up approach that starts with a large γ (small d eff (γ) and dictionary) and increasingly shrinks it (large d eff (γ) and dictionary), Recursive-RLS has both a top-down and bottom-up phase.

In the top-down phase it recursively randomly halves the number of columns in the map Φ: at each level t for each column in Φ t-1 a 1/2 coin flip is performed, and if it is successful the column is included in Φ t . As a consequence we have a top-down chain of maps where Φ t w.h.p. contains O(n/2 t ) atoms, and all atoms in Φ t are included in Φ t-1 . The goal of the recursive chain is to reduce the size of Φ t sufficiently so that it can be used to compute approximate RLSs with a small computational cost. This base-case is encountered after log(n) recursion, where the algorithm constructs a dictionary

I log 2 (n) = {(1, φ i )} based on Φ log 2 (n)
that contains only a constant number of atoms.

Starting from there, the algorithm performs a bottom-up pass. At each level t the dictionary I t+1 from the lower level t + 1 is used to estimate approximate RLSs τ t,i for Φ t . The RLSs estimator used by [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] is identical to Eq. 3.2, although in their independent discovery [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] derived it starting from P, C and c i (Definition 2.8) rather than Γ, Φ, and φ i (our definition). This makes it hard to see

Set I = {(1, φ i )} for all φ i in Φ 4: else 5:
Invoke Algorithm 6 on Φ adjusting δ ← δ/2 6:

Set I to the result returned by the recursive invocation 7: end if 8: For all φ i in Φ compute τ t,i using I and Eq. 3.2 with ε = 1/2. 9: Compute approximate probabilities p t,i = min{1, 16 τ t,i log( τ t,i /δ) 10: Construct I of Φ by approx. RLSs sampling (Algorithm 2) with p i = p t,i and q = 1 Algorithm 6 Recursive-RLS sampling [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]

(Iterative version) Input: map Φ, regularization γ, budget q Output: Dictionary I 1 1: for t = {1, . . . , log 2 (n)} do Top-down 2:
Sample Φ t by randomly including each column of Φ t-1 w.p. 1/2 3: end for

4: Set I log 2 (n) = {(1, φ i )} for all φ i in Φ log 2 (n) Base case 5: for t = {log 2 (n) -1, . . . , 1} do Bottom-up 6:
For all φ i in Φ t compute τ t,i using I t+1 and Eq. 3.2 with ε = 1/2.

7:

Compute approximate probabilities p t,i = min{1, 32 τ t,i log( i τ t,i /(2 -t δ))

8:

Construct I of Φ by approx. RLSs sampling (Algorithm 2) with p i = p t,i and q = 1 9: end for that Eq. 3.2 is actually a generalization (and a member) of a well known family of RLSs estimators from the linear (R d ) setting [START_REF] Cohen | Ridge Leverage Scores for Low-Rank Approximation[END_REF][START_REF] Cohen | Online Row Sampling[END_REF], and harder to see variations such as Lemma 3.5 or Lemma 3.14. Once the approximate RLSs are computed, the dictionary I t is constructed using approximate RLS sampling (Algorithm 2) with ε = 1/2, p i = p t,i = 32 τ t,i log( i τ t,i /(2 -t δ)) and q = 1. In other words, each atom in Φ t is added to I t with probability p t,i and weight 1/ p t,i . This is essentially the approach of drawing a Bernoulli Ber(32 τ t,i log( i τ t,i /(2 -t δ))) rather than a Binomial B( τ t,i , 32 log( i τ t,i /(2 -t δ))), which as we saw in Section 2.1.2 generates slightly larger dictionaries. Note that in the run of the algorithm it is possible for an atom φ i to be present in dictionary I t , not present in I t-1 , and again present in I t-2 , since the drawings at each level are performed independently.

Proposition 3.7 - [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I = I 1 be the final dictionary generated by Recursive-RLS (Algorithm 6). With probability 1 -2δ Accuracy: the dictionary I is (1/2, γ)-accurate Space: the size of each of the dictionaries is bounded by |I| ≤ 384d eff (γ) log(d eff (γ)/δ) Time: the runtime of the algorithm is O(nd eff (γ) 2 ) Passes: the algorithm requires 2 log(n) passes over Φ or 4n random accesses Proof sketch. We provide a sketch of the proof for comparison, and refer the reader to [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] for details. At its core, Recursive-RLS is a recursive scheme where the top-down pass uses Φ t to approximate Φ t-1 through uniform sampling, and a bottom-up pass where I t is used to approximate Φ t through approximate RLSs sampling. The goal is to construct more and more accurate approximations τ t,i during the bottom-up pass, in the same spirit of Two-Step-RLS and Many-Step-RLS. Note that none of the Φ t are reweighted, therefore E[Φ t ] = (1/2 t )Φ and the top-down sampling process that generates Φ t is not unbiased. Therefore, we cannot use concentration inequalities to show that Φ t is an (ε, γ)-accurate approximation of Φ t-1 . Moreover, even if we did reweight Φ t , our results on uniform sampling (Proposition 2.3) suggest that n/2 uniform samples are not sufficient to construct an (ε, γ)-accurate dictionary w.h.p. For the bottom-up pass, since (1) we use I t+1 to approximate RLSs, (2) I t+1 approximates Φ t+1 , and (3) Φ t+1 does not approximate well Φ t , the approximate RLSs τ t,i are not ρ-accurate w.r.t. Φ t and the sampling guarantees of Corollary 3.1 do not apply. Surprisingly and non-trivially, [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] could transfer the proof proposed in Cohen et al., (2015a) for a similar RLSs sampling algorithm in R d to the RKHS setting. With a very technical argument they show that although each τ t,i does not approximate each τ t,i the following invariant holds at each level (1) τ t,i ≤ τ t,i which guarantees that we will never undersample.

(2) i τ t,i ≤ d eff (γ) log(d eff (γ)/δ) which guarantees that across all atoms we will not oversample too much. This is sufficient to show that w.h.p. all dictionaries satisfy

|I t | ≤ O( i τ t,i ) ≤ O(d eff (γ))
and will not be too large, and that although all intermediate matrix Φ t and dictionary I t are not close to Φ, the final dictionary I is (ε, γ)-accurate w.r.t. Φ.

Like Many-Step-RLS, Recursive-RLS produces an accurate dictionary in an small space and time complexity, with Proposition 3.7 improving by a log(log(n/γ)) factor over Theorem 3.6 in space (and polylog(log(n/γ)) in time). We will again postpone a full evaluation of this result to Section 3.3.3, where we compare all near-linear time oracles.

Sequential RLS sampling without removal

While both Many-Step-RLS and Musco and Musco's Recursive-RLS achieve near-linear runtime, they have two major drawbacks: (1) they require multiple passes over the data or alternatively random access to the dataset, and (2) they have inherent bottlenecks that make it difficult to parallelize them. In this section we focus on solving the first of these drawbacks with a simple sequential sampling approach that can operate in a single pass over the dataset, processing the dataset as if it was coming from a continuous stream of data. The parallelization problems will be addressed in the following section by showing how to generalize sequential sampling to operate on multiple streams at the same time.

To describe this new streaming setting, we will slightly modify our notation. Given our dataset D = {φ t } n t=1 we can define the partial dataset D t that contains the first t samples. Associated with D t we have all the corresponding quantities Φ t , Ψ t , Π t , Γ t . Similarly, our goal is now to generate dictionaries I t that are (ε, γ)-accurate w.r.t. to their respective Φ t . We can also define RLSs τ t,i and effective dimension d t eff (γ) w.r.t. the partial dataset Φ t . For notational convenience, we still refer to τ t,i as a RLSs when t < i (when we have not Algorithm 7 Streaming Oracle RLS sampling without removal (SOracle-RLS-NoRem) Input: Regularization ρ, accuracy ε, budget q 1: Initialize I 0 = ∅ 2: for t = {1, . . . , n} do 3:

receive φ t 4:

compute p t = τ t,t using an oracle limited to D t 5:

draw q t ∼ B (p t , q)

6:

if q t = 0, add 1 pt qt q , φ t to I t Expand 7: end for seen the sample yet), but in this case we will arbitrarily set τ t,i = 1 for all t < i. Remember also that in the streaming setting if a past sample φ i is not explicitly stored in the current dictionary I t , we cannot access it, i.e., we delete from memory all samples that are not explicitly stored at each step.

Oracle RLS sampling in the streaming setting

From the previous sections' results on RLS sampling, we know that choosing whether to include sample φ t or not in I t with probability p t based on the sample's final RLS τ n,t will be enough to generate an (ε, γ)-accurate dictionary. This is essentially what Oracle-RLS does, with a computationally unbounded oracle providing the exact RLSs τ n,t . Unfortunately, in the streaming setting at time t we did not yet see the nt future samples remaining in the stream, and not even the oracle can compute these quantities. We will now present SOracle-RLS-Rem, a simple oracle-based algorithm, reported in Algorithm 7, that generalizes Oracle-RLS to the streaming setting. We call this approach sequential RLS sampling without removal because once an atom is added to the dictionary I t , it is never subsequently removed. The basic idea behind SOracle-RLS-NoRem is to use the sample's current RLS τ t,t to decide whether or or not to include it without waiting to see the rest of the stream. Even when using exact RLS τ t,t , this approach can be grouped in the family of approximate RLS sampling methods: since we cannot (this time in an absolute sense) compute the probabilities τ n,t , and we try to approximate them with τ t,t . Due to the oracle, all the samples in the dictionary remain independent, and if we can satisfy the invariant p i = p t = τ t,t ≥ τ n,t required by Proposition 2.5 we will generate an (ε, γ)-accurate dictionary. Moreover, if the sum t i=1 τ i,i is not much larger than d t eff (γ), we can use Proposition 2.2 to show that we generate a small dictionary. We now focus on characterizing how RLSs and d t eff (γ) evolve over time as new samples arrive (e.g., the relationship between τ t,i and τ t+1,i ).

Lemma 3.8 - (Calandriello et al., 2017a). For any map Φ t-1 at time t -1 and its extension Φ t at time t, we have that the RLS are monotonically decreasing and the effective dimension is monotonically increasing,

1 τ t-1,i + 1 τ t-1,i ≤ τ t,i ≤ τ t-1,i , d t eff (γ) ≥ d t-1 eff (γ).
It is easier to show that RLSs shrink over time once we define them in terms of Φ and φ i Chapter 3. Sequential RLS Sampling rather than C and c i 3 [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] which change size when we add the new sample. It suffices to see that

τ t,i = φ i t s=1 φ s φ T s + γΠ t -1 φ i ≤ φ i t-1 s=1 φ s φ T s + γΠ t -1 φ i = τ t-1,i , since t-1 i=1 φ i φ T i t i=1 φ i φ T
i has one less term in the summation. Intuitively, adding a new sample φ t to Φ t-1 can either decrease the importance of samples observed before (i.e., if they are correlated with the new point) or leave it unchanged (i.e., if they are orthogonal) and thus for any i ≤ t, the RLSs will shrink. Each RLS will shrink by a different factor, depending on the degree of orthogonality of each sample with the new point and the rest of the dataset, but there is a maximum shrinkage factor

1 τ t-1,i +1 ≤ 1
2 , that roughly corresponds to the case where the new sample φ i added to Φ t is identical to a sample φ i already present in Φ t . As for the effective dimension, relating

d t eff (γ) = t i=1 τ t,i with d t-1 eff (γ) = t-1 i=1 τ t-1,i
is not as easy, since adding τ t,i increases the total sum, but each of the other τ t,i might shrink compared to τ t-1,i . We can nonetheless show that the effective dimension d t eff (γ) always grows over time, since it represents the total amount of directions larger than the regularization γ present in Φ t and adding a sample will either strengthen existing directions or add a new one. Quantifying or even bounding this increase is much more difficult than in the case of RLSs, and therefore estimating the changes in effective dimension without estimating each RLSs is much more difficult than estimating changes in RLSs [START_REF] Calandriello | Analysis of Nyström method with sequential ridge leverage score sampling[END_REF]. Using the fact that τ t,t ≥ τ n,t , and that all sampling is performed independently accordingly to the oracle, we can invoke Proposition 2.5 with p t = τ t,t and q ≥ 4 log(2r/δ)/ε 2 to guarantee that I n is (ε, γ)-accurate w.r.t. Φ n . Unfortunately, to union bound the failure (generating an inaccurate or too large dictionary) probability across n iteration, compared to Oracle-RLS we must choose a larger q ≥ 4 log(2n/δ)/ε 2 ≥ 4 log(2r/δ)/ε 2 . This is necessary to guarantee that each of the dictionaries I t is (ε, γ)-accurate w.r.t. Φ t with 1δ/n probability, and provide accuracy guarantees for the overall process.

Following a similar path as the one used for Corollary 2.9 in the batch setting, we now need to use Proposition 2.2 to bound the size of the dictionaries I t . Unfortunately, to apply it we must look at the sum t i=1 p i = t i=1 τ i,i of the probabilities used by SOracle-RLS-NoRem, and it is not clear how this quantity relates to d t eff (γ). Knowing that τ t,i ≥ τ t-1,i , it is easy to give a lower bound

d t eff (γ) = t i=1 τ t,i = τ t,t + t-1 i=1 τ t,i ≤ τ t,t + t-1 i=1 τ t-1,i = τ t,t + d t-1 eff (γ) ≤ t i=1 τ i,i ,
which shows that we may pay a price in space complexity when we replace τ n,i with τ i,i .

Finding an upper bound to quantify this price is much more difficult. The simplest case is when all samples are orthogonal. Then τ 1,1 = τ 2,1 = τ t,1 = 1 1+γ , and the two quantities across all samples,

t i=1 τ i,i = t i=1 1 i + γ ≤ log(t + γ) + 1, d t eff (γ) = t t + γ ,
then one quantity is constant while the other is only log(n) times larger. Fortunately, we can show that log(t) is the largest gap that we can incur.

Definition 3.2 The online effective dimension of a map Φ : R t → H is

d t onl (γ) := t i=1 τ i,i (γ).
We call d t onl (γ) the online effective dimension because it is influenced by the order in which we see the samples.

Lemma 3.9 - (Calandriello et al., 2017c). For any map Φ : R t → H and any γ > 0 we have that its online effective dimension d t onl (γ) = t i=1 τ t,t is bounded by

d t onl (γ) ≤ log(Det(Φ T t Φ t /γ + I t )) ≤ d t eff (γ)(1 + log( Φ T t Φ t /γ + 1)) ≤ 2d t eff (γ) log(n/γ).
The first inequality relates d t onl (γ) to the log-determinant of the matrix Φ T t Φ t . As we mentioned when we introduced the concept of effective dimension (see Section 2.2.1), this quantity appears in a large number of works on linear models, where it is connected to the maximal mutual information gain in Gaussian processes [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design[END_REF], or the regret in online optimization [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF]. Finally, the second inequality shows that in general the complexity of measuring this capacity sequentially, i.e., the ratio between the online and offline sum of the RLS, is only a log(n) factor (in the worst case). This allows us to prove the following corollary Corollary 3.10 -(Lemma 3.9, Corollary 2.9). Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I t be the dictionaries generated by SOracle-RLS-NoRem (Algorithm 7). If

p t = τ t,t , q ≥ 4 log(4n/δ) ε 2 , with n i=1 p i = d onl (γ), then for any 0 < ε < 1 and w.p.1 -δ Accuracy: Each I t is (ε, γ)-accurate w.r.t. Φ t Space: the size of each of the dictionaries is bounded by |I t | ≤ 3qd t onl (γ) ≤ 6qd t eff (γ) log(n)
With this result, we have essentially matched the O(d n eff (γ)) space complexity of oracle batch RLS sampling (i.e., sampling according to τ n,i ), up to a small increase in the logarithmic factors (from log(r) to log(n)) due to the union bound across iterations, and an extra log(n) factor due to the gap between online and offline effective dimension. At the same time, SOracle-RLS-NoRem can be run in a streaming setting, since the algorithm does not need to know in advance the final RLSs τ n,i to execute.

Chapter 3. Sequential RLS Sampling Algorithm 8 Kernel online row sampling (KORS) Input: Regularization ρ, accuracy ε, budget q 1: Initialize I 0 = ∅ 2: for t = {1, . . . , n} do 3:

receive φ t 4:
construct temporary dictionary I t, * :

= I t-1 ∪ (1, φ t ) 5: compute τ t,t = (1 -ε)φ T i (Φ t S t, * S T t, * Φ T t + γΠ t ) -1 φ i using the Eq. 3.2 estimator 6: compute p t = τ t,t 7:
draw q t ∼ B ( p t , q)

8:

if q t = 0, add 1 pt qt q , φ t to I t Expand 9: end for

Implementing the oracle and guarantees

We can now remove the necessity of an oracle, and introduce KORS, reported in Algorithm 8, our first single-pass RLS sampling algorithm that achieves near-linear runtime and is not based on an oracle. Note that since computing exact RLSs τ t,t would be too expensive, we will again resort to approximate RLSs τ t,t and approximate probabilities p t = τ t,t . Starting from an empty dictionary I 0 , at each time step the algorithm loads sample φ t into memory and combines the new sample (1, φ t ) to the dictionary I t-1 to constructs the temporary dictionary I t, * . Using the associated selection matrix S t, * , this augmented dictionary can be effectively used to compute τ t,t using the Eq. 3.2 RLS estimator,

τ t,i = (1 -ε)φ T i (Φ t S t, * S T t, * Φ T t + γΠ) -1 φ i = 1-ε γ (φ T i φ i -φ T i Φ t S t, * (S T t, * Φ T t Φ t S t, * + γI It, * ) -1 S T t, * Φ T t φ i ).

3.4

Note that combining two dictionaries I and I , each (ε, γ)-accurate w.r.t. to disjoint datasets D and D generates a dictionary that is (ε, γ)-accurate w.r.t. D ∪ D . Therefore, if I t-1 is (ε, γ)-accurate, combining it with (1, φ t ) makes I * an (ε, γ)-accurate dictionary, because (1, φ t ) can be seen as a (0, 0)-accurate dictionary w.r.t. to φ t . Taking into account Lemma 3.3, we see that if I * is (ε, γ)-accurate, then τ t,t is ρ-accurate. This cycle of using an accurate dictionary to estimate accurate approximate RLSs, and accurate approximate RLSs to compute accurate dictionaries will be at the core of the analysis of the algorithm in Theorem 3.11.

After computing the approximate RLS, KORS sets the approximate probability p t = τ t,t and draws a Binomial q t ∼ B( p t , q). If q t = 0, we choose to Expand I t-1 and φ t is added with weight 1 pt qt q to the next dictionary I t . If q t = 0, the sample is not added and will be discarded at the end of the iteration to free space. Similarly to Two-Step-RLS or Many-Step-RLS, all rows and columns for which S t, * is zero (all points outside the temporary dictionary I t, * ) do not influence the estimator, so they can be excluded from the computation. Therefore, once a sample φ t is dropped from memory we do not need to re-load it anymore for the rest of the algorithm's execution. This means that we only need to load the dataset once and sequentially, making KORS a single-pass algorithm. This is vital in the streaming setting where we cannot go back and resample a discarded atom. Moreover, since the difference between I t-1 and I t-1, * or I t is a single atom φ t , τ t,i can be efficiently computed in O(|I t, * | 2 ) space and O(|I t, * | 2 ) time using an incremental version of Eq. 3.4 that exploits rank-1 updates. Overall, the algorithm will take O(n max t |I t | 2 ) time and O(max t |I t | 2 ) space to run.

We are now ready to state the main result of this section.

Theorem 3.11 - (Calandriello et al., 2017c). Given parameters 0

< ε ≤ 1, 0 < γ, 0 < δ < 1, let ρ = 1+ε
1-ε and run KORS (Algorithm 8) with budget q ≥ 4 log(2n/δ)/ε 2 . Then w.p. 1δ, Accuracy: each I t is (ε, γ)-accurate w.r.t. Φ t and each τ t,t is ρ-accurate w.r.t. τ t,t .

Space: the size of each dictionary is bounded by |I t | ≤ 3qd t onl (γ) ≤ 6d t eff (γ)q log(2n/γ). Time: the algorithm runs in O(d t eff (γ) 2 ) per-iteration time, and O(nd n eff (γ) 2 ) overall time.

Passes: the algorithm requires a single pass over Φ Theorem 3.11 shows that we can construct an (ε, γ)-accurate dictionary containing only

O(d n eff (γ) log 2 (n)) atoms. Although this is a O(log(n))
factor larger than SOracle-RLS-NoRem and batch methods like Many-Step-RLS or Recursive-RLS, KORS does not require an oracle or multiple passes over the data. Moreover, unlike Recursive-RLS, it provides accuracy guarantees for all intermediate dictionaries. Therefore, if we choose to terminate the algorithm early, we still have gained an (ε, γ)-accurate dictionary of the data we have processed up to that point. This is especially important because the size of the dictionary scales with d t eff (γ), which is not known in advance. Therefore, if during KORS's execution we see that the dictionary is growing too large and decide to terminate it, we can use the intermediate dictionary to study the spectral properties of Φ t and choose a better regularization γ for future runs. Finally, note that KORS is a generalization of ORS (online row sampling) by [START_REF] Cohen | Online Row Sampling[END_REF] from Euclidean to Hilbert spaces. While we drew inspiration from their algorithm to derive KORS (e.g., the estimator in Lemma 3.3), moving to an infinite-dimensional space introduces new difficulties, such us finding a way to compute the estimator, how to replace the dimensionality d with d t onl (γ) in the algorithm analysis, and how to bound it.

Proof sketch: The first step in the proof is to reformulate the event failure Γ t -Γ t ≥ ε using the equivalent finite-dimensional P t -P t ≥ ε failure condition. Notice that the matrices P t -P t change size over time. To avoid this, we carefully decompose the failure event across the whole process into separate failure events for each matrix P t -P t ≥ ε, and construct a dedicated random process Y t that models how KORS generates the dictionary P t . These processes are sequential in nature and their steps are not i.i.d., since τ t,t , and therefore P t , depends on τ t-1,t-1 , q t-1 and P t-1 . As a consequence we cannot use concentrations for i.i.d. r.v. such as Proposition 2.4. Instead, following an approach introduced in [START_REF] Cohen | Online Row Sampling[END_REF], we use Freedman's inequality and a deterministic bound on the predictable quadratic variation (variance) W t of the process to bound each failure probability.

Note also that KORS can be seen as a special case of SQUEAK, an algorithm that will be introduced in the next section. Therefore, we will only provide a proof for SQUEAK, highlighting when necessary how things can be specialized to address KORS. For a whole proof dedicated solely to KORS, we direct the reader to Calandriello et al., (2017c).

Sequential RLS Sampling with removal

KORS has two clear drawbacks. First, over-sampling according to τ t,t rather than τ n,t increases the dictionary size by a log(n) factor. This is due to the fact that without a

Chapter 3. Sequential RLS Sampling Algorithm 9 Streaming Oracle RLS sampling with removal (SOracle-RLS-Rem) Input: Regularization ρ, accuracy ε, budget q 1: Initialize I 0 = ∅ 2: for t = {1, . . . , n} do 3:

receive φ t 4:

add (1, q, φ t ) to I t-1 Expand 5:

for s = {i ∈ [t] : q t-1,i = 0} do 6:
compute p t,s = τ t,s using an oracle limited to D t 7:

draw q t,s ∼ B (p t,s /p t-1,s , q t-1,s ) 8: if q t,s = 0, update (p t,s , q t,s , φ s ) in I t 9: if q t,s = 0, remove (p t-1,s , q t-1,s , φ s ) in I t Shrink 10:
end for 11: end for way to remove samples once we notice that they are not as relevant as we thought, we cannot recover from these overestimation mistakes. Second, the algorithm cannot be easily parallelized, due to the sequential nature of the RLSs estimation. In this section, we will show how adding the possibility of removing samples from the dictionary to the sampling process can cope with both drawbacks.

Oracle RLS sampling with removal

We begin with a simple oracle algorithm to illustrate the problem of RLS sampling with removal.

From the analysis of Lemma 3.8 we know that τ t,t ≥ τ t+1,t . Therefore, sampling q t proportionally to τ t,t in SOracle-RLS-NoRem is an overestimate of the current τ t ,t for all step t > t. To correct this, a simple approach is to continue updating q t := q t,t over time, i.e., also for samples already included in I t-1 , and generate q t+1,t , . . . q n,t proportional to the correct τ t+1,t , . . . , τ n,t . We can formalize this using the following adaptive importance sampling chain q t,t ∼ B (p t,t , q) , q t+1,t |q t,t ∼ B (p t+1,t /p t,t , q t,t ) , q t+2,t |q t+1,t ∼ B (p t+2,t /p t+1,t , q t+1,t ) , . . . ,

q n,t |q n-1,t ∼ B (p n,t /p n-1,t , q n-1,t ) ,
where the conditioning q t+1,t |q t,t is necessary to make B (p t+1,t /p t,t , q t,t ) a well defined Binomial. Before continuing, we also extend the dictionary notation 1 pt,s qt,s q , φ s to explicitly keep track of the probabilities and the number of copies {(p t,s , q t,s , φ s )}, where the weights 1 pt,s qt,s q can be recomputed from the dictionary when necessary. We can now introduce our oracle sequential RLSs sampling algorithm with removal, reported in Algorithm 9, where "with removal" highlights that now atoms previously added to the dictionary might be removed by the adaptive importance sampling process. At each step, the algorithm receives a new point φ t , but unlike SOracle-RLS-NoRem it always Expand the previous dictionary I t-1 to include it with probability and number of copies (p t-1,t = 1, q t-1,t = q, φ t ). Remember that our convention is to set τ t-1,t = 1 for samples that we did not see yet. Therefore, we are still initializing our importance sampling chain with a draw from a Binomial B(p t-1,t , q) = B(τ t-1,t , q) = B(1, q) whose outcome is always q. Since SOracle-RLS-Rem, unlike SOracle-RLS-NoRem, always Expand the dictionary at each step, the size of the dictionary and our memory usage always increases. In order to compensate this and decrease the size of the next dictionary I t , the algorithm iterates over all atoms still in the dictionary I t-1 (all atoms with a non-zero q t-1,i ) to adjust their number of copies q t,i using importance sampling. The Shrink step draws a sample from the Binomial B(p t,i /p t-1,i , q t-1,i ), where the fact that p t,i = τ t,i ≥ τ t-1,i = p t-1,i ensures that the Binomial probability is well defined (i.e., p t,i /p t-1,i ≤ 1). This resampling step tracks the changes in the RLS and constructs a new dictionary I t , which is as if it was created from scratch using all the correct RLS τ t,i at time t (with high probability). We can now clarify the relationship between the role of Expand in SOracle-RLS-Rem and the previous SOracle-RLS-NoRem. Since φ t was added with p t-1,t = 1 and q t-1,q , the Shrink step inserts it in the new dictionary I t by drawing from B(p t,t /p t-1,t , q t-1,t ) = B(τ t,t /1, q), which makes the Expand-Shrink combination in SOracle-RLS-Rem equivalent to the Expand of SOracle-RLS-NoRem. In other words, the new element φ t is kept in the dictionary (q t,t = 0) only if its RLS p t,t = τ t,t is high. At the same time, the number of copies of the older atoms already in I t-1 are stochastically reduced using Shrink to reflect the reductions of the RLSs. The lower p t,i is w.r.t. p t-1,i , the lower the number of copies q t,i will be w.r.t. q t-1,i . As the probability p t,i continues to decrease over time, q t,i may become zero, and the atom φ i is completely dropped from the dictionary (by setting its weight to zero).

We show that using this adaptive importance sampling q n,t follows a Binomial distribution B(τ n,t , q). To see this, rewrite q t,i = q j=1 z t,i,j as the sum of q individual coin flip chains z t,t,j . The j-th coin flip chain evolves following the rule

z t,i,j ∼ Ber p t,i p t-1,i if z t-1,i,j = 1, 0 otherwise,
or in other words we continue flipping coins (drawing from a Bernoulli) only if the previous coin flip succeeded, and set the rest of the chain to 0 at the first failure. As a consequence of this definition, we have P(z t+1,t,j = 1) = P(z t+1,t,j = 1|z t,t,j = 1)P(z t,t,j = 1) + P(z t+1,t,j = 1|z t,t,j = 0)P(z t,t,j = 0) = p t+1,t p t,t p t,t + z 0 P(z t+1,t,j = 1|z t,t,j = 0) P(z t,t,j = 0), and iterating P(z n,t,j = 1) = P(z t,t,j = 1)P(z t+1,t,j = 1|z t,t,j = 1) . . . P(z n,t,j = 1|z

I4 I3 + φ4 I3 I2 + φ3 I2 I1 + φ2 I1 φ1 φ2 φ3 φ4
(a) Expand-Shrink cycle, chain view. unconditionally distributed as B(p t,i , q). Combining this with the fact that the probabilities p t,i = τ t,i are provided by an oracle, and the fact that the RLSs p t,i = τ t,i are decided by the data and not the random process, we have that all of the q t,i are appropriately distributed independent Binomials, and we can simply invoke Proposition 2.5 and a union bound over n steps to guarantee the following.

I4 I3 + {(1, φ4)} I3 I2 + {(1, φ3)} I2 I1 + {(1, φ2)} I1 {(1, φ1)} {(1, φ2)} {(1, φ3)} {(1, φ4)} (b) Expand-Shrink cycle, tree view.
Corollary 3.12 -Lemma 2.10, Proposition 2.5, Proposition 2.13. Consider an arbitrary map Φ : R n → H with φ i as its i-th column and rank r. Given γ > 0, let Γ and Γ be defined according to Definition 2.5, and the RLS τ i according to Definition 2.7. Let I t be the dictionaries generated by SOracle-RLS-Rem. If

p t,t = τ t,t , q ≥ 4 log(4n/δ) ε 2 ,
with n i=1 p t,i = d t eff (γ), then for any 0 < ε < 1 and w.p.1δ Accuracy: Each I t is (ε, γ)-accurate w.r.t. Φ t Space: the size of each of the dictionaries is bounded by

|I t | ≤ 3qd t eff (γ)
The space and accuracy of oracle sequential RLS sampling with removal (Corollary 3.12) is almost identical to the space and accuracy of oracle batch RLS sampling (Corollary 2.9), with the only difference that we are forced to increase again our O(log(r)) dependency to O(log(n)) to balance a union bound over n steps. This is not surprising since SOracle-RLS-Rem is equivalent to running Oracle-RLS over the sequence of maps Φ t , but paying attention to do so in a single pass over the dataset, thanks to the sequential adaptive importance sampling rather than repeatedly drawing fresh Binomials. Nonetheless, it shows that it is possible (given the oracle) to construct dictionaries as small and accurate as in the batch setting, but using only a single pass on the data.

Before showing how to implement the oracle, we first show how a simple modification of SOracle-RLS-Rem leads to an algorithm amenable to parallel and distributed computing. We begin by highlighting (Fig. 3.1(a)) that SOracle-RLS-Rem can be seen as a chain of n steps that at each step t combines an (ε, γ)-accurate dictionary I t-1 and a new sample φ t to construct a new dictionary I t .

Although adding an extra Shrink pass improved the space complexity of the algorithm, this is still the same sequential approach of KORS of continuously combining an old dictionary Algorithm 10 SeQUential Approximation for Kernel sampling (SQUEAK)

Input: Dataset D, parameters k, γ, ε, δ Output: I D 1: Partition D into k disjoint sub-datasets D i 2: Initialize I D i = {( p 0,j = 1, q 0,j = q, φ j ) : j ∈ D i } 3: Build set S 1 = {I D i } k i=1 4: for h = 1, . . . , k -1 do 5:
Pick two dictionaries I D , I D from S h 6:

Set D * = D ∪ D 7:

I * = I D ∪ I D Expand 8:
for all ( p i , q i , φ i ) ∈ I * do 9:

Compute

τ i = (1 -ε)φ T i (Φ D * S * S T * Φ T D * + (1 + ε)γΠ D * ) -1 φ i 10: Set p i, * = min{ τ i , p i } 11: Set q i, * ∼ B( p i, * / p i , q i ) 12: If q i, * = 0, update ( p i, * , q i, * , φ i ) ← $ $ $ $ $ ( p i , q i , φ i ) in I * 13: If q i, * = 0, remove ( p i , q i , φ i ) from I * Shrink 14:
end for 15:

Place I D∪D := I * back into S h+1 16: end for 17: Return I D , the last dictionary in S k

Dict-Merge

and new data, as the new data arrives. But in the discussion of Eq. 3.4, we previously argued that what KORS is actually doing is combining two (ε, γ)-accurate dictionaries, since φ t can be seen as a perfect (0, 0)-accurate dictionary {(1, φ t )} w.r.t. the singleton dataset {φ t }. If we take this generalized approach to SOracle-RLS-Rem, we see that we can generalize the chain structure of Fig. 3.1(a) into the tree structure of Fig. 3.1(b), where at each step we merge the previous dictionary I t-1 with another dictionary {(1, φ i )}. It is therefore reasonable to imagine an even more general algorithm that follows an arbitrary merge tree structure, and recursively performs dictionary merges were the two dictionaries being merged can contain either fresh data or the result of a previous merge. In the next section, we introduce a new approximate sequential RLS sampling algorithm, based on this dictionary merging strategy, that is amenable to parallelization, and a more sophisticated analysis that shows that it achieves almost the same space and accuracy as SOracle-RLS-Rem without the need of an oracle.

Implementing the oracle and guarantees

Replacing the sequential merges of dictionaries and a single sample in SOracle-RLS-Rem with general dictionary-dictionary merges allows us to parallelize and distribute the computation of the dictionary I n over multiple machines, thus reducing even further its time complexity. Beside the computational advantage, a distributed architecture is needed as soon as the input dimension d and the number of points n is so large that having the dataset on a single machine is impossible. Furthermore, distributed processing can reduce contention on bottleneck data sources such as databases or network connections. SQUEAK (Algorithm 10) partitions D over multiple machines and the (small) dictionaries The initial dataset is partitioned over k disjoint sub-datasets D i with i = 1, . . . , k and k dictionaries I D i = {( p 0,j = 1, q 0,j = q, φ j ) : j ∈ D i } are initialized simply by placing all samples in D i into I with weight 1 and multiplicity q. Alternatively, if the datasets D i are too large to fit in memory, we can run KORS on each of them separately to generate the initial dictionaries. The dictionaries I D i are added to a dictionary collection S, and following any predefined binary merge tree as in Fig. 3.2, these dictionaries are progressively merged together. To describe the merge tree, we introduce additional notation: we index each node in the merge tree by its height h and position l, where the root (top) of the tree is at height k -1, and the leaves (original datasets) are at height 1. We denote the dictionary associated to node {h, l} by I {h,l} and the collection of all dictionaries available at height h of the merge 3.3 Sequential RLS Sampling with removal 57 tree correspond to S h = {I {h,l} }. We also use Φ {h,l} to refer to the map constructed from the datasets D {h,l} , which contains all points present in the leaves reachable from node {h, l}. For instance in Fig. 3.2, node {3, 1} is associated with I {3,1} , which is (as we will prove) an (ε, γ)-accurate dictionary w.r.t. to the map Φ {3,1} constructed from the dataset D {3,1} . D {3,1} contains D 1 , D 2 , D 3 (descendent nodes are highlighted in red) and it has dimension

I {4,1} I {3,1} + I {3,2} I {3,1} I {2,1} + I {2,2} I {2,1} I {1,1} D 1 I {2,2} I {1,2} + I {1,3} I {1,2} D 2 I {1,3} D 3 I {3,2} I {2,3} I {2,3} I {1,4} + I {1,5} I {1,4} D 4 I {1,5} D 5 h = 4 h = 3 h = 2 h = 1
(|D 1 |+|D 2 |+|D 3 |).
At each iteration of SQUEAK, one node in the tree is scheduled to be merged. Given the two input dictionaries I D and I D contained in the two children of the node, we combine them into a single dictionary I * (the equivalent of the Expand phase in KORS) and then we Shrink the merged dictionaries to create an updated dictionary I D∪D , which is placed back in the dictionary collection S. Overall we call this dictionary merging operation Dict-Merge.

To perform the Shrink part of Dict-Merge, we need to estimate how the RLSs have changed when combining D and D . Unlike in KORS, Dict-Merge is run on the union of two distinct dictionaries rather than one dictionary and a new single point, and must track the changes following the merge of two datasets rather than a dataset and a new single point. As a result, we need to derive the "distributed" counterparts of Lemma 3.8 and Lemma 3.3 to analyze the behavior of the RLSs and the quality of the estimator used in the algorithm.

Lemma 3.13 - (Calandriello et al., 2017a). Given two disjoint datasets D, D , for every

i ∈ D ∪ D , τ D,i ≥ τ D∪D ,i and 2d D∪D eff (γ) ≥ d D eff (γ) + d D eff (γ) ≥ d D∪D eff (γ).
While in KORS we were merging an (ε, γ)-accurate dictionary I t and a new point, which is equivalent to a perfect, (0, 0)-accurate dictionary, in SQUEAK both dictionaries used in a merge are only (ε, γ)-accurate. We need to slightly adjust our RLS estimator to compensate for this loss in accuracy.

Lemma 3.14 - (Calandriello et al., 2017a)

τ i = τ D∪D ,i = (1 -ε)φ T i (Φ D * S * S T * Φ T D * + (1 + ε)γΠ D * ) -1 φ i = 1-ε (1+ε)γ (φ T i φ i -φ T i Φ D * S * (S * T Φ T D * Φ D * S * + (1 + ε)γI I * ) -1 S * T Φ T D * φ i ), 3.5
Then for all φ i in D * , the estimator in Eq. 3.5 is ρ-accurate w.r.t. τ D∪D ,i (γ), with ρ = (1 + 3ε)/(1ε). Moreover, τ i can be computed in O(|I| 3 ) time and space using only φ i and atoms present in I * itself.

We denoted again with I I * the R |D * |×|D * | diagonal matrix with [I I * ] i,i = 1 if q i = 0 in I * and 0 otherwise. Notice that Eq. 3.5 is (except for the specific dictionaries involved) identical to Eq. 3.3 that we used in the multi-step batch RLS sampling Algorithm 5 when we had a single (ε, 2γ)-accurate dictionary and we wanted to estimate γ-RLSs.

We can now make a comparison: in our first proposed estimator Eq. 3.2 we had access to a dictionary with γ error and a new sample with 0 error and achieved 1+ε 1-ε -accuracy, in Eq. 3.3 we had a single dictionary with double error 2γ and a new sample with 0 error and achieved 1+3ε 1-ε -accuracy here we have two dictionaries with γ error and achieve the same 1+3ε 1-ε -accuracy. This shows the flexibility of Eq. 3.2. For example, it is easy to generalize both Eq. 3.5 and Eq. 3.3 to be able to handle k-way merges of (ε, γ)-accurate dictionaries, or the use of a single (ε, kγ)-accurate dictionary and achieve 1+(kn-1)ε 1-ε -accuracy. To conclude the RLSs estimation step, we also take the minimum p i, * = min{ τ i , p i }, or in other words the minimum between the current RLS estimate and the one at the lower level of the tree. This is necessary to guarantee that the following importance sampling step B( p i, * / p i , q i ) remains well defined, and the estimator is not ruined since taking the minimum of two ρ-accurate approximate RLSs w.r.t. to a growing dataset always preserves ρ-accuracy.

Lemma 3.15 Given two approximate RLSs τ D∪D ,i and τ D,i , ρ-accurate RLS w.r.t. τ D∪D ,i and τ D,i , the quantity min τ D∪D ,i , τ D,i is also an ρ-accurate RLS w.r.t. τ D∪D ,i .

Since each merge takes as input two dictionaries and returns one, starting from k subdatasets D i , after k merges we will be left with a single dictionary I D that approximates the whole dataset. Note that Dict-Merge only takes the two dictionaries as input and does not require any information on the dictionaries in the rest of the tree. Therefore, we do not need to run the k merges sequentially, but separate branches can be run simultaneously on different machines, and only the resulting (small) dictionary needs to be propagated to the parent node for the future Dict-Merge.

We can now show that the final dictionary I D is small and (ε, γ)-accurate w.r.t. Φ D .

Theorem 3.16 Given parameters 0 < ε ≤ 1, 0 < γ, 0 < δ < 1, an arbitrary dataset D, and a arbitrary merge tree structure of height k, let ρ = 1+3ε 1-ε and run Algorithm 10 with budget q = 26ρ log(3n/δ) ε 2 . Then w.p. 1δ, Accuracy: each dictionary I {h,l} is (ε, γ)-accurate w.r.t. Φ {h,l} , and each estimate τ D,i is ρ-accurate w.r.t. τ D,i .

Space: the size of each dictionary is bounded by |I {h,l} | ≤ 3qd D {h,l} eff (γ), the overall space requirements of the algorithm depend on the exact shape of the merge tree.

Time: the algorithm runs in O(d D eff (γ) 3 ) per-merge time, the overall time requirements of the algorithm depend on the exact shape of the merge tree.

Passes: the algorithm requires a single pass over Φ Theorem 4.7 gives approximation and space guarantees for every node of the tree. In other words, it guarantees that each intermediate dictionary processed by SQUEAK is both an (ε, γ)-accurate approximation of the datasets used to generate it, and requires a small space proportional to the effective dimension of the same dataset. From an accuracy and space perspective, up to a small ρ constant factor SQUEAK provides exactly the same guarantees as the original batch oracle Nyström sampling from Oracle-RLS, without knowing the final RLSs in advance and without multiple passes over the data. Analysing the runtime space and computational complexity of SQUEAK is however more complex, since the order and arguments of the Dict-Merge operations are determined by the merge tree. We distinguish between the time and work complexity of a tree by defining 3.3 Sequential RLS Sampling with removal 59 I 1,2,3,4,5 the time complexity as the amount of time necessary to compute the final solution, and the work complexity as the total amount of operations carried out by all machines in the tree in order to compute the final solution. We consider two special cases, reported in Fig. 3.3, a fully balanced tree (all inner nodes have either two inner nodes as children or two leaves), and a fully unbalanced tree (all inner nodes have exactly one inner node and one leaf as children). For both cases, we consider trees where each leaf dataset contains a single point D i = {φ i }, and the tree will contain n merges.

I 1,2,3 + I 4 I 1,2,3 I 1 + I 2,3 I 1 I 1 D 1 I 2,3 I 2 + I 3 I 2 D 2 I 3 D 3 I 4 I 4 I 4 D 4 s = 4 s = 3 s = 2 s = 1 (a) arbitrary tree I 1,2,3,4 I 1,2,3 + I 4 I 1,2,3 I 1,2 + I 3 I 1,2 I 1 + I 2 I 1 D 1 I 2 D 2 I 3 I 3 D 3 I 4 I 4 I 4 D 4 s = 4 s = 3 s = 2 s = 1 (b) fully unbalanced (sequential) tree I 1,2,3,4 I 1,2 + I 3,4 I 1,2 I 1,2 I 1 + I 2 I 1 D 1 I 2 D 2 I 3,4 I 3 + I 4 I 3 I 3 D 3 I 4 I 4 D 5 s = 4 s = 3 s = 2 s = 1 (c) fully balanced (minimum depth) tree
In the fully unbalanced tree, we always merge the current dictionary with a new dataset (a single new point) and no Dict-Merge operation can be carried out in parallel. Unsurprisingly, the sequential algorithm induced by this merge tree is strictly equivalent to sequential RLSs sampling with removal, i.e., SOracle-RLS-Rem without the need of an oracle. The runtime is dominated by the n matrix inversions used to compute Eq. 3.5, each requiring O(|I {h,l} | 3 ) time, and by the RLSs estimation also requiring O(|I {h,l} | 3 ) time. Therefore, computing a solution in the fully unbalanced tree takes O(n max h,l |I {h,l} |

3 ) ≤ O(nd D eff (γ) 3 q 3 ) time with a total work that is also O(nd D eff (γ) 3 q 3 ). On the opposite end, in the fully balanced tree we need to invert multiple d D {h,l} eff (γ) dimensional matrices in parallel at each layer of the tree for a total of log(n) layers. Bounding all d D {h,l} eff (γ) with d D eff (γ), gives a complexity for computing the final solution of O(log(n)q 3 d D eff (γ) 3 ) time, with a huge improvement on the unbalanced tree. This is the first result that not only breaks the quadratic time barrier of previous methods, but can extract enough parallelism to break the linear barrier, running in near-constant time in n. Surprisingly, the total work is only twice O(nq 3 d D eff (γ) 3 ), since at each layer h we perform n/2 h inversions (on n/2 h machines), and the sum across all layers is log(n) h=1 n/2 h ≤ 2n. Therefore, we can compute a solution in a much shorter time than in the sequential sampling case, with a comparable amount of work, but at the expense of requiring much more memory across multiple machines, since at layer h, the sum (γ) 2 ≤ d D eff (γ) 2 memory. Finally, note that independently from this thesis, [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] propose an algorithm that is almost equivalent to SQUEAK in its fully sequential variant. Their work focuses on the specific case of sequential graph sparsification (i.e., when the atoms φ i are edges in a graph), and therefore many choices, such as how to estimate RLSs and how to parallelize the algorithm, are tailored to this setting and differ from our work. We will present a more accurate comparison in Section 4.3.2 where we apply SQUEAK to graph spectral sparsification and consider the problem of learning on graph.

|S h | l=1 d D {h,
Proof sketch: Although SQUEAK is conceptually simple, providing guarantees on its space/time complexity and accuracy is far from trivial. The first step in the proof is again to carefully decompose the failure event across the whole merge tree into separate failure events for each merge node {h, l}, and for each node construct a random process Y that models how Algorithm 10 generates the dictionary I {h,l} . Notice that these processes are again sequential in nature and the various steps (layers in the tree) are not i.i.d.. Furthermore, compared to KORS, the variance of Y is now much larger due to the additional randomness of updating q t,i , and cannot be bounded uniformly. Instead, we take a more refined approach, inspired by [START_REF] Pachocki | Analysis of Resparsification[END_REF], that 1) uses Freedman's inequality to treat W, the variance of process Y, as a random object itself, 2) applies a stochastic dominance argument to W to reduce it to a sum of i.i.d. r.v.-s and only then we can 3) apply i.i.d. concentrations to obtain the desired result.

One of the core component of the proof is a stochastic dominance argument for adaptive importance sampling processes that might be of independent interest. Using the notation from Section 3.3.1, consider an adaptive importance sampling chain where the adaptive probabilities p t,i are not i.i.d (i.e., computed using the chain itself), but we can guarantee that the final probability p n,i is lower bounded by a fixed (non-random) quantity τ n,i . Then, no matter which sequence we use for the sampling or the length of the sampling process, we can show that the maximum weight (minimum probability) reached in the chain (which drives the variance of the importance sampling process) is stochastically dominated by a truncated Pareto random variable. This can help us analyse future importance sampling algorithm. Since the importance sampling chain has correlated steps, it is hard to bound its maximum weight with anything other than its range 1/τ n,i . On the other hand it is easy to see that the dominating Pareto r.v. has a much smaller mean 1 + log(1/τ n,i ), and a small, bounded variance.

Comparison of all RLS sampling algorithms

We can now recap how all the sampling algorithm we presented so far compare with each other. The results are reported in Table 3.1. For all algorithms, the comparison is done using the minimum q (or equivalent parameter) that guarantees generating an (ε, γ)-accurate dictionary in the end. We report the runtime O(Runtime) of the algorithm, ignoring constant and logarithmic terms in the runtime, the size of the dictionary O(|I n |), ignoring constants (we treat ε as a constant), and for multi-pass algorithms the number of passes over the data they require to execute. For algorithms that require an oracle to run, we report an extra runtime cost in the runtime column. Finally, note that for all algorithms, except for Exact-RLS, the memory required to execute

nd n eff (γ) 2 d n eff (γ) log 2 (n) Seq. Oracle RLS Sampl. (w/ removal) n + d n eff (γ) log(n) Seq. RLS Sampl. (w/ removal, fully sequential) (Calandriello et al., 2017a) nd n eff (γ) 3 d n eff (γ) log(n)
Seq. RLS Sampl. (w/ removal, parallel on k machines) (Calandriello et al., 2017a) the algorithm scales as O(|I t | 2 ), necessary to store the matrix S T t Φ T t Φ t S t . In Exact-RLS the gap is due to the fact that it takes O(n 2 ) space to construct, and store the Φ T n Φ n matrix necessary to compute RLS, but the final dictionary size is only O(d n eff (γ) log(n)).

(n/k)d n eff (γ) 3 d n eff (γ) log(n)

Multi-pass algorithms

Among batch algorithms, we begin with Oracle-RLS, where we sample proportionally to the true RLS provided by an oracle. Note that even with the RLS known in advance, sampling without parallelism still require n steps4 , and the final dictionary size is

O(d n eff (γ) log(n)
). This will be the minimum complexity and space that we will refer to as "optimal". Note also that in order for RLS sampling to improve performance in downstream tasks, the dictionary size O(d n eff (γ) log(n)) must be small, thus during the comparison we will restrict ourselves to the d n eff (γ) n regime. Exact-RLS, which uses Definition 2.4 to compute the RLS and then sample, achieve optimal O(d n eff (γ) log(n)) dictionary size, but takes O(n 3 ) time and O(n 2 ) memory to run, and it does not scale to large datasets. Two-Step-RLS, introduced by [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF], was the first batch approximate RLS sampling to provably improve on Exact-RLS. Space wise, the first-step uniformly sampled dictionary has to contain n/γ samples to be (ε, γ)-accurate, much larger than the optimal O(d n eff (γ) log(n)) which is achieved only by the second-step dictionary sampled using approximate RLS. Time wise, the first-step uniform sampling has a small O(n) computational cost, but the O(n 3 /γ 2 ) cost for the RLS estimation phase dominates the time complexity of the algorithm. Although this is an improvement over O(n 3 ) it it still quite far away from O(n). The first big jump in space and runtime arrives when we extend Two-Step-RLS to make multiple RLS sampling steps, each increasingly more accurate, resulting in Many-Step-RLS and Recursive-RLS by [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]. Space wise, Recursive-RLS exactly matches the optimal d n eff (γ) log(n) dictionary size, with Many-Step-RLS getting close and only a log(log(n/γ)) factor away from it. Time wise, they both achieve near-linear in n runtime O(nd n eff (γ) 2 ), which in our d n eff (γ) n setting essentially matches the optimal O(n) runtime of Oracle-RLS. Among the two, Recursive-RLS requires slightly less passes, log(n) vs log(n/γ), when γ < 1. Nonetheless, for many downstream tasks Many-

Step-RLS can improve on the number of passes using a computational regularization approach [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF]. In particular, we can start with q = O(log(n log(n))) and γ = n, and keep reducing γ at each pass until cross-validation on the downstream task indicates that we should stop. This way, we avoid performing passes unnecessary for the downstream task's performance. Recap: in the batch setting, we went from the O(n 3 ) time complexity of Exact-RLS, to the near-linear O(nd n eff (γ) 2 ) time complexity of Many-Step-RLS and Recursive-RLS, breaking the quadratic barrier required to compare each sample to all other. We also reduce the O(n 2 ) memory requirement of Exact-RLS, necessary to compute the exact RLS, to a O(d n eff (γ) 2 log 2 (n)) memory requirement to execute Many-Step-RLS and Recursive-RLS.

Single-pass algorithms w/o removal

To eliminate the need for passes altogether, we moved from RLS sampling to sequential RLS sampling. In this category, we first consider algorithms that take an insertion-only approach to dictionary construction. Our optimal baseline is SOracle-RLS-NoRem, where we sample proportionally to the current RLS provided by an oracle restricted by the streaming setting to seeing only the partial dataset D t . The total runtime of the algorithm is n, and it generates a dictionary that contains O(d n onl (γ) log(n)) ≤ O(d n eff (γ) log 2 (n) atoms. The first algorithm that can operate both in the batch and streaming setting is uniform sampling. Uniform takes only O(q) time5 to sample its dictionary. Unfortunately, [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF] shows that to guarantee (ε, γ)-accuracy we must choose q = O(nµ(γ)) proportional to the regularized coherence µ(γ) = max n i=1 τ n,i (γ), resulting in a O(nµ(γ)) runtime. Moreover, we need assumptions or an oracle to tell us in advance the value of the coherence µ(γ) to implement the algorithm. Alternatively, we can use the bound µ(γ) ≤ 1/γ to run the algorithm with q = n/γ, resulting in a O(n/γ) runtime and dictionary size, which matches the n optimal runtime, but is far away from the optimal d n onl (γ) log(n) dictionary size. Using instead our proposed algorithm KORS (Calandriello et al., 2017c) we can achieve (up to constants) the optimal O(d n onl (γ) log(n)) dictionary size in a near-linear O(nd n onl (γ) 2 ) runtime. Therefore, we can achieve essentially the same guarantees as SOracle-RLS-NoRem, without the need of an oracle. Moreover, we get not only guarantees on the final dictionary I n , but also on all intermediate dictionaries I t . Therefore, if the underlying d n eff (γ) of the dataset is large and the size of the dictionary grows so much that we have to interrupt the algorithm, we still know that the intermediate dictionary I t is (ε, γ)-accurate w.r.t. to the data seen so far, and we can use it to choose a better γ. Similarly, in streaming application we can at any moment take a snapshot of the dictionary to start using it in a downstream task, and continue processing the stream without restarting. Recap: in the streaming setting with insertion-only updates, we went from the log(n) passes of Many-Step-RLS and Recursive-RLS to the single pass of KORS. The O(nd n eff (γ) 2 ) time complexity of KORS remain near-linear, although with a worse log 4 (n) dependency when compared to the log 2 (n) dependency of Many-Step-RLS and Recursive-RLS. Similarly, the O(d n eff (γ) 2 log 4 (n)) memory requirement to execute KORS is only log 2 (n) larger than Many-Step-RLS and Recursive-RLS.

Single-pass algorithms w/ removal

Adding the possibility of removing atoms from intermediate dictionaries brings the requirements of single-pass algorithms close to the ones of multi-pass algorithms. This is clear from looking at SOracle-RLS-Rem, a sequential oracle algorithm, that matches the n time complexity and O(d n eff (γ) log(n)) dictionary size of Oracle-RLS, a batch oracle algorithm. In this setting SQUEAK, in its fully sequential variant, runs in a near-linear O(nd n eff (γ) 3 ) time and construct optimally sized O(d n eff (γ) log(n)) dictionaries, essentially matching again both the batch and sequential oracle but without knowing the RLS in advance. The time complexity of the algorithm is O(nd n eff (γ) 3 ), a small increase compared to the O(nd n eff (γ) 2 ) complexity of of KORS, Many-Step-RLS and Recursive-RLS. Compared to KORS, this is because sequential SQUEAK cannot use efficient O(d n eff (γ) 2 ) rank-1 updates when estimating RLSs, due to the fact that it changes many atoms in the dictionary at the same time. Compared to Many-Step-RLS and Recursive-RLS, this is because at each of the n steps KORS needs to perform a O(d n eff (γ) 3 ) inversion followed by O(d n eff (γ)) RLS estimations each costing O(d n eff (γ) 2 ), while the batch algorithms need to perform a O(d n eff (γ) 3 ) inversion only once for each of the log(n) passes, while the n RLS estimations cost a cheaper O(d n eff (γ) 2 ) time each. Nonetheless, if we distribute the computation across k machines, the parallel variant of SQUEAK can compute a solution in O((n/k)d n eff (γ) 3 ) time, breaking even the near-linear time barrier. This is even better than the batch oracle RLS algorithm, and can only be compared to a parallel oracle algorithm. Note also out of all the near-linear time sampling algorithms, SQUEAK is the only one that can be naturally parallelized. Finally, compared to the multi-pass algorithms, KORS and SQUEAK are the only nearlinear time algorithms that can continue to update their dictionary even beyond the first n samples, without having to restart the computation from scratch. Recap: the addition of dictionary removal improves the space and time complexity of streaming algorithms. SQUEAK requires only O(d n eff (γ) 2 log 2 (n)) memory to run, matching Many-Step-RLS and Recursive-RLS and improving on KORS. When run on a single machine, it requires O(nd n eff (γ) 3 ) time to execute, slightly more than the other near-linear algorithms. When run on k machines, we can improve runtime to O((n/k)d n eff (γ) 3 ), surpassing all other near-linear time algorithms and reaching near-constant O(d n eff (γ) 3 ) runtime when enough machines are used.

Straightforward extensions and practical tricks

To simplify the exposition, so far we did not use the tightest possible bounds or the most efficient update rules. In this section we highlight several simple extension to our analysis that can either sharpen our bounds or improve computation.

Improve log(n) dependency in q. It is possible to improve the log(n) dependency in SQUEAK with a milder max{log(d n eff (γ)), log(h)}, where h is the height of the merge tree, i.e., the maximum number of merges that a single sample has to go through until it reaches the root. Proving this requires (1) replacing the concentration inequality used in Proposition 3.18 that scales with n with a stronger version by Minsker, (2011) that scales only with d n eff (γ), and (2) defining more carefully the union bound over the failure events to scale with the actual path each sample follows in the merge tree rather than the number of nodes in the tree. When h = log(n) (fully balanced tree), this means that, if we set q = max{log(d n eff (γ)), log(h)}, we can improve SQUEAK's final dictionary size to

O(d n eff (γ) log(d n eff (γ)) log log(n)) from the original O(d n eff (γ) log(n))
, with associated runtime improvements. Similarly, we can tune KORS's q to improve final dictionary size to

O(d n eff (γ) log(d n eff (γ)) log(n)) from O(d n eff (γ) log 2 (n)). Unfortunately, estimating log(d n eff (γ))
is not easy, and it is impossible to do in a streaming setting since q must be set in advance. Thus in practice it is simpler to just set q proportional to log(n).

Time-dependent regularization γ. All the analysis remains identical if we strictly increase γ over time. This is helpful, especially when processing large datasets sequentially. For example, in many kernel-related problem the γ that gives us the optimal learning rate is

√ n (as we will see in the discussion of Proposition 4.5). In a dataset with a million samples this gives us a γ in the thousands. If we split our dataset in small subset of a few hundreds samples, we might end up with extremely small RLS that are numerically hard to estimate. In this case starting with a smaller γ and increasing it as we continue up the merge tree (e.g., |D {h,l} |improves numerical stability, without costing too much space.

Use overestimates of the RLS instead of underestimates. For the definition of ρ-accurate RLS estimate τ t,i we chose to make them strict underestimates of the true RLS τ t,i . This was done in order to reinforce their interpretation as probabilities in coin flips. Another possible solution is to define an estimate as ρ-accurate when it satisfies τ t,i ≤ τ t,i ≤ ρτ t,i , i.e., when it overestimates τ t,i . To maintain the algorithm well defined, we can compensate this change using thresholding and define the coin flip probabilities as p t,i = min{ τ t,i , 1}. Since we know that the true τ t,i are always smaller than 1, this does not affect ρ-accuracy and our analysis, except for a more complex notation.

On the positive side, we now get more accurate RLS estimates, because we are using prior knowledge on τ t,i to refine our estimator τ t,i . A similar argument can be made for other refinements such as p t,i = max{ τ t,i , 1/(t + γ)} (we know the minimum τ t,i ) and for taking the minimum p t,i = min{ τ t,i , τ t-1,i } (we know τ t,i ≤ τ t-1,i ).

Open questions: lifelong learning and deterministic dictionary learning

Two important questions that remain open at the end of this chapter is whether we can continue updating our dictionary indefinitely, and whether randomness is truly necessary for accuracy.

The first question is especially relevant in the context of online learning and learning on streams, where knowing log(n) in advance might not be possible. It is also central to the problem of lifelong learning [START_REF] Thrun | Lifelong learning algorithms[END_REF], where the learning algorithm is faced with an open-ended goal and must continue to update its model forever.

Unfortunately, both KORS and SQUEAK require to know log(n) in advance to tune q. Note that our guarantees apply for any q larger than log(n) and therefore if we know that our dataset might grow over time we can "overbudget" (which is made easy by the logarithmic dependency). For a fixed q the guarantees also hold for any ε > ε, and therefore if we continue processing samples after the first n our analysis degrades gracefully in guaranteeing (ε , γ)-accuracy.

The main technical obstacle to obtain an algorithm that can truly operate on unbounded streams is the union bound over failure events (total number of dictionary merges). This quantity naturally grows with the number of samples processed, so it is not easy to see if it is possible to avoid this dependency.

Alternatively, we could try to modify our proof to take into account a time-dependant budget q t , and use an adaptive schedule log(t) for it. Unfortunately, this would require going back and re-evaluating already dropped samples when q t increases, violating our single-pass constraint.

Another interesting question is whether it is possible to deterministically select the atoms with the highest RLS (e.g., estimated using SQUEAK) and prove that this constructs an (ε, γ)-accurate dictionary. While some early results are available [START_REF] Papailiopoulos | Provable deterministic leverage score sampling[END_REF][START_REF] Patel | Deterministic Column Sampling for Low-Rank Matrix Approximation: Nyström vs. Incomplete Cholesky Decomposition[END_REF], they are still very far from the accuracy, space and time performance of randomized methods such as SQUEAK.

Proof of main results

We will provide a single proof of Theorem 4.7, since Algorithm 8 and Theorem 3.11 can be seen as a special case of Algorithm 10 with a fully sequential merge tree, and where we do not update the approximate RLSs after estimating them the first time. It is easy to see that if τ t,t = τ t ,t for all t > t, then the importance sampling will always return q t,t = q t ,t , and we perform sequential RLS sampling without removal.

Decomposing the merge tree

We begin by describing more in detail some notation introduced in the main paper and necessary for this proof.

Merge trees

We first restate and formalize more in detail the random process induced by SQUEAK.

We partition D into k disjoint sub-datasets D i of size n i , such that D = ∪ k i=1 D i . For each dataset D i , we construct an initial dictionary I {1,i} = {( p 0,j = 1, q 0,j = q, φ j ) : j ∈ D i } by inserting all points from D i into I D i with weight p 0,j = 1 and number of copies q 0,j = q. It is easy to see that I {1,i} is an (ε, γ)-accurate dictionary, and we can split the data in small enough chunks to make sure that it can be easily stored and manipulated in memory. Alternatively, if we want our initial dictionaries to be small and cannot choose the size of D i , we can run KORS on D i to generate I {1,i} , and the following proof will remain valid. Regardless of construction, the initial (ε, γ)-accurate dictionaries I {1,i} are included into the dictionary pool S 1 .

Chapter 3. Sequential RLS Sampling At iteration h, the inner loop of SQUEAK arbitrarily chooses two dictionaries from S h and merges them into a new dictionary. Any arbitrary sequence of merges can be described by a full binary tree, i.e., a binary tree where each node is either a leaf or has exactly two children. Fig. 3.3 shows several different merge trees corresponding to different choices for the order of the merges. Note that starting from k leaves, a full binary tree will always have exactly k -1 internal nodes, and a maximum depth of k. Therefore, regardless of the structure of the merge tree, we can always stretch it into a tree of depth k, with all the initial dictionaries I 1,i as leaves on its deepest layer. Fig. 3.4 reports an example of such a stretching transformation, where all the extra nodes introduced in the trees do not correspond to merges, and simply propagate the dictionary up the tree (e.g., I {2,3} = I {3,2} ). After this transformation, we index the tree nodes using their height (longest path from the node to a leaf, also defined as depth of the tree minus depth of the node), where all leaves have height 1 and the root has height k. We can also see that at each layer, there is a single dictionary merge, and the size of S h (number of dictionaries present at layer h) is

I 1,2,3,4 I 1,2,3 + I 4 I 1,2,3 I 1,2 + I 3 I 1,2 I 1 + I 2 I 1 D 1 I 2 D 2 I 3 D 3 I 4 D 4 s = 4 s = 3 s = 2 s = 1 (a) merge tree I 1,2,3,4 I 1,2,3 + I 4 I 1,2,3 I 1,2 + I 3 I 1,2 I 1 + I 2 I 1 D 1 I 2 D 2 I 3 I 3 D 3 I 4 I 4 I 4 D 4 s = 4 s = 3 s = 2 s = 1 (b) stretched merge tree
|S h | = k -h + 1.
Therefore, a node corresponding to a dictionary is uniquely identified with two indices {h, l}, where h is the height of the layer and l ≤ |S h | is the index of the node in the layer. For example, in Fig. 3.4, the node containing I 1,2,3 is indexed as {3, 1}, and the highest node containing I 4 is indexed as {3, 2}.

We also define the dataset D {h,l} as the union of all sub-datasets D l that are reachable from node {h, l} as leaves. For example, in Fig. 3.4, dictionary I 1,2,3 in node {3, 1} is constructed starting from all points in D {3,1} = D 1 ∪ D 2 ∪ D 3 , where we highlight in red the descendant tree. We now define P h (see Definition 2.8) as the block diagonal regularized projection matrix where each diagonal block P {h,l} is constructed on D {h,l} . Again, from Fig. 3.3, P 3 is a n × n matrix with two blocks on the diagonal, a first (n

1 + n 2 + n 3 ) × (n 1 + n 2 + n 3 ) block P 3,1 constructed on D {3,1} = D 1 ∪ D 2 ∪ D 3 ,
and a second n 4 × n 4 block P 3,2 constructed on D {3,2} = D 4 . Similarly, we can adapt Eq. 2.15 to define a block diagonal P h , where block P {h,l} is defined using P {h,l} and I {h,l} , which contains the weights and number of copies The statement. Since P h -P h is block diagonal, we have that a bound on its norm implies an equal bound on the norm each matrix on the diagonal, i.e., P h -P h = max l P {h,l} -P {h,l} ≤ ε ⇒ P {h,l} -P {h,l} ≤ ε for all blocks l on the diagonal, and since each block corresponds to a dictionary I {h,l} , this means that if P h -P h ≤ ε, all dictionaries at layer l are ε-accurate approximation of their respective represented datasets. Our goal is to show

P ∃h ∈ [k] : P h -P h 2 ≥ ε ∪ max l=1,...,|S h | |I {h,l} | ≥ 3qd {h,l} eff (γ) = P ∃h ∈ [k] : max l∈[|S h |] P {h,l} -P {h,l} 2 ≥ ε A h ∪ max l∈[|S h |] |I {h,l} | ≥ 3qd {h,l} eff (γ) B h ≤ δ,

3.6

where event A h refers to the case when some dictionary I {h,l} at an intermediate layer h fails to accurately approximate D {h,l} and event B h considers the case when the memory requirement is not met (i.e., too many points are kept in one of the dictionaries I {h,l} at a certain layer h). We can conveniently decompose the previous joint (negative) event into two separate conditions, one for accuracy and one for space, as

P k h=1 A h ∪ B h = P k h=1 A h ∪ k h=1 B h = P k h=1 A h + P   k h=1 B h ∩ k h=1 A h C   = P k h=1 A h + P k h=1 B h ∩ k h=1 A C h = P k h=1 A h + P k h=1 B h ∩ k h =1 A C h .

3.7

We can now apply a union bound to separate each of the k merge processes. Notice that the union bound is over the nodes, since we want to guarantee that each dictionary in each node is accurate. While we know that stretching the tree (Fig. 3.4) can introduce many artificial nodes, we also know that to merge k sub-datasets we need at most k merges, and therefore we only need to bound over k negative events. Let E = {(h, l)} be the set of nodes corresponding to actual merges (i.e., nodes before stretching). Then we can reformulate

P ∃h ∈ [k] : P h -P h 2 ≥ ε ∪ max l=1,...,|S h | |I {h,l} | ≥ 3qd {h,l} eff (γ) = P ∃h ∈ [k] : max l=1,...,|S h | P {h,l} -P {h,l} 2 ≥ ε + P ∃h ∈ [k] : max l=1,...,|S h | |I {h,l} | ≥ 3qd {h,l} eff (γ) ∩ ∀h ∈ [h] : P h -P h 2 ≤ ε ≤ {h,l}∈E P P {h,l} -P {h,l} 2 ≥ ε + {h,l}∈E P |I {h,l} | ≥ 3qd {h,l} eff (γ) ∩ ∀h ∈ [h] : P h -P h 2 ≤ ε ≤ δ.

3.8

As we highlighted many time, the accuracy of the dictionary (first term in the previous bound) is guaranteed by the fact that given an ε-accurate dictionary we obtain RLS estimates which are at least a fraction of the true RLS, thus forcing the algorithm to sample each column enough. On the other hand, the space complexity bound is achieved by exploiting the fact that RLS estimates are always upper-bounded by the true RLS, thus ensuring that SQUEAK does not oversample columns w.r.t. the sampling process following the exact RLS.

In the reminder of the proof, we will show that both events happen with probability smaller than δ/(2k). The main advantage of splitting the failure probability as we did in Eq. 3.8 is that we can now analyze the processes that generated each P {h,l} -P {h,l} (and each dictionary I {h,l} ) separately. Focusing on a single node {h, l} restricts our problem on a well defined dataset D {h,l} , where we can analyze the evolution of I {h,l} sequentially.

Challenges. Due to its sequential nature, the "theoretical" structure of the process generating P {h,l} -P {h,l} is considerably complicated, where each step in the sampling process (layer in the merge tree) is highly correlated with the previous steps.6 This prevents us from using concentration inequalities for i.i.d. processes that are at the basis of the analysis of uniform sampling [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF], Two-Step-RLS by [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF], and Recursive-RLS by [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF].

As a result, we first show that the project error is a martingale process. The main technical difficulty in analyzing how the projection error evolves over iterations is that the projection matrices change dimension every time a new point is processed. In fact, all of the matrices P {h ,l } -P {h ,l } descending from P {h,l} -P {h,l} have potentially a different size since they are based on different datasets D {h ,l } . This requires a careful definition of the martingale process to still use matrix concentration inequalities for fixed-size matrices (see Section 3.4.2).

Once we have defined the martingale structure, we will use a matrix Freedman inequality to bound the deviation of the martingale from its mean as a function of its predictable quadratic variation W. Here the proof of KORS and SQUEAK will differ. In particular, it is possible to find a small deterministic uniform bound on the variance of each step of KORS, and use it to bound the overall process' variance.

Bounding the per-step variance of SQUEAK is much more difficult, since it continues to update the number of copies q t,i after insertion, as well as their weight 1/ p t,i that might grow unbounded over time. As a consequence, uniform bounds on the predictable quadratic variation W become too large. In addition, W itself is a complex object since it is composed of many correlated variables.

In order to provide a tighter bound on the total variance of the martingale process of the projection error, we need to first introduce an i.i.d. stochastically dominant process (Section 3.4.4 (step 2)), which finally allows us to use an i.i.d. matrix concentration inequality to bound the total variance (Section 3.4.4-(step 5)). This finally leads to the bound on the accuracy. The bound on the space complexity (Section 3.4.5) follows similar (but simpler) steps.

3.4.2 Bounding the projection error P {h,l} -P {h,l}

The sequential process. Thanks to the union bound in Eq. 3.6, instead of having to consider the whole merge tree followed by SQUEAK, we can focus on each individual node {h, l} and study the sequential process that generated its dictionary I {h,l} . We will now map more clearly the actions taken by SQUEAK to the process that generated P {h,l} -P {h,l} .

Since the merge tree is decided in advance, the dataset D {h,l} and which samples φ i are involved in the process is not random, and is fixed for the whole process. We begin by focusing on P {h,l} , which is a random matrix defined starting from the fixed map Φ {h,l} and the random dictionary I {h,l} , where the randomness influences both which points are included in I {h,l} , and the weight with which they are added.

Consider now a point i ∈ D {h,l} . Since the starting datasets in the leaves are disjoint, there is a single path in the tree, with length h, from the leaves to {h, l}. This means that for all s < h, we can properly define a unique p s,i and q s,i associated with that point. More in detail, if at layer s point i is present in D {s,l } , it means that either

(1) SQUEAK used I {s,l } to compute p s,i , and p s,i to compute q s,i ,

(2) at layer h SQUEAK did not have any merge scheduled for point i, and we simply propagate p s,i = p s-1,i and q s,i = q s-1,i , (3) SQUEAK dropped the point at a lower layer, and we set p s,i to the last value that was computed by SQUEAK before dropping it, i.e., we do not change the probabilities of dropped points.

Consistently with the algorithm, we initialize p 0,i = 1 and q 0,i = q.

Denote ν {h,l} = |D {h,l} | so that we can use index i ∈ [ν {h,l} ] to index all points in D {h,l} .

Using the matrices C = C {h,l} with vector c i as its i-th column (see Definition 2.8), we can again rewrite the regularized projection matrix as that P {h,l} = CC T = ν {h,l} i=1 c i c T i , and we can see that the norm c i c T i is equal to the RLS of the i-th sample φ i w.r.t. to dataset D {h,l} . Note that since the starting datasets are disjoint, and there is a single path from the leaves to {h, l}, i is present only in node l on layer h. Therefore its RLS is uniquely defined w.r.t. D {h,l} and can be shortened as τ h,i = τ D {h,l} ,i . Using c i , we can also introduce the random matrix P {h,l} s as

P {h,l} s = ν {h,l} i=1 q s,i q p s,i c i c T i = ν {h,l} i=1 q j=1 z s,i,j q p s,i c i c T i .
where z s,i,j are {0, 1} r.v. such that q s,i = q j=1 z s,i,j , and correspond to the coin flip chain (see Section 3.3.1) associated with sample i's j-th copy. Note that when s = h, we have that P {h,l} h = P {h,l} and we recover the definition of the approximate projection matrix from SQUEAK. But, for a general s = h P {h,l} s does not have a direct interpretation in the context of SQUEAK. It combines the vectors c i , which are defined using Φ {h,l} at layer h, with the weights p s,i computed by SQUEAK across multiple nodes at layer s, which are potentially stored in different machines that cannot communicate. Nonetheless, P {h,l} s is a useful tool to analyze SQUEAK.

Taking into account that we are now considering a specific node {h, l}, we can drop the index from the dataset D {h,l} = D, RLS τ D {h,l} ,i = τ h,i , and size ν {h,l} = ν. Using this shorter notation, we can reformulate our objective as bounding P {h,l} -P {h,l} 2 = P {h,l} -P {h,l} h 2 , and reformulate the process as a sequence of matrices {Y s } h s=1 defined as

Y s = P {h,l} -P {h,l} s = 1 q ν i=1 q j=1 1 - z s,i,j p s,i c i c T i ,
where Y h = P {h,l} -P {h,l} h = P {h,l} -P {h,l} , and Y 1 = P {h,l} -P {h,l} 0 = 0 since p 0,i = 1 and q 0,i = q.

Bounding the martingale Y h

We transformed the problem of bounding P {h,l} -P {h,l} into the problem of bounding Y h , which we modeled as a random matrix process, connected to SQUEAK by the fact that both algorithm and random process Y h make use of the same weight p s,i and multiplicities q s,i . There are two main issues in analyzing the process {Y s } h s=1 : (1) The overall algorithm may fail in generating an accurate dictionary at some intermediate iteration and yet return an accurate dictionary at the end. Moreover, whenever one of the intermediate Y s is larger than ε we lose our guarantees for p s,i for the whole process, since an inaccurate p s,i that underestimates too much p s.i will influence all successive p s ,i through the minimum.

To solve this, we consider an alternative (more pessimistic) process which is "frozen" as soon as it constructs an inaccurate dictionary. Freezing the probabilities at the first error gives us a process that fails more often, but that provides strong guarantees up to the moment of failure.

(2) While Y h = P {h,l} -P {h,l} h ≤ ε guarantees that I {h,l} is an ε-accurate dictionary of Φ {h,l} , knowing Y s = P {h,l} -P {h,l} s ≤ ε for s < h does not guarantee that all descendant I {s,l } are (ε, γ)-accurate w.r.t. their Φ {s,l } . Intuitively, this is because

C t is defined w.r.t. (Φ T t Φ t + γI t ) -1 , while C s is defined w.r.t. (Φ T s Φ s + γI s ) -1 , with (Φ t Φ T t + γΠ t ) -1 (Φ s Φ T s + γΠ s ) -1 .
Since the inverse at step t is "stronger" than at step s, it is possible for C T t (I t -S t s (S t s ) T )C T t to be smaller than ε, while C T s (I s -S s (S s ) T )C T s is larger than ε. Nonetheless, we will show that Y s ≤ ε is enough to guarantee that the intermediate estimate p s,i computed in SQUEAK, and used as weights in P {h,l} s , are never too small.

The frozen process. We will now replace the process Y s with an alternative process Y s defined as

Y s = Y s-1 I Y s-1 ≤ ε + Y s-1 I Y s-1 ≥ ε ,
This process starts from Y 0 = Y 0 = 0, and is identical to Y s until a step s where for the first time Y s ≤ ε and Y s+1 ≥ ε. After this failure happen the process Y s is "frozen" at s and Y s = Y s+1 for all s + 1 ≤ s ≤ h. Consequently, if any of the intermediate elements of the sequence violates the condition Y s ≤ ε, the last element will violate it too. For the rest, Y s behaves exactly like Y s . Therefore,

P ( Y h ≥ ε) ≤ P Y h ≥ ε ,
and if we can bound P Y h ≥ ε we will have a bound for the failure probability of SQUEAK, even though after "freezing" the process Y h does not make the same choices as the algorithm.

We will see now how to construct the process Y s starting from z s,i,j and p s,i,j . We recursively define the indicator ({0, 1}) random variable z s,i,j as z s,i,j = I u s,i,j ≤ p s,i,j p s-1,i,j z s-1,i,j , where u s,i,j ∼ U(0, 1) is a [0, 1] uniform random variable and p s,i,j is defined as

p s,i,j = p s,i I Y s-1 ≤ ε ∩ z s-1,i,j = 1 + p s-1,i,j I Y s-1 ≥ ε ∪ z s-1,i,j = 0 .
This definition of the process satisfies the freezing condition, since if Y s+1 ≥ ε (we have a failure at step s), for all s ≥ s + 1 we have z s ,i,j = z s+1,i,j with probability 1 (p s+1,i,j /p s,i,j = p s,i,j /p s,i,j = 1), and the weights 1/(qp s+1,i,j ) = 1/(qp s,i,j ) never change.

Introducing a per-copy weight p s,i,j and enforcing that p s+1,i,j = p s,i,j when z s,i,j = 0 avoids subtle inconsistencies in the formulation. In particular, not doing so would semantically correspond to reweighting dropped copies. Although this does not directly affect Y s (since the ratio z s,i,j / p s,i is zero for dropped copies), and therefore the relationship P ( Y h ≥ ε) ≤ P Y h ≥ ε still holds, we will see later how maintaining consistency helps us bound the second moment of our process.

We can now arrange the indices s, i, and j into a linear index r = s in the range [1, . . . , ν 2 q], obtained as r = {s, i, j} = (s -1)νq + (i -1)q + j. We also define the difference matrix as

X {s,i,j} = 1 q z s-1,i,j p s-1,i,j - z s,i,j p s,i,j c i c T i ,
which allows us to write the cumulative matrix as Y {s,i,j} = {s,i,j} r=1

X {s,i,j} where the checkpoints {s, ν, q} correspond to Y s ,

Y {s,ν,q} = Y s = 1 q ν i=1 q j=1 1 - z s,i,j p s,i,j c i c T i .
Let F s be the filtration containing all the realizations of the uniform random variables u s,i,j up to the step s, that is F s = {u s ,i ,j , ∀{s , i , j } ≤ s}. Again, we notice that F s defines the state of the algorithm after completing iteration s because, unless a "freezing" happened, SQUEAK and Y s flip coins with the same probability, and generate the same dictionaries. Since τ s,i and p s,i,j are computed at the beginning of iteration s using the dictionary I {s,l } (for some l unique at layer s), they are fully determined by F s-1 . Furthermore, since F s-1 also defines the values of all indicator variables z s ,i,j up to z s-1,i,j for any i and j, we have that all the Bernoulli variables z s,i,j at iteration s are conditionally independent given F s-1 . In other words, we have that for any i , and j such that {s, 1, 1} ≤ {s, i , j } < s the following random variables are equal in distribution, z s,i,j F {s,i ,j } = z s,i,j F {s-1,ν,q} ∼ Ber p s,i,j p s-1,i,j , 3.9 and for any i , and j such that {s, 1, 1} ≤ {s, i , j } ≤ {s, ν, q} and s = {s, i , j } we have the independence z s,i,j F {s-1,ν,q} ⊥ z s,i ,j F {s-1,ν,q} .

3.10

While knowing that Y s ≤ ε is not sufficient to provide guarantees for the approximate probabilities p s,i , we can show that it is enough to prove that the frozen probabilities p s,i,j are never too small.

Proof of Lemma 3.17. Let s be the step where the process freezes (s = h if it does not freeze), or, in other words, Y s < ε and Y s+1 ≥ ε. From the definition of p s,i,j , we have that

p s,i,j ≥ p s,i = p s,i = max {min { τ s,i , p s-1,i } , p s-1,i /2} ≥ min { τ s,i , p s-1,i } = min { τ s,i , p s-2,i } = min { τ s,i , p s-3,i } . . . = min { τ s,i , p 0,i } = τ s,i ,
and therefore p s,i,j ≥ τ s,i . Now let {s, l } be the node where τ s,i was computed. We will again drop the {h, l} index from D {h,l} , and simply refer to it as D. Similarly, we will refer with D i to D {s,l } (as in, the dataset used to compute τ s,i ), and with D i to the samples in

D not contained in D i (complement of D i ).
Define A as the |D| × |D i | matrix that contains the columns of S s related to points in D i , and similarly define B as the |D| × |D i | matrix that contains the columns of S s related to points in D i , where S s can be reconstructed by interleaving columns of A and B. From its definition in Eq. 3.5, we know that τ s,i is computed by SQUEAK as

τ s,i = (1 -ε)φ T i (Φ D AA T Φ T D + (1 + ε)γΠ D i ) -1 φ i ,
using only the points in A that are available at node {s, l }. From Lemma 2.10 we know that

Y s = P {h,l} -P {h,l} s 2 = (Φ D Φ T D + γΠ D ) -1/2 (Φ D Φ T D -Φ D S s S T s Φ T D )(Φ D Φ T D + γΠ D i ) -1/2 2 = (Φ D Φ T D + γΠ D ) -1/2 (Φ D Φ T D -Φ D AA T Φ T D -Φ D BB T Φ T D )(Φ D Φ T D + γΠ D ) -1/2 2 ≤ ε
and we know that this implies

Φ D AA T Φ T D Φ D Φ T D + ε(Φ D Φ T D + γΠ D ) -Φ D BB T Φ T D Φ D Φ T D + ε(Φ D Φ T D + γΠ D ).
Plugging it in the initial definition,

τ s,i = (1 -ε)φ T i (Φ D AA T Φ T D + (1 + ε)γΠ D i ) -1 φ i ≥ (1 -ε)φ T i (Φ D AA T Φ T D + (1 + ε)γΠ D ) -1 φ i ≥ (1 -ε)φ T i (Φ D Φ T D + ε(Φ D Φ T D + γΠ D ) + (1 + ε)γΠ D ) -1 φ i ≥ (1 -ε) 1 1 + 3ε φ T i (Φ D Φ T D + γΠ D ) -1 φ i ≥ 1 -ε 1 + 3ε τ h,i ≥ τ h,i /ρ.
This result is weaker then Lemma 3.3 and Lemma 3.14, since we do not provide an upper bound, and only show that p s,i ≥ p h,i /ρ and not p s,i ≥ p s,i /ρ, but it guarantees that the probabilities used at any intermediate layer s are bigger than a fraction 1/ρ of the exact final p h,i probabilities that we would use at layer h, which will suffice for our purpose.

Proof of main results

73

We now proceed by studying the process {Y s } h s=1 and showing that it is a bounded martingale. In order to show that Y s is a martingale, it is sufficient to verify the following (equivalent) conditions

E Y s F s-1 = Y s-1 ⇔ E X {s,i,j} F s-1 = 0.
We begin by inspecting the conditional random variable X {s,i,j} |F s-1 . Given the definition of X {s,i,j} , the conditioning on F s-1 determines the values of z s-1,i,j and the approximate probabilities p s-1,i,j and p s,i,j . In fact, remember that these quantities are fully determined by the realizations in F s-1 which are contained in F s-1 . As a result, the only stochastic quantity in X {s,i,j} is the variable z s,i,j . Specifically, if Y s-1 ≥ ε, then we have p s,i,j = p s-1,i,j and z s,i,j = z s-1,i,j (the process is stopped), and the martingale requirement

E X {s,i,j} F s-1 = 0 is trivially satisfied. On the other hand, if Y s-1 ≤ ε we have E u s,i,j 1 q z s-1,i,j p s-1,i,j - z s,i,j p s,i,j c i c T i F s-1 = 1 q z s-1,i,j p s-1,i,j - z s-1,i,j p s,i,j E I u s,i,j ≤ p s,i,j p s-1,i,j F s-1 c i c T i = 1 q z s-1,i,j p s-1,i,j - z s-1,i,j p s,i,j p s,i,j p s-1,i,j c i c T i = 0,
where we use the recursive definition of z s,i,j and the fact that u s,i,j is a uniform random variable in [0, 1]. This proves that Y s is indeed a martingale. We now compute an upper-bound R on the norm of the values of the difference process as

X {s,i,j} = 1 q z s-1,i,j p s-1,i,j - z s,i,j p s,i,j c i c T i ≤ 1 q 1 p s,i,j c i c T i = 1 q 1 p s,i,j τ h,i ≤ 1 q ρ τ h,i τ h,i = ρ q def = R,
where we used Lemma 3.17 to bound p s,i,j ≤ τ h,i /ρ. If instead, Y s-1 ≥ ε, the process is stopped and X s = 0 = 0 ≤ R.

We are now ready to use a Freedman matrix inequality from [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF] to bound the norm of Y.

Proposition 3.18 - [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF], Theorem 1.2). Consider a matrix martingale {Y k : k = 0, 1, 2, . . . } whose values are self-adjoint matrices with dimension d, and let {X k : k = 1, 2, 3, . . . } be the difference sequence. Assume that the difference sequence is uniformly bounded in the sense that

X k 2 ≤ R almost surely for k = 1, 2, 3, . . . .
Define the predictable quadratic variation process of the martingale as

W k def = k j=1 E X 2 j {X s } j-1 s=0 , for k = 1, 2, 3, . . . .

Chapter 3. Sequential RLS Sampling

Then, for all ε ≥ 0 and σ 2 > 0,

P ∃k ≥ 0 : Y k 2 ≥ ε ∩ W k ≤ σ 2 ≤ 2d • exp - ε 2 /2 σ 2 + Rε/3 •
In order to use the previous inequality, we develop the probability of error for any fixed h as

P ( Y h ≥ ε) ≤ P Y h ≥ ε = P Y h ≥ ε ∩ W h ≤ σ 2 + P Y h ≥ ε ∩ W h ≥ σ 2 ≤ P Y h ≥ ε ∩ W h ≤ σ 2
(a)

+ P W h ≥ σ 2 (b)
.

Using the bound on X {s,i,j} 2 , we can directly apply Proposition 3.18 to bound (a) for any fixed σ 2 . To bound the part (b), we need to bound w.h.p. the norm W h . The proofs for KORS and SQUEAK will now diverge. For KORS we can show the following deterministic result, proved later in Section 3.4.4.

Lemma 3.19 -Uniform bound on norm of the predictable quadratic variation process of KORS.

W h ≤ ρ q
Unfortunately, in the worst case W h for SQUEAK might grow as n, and finding a small deterministic bound is impossible. Instead we use the following high probability result, proved later in Section 3.4.4.

Lemma 3.20 -Low probability of the large norm of the predictable quadratic variation process of SQUEAK.

P W h ≥ 6ρ q ≤ n • exp -2 q ρ
Therefore for SQUEAK, combining Proposition 3.18 with σ 2 = 6ρ/q, Lemma 3.20, the fact that 2ε/3 ≤ 1 and the value used by SQUEAK q ≥ 26ρ log(3n/δ)/ε 2 we obtain

P P {h,l} -P {h,l} 2 ≥ ε = P ( Y h ≥ ε) ≤ P Y h ≥ ε ∩ W h ≤ σ 2 + P W h ≥ σ 2 ≤ 2ν • exp - ε 2 q ρ 1 12 + 2ε/3 + n • exp -2 q ρ ≤ 3n • exp - ε 2 13ρ q = 3n • exp -2 log 3n δ = 3n • exp -log 3n δ 2 = 3n δ 2 9n 2 ≤ δ 2n •
This, combined with the fact that k ≤ n since at most we can split our dataset in n parts, concludes this part of the proof.

For KORS, using σ 2 = ρ/q, replacing Lemma 3.20 with Lemma 3.19 and the value used by KORS q ≥ 4 log(2n/δ)/ε 2 we obtain the same result.

3.4 Proof of main results 75

Bound on predictable quadratic variation W h

Step 1 (applying the definition). We start by writing out W r for the process Y s ,

W r = 1 q 2 {s,i,j}≤r E z s-1,i,j p s-1,i,j - z s,i,j p s,i,j 2 F {s,i,j}-1 c i c T i c i c T i .
We rewrite the expectation terms in the equation above as

E z s-1,i,j p s-1,i,j - z s,i,j p s,i,j 2 F {s,i,j}-1 = E z 2 s-1,i,j p 2 s-1,i,j -2 z s-1,i,j p s-1,i,j z s,i,j p s,i,j + z 2 s,i,j p 2 s,i,j F {s,i,j}-1 (a) = E z 2 s-1,i,j p 2 s-1,i,j -2 z s-1,i,j p s-1,i,j z s,i,j p s,i,j + z 2 s,i,j p 2 s,i,j F s-1 = z 2 s-1,i,j p 2 s-1,i,j -2 z s-1,i,j p s-1,i,j 1 p s,i,j E [z s,i,j | F s-1 ] + 1 p 2 s,i,j E z 2 s,i,j F s-1 (b) = z s-1,i,j p 2 s-1,i,j -2 z s-1,i,j p s-1,i,j z s-1,i,j p s-1,i,j + 1 p 2 s,i,j E [z s,i,j | F s-1 ] = 1 p 2 s,i,j E [z s,i,j | F s-1 ] - z s-1,i,j p 2 s-1,i,j (c) = 1 p s,i,j z s-1,i,j p s-1,i,j - z s-1,i,j p 2 s-1,i,j = z s-1,i,j p s-1,i,j 1 p s,i,j - 1 p s-1,i,j ,
where in (a) we use the fact that the approximate probabilities p s-1,i,j and p s,i,j and z s-1,i,j are fixed at the end of the previous iteration, while in (b) and (c) we use the fact that z s,i,j is a Bernoulli of parameter p s,i,j /p s-1,i,j (whenever z s-1,i,j is equal to 1). Therefore, we can write W r at the end of the process as

W h = W {h,m,q} = 1 q 2 q j=1 ν i=1 h s=1 z s-1,i,j p s-1,i,j 1 p s,i,j - 1 p s-1,i,j c i c T i c i c T i .
Before continuing with the other steps necessary to bound W h for SQUEAK, we take a moment to analyze a simpler problem, showing the deterministic bound on the variance of KORS.

Proof of Lemma 3.19. Since KORS does not update its weights after the insertion, i.e., z s,i,j = z s ,i,j and p s,i,j = p s ,i,j for all s > s, we can greatly simplify the notation. We only flip coins for atom i at step i, so we can simplify the notation z s,i,j to z s,j = z s,s,j .

Similarly, the probabilities do not change at all across iteration, so we can drop the percopy index, leaving only the time index that will serve as both time and atom index p s,i,j = p s,i = p s,s = p s .

Chapter 3. Sequential RLS Sampling

We can therefore simplify the definition of the predictable quadratic variation

W h = 1 q 2 q j=1 ν i=1 h s=1 z s-1,i,j p s-1,i,j 1 p s,i,j - 1 p s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 h s=1 z i,i,j 1 p s,i,j p s-1,i,j - 1 p 2 s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 z i,j p i,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 z i,j p i τ h,i c i c T i ,
where we used the fact that the probabilities p h,i = p h-1,i = p i,i are not updated for s > i, and therefore

1 p s,i,j p s-1,i,j = 1 p 2 s-1,i,j
, and that for s = i we have p i-1,i = 1 by definition. We can now bound this quantity as

W h = 1 q 2 q j=1 ν i=1 z i,j p i τ h,i c i c T i ≤ 1 q 2 q j=1 ν i=1 1 p i τ h,i c i c T i ≤ 1 q 2 q j=1 ν i=1 ρ τ h,i τ h,i c i c T i ≤ ρ q ν s=1 c s c T s = ρ q C t C T t = ρ q P t ≤ ρ q := σ 2 .
Where we used Lemma 3.17 to bound 1/p i ≤ ρ/τ h,i .

In a nutshell, this bound show us that if we can bound the chain h s=1 z s-1,i,j p s-1,i,j

1 p s,i,j -1 p s-1,i,j
with 1/τ h,i , then we can find a good bound on the variance.

Note that we could unroll and simplify

h s=1 z s-1,i,j p s-1,i,j 1 p s,i,j -1 p s-1,i,j
only because KORS does not change the probabilities p s,i,j , and z s,i,j is constant after s = i. In SQUEAK these probabilities continue to grow, and if the variable z h,i,j is still non-zero at the end of the chain, after unrolling and simplifying we are left with

1 p h,i,j p h-1,i,j - 1 p 2 1,i,j = 1 p h,i,j p h-1,i,j -1 ≤ 1 p 2 h,i,j
, which is usually much larger than the 1/p i,i,j that we obtained for KORS, since τ h,i ≤ τ i,i . Therefore, in the worst case we have a variance that scales with 1/p 2 h,i,j ≤ ρ 2 /τ 2 h,i , making it impossible to prove a deterministic bound of the order of 1/τ h,i . To tame the larger variance of SQUEAK we continue our derivation looking for a more sophisticated bound that holds not uniformly but in high probability.

Step 2 (a preliminary bound). We can now upper-bound W h as

W h 1 q 2 q j=1 ν i=1 h s=1 z s-1,i,j p s-1,i,j 1 p s,i,j - 1 p s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 z h,i,j p 2 h,i,j - z h,i,j p 2 h,i,j + h s=1 z s-1,i,j p s-1,i,j 1 p s,i,j - 1 p s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 z h,i,j p 2 h,i,j + h s=1 - z s,i,j p 2 s,i,j + z s-1,i,j p s,i,j p s-1,i,j - z 0,i,j p 2 0,i,j c i c T i c i c T i 1 q 2 q j=1 ν i=1 z h,i,j p 2 h,i,j + h s=1 z s-1,i,j p s,i,j p s-1,i,j - z s,i,j p s,i,j p s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 z h,i,j p 2 h,i,j + h s=1 z s-1,i,j (1 -z s,i,j ) p s,i,j p s-1,i,j c i c T i c i c T i ,
where in the inequality we use the fact p s,i,j ≤ p s-1,i,j . From the definition of p s,i,j , we know that when z s,i,j = 0, p s,i,j = p s-1,i,j . Therefore

z s-1,i,j (1-z s,i,j ) p s,i,j p s-1,i,j = z s-1,i,j (1-z s,i,j ) p 2 s-1,i,j
, since the term is non-zero only when z s,i,j = 0, i.e., dropped copies are not reweighted. Finally, we see that only one of the z s-1,i,j (1z s,i,j ) terms can be active for s ∈ [h] and thus

W h 1 q 2 q j=1 ν i=1 z h,i,j p 2 h,i,j + h s=1 z s-1,i,j (1 -z s,i,j ) p 2 s-1,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 max max s=1,...,h z s-1,i,j (1 -z s,i,j ) p 2 s-1,i,j ; z h,i,j p 2 h,i,j c i c T i c i c T i = 1 q 2 q j=1 ν i=1 c i c T i c i c T i max s=0,...,h z s,i,j p 2 s,i,j • 3.11
Step 3 (introduction of a stochastically dominant process). We want to study max s=0,...,h z s,i,j p 2 s,i,j

. To simplify notation, we will consider max s=0,...,h z s,i,j p s,i,j

, where we removed the square, which will be re-added in the end. We know trivially that this quantity is larger or equal than, 1 because z 0,i,j /p 0,i,j = 1, but upper-bounding this quantity is not trivial as the evolution of the various p s,i,j depends in a complex way on the interaction between the random variables z s,i,j . Nonetheless, whenever p s,i,j is significantly smaller than p s-1,i,j , the probability of keeping a copy of point i at iteration s (i.e., z s,i,j = 1) is also very small. As a result, we expect the ratio z s,i,j p s,i,j to be still small with high probability.

Unfortunately, due to the dependence between different copies of the point at different iterations, it seems difficult to exploit this intuition directly to provide an overall highprobability bound on W h . For this reason, we simplify the analysis by replacing each of the (potentially dependent) chains {z s,i,j /p s,i,j } h s=0 with a set of (independent) random variables w 0,i,j that will stochastically dominate them. We define the random variable w s,i,j using the following conditional distribution,7

P 1 w s,i,j ≤ a F s =        0 for a < 1/p s,i,j 1 -1 p s,i,j a for 1/p s,i,j ≤ a < ρ/p h,i 1 for ρ/p h,i ≤ a .
To show that this distribution is well defined, we use Lemma 3.17 to guarantee that 1/p s,i,j ≤ a < ρ/p h,i . Note that the distribution of 1 w s,i,j conditioned on F s is determined by only p s,i,j , p h,i , and ρ, where p h,i and ρ are fixed. Remembering that p s,i,j is a function of F s-1 (computed using the previous iteration), we have that

P 1 w s,i,j ≤ a F s = P 1 w s,i,j ≤ a F s-1 .
Notice that in the definition of w s,i,j , none of the other w s ,i ,j (for any different s , i , or j ) appears and p s,i,j is a function of F s-1 . It follows that given F s-1 , w s,i,j is independent from all other w s ,i ,j (for any different s , i , or j ). This is easier to see in the probabilistic graphical model reported in Fig. 3.5, which illustrates the dependence between the various variables.

Finally for the special case w 0,i,j the definition above reduces to

P 1 w 0,i,j ≤ a =      0 for a < 1 1 -1 a for 1 ≤ a < ρ/p h,i 1
for ρ/p h,i ≤ a , 3.12 since p 0,i,j = 1 by definition. Due to its definition, 1/w 0,i,j is a truncated Pareto r.v., with scale 1 and shape 1, that gets truncated at ρ/p h,i . We can also see from this definition that w 0,i,j and w 0,i ,j are all independent, and this will allow us to use stronger concentration inequalities for independent random variables.

Step 4 Proving the dominance. We remind the reader that a random variable A stochastically dominates random variable B, if for all values a the two equivalent conditions are verified,

P(A ≥ a) ≥ P(B ≥ a) ⇔ P(A ≤ a) ≤ P(B ≤ a).
As a consequence, if A dominates B, the following implication holds,

P(A ≥ a) ≥ P(B ≥ a) =⇒ E[A] ≥ E[B],
while the reverse (A dominates B, if

E[A] ≥ E[B]
) is not true in general. Following this definition of stochastic dominance, our goal is to prove

P max s=0,...,h z s,i,j p s,i,j ≤ a ≥ P 1 w 0,i,j ≤ a .
We prove this inequality by proceeding backwards with a sequence of conditional probabilities. We first study the distribution of the maximum conditional to the state of the algorithm at the end of iteration h, i.e., F h . From the definition of w h,i,j , we know that, w.p. 1, 1/p h,i ≤ 1/w h,i,j . Therefore, P max s=0,...,h z s,i,j p s,i,j ≤ a ≥ P max max s=0,...,h-1 z s,i,j p s,i,j ; z h,i,j w h,i,j ≤ a .

Now focus on an arbitrary intermediate step 1 ≤ k ≤ h, where we fix F k-1 . Since u k,i,j and w k,i,j are independent given F k-1 , we have

P z k,i,j w k,i,j ≤ a F k-1 = P I u k,i,j ≤ p k,i,j p k-1,i,j 1 w k,i,j ≤ a F k-1 =              0 for a ≤ 0 1 - p k,i,j p k-1,i,j for 0 ≤ a < 1/p k,i,j 1 - p k,i,j p k-1,i,j + p k,i,j p k-1,i,j 1 -1 p k,i,j a = 1 - 1 p k-1,i,j a for 1/p k,i,j ≤ a < ρ/p h,i 1 for ρ/p h,i ≤ a ≥            0 for a < 1/p k-1,i,j 1 - 1 p k-1,i,j a for 1/p k-1,i,j ≤ a < 1/p k,i,j 1 - 1 p k-1,i,j a for 1/p k,i,j ≤ a < ρ/p h,i 1 for ρ/p h,i ≤ a 3.13 = P 1 w k-1,i,j ≤ a F k-2 = P 1 w k-1,i,j ≤ a F k-1 ,
where the inequality is also represented in Fig. 3.6. We now proceed by peeling off layers from the end of the chain one by one, taking advantage of the dominance we just proved. Fig. 3.6 visualizes one step of the peeling when z k-1,i,j = 1 (note that the peeling is trivially

1 1/ pk-1 1/ pk α 2 /p h 0 1-pk pk-1 1-p h α 2 pk 1 P max zk-1 pk-1 ; zk w k ≤a | F k-1 P zk-1 w k-1 ≤a|F k-1 zk-1 =1 Figure 3.6: C.d.f
. of max z k-1,i,j /p t,i,j ; z k,i,j /p k,i,j and z k-1,i,j /w k-1,i,j conditioned on F {k-1} . For conciseness, we omit the i, j indices.

true when z k-1,i,j = 0 since the whole chain terminated at step z k-1,i,j ). We show how to move from an iteration k ≤ h to k -1.

P max max s=0...k-1 z s,i,j p s,i,j ; z k,i,j w k,i,j ≤ a = E F k-1 P max max s=0...k-1 z s,i,j p s,i,j ; z k,i,j w k,i,j ≤ a F k-1 (a) ≥ E F k-1 P max max s=0...k-1 z s,i,j p s,i,j ; z k-1,i,j w k-1,i,j ≤ a F k-1 = E F k-1 P max max s=0...k-2 z s,i,j p s,i,j ; z k-1,i,j p k-1,i,j ; z k-1,i,j w k-1,i,j ≤ a F k-1 = E F k-1 P max max s=0...k-2 z s,i,j p s,i,j ; z k-1,i,j max 1 p k-1,i,j ; 1 w k-1,i,j ≤ a F k-1 (b) = E F k-1 P max max s=0...k-2 z s,i,j p s,i,j ; z k-1,i,j w k-1,i,j ≤ a F k-1 = P max max s=0...k-2 z s,i,j p s,i,j ; z k-1,i,j w k-1,i,j ≤ a ,
where in (a), given F k-1 , everything is fixed except u k,i,j and w k,i,j and we can use the stochastic dominance in Eq. 3.13, and in (b) we use the fact that the inner maximum is always attained by 1/w k,i,j since by definition 1/w k-1,i,j is lower-bounded by 1/p k-1,i,j . Applying the inequality recursively from k = h to k = 1 removes all z s,i,j from the maximum and we are finally left with only w 0,i,j as we wanted, P max s=0,...,h z s,i,j p s,i,j ≤ a ≥ P max z 0,i,j p 0,i,j ; z 0,i,j w 0,i,j ≤ a ≥ P 1 w 0,i,j ≤ a ,

where in the last inequality we used that z 0,i,j = 1 from the definition of the algorithm and p 0,i,j = 1 while w 0,i,j ≤ 1 by Eq. 3.12.

Step 5 (stochastic dominance on W h ). Now that we proved the stochastic dominance of 1/w 0,i,j , we plug this result in the definition of W h . For the sake of notation, we introduce the term p max h ,i,j to indicate the maximum over the first h step of copy i, j such that max s=0,...,h z s,i,j p s,i,j = 1 p max h ,i,j

•

We first notice that while Y h is not necessarily PSD, W h is a sum of PSD matrices. Introducing the function Λ({1/p max h,i,j } i,j ) we can restate Eq. 3.11 as

W h = λ max (W h ) ≤ Λ({1/p max h,i,j } i,j ) def = λ max   1 q 2 q j=1 ν i=1 1 p max h,i,j 2 c i c T i c i c T i   .
In Step 4, we showed that 1/p max h,i,j is stochastically dominated by 1/w 0,i,j for every i and j. In order to bound Λ({1/p max h,i,j } i,j ), we need to show that this dominance also applies to the summation over all columns inside the matrix norm. We can reformulate Λ({1/p max h,i,j } i,j ) as

λ max   1 q 2 q j=1 ν i=1 1 p max h,i,j 2 c i c T i c i c T i   = max x: x =1 x T   1 q 2 q j=1 ν i=1 1 p max h,i,j 2 c i c T i c i c T i   x = max x: x =1 1 q 2 q j=1 ν i=1 1 p max h,i,j 2 c i 2 2 x T c i c T i x = max x: x =1 1 q 2 q j=1 ν i=1 1 p max h,i,j 2 ( c i 2 c T i x) 2 .
From this reformulation, it is easy to see that, because 1/p max h,i,j is strictly positive, the function Λ({1/p max h,i,j } i,j ) is monotonically increasing w.r.t. the individual 1/p max h,i,j , or in other words that increasing an 1/p max h,i,j without decreasing the others can only increase the maximum. Introducing Λ({1/w 0,i,j } i,j ) as

Λ({1/w 0,i,j } i,j ) def = max x: x =1 1 q 2 q j=1 ν i=1 1 w 0,i,j 2 ( c i 2 c T i x) 2 ,
we now need to prove the stochastic dominance of Λ({1/w 0,i,j } i,j ) over Λ({1/p max h,i,j } i,j ). Using the definition of 1/p max h,i,j , w h,i,j , and the monotonicity of Λ we have

P   Λ   1 p max h,i,j i,j   ≤ a   = P Λ max max s=0,...,h-1 z s,i,j p s,i,j ; z h,i,j p h,i,j i,j ≤ a ≥ P Λ max max s=0,...,h-1 z s,i,j p s,i,j ; z h,i,j w h,i,j i,j ≤ a . Now pick 1 ≤ k ≤ h, for a fixed F k-1 , 1 p max k-1,i,j
is a constant and max

1 p max k,i,j
; x is a monotonically increasing function in x, making Λ max

1 p max k,i,j
; x also an increasing function. Therefore, we have

P   Λ   max 1 p max k-1,i,j ; z k,e,j w k,i,j i,j   ≤ a   = E F k-1   P   Λ   max 1 p max k-1,i,j ; z k,e,j w k,i,j i,j   ≤ a F k-1     (a) ≥ E F k-1   P   Λ   max 1 p max k-1,i,j ; z k-1,i,j w k-1,i,j i,j   ≤ a F k-1     (b) = E F k-1   P   Λ   max 1 p max k-2,i,j ; z k-1,i,j w k-1,i,j i,j   ≤ a F k-1     ,
where inequality (a) follows from the fact that stochastic dominance is preserved by monotonically increasing functions [START_REF] Levy | Stochastic dominance: Investment decision making under uncertainty[END_REF], such as Λ, combined with the fact that for a fixed F k-1 the variables z k,i,j and w k,i,j are all independent and (b) from the definition of 1/p max k-1,i,j and the fact that by definition 1/w k-1,i,j is lower-bounded by 1/p k-1,i,j . We can iterate this inequality to obtain the desired result

P( W h ≥ σ 2 ) ≤ P   Λ   1 p max h,i,j i,j   ≥ σ 2   ≤ P   λ max   1 q 2 q j=1 ν i=1 1 w 0,i,j 2 c i c T i c i c T i   ≥ σ 2   .
Step 6 (concentration inequality). Since all w 0,i,j are (unconditionally) independent from each other, we can apply the following theorem. Introduce the random matrix V def = k X k , and the maximum eigenvalue of its expectation

µ max def = λ max (E [V]) = λ max k E [X k ] .

Proof of main results

83

Then, for all h ≥ 0,

P (λ max (V) ≥ (1 + h)µ max ) ≤ d • e h (1 + h) 1+h µmax L ≤ d • exp - µ max L ((h + 1) log(h + 1) -h) •
In our case, we have

X {i,j} = 1 q 2 1 w 2 0,i,j c i c T i c i c T i 1 q 2 ρ 2 p 2 h,i c i c T i c i c T i 1 q 2 ρ 2 p 2 h,i c i c T i 2 I ρ 2 q 2 I,
where the first inequality follows from the definition of w 0,i,j in Eq. 3.12, the second from the PSD ordering, and the third from the definition of c i c T i .

Therefore, we can use L def = ρ 2 /q 2 for the purpose of Proposition 3.21. We need now to compute E [X k ], that we can use in turn to compute µ max . We begin by computing the expected value of 1/w 0,i,j . Let us denote the c.d.f. of 1/w 2 0,i,j as

F 1/w 2 0,i,j (a) = P 1 w 2 0,i,j ≤ a = P 1 w 0,i,j ≤ √ a =      0 for a < 1 1 -1 √ a for 1 ≤ a < ρ 2 /p 2 h,i 1 for ρ 2 /p 2 h,i ≤ a .
Since P 1/w 2 0,i,j ≥ 0 = 1, we have that

E 1 w 2 0,i,j = ∞ a=0 1 -F 1/w 2 0,i,j (a) da = 1 a=0 1 -F 1/w 2 0,i,j (a) da + ρ 2 p 2 h,i a=1 1 -F 1/w 2 0,i,j (a) da + ∞ a= ρ 2 p 2 h,i 1 -F 1/w 2 0,i,j (a) da = 1 a=0 (1 -0) da + ρ 2 /p 2 h,i a=1 1 -1 - 1 √ a da + ∞ a=ρ 2 /p 2 h,i (1 -1) da = 1 a=0 da + ρ 2 /p 2 h,i a=1 1 √ a da = 1 + [2 √ a] ρ 2 /p 2 h,i 1 = 2ρ/p h,i -1.
Therefore,

µ max = λ max (E [V]) = λ max {i,j} E X {i,j} = λ max   1 q 2 q j=1 ν i=1 E 1 w 2 0,i,j c i c T i c i c T i   = λ max 1 q ν i=1 2ρ p h,i -1 p h,i c i c T i ≤ λ max 2ρ q ν i=1 c i c T i = 2ρ q λ max (P) ≤ 2ρ q def = L.
Therefore, selecting h = 2, σ 2 = 6ρ/q and applying Proposition 3.21 we have

P W h ≥ σ 2 ≤ P   λ max   1 q 2 q j=1 ν i=1 1 w 2 0,i,j c i c T i c i c T i   ≥ (1 + 2) 2ρ q   ≤ ν • exp - 2ρ q q 2 ρ 2 (3 log(3) -2) ≤ n • exp - 2q ρ •
Chapter 3. Sequential RLS Sampling

Space complexity bound

Denote with A the event A = ∀h ∈ {1, . . . , h} : P h -P h 2 ≤ ε , and again ν = |D {h,l} |. Letting q = |I {h,l} | = ν i=1 q h,i = q j=1 ν i=1 z h,i,j be the random number of points in I {h,l} , we reformulate

P |I {h,l} | ≥ 3qd {h,l} eff (γ) ∩ ∀h ∈ {1, . . . , h} : P h -P h 2 ≤ ε ≤ ε = P |I {h,l} | ≥ 3qd {h,l} eff (γ) ∩ A = P   q j=1 ν i=1 z h,i,j ≥ 3qd {h,l} eff (γ) ∩ A   = P   q j=1 ν i=1 z h,i,j ≥ 3qd {h,l} eff (γ) A   P (A) .
While we do know that the z h,i,j are Bernoulli random variables (since they are either 0 or 1), it is not easy to compute the success probability of each z h,i,j , and in addition there could be dependencies between z h,i,j and z h,i ,j . Similarly to Lem. 3.20, we are going to find a stochastic variable to dominate z h,i,j . Denoting with u s,i,j ∼ U(0, 1) a uniform random variable, we will define w s,i,j as

w s,i,j |F {s,i ,j } = w s,i,j |F s-2 def = I u s,i,j ≤ p h,i p s-1,i ∼ Ber p h,i p s-1,i
for any i and j such that {s, 1, 1} ≤ {s, i , j } < {s, i, j}. Note that w s,i,j , unlike z s,i,j , does not have a recursive definition, and its only dependence on any other variable comes from p s-1,i . First, we peel off the last step

P   q j=1 ν i=1 z h,i,j ≥ g A   = E F t-1 |A   P   q j=1 ν i=1 I u h,i,j ≤ p h,i p t-1,i z h-1,i,j ≥ g F t-1 ∩ A     ≤ E F t-1 |A   P   q j=1 ν i=1 I u h,i,j ≤ p h,i p t-1,i z h-1,i,j ≥ g F t-1 ∩ A     = P   q j=1 ν i=1 w h,i,j z h-1,i,j ≥ g A   ,
where we used the fact that conditioned on A, I {h,l} is accurate w.r.t. D {h,l} , which guarantees that p h,i ≤ p h,i . Plugging this in the previous bound,

P   q j=1 ν i=1 z h,i,j ≥ g A   P (A) ≤ P   q j=1 ν i=1 w h,i,j z h-1,i,j ≥ g ∩ A   ≤ P   q j=1 ν i=1 w h,i,j z h-1,i,j ≥ g   .
3.5 Extended proofs 85

We now proceed by peeling off layers from the end of the chain one by one. We show how to move from an iteration s ≤ h to s -1.

P   q j=1 ν i=1 w s,i,j z s-1,i,j ≥ g   = E F s-2   P   q j=1 ν i=1 I u s,i,j ≤ p h,i p s-1,i z s-1,i,j ≥ g F s-2     = E F s-2   P   q j=1 ν i=1 I u s,i,j ≤ p h,i p s-1,i I u s-1,i,j ≤ p s-1,i p s-2,i z s-2,i,j ≥ g F s-2     = E F s-2   P   q j=1 ν i=1 I u s-1,i,j ≤ p h,i p s-2,i z s-2,i,j ≥ g F s-2     = P   q j=1 ν i=1 w s-1,i,j z s-2,i,j ≥ g  
Applying this repeatedly from s = h to s = 2 we have,

P   q j=1 ν i=1 w h,i,j z h-1,i,j ≥ g   = P   q j=1 ν i=1 w 1,i,j z 0,i,j ≥ g   = P   q j=1 ν i=1 w 1,i,j ≥ g   .
Now, all the w 1,i,j are independent Bernoulli random variables, and we can bound their sum with Proposition 2.13.

Extended proofs

Proof of Lemma 3.8 and Lemma 3.13. It is easy to show that τ t,i ≤ τ t-1,i for all t > i (τ t,i is not defined for t < i).

Since Φ t Φ T t + Π t Φ t-1 Φ T t-1 + Π t-1 and φ i = Π t φ i = Π t-1 φ i we have τ t,i = φ T i (Φ t Φ T t + γΠ t ) -1 φ i ≤ φ T i (Φ t-1 Φ T t-1 + γΠ t-1 ) -1 φ i = τ t-1,i .
We will now prove the lower bound τ t,i ≥ τ t-1,i /(τ t-1,1 + 1).

Note that since both Π t-1 and Π t include φ i , we have

τ t-1,i = φ T i (Φ t-1 Φ T t-1 +γΠ t-1 ) -1 φ i = φ T i (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i .
Considering the definition of τ t,i in terms of φ i and Φ t , and applying the Sherman-Morrison formula we obtain

τ t,i = φ T i (Φ t Φ T t + γ t Π) -1 φ i = φ T i (Φ t-1 Φ T t-1 + φ t φ T t + γ t Π t ) -1 φ i = φ T i (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i - φ T i (Φ t-1 Φ T t-1 + γΠ t ) -1 φ t φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i 1 + φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ t = τ t-1,i - φ T i (Φ t-1 Φ T t-1 + γΠ t ) -1 φ t φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i 1 + φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ t • Let x = (Φ t-1 Φ T t-1 + γΠ t ) -1/2 φ i and y = (Φ t-1 Φ T t-1 + γΠ t ) -1/2 φ t .
Chapter 3. Sequential RLS Sampling

Then τ t,i /τ t-1,i is equal to τ t,i τ t-1,i = 1 - (φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i ) 2 (1 + φ T t (Φ t-1 Φ T t-1 + γΠ t ) -1 φ t )φ T i (Φ t-1 Φ T t-1 + γΠ t ) -1 φ i = 1 - (y T x) 2 (1 + y T y) x T x •
Defining the cosine between y and x as cos(y, x) = y T x/( x y ), we have that

1 - (y T x) 2 (1 + y T y) x T x = 1 - y T yx T xcos(y, x) 2 (1 + y T y) x T x = 1 - y 2 1 + y 2 cos(y, x) 2 ,
where y 2 1+ y 2 depends only on the norm of y and not its direction, and cos(y, x) depends only on the direction of y and is maximized when y = x. Therefore,

τ t+1,i τ t,i = 1 - (y T x) 2 (1 + y T y) x T x = 1 - y 2 1 + y 2 cos(y, x) 2 ≥ 1 - x 2 1 + x 2 = 1 1 + x 2 = 1 1 + τ t,i
.

For the effective dimension, we can simply use the fact [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] that the map

ΦΦ T → Tr(ΦΦ T (ΦΦ T + γΠ)),
is monotone in ΦΦ T to see that

d t-1 eff (γ) = Tr(Φ t-1 Φ T t-1 (Φ t-1 Φ T t-1 + γΠ t-1 )) ≤ Tr(Φ t Φ T t (Φ t Φ T t + γΠ t )) = d t eff (γ).
For the more general Lemma 3.13, the first point τ D,i ≥ τ D∪D ,i can be easily proven by fixing the first D, and then invoke Lemma 3.8 as we add one sample at a time from D . Also as easily for d D eff (γ) + d D eff (γ) we have

d D eff (γ) + d D eff (γ) ≤ 2 max{d D eff (γ); d D eff (γ)} ≤ 2 max{d eff (γ) D∪D ; d eff (γ) D∪D } = 2d eff (γ) D∪D .
Finally, we prove the other side of the inequality for d D eff (γ) + d D eff (γ). Let Φ D , Φ D be the maps constructed using the samples in D and D respectively. Then,

d D eff (γ) + d D eff (γ) = i∈D τ D,i + i∈D τ D ,i = i∈D φ T i (Φ D Φ T D + γΠ D ) -1 φ i + i∈D φ T i (Φ D Φ T D + γΠ D ) -1 φ i ≥ i∈D φ T i (Φ D∪D Φ T D∪D + γΠ D∪D ) -1 φ i + i∈D φ T i (Φ D∪D Φ T D∪D + γΠ D∪D ) -1 φ i = i∈D∪D φ T i (Φ D∪D Φ T D∪D + γΠ D∪D ) -1 φ i = i∈D∪D τ D∪D ,i = d D∪D eff (γ).
Proof of Lemma 3.3 and Lemma 3.14. We give a proof for a generalized estimator that instead of using a single dictionary I uses k ε-accurate dictionaries I k . Given the dataset D and its associated Φ and Π we split it in disjoint datasets {D i } k i=1 with associated maps Φ i . From each dataset, construct an (ε, γ)-accurate dictionary I i , with its associated selection matrix S i .

To estimate the RLS τ i of point i w.r.t. the whole dataset D = ∪ k j=1 D j , and corresponding feature matrix Φ, we set the estimator to be

τ i = (1 -ε)φ T i   k j=1 Φ j S j S T j Φ T j + (1 + (k -1)ε)γΠ   -1 φ i .
Part 1: accuracy of the RLS estimator τ i . Since each of the dictionaries I i used to generate S i is (ε, γ)-accurate, we have from Definition 2.6 and Lemma 2.7 that

(1 -ε)Φ i Φ T i -εγΠ Φ i S i S T i Φ T i (1 + ε)Φ i Φ T i + εγΠ. Therefore, we have τ i = (1 -ε)φ T i   k j=1 Φ j S j S T j Φ T j + (1 + (k -1)ε)γΠ   -1 φ i ≤ (1 -ε)φ T i   k j=1 (1 -ε)Φ j Φ T j -εγΠ + (1 + (k -1)ε)γΠ   -1 φ i ≤ (1 -ε)φ T i ((1 -ε)ΦΦ T -kεγΠ + (1 + (k -1)ε)γΠ) -1 φ i = (1 -ε)φ T i ((1 -ε)(ΦΦ T + γΠ)) -1 φ i = (1 -ε) (1 -ε) φ T i (ΦΦ T + γΠ) -1 φ i = τ i ,
and

τ i = (1 -ε)φ T i   k j=1 Φ j S j S T j Φ T j + (1 + (k -1)ε)γΠ   -1 φ i ≥ (1 -ε)φ T i   k j=1 (1 + ε)Φ j Φ T j + εγΠ + (1 + (k -1)ε)γΠ   -1 φ i = (1 -ε)φ T i ((1 + ε)ΦΦ T + kεγΠ + (1 + (k -1)ε)γΠ) -1 φ i = (1 -ε)φ T i ((1 + ε)ΦΦ T + (1 + (2k -1)ε)γΠ)) -1 φ i ≥ (1 -ε)φ T i ((1 + (2k -1)ε)ΦΦ T + (1 + (2k -1)ε)γΠ)) -1 φ i = (1 -ε) (1 + (2k -1)ε) φ T i (ΦΦ T -γΠ) -1 φ i = (1 -ε) (1 + (2k -1)ε) τ i .
Then, we can instantiate this result with k = 1 to prove the accuracy claim in Lem. 3.3, and with k = 2 to prove the accuracy claim in Lem. 3.14.

Part 2: accuracy of min { τ t , τ t-1 }. To simplify the notation, for this part of the proof we indicate with τ t ≤ τ t-1 that for each i ∈ {1, . . . , t -1} we have τ t,i ≤ τ t-1,i . From Lem. 3.8, we know that τ t-1 ≥ τ t . Given α-accurate τ t and τ t-1 we have the upper bound

min { τ t , τ t-1 } ≤ min {τ t , τ t-1 } = τ t ,
and the lower bound,

min { τ t , τ t-1 } ≥ 1 α min {τ t , τ t-1 } = 1 α τ t ,
which combined gives us 1 α τ t ≤ min { τ t , τ t-1 } ≤ τ t as required by the definition of αaccuracy.

Proof of Lemma 3.5. We will prove this result for a generic k ≥ 1 and (ε, kγ)-accurate dictionary I, with (ε, 2γ)-accuracy as a special case when k = 2. Starting from the proof of Lemma 3.3 and Lemma 3.14 we can adapt some passages. In particular we have only one single (ε, kγ)-accurate dictionary I and a single dataset D, and we want to show that

τ i = (1 -ε)φ T i (ΦSS T Φ T + (1 + (k -1)ε)γΠ) -1 φ i . is α-accurate with α = 1+2(k-1)ε 1-ε .
Since I used to generate S is (ε, kγ)-accurate, The identically to the proof of Lemma 3.3 we have

τ i = (1 -ε)φ T i (ΦSS T Φ T + (1 + (k -1)ε)γΠ) -1 φ i ≤ (1 -ε)φ T i (((1 -ε)ΦΦ T -εγΠ) + (1 + (k -1)ε)γΠ) -1 φ i ≤ (1 -ε)φ T i ((1 -ε)ΦΦ T -kεγΠ + (1 + (k -1)ε)γΠ) -1 φ i = (1 -ε)φ T i ((1 -ε)(ΦΦ T + γΠ)) -1 φ i = (1 -ε) (1 -ε) φ T i (ΦΦ T + γΠ) -1 φ i = τ i , and
τ i = (1 -ε)φ T i (ΦSS T Φ T + (1 + (k -1)ε)γΠ) -1 φ i ≥ (1 -ε)φ T i ((1 + ε)ΦΦ T + kεγΠ + (1 + (k -1)ε)γΠ) -1 φ i = (1 -ε)φ T i ((1 + ε)ΦΦ T + (1 + (2k -1)ε)γΠ)) -1 φ i ≥ (1 -ε)φ T i ((1 + (2k -1)ε)ΦΦ T + (1 + (2k -1)ε)γΠ)) -1 φ i = (1 -ε) (1 + (2k -1)ε) φ T i (ΦΦ T -γΠ) -1 φ i = (1 -ε) (1 + (2k -1)ε) τ i .
Then, we can instantiate this result with k = 2 to prove the accuracy claim in Lem. 3.5.

Proof of Lem. 3.9. From the definition of τ t,t we have

T t=1 τ t,t = T t=1 φ T t (Φ t Φ T t + αΠ T ) -1 φ t = T t=1 (φ T t / √ α) (Φ t Φ T t /α + Π T ) -1 (φ t / √ α) ≤ log(Det(Φ T Φ T T /α + Π T )),
where the last passage is proved by [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF]. Using Sylvester's determinant identity,

Det(Φ T Φ T T /α + Π T ) = Det(Φ T T Φ T /α + I T ) = T t=1 (λ t /α + 1),
where λ t are the eigenvalues of Φ T T Φ T . Then,

T t=1 τ t,t ≤ log T t=1 (λ t /α + 1) = T t=1
log(λ t /α + 1).

We can decompose this as

T t=1 log(λ t /α + 1) = T t=1 log(λ t /α + 1) λ t /α + 1 λ t /α + 1 = T t=1 log(λ t /α + 1) λ t /α λ t /α + 1 + T t=1 log(λ t /α + 1) λ t /α + 1 ≤ log( Φ T T Φ T /α + 1) T t=1 λ t λ t + α + T t=1 log(λ t /α + 1) λ t /α + 1 ≤ log( Φ T T Φ T /α + 1)d T eff (α) + T t=1 (λ t /α + 1) -1 λ t /α + 1 = log( Φ T T Φ T /α + 1)d T eff (α) + d T eff (α),
where the first inequality is due to Φ T T Φ T ≥ λ t for all t and the monotonicity of log(•), and the second inequality is due to log(x) ≤ x -1.

Sequential RLS Sampling in the Batch Setting

In this section, we show how ε-accurate and (ε, γ)-accurate dictionaries can be used for important downstream tasks, and how to instantiate our sequential RLS sampling algorithms to each setting. In particular, we focus on low-rank kernel matrix approximation and spectral graph sparsification.

Sequential RLS sampling for kernel matrix approximation

One of the major limits of kernel ridge regression (KRR), kernel PCA [START_REF] Schölkopf | Kernel principal component analysis[END_REF], and other kernel methods is that for n samples storing and manipulating the empirical kernel matrix K n requires O(n 2 ) space, which becomes rapidly infeasible for even a relatively small n. A popular approach [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF][START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF][START_REF] Williams | Using the Nystrom method to speed up kernel machines[END_REF] is to construct low-rank approximations of the kernel matrix by randomly selecting a subset (dictionary) of columns from K n , i.e., dictionary learning where the atoms are the columns of K n . Since they are also based on Nyström sampling, these methods are often referred to as Nyström approximations. It is also easy to see that the dictionary of columns can be replaced with a dictionary of atoms. For example, the standard Nyström approximation

K n = K n S(S T K n S) + S T K n can be rewritten as K n = Φ T ΦS(S T KS T ) + S T Φ T Φ using the relationship K n = Φ T n Φ n ,
or in other words it is equivalent to projecting the map Φ n on the subspace Π In spanned by the dictionary I n . Therefore, we can use KORS or SQUEAK to find a good dictionary I n to approximate K n .

More formally, we have again that our atoms φ i are point in an H and our goal is to approximate the empirical kernel matrix K = Φ T Φ. Note that this is a different goal than approximating the covariance operator ΦΦ T , although the equivalence of Lemma 2.10 showed us how the two objects are closely related. We will not consider the simpler case where H = R d is a euclidean space (i.e., we will stick to the d n setting), since in this case the atoms φ i = x i can be explicitly stored and manipulated, and there exists a plethora of different efficient methods that can be applied.

For example, we saw already that in this special case KORS is equivalent to a method proposed by [START_REF] Cohen | Online Row Sampling[END_REF]. For any curiosity on to the topic of matrix approximation in Euclidean spaces, we refer the reader to the excellent survey by Woodruff, (2014) on randomized linear algebra.

We begin by introducing additional notation. In particular, using the kernel trick [START_REF] Aizerman | Theoretical foundations of the potential function method in pattern recognition learning[END_REF] we can introduce the kernel function K so that the inner product between two points in H can be expressed as

K(x, x ) = K(x, •), K(x , •) H = ϕ(x), ϕ(x ) H = ϕ(x) T ϕ(x ).
Given a dataset of samples D = {x t } n t=1 , we define the (empirical) kernel matrix K t ∈ R t×t as the application of the kernel function on all pairs of input values (i.e.,

[K t ] ij = k i,j = K(x i , x j ) for any i, j ∈ [t]
), with k t,i = K t e t,i as its i-th column.

Kernel matrix reconstruction

We begin by showing how to get an accurate approximation K from an (ε, γ)-accurate dictionary. For this task [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF] shows the following reconstruction guarantees Proposition 4.1 - [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF]. Given an (ε, γ)-accurate dictionary I of map Φ : R n → H, and its selection matrix S ∈ R n×n the regularized Nyström approximation

K of K K = KS(S T KS + γI I ) -1 S T K = Φ T ΦS(S T KS T + γI I ) + S T Φ T Φ, satisfies 0 K -K γ 1 -ε K(K + γI n ) -1 γ 1 -ε I n .
Since K is the result of passing Φ trough a regularized projection on the atoms in the dictionary I Γ I = ΦS(S T KS T + γI I ) + S T Φ T , it is easy to see that K K no matter the dictionary. Proposition 4.1 gives us a also an upper bound, showing that if the dictionary is (ε, γ)-accurate the projection preserves the matrix K up to a small additive error γ 1-ε K(K + γI n ) -1 , which can be further upper bounded by γ 1-ε I n . [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] show a slightly tighter bound for the unregularized projection matrix construed using the dictionary. Proposition 4.2 - [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]. Given an (ε, γ)-accurate dictionary I of map Φ : R n → H, and its selection matrix S ∈ R n×n the unregularized Nyström approximation K of K satisfies

K = KS(S T KS) + S T K = Φ T ΦS(S T KS T ) + S T Φ T Φ, 0 K -K εγ 1 -ε I n .
Note that this time the we do not have a bound in terms of K(K + γI n ) -1 , but directly in terms of I n . We could have obtained a similar result, but with a larger γ/(1ε) bound, simply by observing that Alaoui and Mahoney, (2015)'s bound for the regularized approximation implies

K KS(S T KS + γI I ) -1 S T K + γ 1 -ε I n KS(S T KS) + S T K + γ 1 -ε I n .
4.1 Sequential RLS sampling for kernel matrix approximation 93

Removing the regularization γ has several advantages. Computationally, we do not need anymore to reweight the atoms using S when constructing the approximation, which in practice saves us time and space when computing K and improves numerical stability. Moreover the operator Π I = ΦS(S T KS T ) + S T is the projection on Im(ΦS), and this gives us an intuition that the subspaces spanned by the dataset and the dictionary are close. Finally, since Π I is a projection, it defines an accurate finite-rank subspace H I , which will be useful in Chapter 5.

We can simplify both of the original proofs, and slightly improve the bound of [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF], by showing that they are special cases of the following result.

Lemma 4.3 Given a dictionary I that is (ε, γ)-accurate w.r.t. map Φ : R n → H, and its selection matrix S ∈ R n×n , for any α ≥ εγ we have

Π ΦS(S T Φ T ΦS + αI I ) -1 S T Φ T + α 1 -ε ΦΦ T + α -εγ 1 -ε Π + . 4.1 Therefore K KS(S T KS + αI I ) -1 S T K + α 1 -ε K K + α -εγ 1 -ε I n + KS(S T KS) + S T K + α 1 -ε K (K) + . Proof of Lemma 4.3. Using Π's definition Π = (ΦSS T Φ T + αΠ) -1/2 (ΦSS T Φ T + αΠ)(ΦSS T Φ T + αΠ) -1/2 = (ΦSS T Φ T + αΠ) -1/2 ΦSS T Φ T (ΦSS T Φ T + αΠ) -1/2 + α(ΦSS T Φ T + αΠ) -1/2 Π(ΦSS T Φ T + αΠ) -1/2 = (ΦSS T Φ T + αΠ) -1/2 ΦSS T Φ T (ΦSS T Φ T + αΠ) -1/2 + α(ΦSS T Φ T + αΠ) -1 .
Using the (ε, γ)-accuracy

(ΦSS T Φ T + γΠ) -1 ((1 -ε)ΦΦ T + (α -εγ)Π) -1 = 1 1 -ε ΦΦ T + α -εγ 1 -ε I n -1
.

We finish by reformulating

ΦS(S T Φ T ΦS + αI I ) -1 S T Φ T = (ΦSS T Φ T + αΠ) -1/2 ΦSS T Φ T (ΦSS T Φ T + αΠ) -1/2 .
Therefore, setting α = γ gives us Alaoui and Mahoney, (2015)'s bound, and setting α = εγ gives us

0 K -KS(S T KS + εγI I ) -1 S T K εγ 1 -ε KK + εγ 1 -ε I n . Noting K KS(S T KS + εγI I ) -1 S T K + εγ 1 -ε I n KS(S T KS) + S T K + εγ 1 -ε I n
recovers [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF]'s bound.

Computing

(S T n K n S n + λ(S T n K n S n + γI In )) -1 takes O(n|I n | 2
) time to construct the matrix and O(|I n | 3 ) to invert it, while the other matrix-matrix multiplication take at most O(n|I n | 2 ) time. Overall, these operations require to store at most an n × |I n | matrix. Therefore the final complexity of computing a KRR using the dictionary is reduced from O(n 3 ) to a linear O(n|I n | 2 + |I n | 3 ) time, and from O(n 2 ) to again linear O(n|I n |) space.

The following corollary provides guarantees for the empirical risk of the solution ω n in a fixed design setting.

Corollary 4.4 - (Alaoui and Mahoney, 2015, Thm. 3). For an arbitrary dataset D, let K be the kernel matrix constructed on D. Given an (ε, γ)-accurate dictionary I of D, the solution ω computed using the regularized Nyström approximation K satisfies

R D ( ω) ≤ 1 + γ λ ε 1 -ε 2 R D ( ω),
where λ is the regularization of kernel ridge regression problem.

Therefore, if we choose λ = γ, i.e., if the additive error that we introduce is of the same order as the regularization, the approximate solution that can be computed in O(nd n eff (γ) 2 ) time (including computing the dictionary with KORS or SQUEAK), and will have an empirical risk that is only a constant factor 1/(1ε) 2 larger. In other words, using RLS sampling we can avoid expending computational power to compute a solution along directions that are not relevant to the problem, since all directions with an influence smaller than λ will be suppressed by the regularization.

If we also assume that the samples x i are being generated according to a distribution µ(x i ) we can move from bounding the empirical risk R D to bounding the expected risk R µ . To do so, we will adapt a result by [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF]. We now consider the case where we are given a dataset D = {(x i , y i )} n i=1 with x i sampled according to µ and where the target variables y i = φ T i w * + i are computed using an arbitrary function w * in H with bounded norm. Given D the usual estimator used in KRR, i.e., Eq. 4.3 is w = Φ n ω n . [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF] propose instead to use the estimator w n = Φ I ω I where Φ I are the atoms contained in an (ε, γ)-accurate dictionary I, and the weights ω I ∈ R |I| are computed as

ω I = (Φ T I Φ n Φ T n Φ I + λΦ T I Φ I ) + Φ T I Φ n y n . 4.6 Note that since (Φ T I Φ n Φ T n Φ I + λΦ T I Φ I ) + is again a |I| × |I| matrix, computing ω I takes only O(n|I n | 2 + |I n | 3 ) time.
If the (ε, γ)-accurate dictionary is constructed using SQUEAK or KORS, this means that overall we can compute a solution using only O(nd n eff (γ) 3 ) time, and in a single pass over the dataset, without ever constructing or storing the full matrix K n . Another advantage of the estimator ω I ∈ R |I| compared to ω ∈ R n is that at evaluation time we will only require O(d n eff (γ)) kernel evaluations, regardless of n. We can now define the expected risk of the estimator ω I as

R µ (Φ I ω I ) = E x∼µ; (ϕ(x) T w * + -ϕ(x) T Φ I ω I ) 2 .
We can also define an analogous of the empirical effective dimension d n eff (γ) based on the distribution µ rather than a dataset.

Chapter 4. Sequential RLS Sampling in the Batch Setting Definition 4.1 - [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF]. Given a feature map ϕ(•) : X → H and a distribution µ defined over X the effective dimension of H w.r.t. µ is

d µ eff (γ) = E x∼µ ϕ(x) T E x ∼µ [ϕ(x )ϕ(x ) T ] + γΠ -1 ϕ(x) .
To obtain their expected risk bound, [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF] make several technical assumption.

We report the main one, in the statement of Proposition 4.5 and refer the reader to the full paper for details.

Proposition 4.5 - [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF]. Given a dataset D sampled according to distribution µ, and a dictionary I (ε, γ)-accurate w.r.t. D, let ω I be computed according to Eq. 4.6 with λ = n -1/(1+q) .

If the eigenvalues of the covariance operator E x ∼µ [ϕ(x )ϕ(x ) T ] decay at least polynomially, i.e., λ i (E x ∼µ [ϕ(x )ϕ(x ) T ]) ≤ i -q , with polynomial degree q ≥ 1 then w.h.p. there exist some constants R and R such that

d µ eff (λ) ≤ Rλ -1/q = Rn 1/(1+q) , R µ (Φ I ω I ) -R µ (w * ) ≤ R n -1/(1+q)
As [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF] remarks, this holds for q = 1 whenever the covariance operator is trace class, which includes our bounded norm φ i case. Therefore, running KORS or SQUEAK to construct the dictionary, we can efficiently compute a solution with expected risk guarantees that achieve the optimal [START_REF] Caponnetto | Fast Rates for Regularized Least-squares Algorithm[END_REF] non-parametric rate n -1/(1+q) , and falls back to the standard non-parametric rate n -1/2 in the worst case. Moreover, [START_REF] Rudi | Less is more: Nystrom computational regularization[END_REF] show that w.h.p. O(nd n eff (λ) 3 ) ≤ O(nd µ eff (λ)) and therefore our computational complexity will be smaller than O(nd n eff (λ) 3 ) ≤ O(nd µ eff (λ) 2 ) ≤ O(n 3/(q+1) ), which is always a strict improvement over the O(n 3 ) of exact KRR, and even over the O(n 2 ) time required to construct the matrix K n . Computationally, we also outperform simpler strategies like uniform sampling [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF], and if we take into account that SQUEAK can be run in less than linear time using a parallel merge tree, we outperform other state of the art RLS sampling methods such as Recursive-RLS [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF], or even distributed averaging approaches such as the one in [START_REF] Zhang | Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates[END_REF].

Kernel principal component analysis

Starting again from Lemma 4.3 we can provide guarantees for Kernel principal component analysis (K-PCA). In K-PCA, given a dataset D = {x i } n i=1 we are interested in finding a k-rank projection Π k : H → H that minimizes

Π k = arg min Π:Rank(Π)=k Φ n -ΠΦ n 2 F = arg min Π:Rank(Π)=k Tr(Φ T n Φ n -Φ T n ΠΦ n ),
where • F is the Frobenius norm. Given an (ε, γ)-accurate dictionary I, using Lemma 4.3 with α = γ it easy to see that the regularized projection on the dictionary I satisfies

Π n - γ 1 -ε (ΦΦ T + γΠ n ) -1 ΦS(S T Φ T ΦS + γI I ) -1 S T Φ T ΦS(S T Φ T ΦS) + S T Φ T .
4.2 Efficient learning with kernels 97 and therefore using the columns in the dictionary as a projection

Tr(Φ T n Φ n -Φ T n ΦS(S T Φ T ΦS) + S T Φ T Φ n ) ≤ Tr Φ T n Φ n -Φ T n Π n Φ n + γ 1 -ε Φ T n (ΦΦ T + γΠ n ) -1 Φ n = γ 1 -ε d n eff (γ).
All that is left is to see how far away the optimal Π k is, in terms of Frobenius norm, from d n eff (γ), where we will fix k and optimize over γ. Let again Φ n = VΣU T be the SVD decomposition of Φ n . We know that the K-PCA objective is minimized by V k , the map containing the first k columns v i of V [START_REF] Schölkopf | Kernel principal component analysis[END_REF], and therefore

Π k = V k V T k .
Similarly, let Σ k be the diagonal matrix with only the first k singular value on the diagonal. The cost of the optimal solution is

Tr(Φ T n Φ n -Φ T n Π k Φ n ) = Tr(Φ T n Φ n -Φ T n V k V T k Φ n ) = Tr(Σ T Σ -Σ T k Σ k ) = n i=k+1 σ 2 i . Setting γ = 1 k n i=k+1 σ 2 i we have d n eff 1 k n i=k+1 σ 2 i = n j=1 σ 2 j σ 2 j + 1 k n i=k+1 σ 2 i = k j=1 σ 2 j σ 2 j + 1 k n i=k+1 σ 2 i + n j=k+1 σ 2 j σ 2 j + 1 k n i=k+1 σ 2 i ≤ k j=1 1 + n j=k+1 σ 2 j 1 k n i=k+1 σ 2 i = k + k n j=k+1 σ 2 j n i=k+1 σ 2 i = 2k.
Putting it all together we have the final result.

Theorem 4.6 Given a dictionary I that is (ε, γ)-accurate w.r.t. map Φ : R n → H, and its selection matrix S ∈ R n×n the projection Π I = Φ T n ΦS(S T Φ T ΦS) + S T Φ T on the atoms contained in the dictionary satisfies

Φ n -Π I Φ n 2 F ≤ γ (1 -ε) d n eff (γ). If γ = 1 k n i=k+1 σ 2 i = Φ n -Π k Φ n 2 F /k we have Φ n -Π I Φ n 2 F ≤ 2 1 -ε Φ n -Π k Φ n 2 F .
While in general it is not easy to compute the right γ that satisfies the condition, it is possible to invert the relationship, i.e., express k in function of γ and the spectrum of the matrix, and use additional information on Φ n (e.g., bounded trace) to know how many accurate eigenfunctions we can extract.

Computationally, using SQUEAK with a sequential merge tree to compute the (ε, γ)accurate dictionary matches the near-linear time and O(d n eff (γ)) space complexity of other state of the art batch approximate K-PCA methods [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF], or outperforms them using a parallel merge tree that requires less than linear runtime. For streaming approximate K-PCA methods, existing methods that provide guarantees, such as the one proposed in Ghashami et al., (2016b), all require at least linear runtime, and parallelizing SQUEAK gives us again an improvement. This despite the fact that many of these method require the input space to be Euclidean, and SQUEAK makes no such assumption.

Sequential RLS sampling for graph spectral sparsification

We will now look at the special case when the atoms that we want to select for our dictionary are edges in a graph G.

Graphs are a useful tool in machine learning to study the relationship between heterogeneous entities. In many settings, the graph structure arises naturally from the data (e.g., social or collaboration networks, influence graphs), and we have only access to the nodes and edges without knowing in detail the process that generated them. Even if we have access to a flat vectorial representation for the entities (nodes), it is common to define a similarity function (e.g., exponential kernel), and construct a similarity graph out of the vectorial data. Regardless of the origin of the graph G (natural or constructed ), many important inference problems are defined starting from the Laplacian representation of the graph, for example, graph semi-supervised learning (SSL) [START_REF] Chapelle | Semi-Supervised Learning[END_REF][START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF], graph regularized least squares [START_REF] Belkin | Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples[END_REF], laplacian embedding [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF] and spectral clustering [START_REF] Luxburg | A tutorial on spectral clustering[END_REF].

The Laplacian of a graph can be expressed as the covariance matrix of the edge-vertex incidence vectors. Therefore, we can apply KORS and SQUEAK to approximate it. Unlike in the dictionary learning for H approximation problem, where KORS and SQUEAK greatly improve on existing methods, there are many efficient algorithms to approximate Laplacian matrices. In particular, the whole field of graph spectral sparsification is focused on this problem. Similarly to the case of RLS sampling in Euclidean spaces, these methods can exploit particular properties of the graph setting, and are comparable or more efficient than the methods we proposed. For example, if we specialize KORS to this setting we recover an equivalent algorithm from [START_REF] Cohen | Online Row Sampling[END_REF], and both [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] and [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] proposed algorithms similar to SQUEAK with a fully sequential merge tree. Nonetheless, looking in detail at the similarities between graph sparsification and learning an ε-accurate dictionary, will prepare us for the next section, where we show how graph sparsifiers can be applied to efficiently solve machine learning problems, and introduce new results for semi-supervised learning on graphs.

Graph spectral sparsification in the semi-streaming setting

We denote with G = (X , E) an undirected weighted graph with n vertices X and m edges E. Associated with each edge e i,j ∈ E there is a weight a e i,j (shortened a e ) measuring the "distance" between vertex i and vertex j.1 Given two graphs G and G over the same set of nodes X , we denote by G + G the graph obtained by summing the weights of the edges of G and G.

Given the weighted adjacency matrix A G and the degree matrix D G , the Laplacian of G is the PSD matrix defined as L G = D G -A G . Furthermore, we assume that G is connected and thus has only one eigenvalue equal to 0 and Ker(L G ) = 1. Let L + G be the pseudoinverse of L G , and

L -1/2 G = (L + G ) 1/2
. For any node i = 1, . . . , n, we denote with χ i ∈ R n the indicator vector so that b e = χ iχ j is the "edge" vector. If we denote with B G the m × n signed edge-vertex incidence matrix, then the Laplacian matrix can be written as

L G = e a e b e b T e = B T G E G B G , where E G is the m × m diagonal matrix with E G (e, e) = a e .
We indicate with Π = L G L + G the matrix of the orthogonal projection on the n-1 dimensional space orthogonal to the all one vector 1. Since the Laplacian of any connected graph G has a null space equal to 1, then Π is invariant w.r.t. the specific graph G on n vertices used to defined it. Alternatively, the projection matrix Π can be obtained as

Π = L + G L G = L -1/2 G L G L -1/2 G . Finally, let π e = √ a e L -1/2 G
b e , then we have Π = e π e π T e .

A graph H is a spectral sparsifier of G if the whole spectrum of the original graph is well approximated by using only a small portion of its edges. More formally,

Definition 4.2 A 1 ± ε spectral sparsifier of G is a graph H ⊆ G such that for all x ∈ R n (1 -ε)x T L G x ≤ x T L H x ≤ (1 + ε)x T L G x.
It is easy to see that this is equivalent to the definition of an ε-accurate dictionary I H w.r.t. the signed edge-vertex incidence matrix

B T G E 1/2 G ∈ R n×m . Then definition Definition 4.2 is just equivalent to Lemma 2.7 applied to L H = B T G E 1/2 G SS T E 1/2 G B G .
In the context of Laplacian sparsification, the leverage scores τ e = π e 2 are usually replaced by the effective resistances.

G B G L + G B T G ) = Tr(L G L + G ) = n -1.
The effective resistance r e = τ e /a e is defined as the leverage scores of edge e dived by its weight. It also has particular interpretations that are specific to the graph setting.

Intuitively, the effective resistance encodes the importance of an edge in preserving the minimum distance between two nodes. If an edge is the only connection between two parts of the graph, its r e is large. On the other hand, if there are multiple parallel paths across many edges to connect two nodes, the effective resistance of an edge between the two nodes will be small, similarly to actual resistances in parallel in an electrical network.

An important consequence of this intuition is that it makes it easy to see that adding edges to a graph can only reduce the effective resistance of other edges, because it can only introduce new alternative (parallel) paths in the graph. This is yet another reformulation of Lemma 3.8 on the evolution of LS and RLS. Also as expected, the weighted sum of the effective resistance is the rank of the Laplacian, and differently to our previous settings, it is always n -1.

Similarly to what [START_REF] Drineas | Relative-error CUR matrix decompositions[END_REF] did for LS sampling (i.e., Corollary 2.6), [START_REF] Spielman | Graph sparsification by effective resistances[END_REF] could prove that sampling the edges of G with replacement using a distribution proportional to their effective resistance (i.e., oracle Nyström sampling with LS) produces a spectral sparsifier H of size O(n log(n)/ε 2 ) with high probability, regardless of the number of edges in the original graph.

As in Chapter 2 and Chapter 3, the main issue of this approach is that we want to compute a sparsifier to avoid storing and processing the whole Laplacian, but we need to store and (pseudo-)invert the Laplacian to compute exact effective resistances to construct the sparsifier. But, as in the case of Nyström sampling in Euclidean spaces, there have been many approximate methods proposed over the years to approximate effective resistances. In particular, it is possible to estimate them using approaches unique to graphs, which fall mostly in two broad categories

• combinatorial methods, that exploit the interpretation of effective resistances as distances over parallel paths in the graphs and try to approximate this quantity, for example building ensembles of low-stretch trees [START_REF] Kapralov | Spectral sparsification via random spanners[END_REF][START_REF] Koutis | Simple parallel and distributed algorithms for spectral graph sparsification[END_REF], or recursive update rules (Kelner et al., 2013). • solver based methods, that exploit the fact that effective resistances can be expressed as the solution of a linear system, i.e., computing min

B T G E 1/2 x=be x 2 (Koutis et al., 2012),
We will focus on solver based methods, which are based on fast, graph-specific algorithms to solve Laplacian linear systems, such as the one proposed in [START_REF] Koutis | Improved spectral sparsification and numerical algorithms for SDD matrices[END_REF]. Note that this fast solvers also allowed [START_REF] Cohen | Online Row Sampling[END_REF] to directly adapt efficient leverage score estimation techniques for Euclidean spaces to the graph setting and obtain efficient methods. The details of these fast solvers are beyond the scope of this thesis, and we refer the reader to Spielman, (2017) for further reference. For our purposes, we just assume we have access to a fast method that can estimate the effective resistance of a graph. For the rest of the thesis we will focus on the one described by [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] that internally uses the Laplacian solver introduced by [START_REF] Koutis | Improved spectral sparsification and numerical algorithms for SDD matrices[END_REF]. [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] show that given as input a graph with m edges, we can compute an ρ-accurate estimate of the effective resistance of all in edges in the graph in O(m log(n)) time and O(m) space, or in other words in near-linear time and space in m.

As [START_REF] Spielman | Graph sparsification by effective resistances[END_REF] note, computing all effective resistances in near-linear time is already optimal, since we need anyway O(m) time to sample or even to just look at all edges. Conversely, the O(m) space requirement is much larger than the optimal O(n log(n)) sparsifier size, and is not feasible when m is large.

SQUEAK for graph Laplacian approximation

To reduce the space complexity of graph sparsification, [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] introduced a simple single-pass approach to generate a spectral sparsifier of a graph in the semistreaming setting, where edges are received one at a time. They store only an intermediate, approximate sparsifier and every time a new edge arrives, it is added to it. Whenever the sparsifier gets too large, they apply a resparsification algorithm to reduce its size, without compromising its spectral guarantees.

Although the algorithm is intuitive and simple to implement, the original proof presented in their paper is incomplete, as originally pointed out in [START_REF] Cohen | Online Row Sampling[END_REF]. In particular, [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] relies on a concentration inequality for independent random variables, while in the sparsification algorithm the probability of keeping edge e in the sparsifier at step s does depend on whether other edges e have been included in the sparsifier at previous iterations. This structure introduces subtle statistical dependencies through different iterations of the algorithm, and a more careful analysis is necessary. In this section, we will show that the analysis of SQUEAK can be directly applied to complete the analysis of [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF].

In addition to pointing out the problems with the original proof in [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF], Algorithm 11 Kelner and Levin [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] stream sparsification algorithm. Input: Graph G, weights a e for all edges in G. Output: H τ , a 1 ± ε sparsifier of G 1: Set space budget q = 40ρ log(3m/δ)/ε 2 2: Partition the graph (dataset) G into k = m/q edge-disjoint sub-graphs (sub-datasets)

{G s } k s=0 such that G = k s=0 G s 3: Initialize the sparsifier (dictionary) H 0 = ∅ 4: for all s ∈ [1, . . . , k] do 5: Receive G s 6: Set H * = H s-1 + G s 7:
Compute ρ-accurate estimates of all effective resistances r s,e in H * using a fast SDD solver (Kelner and Levin, 2013, Theorem 3) 8:

Compute new probabilities p s,e = a e r s,e

9:

for all edges e ∈ H s-1 do for all edges e ∈ H s-1 do 14:

Set q s,e ∼ B( p s,e / p s-1,e , q s-1,e ) 15:

If q s,e = 0 add edge e to H s with weight ae ps,e qs,e q 16:

If q s,e = 0 discard edge from memory{ Shrink } 17:

end for 18:

for all edges e ∈ G s do 19:

Set q s,e ∼ B( p s,e , q) 20:

If q s,e = 0 add edge e to H s with weight ae ps,e qs,e q { Expand } 21:

end for

22:

Return H k 23: end for [START_REF] Cohen | Online Row Sampling[END_REF] introduces a new algorithm to construct a sparsifier in a semi-streaming setting but, differently from the original algorithm in [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF], interactions between iterations are avoided because the algorithm proposed in [START_REF] Cohen | Online Row Sampling[END_REF] never drops an edge once it is introduced in the sparsifier. In particular, [START_REF] Cohen | Online Row Sampling[END_REF] is equivalent to specializing KORS to the problem of graph sparsification, and therefore only able to generate a suboptimal O(n log 2 (n)) number of edges (Theorem 3.11).

If we specialize SQUEAK to the specific setting of effective resistance sampling and graph sparsification, and use a fully sequential tree, we recover exactly the original algorithm of [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF], reported in Algorithm 11. Using the analysis of SQUEAK, we can also prove that w.h.p. it generates an ε-accurate dictionary, or in other words a graph sparsifier.

Theorem 4.7 Given parameters 0 < ε ≤ 1, 0 < δ < 1, an arbitrary graph G, and a fully sequential merge tree of height m/q , let ρ = 1+3ε 1-ε and run Algorithm 11 with budget q ≥ 26ρ log(3m/δ)/ε 2 . Then w.p. 1δ, Accuracy: each graph sparsifier H s is ε-accurate w.r.t. s t=1 G t , and each r s,e is ρaccurate w.r.t. r s,e .

Space: the size of each sparsifier is bounded by |H s | ≤ 3qd s eff (0) ≤ 3qn.

Time: the algorithm runs in O(n log 2 (n)) per-merge time, and O(m log(n)) overall time.

Passes: the algorithm requires a single pass over G.

Unlike in the analysis of SQUEAK, here we have fixed the merge tree (fully sequential) and we have a closed form solution for the effective dimension d eff (0) = Rank(L G ) = n -1, and therefore a O(n log(n)) bound on the size of the dictionary and the memory requirement of the algorithm.

To compute the time complexity, we combine the runtime analysis of SQUEAK, the number of merges m/q , the O(n log(n)) sparsifier (dictionary) size bound, and the

O(|I s | log(n)) ≤ O(n log 2 (n))
cost of estimating the effective resistances (LS) using a near-linear time solver. The final result is a closed-form O(m log(n)) bound on the runtime of the algorithm.

Compared to [START_REF] Kelner | Spectral Sparsification in the Semi-streaming Setting[END_REF] this is a log(n) factor improvement in space and runtime, not due to any change in the algorithm, but simply because of a tighter analysis. We can also compare our result with the independent analysis of [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] of sequential graph resparsification, which also resolves the problems in Kelner and Levin, (2013)'s analysis. [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] can be seen as a special case of our sequential RLS sampling with removal (i.e., SQUEAK), and both works take a similar analysis approach based on the result of [START_REF] Pachocki | Analysis of Resparsification[END_REF]. As the main tool for the proof, [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] uses an adversarial game argument to bound the variance of the process W t (see Section 3.4.4). For SQUEAK's proof instead we derived an argument based on stochastic dominance that essentially tells us how the adversary of [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] can be constructed. Moreover, since they focus on the graph setting and effective resistances (LS), they do not generalize their result to RLS, or have the need to derive an estimator that can work both in Euclidean and RKH spaces (i.e., Lemma 3.14), relying instead on fast solvers. Conversely, when [START_REF] Kyng | A framework for analyzing resparsification algorithms[END_REF] move from sequential to parallel graph sparsification, they take an approach completely different from the one we used for SQUEAK. Although the resulting algorithm is only applicable to graphs, since it is based on combinatorial arguments, it can extract more parallelism than SQUEAK and construct a graph sparsifier in a smaller time.

Efficient graph semi-supervised learning

We can now show how using graph sparsification, in particular fully sequential SQUEAK, will allow us to greatly reduce the computational cost of solving semi supervised learning problems.

In many classification and regression tasks, obtaining labels for large datasets is expensive.

When the number of labeled samples is too small, traditional supervised learning algorithms fail in learning accurate predictors. Semi-supervised learning (SSL) [START_REF] Chapelle | Semi-Supervised Learning[END_REF][START_REF] Zhu | Semi-Supervised Learning Literature Survey[END_REF] effectively deals with this problem by integrating the labeled samples with an additional set of unlabeled samples, which are abundant and readily available in many applications (e.g., set of images collected on web sites [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF]).

The intuition behind SSL is that unlabeled data may reveal the underlying structure of the problem (e.g., a manifold) that could be exploited to compensate for the limited number of labels and improve the prediction accuracy. Among different SSL settings, in this thesis we focus on the case where data are embedded in a graph.

The graph is expected to effectively represent the geometry of data and graph-based SSL [START_REF] Belkin | Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples[END_REF][START_REF] Subramanya | Graph-Based Semi-Supervised Learning[END_REF][START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF] methods leverage the intuition that nodes that are similar according to the graph are more likely to be labeled similarly. A popular approach is the harmonic function solution (HFS) [START_REF] Belkin | Regularization and Semi-Supervised Learning on Large Graphs[END_REF][START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF][START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF], whose objective is to find a solution where each node's value is the weighted average of its neighbors. Computing the HFS solution requires solving a Laplacian regularized least-squares problem. While the resulting solution is both empirically effective [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF] and enjoys strong performance guarantees [START_REF] Belkin | Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples[END_REF][START_REF] Cortes | Stability of transductive regression algorithms[END_REF], solving exactly the least-squares problem on a graph with n nodes amounts to O(n 3 ) time and O(n 2 ) space complexity.

Using a general iterative solver on a graph with m edges to obtain a comparable solution requires only O(mn) time and O(m) space, but this is still practically unfeasible in many applications of interest. In fact, many graphs have naturally a large number of edges, so that even if the n nodes could fit in memory, storing m edges largely exceeds the memory capacity. For instance, Facebook's graph of relationships [START_REF] Ching | One trillion edges: graph processing at Facebook-scale[END_REF] counts about n = 1.39e9 users connected by a trillion (m = 1e12) edges. While n is still in the order of a normal memory capacity, the edges cannot be stored in a single computer. A similar issue is faced when the graph is built starting from a dataset, for instance using a k-nn graph. In this case m = kn edges are created, and sometimes a large k is necessary to obtain good performance [START_REF] Saluja | Graph-based semi-supervised learning of translation models from monolingual data[END_REF], or artificially adding neighbours can improve the stability of the method [START_REF] Gleich | Using Local Spectral Methods to Robustify Graph-Based Learning Algorithms[END_REF].

In such problems, a direct application of HFS is not possible and thus some form of approximation or graph sketching is required. A straightforward approach is to distribute the graph over multiple machines in a cluster and resort to an iterative solver for the solution of the least-squares problem [START_REF] Ching | One trillion edges: graph processing at Facebook-scale[END_REF]. Distributed algorithms require infrastructure and careful engineering in order to deal with communication issues [START_REF] Cai | A comparison of platforms for implementing and running very large scale machine learning algorithms[END_REF]. But even assuming that these problems are satisfactorily dealt with, all known iterative solvers that have provably fast convergence do not have known distributed implementations, as they assume random, constant time access to all the edges in the graph. Thus one would have to resort to distributed implementations of simpler but much slower methods, in effect trading-off space for a significant reduction in overall efficiency.

More principled methods try to address the memory bottleneck by directly manipulating the structure of the graph to reduce its size. These include subsampling the nodes of the original graph, quantization, approaches related to manifold learning, and various approximation strategies. The most straightforward way to reduce the complexity in graph-based method is to subsample the nodes to create a smaller, backbone graph of representative vertices, or landmarks [START_REF] Talwalkar | Large-Scale Manifold Learning[END_REF]. Nyström sampling methods [START_REF] Kumar | Sampling Methods for the Nyström Method[END_REF] randomly select s nodes from the original graph and compute q eigenvectors of the smaller graph, which can be later used to solve the HFS regularized problem. It can be shown [START_REF] Kumar | Sampling Methods for the Nyström Method[END_REF] that the reconstructed Laplacian is accurate in 2 -norm and thus only its largest eigenvalue is preserved. Unfortunately, the HFS solution does not depend only on the largest eigenvalues, both because the largest eigenvectors are the ones most penalized by HFS's regularizer ( [START_REF] Belkin | Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples[END_REF]) and because theoretical analysis shows that preserving the smallest eigenvalue is important for generalization bounds ( [START_REF] Belkin | Regularization and Semi-Supervised Learning on Large Graphs[END_REF]). As a result, subsampling methods can completely fail when the sampled nodes compromise the spectral structure of the graph [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF]. Although alternative techniques have been developed over years (see e.g., [START_REF] Garcke | Semi-supervised learning with sparse grids[END_REF][START_REF] Jebara | Graph construction and b-matching for semi-supervised learning[END_REF][START_REF] Liu | Large graph construction for scalable semisupervised learning[END_REF][START_REF] Tsang | Large-Scale Sparsified Manifold Regularization[END_REF][START_REF] Yu | Blockwise supervised inference on large graphs[END_REF][START_REF] Zhu | Harmonic mixtures: combining mixture models and graph-based methods for inductive and scalable semi-supervised learning[END_REF]), this drawback is common to all backbone graph methods. Motivated by this observation, other approaches focus on computing a more accurate approximation of the spectrum of the Laplacian. Fergus et al. [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF] build on the observation that when the number of unlabeled samples n tends to infinity, the eigenvectors of the Laplacian tend to the eigenfunctions of the sampling distribution P. Thus instead of approximating eigenvectors in R n , they first compute empirical eigenfunctions of the estimated sampling distribution (defined on the d-dimensional feature space) obtained by assuming that P is factorized and by using a histogram estimation over b bins over each dimension separately. While the method scales to the order of million nodes, it still requires d and b to be small to be efficient. Furthermore, no theoretical analysis is available, and the method may return poor approximations whenever the sampling distribution is not factorized.

Motivated by the empirical success of [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF], Ji et al. [START_REF] Ji | A Simple Algorithm for Semi-supervised Learning with Improved Generalization Error Bound[END_REF] proposed a similar algorithm, Simple-HFS, for which they provide theoretical guarantees. However, in order to prove bounds on the generalization error, they need to assume several strong and hard to verify assumptions, such as a sufficiently large eigengap. On the contrary, the guarantees for our method work for any graph.

Our contribution

In this section, we focus on reducing the space complexity of graph-based SSL while matching the smallest possible computational complexity of Ω(m) up to logarithmic factors2 and providing strong guarantees about the quality of the solution. In particular, we introduce a novel approach which employs efficient spectral graph sparsification techniques to incrementally process the original graph. This method, coupled with dedicated solvers for symmetric diagonally dominant (SDD) systems [START_REF] Koutis | A Nearly-m log n Time Solver for SDD Linear Systems[END_REF], allows to find an approximate HFS solution without storing the whole graph in memory and to control the computational complexity as n grows.

In fact, we show that our proposed method, called Sparse-HFS, requires only fixed O(n log(n)) space to run, and allows to compute solutions to large HFS problems in memory. For example, in the experimental section we show that the sparsifier can achieve an accuracy comparable to the full graph, using one order of magnitude less edges. With a careful choice of the frequency of resparsification, the proposed method does not increase significantly the running time. Given a minimum amortized cost of Ω(1) per edge, necessary to examine each edge at least once, our algorithm only increases this cost to O(log(n)). Furthermore, using the approximation properties of spectral sparsifiers and results from algorithmic stability theory [START_REF] Bousquet | Stability and generalization[END_REF][START_REF] Cortes | Stability of transductive regression algorithms[END_REF] we provide theoretical guarantees for the generalization error for Sparse-HFS, showing that the performance is asymptotically the same as the exact solution of HFS.

Finally, we report empirical results on both synthetic and real data showing that Sparse-HFS is competitive with subsampling and the EigFun method in [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF].

Semi-supervised learning with graph sparsifiers

Notation. We consider the problem of regression in the semi-supervised setting, where a large set of n points X = (x 1 , . . . , x n ) ⊂ R d is drawn from a distribution P and labels {y i } l i=1 are provided only for a small (random) subset S ⊂ X of l points. Graph-based SSL builds on the observation that P is often far from being uniform and it may display a specific structure that could be exploited to "propagate" the labels to similar unlabeled points. Building on this intuition, graph-based SSL algorithms consider the case when the points in X are embedded into an undirected weighted graph G = (X , E) with |E| = m edges. Associated with each edge e i,j ∈ E there is a weight a e i,j measuring the "distance" between x i and x j .

A graph-based SSL algorithm receives as input G and the labels of the nodes in S and it returns a function f : X → R that predicts the label for all nodes in X . The objective is to minimize the prediction error over the set T of u = nl unlabeled nodes. In the following we denote by y ∈ R n the full vector of labels.

Stable-HFS. HFS directly exploits the structure embedded in G to learn functions that are smooth over the graph, thus predicting similar labels for similar nodes. We assume that G is connected and thus has only one eigenvalue at 0. Let L + G be the pseudoinverse of L G , and

L -1/2 G = (L + G ) 1/2
. The HFS method [START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF] can be formulated as the Laplacian-regularized least-squares problem

f = arg min f ∈R n 1 l (f -y) T I S (f -y) + ηf T L G f , 4.7
where I S ∈ R n×n is the identity matrix with zeros corresponding to the nodes not in S and η is a regularization parameter. The solution can be computed in closed form as f = (ηlL G + I S ) + y S , where y S = I S y ∈ R n .

The singularity of the Laplacian may lead to unstable behavior with drastically different results for small perturbations to the dataset. For this reason, we focus on the Stable-HFS algorithm proposed in [START_REF] Belkin | Regularization and Semi-Supervised Learning on Large Graphs[END_REF] where an additional regularization term is introduced to restrict the space of admissible hypotheses to the space F = {f : f , 1 = 0} of functions orthogonal to null space of L G (i.e., centered functions). This restriction can be easily enforced by introducing an additional regularization term µ l f 1 in Eq. 4.7. As shown in [START_REF] Belkin | Regularization and Semi-Supervised Learning on Large Graphs[END_REF], in order to guarantee that the resulting f actually belongs to F, it is sufficient to set µ = ((ηlL G + I S ) + y S ) T 1/((ηlL G + I S ) + 1) T 1, and compute the solution as f = (ηlL G + I S ) + (y S -µ1). Furthermore, it can be shown that if we center the vector of labels y S = y Sy S , with y = 1 l y T S 1, then the solution of Stable-HFS can be rewritten as projected solution

f = (ηlL G + I S ) + ( y S -µ1) = Π F (ηlL G + I S ) + y S , where Π F = L G L +
G is the projection matrix on the n -1 dimensional space F. Indeed, since the Laplacian of any graph G has a null space equal to the one vector 1, then Π F is invariant w.r.t. the specific graph G used to defined it. While Stable-HFS is more stable and thus more suited for theoretical analysis, its time and space requirements remain O(mn) and O(m), and cannot be applied to graph with a large number of edges.

To accelerate Stable-HFS, we introduce a novel variant of HFS, called Sparse-HFS, where the graph G is pre-processed using spectral graph sparsification techniques, and the Stable-HFS solution is computed efficiently on a smaller subgraph H, drastically reducing the time and memory requirements without compromising the resulting accuracy. subset of labeled points S) the resulting function f satisfies,

R( f ) ≤ R( f ) + β + 2β + c 2 (l + u) lu π(l, u) ln 1 δ 2 + εlηλ 2 (G)M ((1 -ε)lηλ 2 (G)-1) 2 2 , 4.8
where f is the solution of exact Stable-HFS on G, π(l, u) = lu l + u -0.5 2 max{l, u} 2 max{l, u} -1 , and

β ≤ 1.5M √ l ((1 -ε)lηλ 2 (G) -1) 2 + 4M (1 -ε)lηλ 2 (G) -1 .
Theorem 4.8 shows how approximating G with H impacts the generalization error as the number of labeled samples l increases. If we set ε = 0, we recover the exact case bound by [START_REF] Cortes | Stability of transductive regression algorithms[END_REF], which depends only on R( f ) and β.

When ε > 0, we see from Eq. 4.8 that the two terms already present in the exact case are either unchanged ( R( f )) or increase only by a constant factor (β). Because of the approximation, a new error term (the last one in Eq. 4.8) is added to the bound, but we can see that it is negligible compared to β. In fact, it converges to zero as O(ε 2 /l 2 (1ε) 4 ) as l grows and it is dominated by β for any constant value of ε. This means that increasing ε corresponds to a constant increase in the bound, regardless of the size of the problem. Consequently, ε can be freely chosen to trade off accuracy and space complexity (Theorem 4.7) depending on the problem constraints. Furthermore running time does not depend on this trade-off, because a larger block size is balanced by less frequent resparsifications.

Finally, because the eigenvalues present in the bound are the ones of the original graph, any external knowledge on the spectral properties of the input graph can be easily included in the analysis. Therefore it is straightforward to provide stronger guarantees for Sparse-HFS when combined with assumptions on the graph generating model. We also remark the level of generality of this result that holds for the integration between HFS and any ε-accurate spectral sparsification method.

Proof of Theorem 4.8.

Step 1 (generalization of stable algorithms). Let β be the stability of Sparse-HFS, then using the result in [START_REF] Cortes | Stability of transductive regression algorithms[END_REF], we have that with probability at least 1δ (w.r.t. the randomness of the labeled set S) the solution f returned by the Sparse-HFS satisfies

R( f ) ≤ R( f ) + β + 2β + c 2 (l + u) lu π(l, u) log(1/δ) 2 .
In order to obtain the final result we first derive an upper bound on the stability of Sparse-HFS and then relate its empirical error to the one of Stable-HFS.

Step 2 (stability). The bound on the stability follows similar steps as in the analysis of Stable-HFS in [START_REF] Belkin | Regularization and Semi-Supervised Learning on Large Graphs[END_REF] integrated with the properties of streaming spectral sparsifiers reported in Definition 4.2 and Proposition 2.1. Let S and S be two labeled sets only differing by one element and f and f be the solutions obtained by running Sparse-HFS on S and S respectively. Without loss of generality, we assume that I S (l, l) = 1 and I S (l + 1, l + 1) = 0, and the opposite for I S . The original proof in [START_REF] Cortes | Stability of transductive regression algorithms[END_REF] showed that the stability β can be bounded as β ≤ ff . In the following we show that the difference between the solutions f and f , and thus the stability of the algorithm, is strictly related to eigenvalues of the sparse graph H. Let A = Π F (lηL H + I S ) and B = Π F (lηL H + I S ), we remind that if the labels are centered, the solutions of Sparse-HFS can be conveniently written as f = A -1 y S and f = B -1 y S . As a result, the difference between the solutions can be written as

f -f = A -1 y S -B -1 y S ≤ A -1 ( y S -y S ) + A -1 y S -B -1 y S .
4.9

Let us consider any vector f ∈ F, since the null space of a Laplacian L H is the one vector 1 and

Π F = L H L + H , then Π F f = f . Thus we have Π F (lηL H + I S )f (1) ≥ Π F lηL H f -Π F I S f (2) ≥ Π F lηL H f -f (3) ≥ (lηλ 2 (H)-1) f 4.10
where (1) follows from the triangle inequality and (2) follows from the fact that Π F I S f ≤ f since the largest eigenvalue of the project matrix Π F is one and the norm of f restricted on S is smaller than the norm of f . Finally (3) follows from the fact that

Π F L H f = L H L + H L H f = L H f and since f is orthogonal to the null space of L H then L H f ≥ λ 2 (H) f
, where λ 2 (H) is the smallest non-zero eigenvalue of L H . At this point we can exploit the spectral guarantees of the sparsified graph L H and from the definition of graph sparsifier, we have that λ 2 (H) ≥ (1ε)λ 2 (G). As a result, we have an upper-bound on the spectral radius of the inverse operator (Π F (lηL H + I S )) -1 and thus

A -1 (y S -y S ) ≤ 4M lη(1 -ε)λ 2 (G) -1 ,
where the first step follows from Eq. 4.10 since both y S and y S are centered and thus (y Sy S ) ∈ F, and the second step is obtained by bounding y Sy S ≤ y Sy S + y Sy S ≤ 4M . The second term in Eq. 4.9 can be bounded as

A -1 y S -B -1 y S = B -1 (B -A)A -1 y S = B -1 Π F (I S -I S )A -1 y S ≤ 1.5M √ l (lη(1 -ε)λ 2 (G) -1) 2 ,
where we used y S ≤ y S + y S ≤ 2M √ l, Π F (I S -I S ) ≤ √ 2 < 1.5 and we applied Eq. 4.10 twice. Putting it all together we obtain the stated bound.

Step 3 (empirical error). The other element effected by the sparsification is the empirical error R( f ). We first recall that

Π F = L + G L G = L -1/2 G L G L -1/2 G
(and equivalently with G replaced by H) and we introduce

Π F = L -1/2 G L H L -1/2 G Let A = Π F (lηL H + I S ), A = Π F (lηL G + I S ), then rewrite the empirical error as R( f ) = 1 l I S f -I S f + I S f -y S 2 ≤ 1 l I S f -y S 2 + 1 l I S f -I S f 2 ≤ R( f ) + 1 l I S ( A -1 -A -1 ) y S 2 ≤ R( f ) + 1 l A -1 ( A -A) A -1 y S 2 = R( f ) + l 2 η 2 l A -1 (Π F (L G -L H )) A -1 y S 2 = R( f ) + lη 2 A -1 (Π F (L G -L H )Π F ) A -1 y S
Where in the last passage we use L G Π F = L G and L H Π F = L H . To bound the second term we derive

A -1 Π F (L G -L H )Π F A -1 y S 2 (1) = A -1 L 1/2 G L -1/2 G (L G -L H )L -1/2 G L 1/2 G A -1 y S 2 (2) = A -1 L 1/2 G Π F L -1/2 G (L G -L H )L -1/2 G Π F L 1/2 G A -1 y S 2 (3) = A -1 L 1/2 G Π F (Π F -Π F )Π F L 1/2 G A -1 y S 2 ≤ A -1 L 1/2 G Π F 2 Π F -Π F 2 Π F L 1/2 G A -1 2 y S 2 (4) ≤ lM 2 ε 2 A -1 L 1/2 G Π F 2 A -1 L 1/2 G Π F 2
where in (1) and (2) we use the definition of Π F , in (3) we use the definition of Π F , while in (4) we use the fact that Definition 4.

2 implies that (1 -ε)Π F Π F (1 + ε)Π F and thus the largest eigenvalue of Π F -Π F is ε 2 . We need now to bound A -1 L 1/2 G Π F 2 . Similarly
to Eq. 4.10 finding a lower bound on AL

-1/2 G Π F
x for all x is equivalent to find a lower bound for all f ∈ F to

Π F (lηL H + I S )L -1/2 G f ≥ Π F lηL H L -1/2 G f -Π F I S L -1/2 G f ≥ Π F lηL H L -1/2 G f -L -1/2 G f ≥ lη λ 2 (H) λ 2 (G) - 1 λ 2 (G) f ≥ 1 λ 2 (G) (lηλ 2 (H)-1) f ≥ 1 λ 2 (G) (lη(1 -ε)λ 2 (G)-1) f
Similarly, we can show that

Π F (lηL G + I S )L -1/2 G f ≥ Π F lηL G L -1/2 G f -Π F I S L -1/2 G f ≥ Π F lηL G L -1/2 G f -L -1/2 G f ≥ lη λ 2 (G) λ 2 (G) - 1 λ 2 (G) f ≥ 1 λ 2 (G) (lηλ 2 (G)-1) f ≥ 1 λ 2 (G) (lη(1 -ε)λ 2 (G)-1) f
Taking this and putting all together shows

R( f ) ≤ R( f ) + l 2 η 2 λ 2 (G) 2 ε 2 M 2 (lη(1 -ε)λ 2 (G)-1) 4
Combining the three steps above concludes the proof.

Experiments on spam detection

In this section we evaluate the empirical accuracy of Sparse-HFS compared to other baselines for large-scale SSL on both synthetic and real datasets.

Synthetic data. The objective of this first experiment is to show that the sparsification method is effective in reducing the number of edges in the graph and that preserving the full spectrum of G retains the accuracy of the exact HFS solution. We evaluate the algorithms on the R 2 data distributed as in Fig. 4.1(a), which is designed so that a large number of Chapter 4. Sequential RLS Sampling in the Batch Setting Guarant.

Space Preprocessing Time Solving Time neighbours is needed to achieve a good accuracy. The dataset is composed of n = 12100 points, where the two upper clusters belong to one class and the two lower to the other. We build an unweighted, k-nn graph G for k = 100, . . . , 12000. After constructing the graph, we randomly select two points from the uppermost and two from lowermost cluster as our labeled set S. We then run Sparse-HFS with ε = 0.8 to compute H and f , and run (exact) Stable-HFS on G to compute f , both with γ = 1. Fig. 4.1(b) reports the accuracy of the two algorithms. Both algorithms fail to recover a good solution until k ≈ 4000. This is due to the fact that until a certain threshold, each cluster remains separated and the labels cannot propagate. Beyond this threshold, Stable-HFS is very accurate, while, as k increases again, the graph becomes almost full, masking the actual structure of the data and thus loosing performance again. We notice that the accuracy of Stable-HFS and Sparse-HFS is never significantly different, and, quite importantly, they match around the value of k = 4500 that provides the best performance. This is in line with the theoretical analysis that shows that the contribution due to the sparsification error has the same order of magnitude as the other elements in the bound. Furthermore, in Fig. 4.1(c) we report the ratio of the number of edges in the sparsifier H and G. This quantity is always smaller than one and it constantly decreases since the number of edges in H is constant, while the size G increases linearly with the number of neighbors (i.e., |H|/|G| = O(1/k)). We notice that for the optimal k the sparsifier contains less than 10% of the edges of the original graph but it achieves almost the same accuracy.

Sparse-HFS N = O(n log(n)) O(m log 2 (n)) O(n log 2 (n)) Stable-HFS O(m) O(m) O(mn) Simple-HFS O(m) O(mq) O(q 4 ) EigFun O(nd + nq + b 2 ) O(qb 3 + db 3 ) O(q 3 + nq) SubSampling O(sk) O(m) O(s 2 k + n)
Spam-filtering dataset. We now evaluate the performance of our algorithm on the TREC 2007 Public Spam Corpus 3 , that contains n = 75419 raw emails labeled as either SPAM or HAM. The emails are provided as raw text and we applied standard NLP techniques to extract features vectors from it. In particular, we computed TF-IDF scores for each of the emails, with some additional cleaning in the form of a stop word list, simple stemming and dropping the 1% most common and most rare words. We ended up with d = 68697 features, each representing a word present in some of the emails. From these features we proceeded to build a graph G where given two emails x i , x j , the weight is computed as a ij = exp(x ix j /2σ 2 ), with σ 2 = 3. We consider the transductive setting, where the graph is fixed and known, but only a small random subset of l = {20, 100, 1000} labels is revealed to the algorithm. As a performance measure, we consider the prediction accuracy over the whole dataset. We compare our method to several baselines. The most basic supervised baseline is 1-NN, which connects each node to the closest labeled node. The SubSampling algorithm selects uniformly s nodes out of n, computes the HFS solution on the induced subgraph of G and assigns to each node outside of the subset the same label as the closest node in the subset. SubSampling's complexity depends on the size s of the subgraph and the number k of neighbors retained when building the k-nn subgraph. The eigenfunction (EigFun) algorithm [START_REF] Fergus | Semi-Supervised Learning in Gigantic Image Collections[END_REF] tries to sidestep the computational complexity of finding an HFS solution on G, by directly approximating the distribution that created the graph. Starting directly from the samples, each of the d feature's density is separately approximated using histograms with b bins. From the histograms, q empirical eigenfunctions (vectors in R n ) are extracted and used to compute the final solution. We did not include Stable-HFS and Simple-HFS in the comparison because their O(m) space complexity made them unfeasible for this dataset. In Fig. 4.2, we report the accuracy of each method against the time and space complexity, where each separate point corresponds to a different choice in metaparameters (e.g. k, q, s). For EigFun, we use the same b = 50 as in the original implementation, but we varied q from 10 to 2000. For SubSampling, s = 15000 and k varies from 100 to 10000. We run Sparse-HFS on G setting ε = 0.9, and changing the size m of the input graph by changing the number of neighbours k from 1000 to 7500. Since the actual running time and memory occupation are highly dependent on the implementation (e.g., EigFun is implemented in Matlab, while Sparse-HFS is Matlab/C), the complexities are computed using their theoretical form (e.g., O(m log(n)) for Sparse-HFS) with the values actually used in the experiment (e.g., m = nk for a k-nn graph). All the complexities are reported in Table 4.1. The only exception is the number of edges in the sparsifier |H| used in the space complexity of Sparse-HFS. Since this is a random quantity that holds only w.h.p. and that is independent from implementation details, we measured it empirically and used it for the complexities. For all methods we notice that the performance increases as the space complexity gets larger, until a peak is reached, while additional space induces the algorithms to overfit and reduces accuracy. For EigFun this means that a large number of eigenfunctions is necessary to accurately model the high dimensional distribution. And as theory predicts, SubSampling's uniform sampling is not efficient to approximate the graph spectra, and a large subset of the nodes is required for good performance. Sparse-HFS's accuracy also increases as the input graph gets richer, but unlike the other methods the space complexity does not change much. This is because the sparsifier is oblivious to the structure of the graph, and even if Sparse-HFS reaches its optimum performance for k = 3000, the sparsifier contains roughly the same number of edges present as k = 1000, and only 5% of the edges present in the input graph. Although preliminary, this experiment shows that the theoretical properties of Sparse-HFS translate into an effective practical algorithm which is competitive with state-of-the-art methods for large-scale SSL.

Recap and open questions: dynamic sparsifiers

We introduced Sparse-HFS, an algorithm that combines sparsification methods and efficient solvers for SDD systems to find approximate HFS solutions using only O(n log 2 (n)) space instead of O(m). Furthermore, we show that the O(m log(n)) time complexity of the methods is only a polylog term away from the smallest possible complexity Ω(m). Finally, we provide a bound on the generalization error that shows that the sparsification does not affect the asymptotic convergence rate of HFS. As such, the accuracy parameter ε can be freely chosen to meet the desired trade-off between accuracy and space complexity.

An interesting feature of Sparse-HFS is that it could be easily employed in online learning problems where edges arrive in a stream and intermediate solutions have to be computed over time. Since Sparse-HFS has a O(log(n)) amortized time per edge, it could compute intermediate solutions every q edges without compromising its overall time complexity.

The fully dynamic setting, where edges can be both inserted and removed, is an important extension where our approach could be further investigated, especially because it has been observed in several domains that graphs become denser as they evolve over time [START_REF] Leskovec | Graphs over time: densification laws, shrinking diameters and possible explanations[END_REF]. While sparsifiers have been developed for this setting (see e.g., [START_REF] Kapralov | Single pass spectral sparsification in dynamic streams[END_REF]), current solutions would require O(n 2 polylog(n)) time to construct the sparsifier, thus making it unfeasible to repeat this computation many times over the stream. Extending sparsification techniques to the fully dynamic setting in a computationally efficient manner is currently an open problem.

Sequential RLS Sampling in the Online Setting

Online learning (OL) represents a family of efficient and scalable learning algorithms for building a predictive model incrementally from a sequence of T data points.

A popular online learning approach, introduced in [START_REF] Zinkevich | Online Convex Programming and Generalized Infinitesimal Gradient Ascent[END_REF], is to learn a linear predictor using gradient descent (GD) in the input space R d . Since we can explicitly store and update the d weights of the linear predictor, the total runtime of this algorithm is O(T d), allowing it to scale to large problems. Unfortunately, it is sometimes the case that no good predictor can be constructed starting from only the linear combination of the input features.

For this reason, online kernel learning (OKL) (Kivinen et al., 2004a) first maps the points into a high-dimensional reproducing kernel Hilbert space (RKHS) using a non-linear feature map ϕ, and then runs GD on the projected points, which is often referred to as functional GD (FGD) (Kivinen et al., 2004b). With the kernel approach, each gradient step does not update a fixed set of weights, but instead introduces the feature-mapped point in the predictor as a support vector (SV). The resulting kernel-based predictor is flexible and data adaptive, but the number of parameters, and therefore the per-step space and time cost, now scales with O(t), the number of SVs included after t steps of GD. This curse of kernelization, results in an O(T 2 ) total runtime, and prevents standard OKL methods from scaling to large problems.

When judging an online learning algorithm, we are not only interested in its computational complexity, but also in its prediction accuracy. Given a function space containing functions with very small prediction loss, the objective of an online learning algorithm is to approach over time the performance of the best predictor in the space and thus minimize the regret, that is the difference in cumulative loss between the OL algorithm and the best predictor in hindsight.

Over time, many methods have been proposed to trade-off minimizing the regret of the online learning algorithm, and the computational cost required to compute the predictions.

Gradient descent. For OL [START_REF] Zinkevich | Online Convex Programming and Generalized Infinitesimal Gradient Ascent[END_REF] showed that simple gradient descent (GD), combined with a smart choice for the stepsize η t of the gradient updates, achieves a O( √ dT ) regret with a O(d) space and time cost per iteration. When the only assumption on the losses is simple convexity, this upper bound matches a corresponding lower bound [START_REF] Luo | Efficient second-order online learning via sketching[END_REF], thus making first-order methods (e.g., GD) essentially unimprovable in a minimax sense. However, if we know that the losses are strongly convex, setting a more aggressive step-size achieves a smaller O(log(T )) regret [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF][START_REF] Zhu | Online Gradient Descent in Function Space[END_REF]. Unfortunately, most common losses, such as the squared loss, are not strongly convex when evaluated for a single point x t . Nonetheless, they possess a certain directional curvature and [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF] show that online Newton step (ONS), an adaptive method that exploits second-order (second derivative) information on the losses, can achieve a logarithmic regret O(d log T ) without strong convexity along all directions. The downside of this adaptive method is the larger O(d 2 ) space and per-step time complexity, since second-order updates require to construct, store, and invert H t , a preconditioner matrix related to the Hessian of the losses used to correct the first-order updates.

Kernel gradient descent. OKL is fundamentally harder than OL, due to 1) the fact that an infinite parametrization makes regret bounds scaling with the dimension d meaningless and 2) the size of the model, and therefore time and space complexities, scales with t itself, making these methods even less performant than OL algorithms. Kernel extensions of first-order OL algorithms have been proposed for OKL, such as functional GD (e.g., NORMA, (Kivinen et al., 2004a)) which achieves a O( √ T ) regret with a O(t) space and time cost per iteration. For second-order methods, the Second-Order Perceptron [START_REF] Cesa-Bianchi | A second-order perceptron algorithm[END_REF] or NAROW [START_REF] Orabona | New adaptive algorithms for online classification[END_REF] for generic curved losses and Recursive Kernel Least Squares [START_REF] Zhdanov | An Identity for Kernel Ridge Regression[END_REF] or Kernel AAR [START_REF] Gammerman | On-line prediction with kernels and the complexity approximation principle[END_REF] for the specific case of 2 losses provide bounds that scale with the log-determinant of the kernel-matrix. As we showed in Lemma 3.9 this quantity can be upper bounded using the effective dimension d T eff (γ) of the of the points x t , and scales as O(d T eff (γ) log(T )), playing a similar role as the O(d log T ) bound from OL.

Approximate GD. To trade off between computational complexity smaller than O(d 2 ) and improved regret (close to O(d log T )), several OL methods try approximate second-order updates, replacing H t with an approximate H t that can be efficiently stored and inverted. AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] and ADAM [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] reweight the gradient updates on a per-coordinate basis using a diagonal H t , but these methods ultimately only improve the regret dependency on d and leave the √ T component unchanged. Sketched-ONS, by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF], uses matrix sketching to approximate H t with a r-rank sketch H t , that can be efficiently stored and updated in O(dr 2 ) time and space, close to the O(d) complexity of diagonal approximations. More importantly, Sketched-ONS achieves a much smaller regret compared to diagonal approximations: When the true H t is of low-rank r, it recovers a O(r log T ) regret bound logarithmic in T . Unfortunately, when the true H t is not low-rank, a new term appears in the bound, due to the sketch approximation, that scales with the spectrum of H t and in some cases can grow much larger than O(log T ).

Approximate kernel GD. Many approximation methods have been proposed to make first-order OKL methods scale to large datasets. They all try to reduce the O(t) per-step complexity to a constant independent from t. Approximate methods usually take one of two approaches:

• budgeted methods perform approximate gradient updates in the true RKHS, where instead of updating the SVs' weights in our predictor using the true gradient direction we use an update that keeps most weight at zero, • functional approximation methods perform exact gradient updates in an approximate RKHS, where the points are embedded in a finite-dimensional space and the curse of kernelization does not apply, allowing us to follow the gradient without increasing the number of SV.

Overall, the goal is to never exceed a budget of SVs in order to maintain a fixed per-step cost.

Among budgeted methods, how to modify the gradient update to maintain zero weights (i.e., weight degradation [START_REF] Wang | Breaking the curse of kernelization: Budgeted stochastic gradient descent for large-scale svm training[END_REF]) can be approached in many different ways, such as removal [START_REF] Cavallanti | Tracking the best hyperplane with a simple budget perceptron[END_REF][START_REF] Dekel | The forgetron: A kernel-based perceptron on a budget[END_REF] or more expensive projection [START_REF] Orabona | The projectron: a bounded kernel-based perceptron[END_REF] and merging strategies. Nonetheless, we will see that as long as the size of the budget is fixed, the adversary can exploit this to increase the regret of the algorithm, and oblivious inclusion strategies such as uniform sampling [START_REF] He | Simple randomized algorithms for online learning with kernels[END_REF] fail.

As for functional approximation methods, the most common approach is to replace the exact feature-map ϕ with a finite-rank approximate feature map ϕ which allows to explicitly represent the mapped points in an Euclidean space, and run OL on this embedding [START_REF] Lu | Large Scale Online Kernel Learning[END_REF][START_REF] Yang | Nyström method vs random fourier features: A theoretical and empirical comparison[END_REF].

When the embedding is oblivious to data, the method is known as random-feature expansion [START_REF] Le | Fastfood -Approximating kernel expansions in loglinear time[END_REF], while a common data-dependent embedding mapping is the Nyström method [START_REF] Williams | Using the Nystrom method to speed up kernel machines[END_REF]. Again, if the embedding is fixed or with a limit in size, the adversary can exploit it. In addition, analyzing a change in embedding during the gradient descent is an open problem, since the underlying RKHS changes with it.

Unfortunately, to guarantee O( √ T ) regret using less than O(t) space and time per round w.h.p., all of these methods require additional assumptions, such as points x t coming from a distribution or strong convexity on the losses. Moreover, as approximate first-order methods, they can at most hope to match the O( √ T ) regret of exact GD, and among second-order kernel methods, no approximation scheme has been proposed that can provably maintain the same O(log T ) regret as exact second-order GD. In addition, approximating H t is harder for OKL. Since we cannot directly access the matrix representation of H t in the feature-space, diagonal approximation is impossible, and low-rank sketching harder.

Contributions:

In this chapter, we provide two second-order OKL algorithms: Sketched-KONS takes the approach of approximate gradient updates, and PROS-N-KONS takes the approach of approximating the feature map. Both are based on an exact second-order method OKL algorithm called Kernelized-ONS (KONS), which is a generalization of [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF]'s online Newton step (ONS) from OL to OKL.

With Sketched-KONS (Calandriello et al., 2017c) our goal is to show that it is possible to approximate the second-order updates of an exact second-order OKL method (i.e., approximate the preconditioner H t ) using only a small budget of SV, and without losing the logarithmic regret rate. In particular, Sketched-KONS chooses which SV to include in the approximate preconditioner H t based on their RLS, using KORS to estimate them. The result is the first approximate second-order OKL algorithm that w.h.p. achieves logarithmic regret, and offers a favorable regret-performance trade-off. For a given factor β ≤ 1, we can increase the regret by a linear 1/β factor to O(d t eff (γ) log(T )/β), while obtaining a quadratic β 2 improvement in runtime compared to exact methods, and achieving a O(t 2 β 2 ) space and time cost per iteration. Unfortunately, to construct a truly scalable OKL algorithm, we need to achieve a near-linear O(T d T eff (γ)) runtime. In Section 5.2.1 we show a counterexample where any SV inclusion rule will fail to preserve both logarithmic regret and O(d T eff (γ)) runtime, highlighting the limits of our approximate gradient update rule.

To resolve the failure mode proposed by the counterexample, we introduce PROS-N-KONS (Calandriello et al., 2017b), an improved second-order OKL method that for the first time guarantees both logarithmic regret and avoids the curse of kernelization, taking only a near-constant O(d T eff (γ) 2 ) per-step time and space cost. To achieve this, instead of replacing the update rule for the preconditioner H t , we replace the exact feature map ϕ with an approximate ϕ and H constructed using a Nyström dictionary approximation, where the dictionary is computed using KORS. While previous methods [START_REF] Le | Fastfood -Approximating kernel expansions in loglinear time[END_REF][START_REF] Lu | Large Scale Online Kernel Learning[END_REF] used fixed embeddings, we adaptively update our dictionary and embedding H, so that it cannot be indefinitely exploited by the adversary. For a dictionary of size j, this non-linearly embeds the points in R j , where we can efficiently perform exact second-order updates in constant O(j 2 ) per-step time, and achieve the desired O(log(T )) regret, with respect to the best competitor in H. Using KORS' (ε, γ)-accuracy guarantees to show that H and H are close, we prove that we never get stuck performing GD in an embedded space that is too distant from the true H, but at the same time the size of the embedding j never grows larger than the effective dimension of the problem. Combined with an adaptive restart strategy, this allows us to bound our regret w.r.t. to the best solution in H.

Online kernel learning and the kernelized online Newton step

We focus on the online kernel learning setting, where an adversary chooses an arbitrary sequence of points {x t } T t=1 and convex differentiable losses { t } T t=1 . The learning protocol is the following. At each round t ∈ [T ]

(1) the adversary reveals the new point x t , (2) the learner chooses a function f wt and predicts f wt (x t ) = ϕ(x t ) T w t , (3) the adversary reveals the loss t , (4) the learner suffers t (ϕ(x t ) T w t ) and observes the associated gradient g t .

The adversary can choose any function f ∈ H that can be represented as a (potentially infinite) set of weights w such that f w (x) = ϕ(x) T w and the norm w is bounded. After t timesteps, we indicate with D t = {x i } t i=1 , the dataset containing the points observed so far. In the rest of the thesis we consider the problem of OKL where H is arbitrary and potentially infinite dimensional (non-parametric setting). We refer to the special case H = R d and φ t = x t as OL (parametric setting).

We are interested in bounding the cumulative regret between the learner and a fixed function w defined as R T (w) = T t=1 (φ t w t ) -(φ t w). Since H is potentially a very large space, we also need to restrict the class of comparators w. As [START_REF] Luo | Efficient second-order online learning via sketching[END_REF], we define the feasible set as S t = {w : |φ T t w| ≤ C} and S = ∩ T t=1 S t . This comparison class contains all functions f w whose output is contained (clipped) in the interval [-C, C] on all points φ 1 , . . . , φ T . Unlike the often used constraint on w [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF][START_REF] Zhu | Online Gradient Descent in Function Space[END_REF], comparing against clipped functions (Gammerman et al.,

compute u t = A -1 t-1 ( t-1 s=0 b s g s ) 6:
compute y t = ϕ(x t ) T u t 7:

predict y t = ϕ(x t ) T w t = y th(y t )

8:

observe g t , update A t = A t-1 + η t g t g T t 9: end for 2004; [START_REF] Luo | Efficient second-order online learning via sketching[END_REF][START_REF] Zhdanov | An Identity for Kernel Ridge Regression[END_REF] has a clear interpretation even when passing from R d to H. Moreover, S is invariant to linear transformations of H and suitable for practical problems where it is often easier to choose a reasonable interval for the predictions y t rather than a bound on the norm of a (possibly non-interpretable and infinite) parametrization w.

We make the following assumptions on the losses Assumption 1 -Scalar Lipschitz. The loss functions t satisfy | t (z)| whenever |z| ≤ C. Assumption 2 -Curvature. There exists σ t ≥ σ > 0 such that for all u, w ∈ S and for all t ∈ [T ],

t (φ T t w) := l t (w) ≥ l t (u) + ∇l t (u) T (w -u) + σ t 2 (∇l t (u) T (w -u)) 2 .
This assumption is weaker than strong convexity as it only requires the losses to be strongly convex in the direction of the gradient. It is satisfied by squared loss, squared hinge loss, logistic loss, and in general all exp-concave losses [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF]. Under this weaker requirement, exact second-order learning methods (Calandriello et al., 2017c;[START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF], can achieve logarithmic regret at the cost of a higher computational complexity w.r.t. first-order methods.

Kernelized online newton step

We now present an simple second-order OKL method, that will serve as a basis for Sketched-KONS and PROS-N-KONS. In particular, we generalize to the OKL setting the online Newton step algorithm, originally introduced by [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF] for OL.

ONS is a projected gradient descent that uses the following update rules

u t = w t-1 -A -1 t-1 g t-1 , w t = Π A t-1 St (u t ), 5.1
where Π

A t-1 St (u t ) = arg min w∈St u tw A t-1 is an oblique projection on a set S t with matrix A t-1 . If S t is the set of vectors with bounded prediction in [-C, C], then as shown by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF] the projection reduces to

w t = Π A t-1 St (u t ) = u t - h(φ T t u t ) φ T t A -1 t-1 φ t A -1 t-1 φ t ,
where h(z) = sign(z) max{|z| -C, 0} computes how much z is above or below the interval [START_REF] Zinkevich | Online Convex Programming and Generalized Infinitesimal Gradient Ascent[END_REF]. In the same setting, [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF] shows that choosing A t = t s=1 η s g s g T s + αΠ T makes ONS an efficient reformulation of follow the approximate leader (FTAL). While traditional follow-the-leader algorithms play the set of weights w t = arg min w∈St t-1 s=1 l t (w), FTAL replaces the loss l t with a convex approximation using Asm. 2, and plays the minimizer of the surrogate function. As a result, under Asm. 1-2 and when σ t ≥ σ > 0, FTAL achieves a logarithmic O(d log T ) regret. FTAL's solution path can be computed in O(d 2 ) time using ONS updates, and further speedups were proposed by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF] using matrix sketching.

[-C, C]. When A t = Π T /η t ,
Unfortunately, in OKL, vectors φ t and weights w t cannot be explicitly represented, and most of the quantities used in vanilla ONS (Eq. 5.1) cannot be directly computed. Instead, we derive a closed form alternative (Alg. 12) that can be computed in practice. Using a rescaled variant of our feature vectors φ t ,

φ t = ġt √ η t φ t = √ η t g t and Φ t = [φ 1 , . . . , φ t ],
we can rewrite

A t = Φ t Φ T t + αΠ T and Φ T t Φ t = K t ,
where the empirical kernel matrix K t is computed using the rescaled kernel K(x i ,

x j ) = ġi √ η i ġj √ η j K(x i , x j ) instead of the original K, or equivalently K t = D t K t D t with D t = Diag({ ġi √ η i } t i=1
) the rescaling diagonal matrix. We begin by noting that

y t = φ T t w t = φ T t u t - h(φ T t u t ) φ T t A -1 t-1 φ t A -1 t-1 φ t = φ T t u t -h(φ T t u t ) φ T t A -1 t-1 φ t φ T t A -1 t-1 φ t = y t -h(y t ).
As a consequence, if we can find a way to compute y t , then we can obtain y t without explicitly computing w t . Before that, we first derive a non-recursive formulation of u t .

Lemma 5.1 - (Calandriello et al., 2017c). In Alg. 12 we introduce

b i = [b t ] i = ġi √ η i y i - h(y i ) φ T i A -1 i-1 φ i - 1 √ η i
and compute u t as

u t = A -1 t-1 Φ t-1 b t-1 .
Then, u t is equal to the same quantity in Eq. 5.1 and the sequence of predictions y t is the same in both algorithms.

While the definition of b t and u t still requires performing operations in the (possibly infinitely dimensional) feature space, in the following we show that b t and the prediction y t can be conveniently computed using only inner products.

Lemma 5.2 - (Calandriello et al., 2017c). All the components b i = [b t ] i of the vector introduced in Lem. 5.1 can be computed as

ġi √ η i y i - αh(y i ) k i,i -k T [i-1],i (K i-1 + αI i-1 ) -1 k [i-1],i - 1 η i .
Then, we can compute

y t = 1 α k T [t-1],t D t-1 (b t-1 -(K t-1 + αI i-1 ) -1 K t-1 b t-1 ).
Since Alg. 12 is equivalent to ONS (Eq. 5.1), existing regret bounds for ONS directly applies to its kernelized version.

Proposition 5.3 - [START_REF] Luo | Efficient second-order online learning via sketching[END_REF]. For any sequence of losses t satisfying Asm. 1-2, the regret R T of Alg. 12 is bounded by

R T ≤ α w * 2 + R G + R D with R G := T t=1 g T t A -1 t g t = T t=1 φ T t (Φ t Φ T t + αΠ T ) -1 φ t /η t R D := T t=1 (w t -w * ) T (A t -A t-1 -σ t g t g T t )(w t -w * ) = T t=1 (η t -σ t ) ġ2 t (φ T t (w t -w * )) 2 .
In the d-dimensional OL, choosing a decreasing step-size η t = d/(C 2 L 2 t) allows ONS to achieve a O(CL √ dT ) regret for the cases where σ t = 0. When σ t ≥ σ > 0 (e.g., when the functions are exp-concave) we can set η t = σ t and improve the regret to O(d log(T )). Unfortunately, these quantities hold little meaning for OKL, where d is not guaranteed to be finite-dimensional, and a O( √ d) regret can be very large or even infinite. Instead, we expect the regret of KONS to depend on quantities that are more strictly related to the kernel K t and its complexity.

By focusing the first regret term in the decomposition of Prop. 5.3, we notice that

R G = T t=1 φ T t (Φ t Φ T t + αΠ T ) -1 φ t /η t = T t=1 τ t,t /η t ,
where τ t,t it the RLS of sample t computed using the rescaled K rather than K (i.e., using φ t and w.r.t. Φ t ). This reveals a deep connection between the regret of KONS and the cumulative sum of the RLS. In other words, the RLS capture how much the adversary can increase the regret by picking orthogonal directions that have not been seen before. While in OL, this can happen at most d times (hence the dependency on d in the final regret, which is mitigated by a suitable choice of η t ), in OKL, R G can grow linearly with time, since large H can have infinite near-orthogonal directions. Nonetheless, the actual growth rate is now directly related to the complexity of the sequence of points chosen by the adversary and the kernel function K. While the sum i τ i,i scales with the online effective dimension of the points, we can use Lemma 3.9 to bound it as d t eff (α) log(t). This inequality shows again that in general the complexity of online learning is only a factor log(T ) (in the worst case) away from the complexity of batch learning. Moreover, notice that the log-determinant of the covariance matrix A T appears in many other online learning problems [START_REF] Cesa-Bianchi | A second-order perceptron algorithm[END_REF][START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design[END_REF].

At this point, we can generalize the regret bounds of ONS for OL to KONS and OKL.

Theorem 5.4 - (Calandriello et al., 2017c). For any sequence of losses t satisfying Asm. 1-2, let σ = min t σ t . If η t ≥ σ ≥ 0 for all t and α ≤ √ T , the regret of Alg. 12 is Chapter 5. Sequential RLS Sampling in the Online Setting Algorithm 13 Sketched-KONS Input: Feasible parameter C, stepsizes η t , regulariz. α 1: Initialize w 0 = 0, g 0 = 0, b 0 = 0, A 0 = αΠ T 2: Initialize independent run of KORS 3: for t = {1, . . . , T } do 4:

receive x t 5: compute u t = A -1 t-1 ( t-1 s=0 b s g s ) 6:
compute yt = ϕ(x t ) T u t 7:

predict y t = ϕ(x t ) T w t = yth(y t ), observe g t 8:

compute τ t,t using KORS (i.e., Algorithm 8 and Eq. 3.4) 9:

compute p t = max{ τ t,t , β} 10:

draw q t ∼ B( p t , q)

11:

update A t = A t-1 + η t qt q g t g T t 12: end for upper-bounded as

R T ≤ α w * 2 + d T onl (α)/η T + 4C 2 L 2 T t=1 (η t -σ).
In particular, if for all t we have σ t ≥ σ > 0, setting η t = σ we obtain

R T ≤ α w * 2 + 2d T eff α/(σL 2 ) log(2σL 2 T ) σ ,
otherwise, σ = 0 and setting η t = 1/(LC √ t) we obtain

R T ≤ α w * 2 + 4LC √ T d T eff (α/L 2 ) log(2L 2 T ).
Comparison to OL algorithms. We first notice that the effective dimension d T eff (α) can be seen as a soft rank for K T and that it is smaller than the rank r for any α.1 For exp-concave functions (i.e., σ > 0), we slightly improve over the bound of [START_REF] Luo | Efficient second-order online learning via sketching[END_REF]

= d T eff (α)/(L 2 C 2 t) gives a regret O( d T eff (α)T ) ≤ O( √ rT ).
Comparison to OKL algorithms. Simple functional gradient descent, e.g., NORMA (Kivinen et al., 2004a), achieves a O( √ T ) regret when properly tuned [START_REF] Zhu | Online Gradient Descent in Function Space[END_REF], regardless of the loss function. For the special case of squared loss, [START_REF] Zhdanov | An Identity for Kernel Ridge Regression[END_REF] show that kernel recursive least squares achieves the same O(log(Det(K T /α + I T ))) regret as achieved by KONS for general exp-concave losses.

Sketched KONS

Building on KORS, we now introduce a sketched variant of KONS that can efficiently trade off between computational performance and regret. Sketched-KONS (Algorithm 13) runs KORS (Algorithm 8) as a black-box estimating RLS τ t , that are then used to sketch the original matrix A t with a matrix A t = t s=1 η t (q s /q)g t g T t + Π T , where at each step we add the current gradient g t g T

t only if the one of the coin flips in q t succeeded (i.e., q t = 0). Unlike KORS, the elements added to A t are not weighted, and the probabilities p t used for the individual coins z t,j are chosen as the maximum between τ t,t , and a parameter 0

≤ β ≤ 1. Let R t be the unweighted counterpart of S t , that is [R t ] i,j = 0 if [S t ] i,j = 0 and [R t ] i,j = 1 if [S t ] i,j = 0.
Then we can efficiently compute the coefficients b t and predictions y t as follows.

Lemma 5.5 - (Calandriello et al., 2017c). Let E t = R T t K t R t + αI t be an auxiliary matrix, then all the components b i = [ b t ] i used in Alg. 13 can be computed as

ġi √ η i y i - αh(y i ) k i,i -k T [i-1],i R i-1 E -1 i-1 R i-1 k [i-1],i - 1 η i .
Then we can compute

yt = 1 α k T [t-1],t D t-1 b t-1 -k T [t-1],t D t-1 R t-1 E -1 t-1 R t-1 K t-1 b t-1 .
Note that since the columns in R t are selected without weights, (R T t K t R t + αI t ) -1 can be updated efficiently using block inverse updates, and only when A t changes. While the specific reason for choosing the unweighted sketch A t instead of the weighted version A It t used in KORS is discussed further in the discussion of Theorem 5.7, the following corollary shows that A t is as accurate as A It t in approximating A t up to the smallest sampling probability p β t .

Corollary 5.6 -Theorem 3.11. Let p β min = min T t=1 p β t . Then w.h.p., we have

(1 -ε) p min A t p min A It t A t .
We can now state the main result of this section. Since for Sketched-KONS we are interested not only in regret minimization, but also in space and time complexity, we do not consider the case σ = 0, because when the function does not have any curvature, standard GD already achieves the optimal regret of O( √ T ) [START_REF] Zhu | Online Gradient Descent in Function Space[END_REF] while requiring only O(t) space and time per iteration.

Theorem 5.7 - (Calandriello et al., 2017c). For any sequence of losses t satisfying Asm. 1-2, let σ = min t σ t and τ min = min T t=1 τ t,t . When η t ≥ σ > 0 for all t, α ≤ √ T , q ≥ 3 log(T /δ)/ε 2 , if we set η t = σ then w.p. 1δ the regret of Alg. 13 satisfies Regret guarantees. From Eq. 5.2 we can see that when τ min is not too small, setting β = 0 we recover the guarantees of exact KONS. Since usually we do not know τ min , we can choose to set β > 0, and as long as β ≥ 1/ polylog T , we preserve a (poly)-logarithmic regret.

R T ≤ α w * 2 + 2 d T eff α/(σL 2 ) log(2σL 2 T ) σ max{β, τ min } , 5 .
Computational speedup. The time required to compute k [t-1],t , k t,t , and k T [t-1],t D t-1 b t-1 gives a minimum O(t) per-step complexity. Note that K t-1 b t-1 can also be computed incrementally in O(t) time. Denoting the size of the dictionary at time t as Therefore, in the case when τ min is not too small, our runtime is of the order O(d t eff (α) 2 + t), which is almost as small as the O(t) runtime of GD but with the advantage of a second-order method logarithmic regret. Moreover, when τ min is small and we set a large β, we can trade off a 1/β increase in regret for a β 2 decrease in space and time complexity when compared to exact KONS (e.g., setting β = 1/10 would correspond to a tenfold increase in regret, but a hundred-fold reduction in computational complexity).

B t = O(d eff (α) t + tβ), computing [ b t ] i and k T [t-1],t D t-1 R t-1 E -1 t-1 R t-1 K t-1 b t-
Asymptotic behavior. Notice however, that space and time complexity, grow roughly with a term Ω(t min t s=1 p s ) ∼ Ω(t max{β, τ min }), so if this quantity does not decrease over time, the computational cost of Sketched-KONS will remain large and close to exact KONS. This is to be expected, since Sketched-KONS must always keep an accurate sketch in order to guarantee a logarithmic regret bound. Note that [START_REF] Luo | Efficient second-order online learning via sketching[END_REF] took an opposite approach for OL, where they keep a fixed-size sketch but possibly pay in regret, if this fixed size happens to be too small. Since a non-logarithmic regret is achievable simply running vanilla GD, we rather opted for an adaptive sketch at the cost of space and time complexity. In batch optimization, where t does not change over time, another possibility is to stop updating the solution once τ min becomes too small. When H s is the Hessian of w.r.t. w s , then the quantity g T t H -1 t g t , in the context of Newton's method, is called Newton decrement and it corresponds up to constant factors to τ min . Since a stopping condition based on Newton's decrement is directly related to the near-optimality of the current w t [START_REF] Nesterov | Interior-point polynomial algorithms in convex programming[END_REF], stopping when τ min is small also provides guarantees about the quality of the solution.

Sampling distribution. Note that although β > 0 means that all columns have a small uniform chance of being selected for inclusion in A t , this is not equivalent to uniformly sampling columns. It is rather a combination of a RLS-based sampling to ensure that columns important to reconstruct A t are selected and a threshold on the probabilities to avoid too much variance in the estimator.

Biased estimator and results in expectation. The random approximation

A t is biased, since E[Φ t R t R T t Φ T t ] = Φ t Diag({τ t,t })Φ T t = Φ t Φ T t .
Another option would be to use a weighted and unbiased approximation A t = t s=1 η s q s /(q p s )g s g T s used in KORS and a common choice in matrix approximation methods, see e.g., [START_REF] Alaoui | Fast randomized kernel methods with statistical guarantees[END_REF]. Due to its unbiasedness, this variant would automatically achieve the same logarithmic regret as exact KONS in expectation, similar to the result obtained by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF], using Gaussian random projection in OL. While any unbiased estimator, e.g., uniform sampling of g t , would achieve this result, RLS-based sampling already provides strong reconstruction guarantees sufficient to bound R G . Unfortunately, and differently from the batch case, the adaptive weights 1/ p s may cause large variations in A t over consecutive steps, thus leading to a large regret R D in high probability.

Limitations of budgeted OKL algorithms

From the discussion above, it appears that a weighted, unbiased dictionary may not achieve high-probability logarithmic guarantee because of the high variance coming from sampling. Therefore, if we want to recover the logarithmic regret guarantee, we may have to pay for it with a large dictionary. This may actually be due to the analysis, the setting, or the algorithm.

• The regret analysis of is tight for the OL setting without any curvature assumption, since [START_REF] Luo | Efficient second-order online learning via sketching[END_REF] shows a lower bound for this case. As OL is a special case of OKL, it does not seem easy to find a tighter bound, although possible data-adaptive bounds remain an interesting open question. • Regarding the difficulty of approximation in the online setting, notice that in the batch setting (Chapter 4), the subsampling does not cause any issue and we can have strong learning guarantees in high probability with a small dictionary. Differently from the batch setting, in OKL we have to balance two opposing goals: we want our solution to change quickly, in order to minimize R G , but we also want our solution to not change too much, to minimize R D . • Algorithmically, an important property of the dictionary learning approach used in KORS is that it can only include atoms coming from the input, re-weight them, and cannot remove an atom from the dictionary after insertion. If we set the weights to have an unbiased estimate, we achieve an accurate R G but suffer a huge regret in R D . On the other hand, we can store unweighted atoms to have small R D but large R G .

To see how this sampling without removal approach could fail, consider the following simple scenario. The adversary always presents to the learner the same point x (with associated φ), but for the loss it alternates between 2t (w t ) = (Cφ T w t ) 2 on even steps and 2t+1 (w t ) = (-Cφ T w t ) 2 on odd steps. Then, σ t = σ = 1/(8C 2 ), and we have a gradient that always points in the same φ direction, but switches sign at each step. The optimal solution in hindsight is asymptotically w = 0 and let this be also our starting point w 0 . We also set η t = σ, since this is what KONS would do, and α = 1 for simplicity.

For this scenario, we can compute R G and R D in closed form,

R G ≤ T t=1 ġ2 t t s=1 ġ2 s σ + α ≤ T t=1 C 2 C 2 σt + α ≤ O(log T ), R D = t s=1 (η t -σ)(w T t g t ) 2 = 0.
Note that although the matrix A t is rank 1 at each time step, vanilla ONS does not take advantage of this easy data, and would store it all with a O(t 2 ) space in OKL.

Notice that in this all-identical samples example, the losses t are effectively strongly convex, and even basic gradient descent with a stepsize η t = 1/t would achieve logarithmic regret [START_REF] Zhu | Online Gradient Descent in Function Space[END_REF] with even smaller space. On the other hand, we show how sampling without removal has difficulties in minimizing the regret bound from Proposition 5.3 in our simple scenario.

In particular, consider an arbitrary (possibly randomized) algorithm that is allowed only to reweight atoms in the dictionary and not to remove them. In our example, this translates to choosing a schedule of weights w s and set A t = t s=1 w s φ s φ s = W t φφ with total weight W = W T = T s=1 w s and space complexity equal to the number of non-zero weights B = |{w s = 0}|. We can show that there is no schedule for this specific class of algorithms with good performance due to the following three conflicting goals.

(1) To maintain R G small, t s=1 w s should be as large as possible, as early as possible.

(2) To maintain R D small, we should choose weights w t > 1 as few times as possible, since we accumulate max{w t -1, 0} regret every time.

(3) To maintain the space complexity small, we should choose only a few w t = 0.

To enforce goal (3), we must choose a schedule with no more than B non-zero entries. Given the budget B, to satisfy goal (2) we should use all the B budget in order to exploit as much as possible the max{w t -1, 0} in R D , or in other words we should use exactly B non-zero weights, and none of these should be smaller than 1. Finally, to minimize R G we should raise the sum t s=1 w s as quickly as possible, settling on a schedule where w 1 = W -B and w s = 1 for all the other B weights. It easy to see that if we want logarithmic R G , W needs to grow as T , but doing so with a logarithmic B would make R D = T -B = Ω(T ). Similarly, keeping W = B in order to reduce R D would increase R G . In particular notice, that the issue does not go away even if we know the RLS perfectly, because the same reasoning applies. This simple example suggests that dictionary-based sketching methods without removal, which are very successful in batch scenarios, may actually fail in achieving logarithmic regret in online optimization.

Algorithmically, this could be potentially fixed if we knew how to remove less important columns from dictionary to gain some slack in R D , or how to create new atoms that can be used to represent multiple inputs, such as Frequent Directions (FD, Ghashami et al., 2016a) do in the Euclidean case.

Removing an atom present in the dictionary reduces our space complexity, but at the same time increases R G , since the dampening effect of A -1 t decreases. Moreover, let g t be the last computed gradient, and let g s with s < t be the gradient we want to remove from A t and our dictionary. Unless g t = g s (i.e., for any arbitrarily small perturbation) the operator A t -A t-1 = g t g T tg s g T s has both a positive and negative eigenvalue, and the adversary can make the corresponding (w tw * ) T ((1σ t )g t g T tg s g T s )(w tw * ) term in R D arbitrarily large. Therefore, removing a gradient from the dictionary will decrease space, but it increases R G , and might cause an arbitrarily large increase in R D due to the adversary. Similar arguments can be derived for schemes that keep reweighting atoms after insertion.

Finally, as our counterexample in the simple scenario hints, creating new atoms (either through projection or merging) allows for better adaptivity. For example, if we have a single gradient g each time slightly perturbed, we can represent it using just the principal components of these perturbed copies without storing each copy separately. This is essentially the approach taken by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF] in the OL setting using FD as a sketching algorithm. However, in OKL we cannot explicitly represent new atoms, and therefore the kernelization of FD does not appear to be straightforward. The most recent step in this direction is proposed again by Ghashami et al., (2016b) for batch kernel PCA. It first embeds the points into a finite-dimensional space using a fixed set of random feature expansions and afterwards uses FD to reduce the computational complexity. Unfortunately, their method only works for an Euclidean input space X , and does not have online or any-time guarantees. Moreover, note that our example can be easily generalized to defeat any approximation strategy that uses a fixed embedding technique (e.g., fixed amount of random features or a fixed dictionary) or a fixed budget of SV. It suffice to replace the single φ vector with a set of repeating orthogonal vectors that exceed the budget.

Nonetheless, this single sample problem is intrinsically simple: its effective dimension d T eff (α) 1 is small, and its induced RKHS H = φ is a singleton. Therefore, following an adaptive embedding approach, we can reduce it to a 1 dimensional parametric problem, and solve it efficiently in this space using exact ONS updates. We can see this approach as constructing an approximate feature map ϕ that after one step will exactly coincide with the exact feature map ϕ, but allows us to run exact KONS updates efficiently replacing K with K. In other words, since correctly approximating second-order updates in the exact RKHS H is hard due to the adversary, we instead resort to make exact second order updates, that the adversary cannot exploit, but in an approximate and smaller RKHS H. Building on this intuition, we propose PROS-N-KONS, a new second-order FGD method that continuously searches for the best embedding space H It and, at the same time, exploits the small embedding space H It to efficiently perform exact second-order updates.

The PROS-N-KONS algorithm

A common approach to alleviate the computational cost of OKL is to replace the highdimensional feature map ϕ with a finite-dimensional approximate feature map ϕ. Let I = {φ i } j i=1 be a dictionary of j points chosen from those revealed by the adversary, and let and Φ I be the associated feature matrix with ϕ(x i ) as columns. We define the embedding ϕ(x) = Σ -1 U T Φ T I ϕ(x) ∈ R j , where Φ I = VΣU T is the singular value decomposition of the feature matrix. While in general Φ I is infinite dimensional and cannot be directly decomposed, we exploit the fact that UΣV T VΣU T = Φ T I Φ I = K I = UΛU T and that K I is a (finite-dimensional) PSD matrix. Therefore it is sufficient to compute the eigenvectors U and eigenvalues Λ of K I and take the square root Λ 1/2 = Σ. Note that with this definition we are effectively replacing the kernel K and H with an approximate K I and H I , such that K

I (x, x ) = ϕ(x) T ϕ(x ) = ϕ(x) T Φ I UΣ -1 Σ -1 U T Φ T I ϕ(x ) = ϕ(x) T Π I ϕ(x ) where Π I = Φ I (Φ T I Φ I ) -1 Φ T
I is the projection matrix on the column span of Φ I . Since ϕ returns vectors in R j , this transformation effectively reduces the computation complexity of kernel operations from t down to the size of the dictionary j. The accuracy of ϕ is directly related to the accuracy of the projection Π I in approximating the projection Π

t = Φ t (Φ T t Φ t ) -1 Φ T t , so that for all s, s ∈ [t], ϕ(x s ) T ϕ(x s ) is close to ϕ(x s ) T Π t ϕ(x s ) = ϕ(x s ) T ϕ(x s ).
All that is left is to find an efficient algorithm to choose a good dictionary I to minimize the error Π I -Π t . Among dictionary-selection methods, we focus on KORS (Section 3.2). While SQUEAK can generate smaller dictionaries, as we will see the frequent insertion and removal of atoms from the dictionary can change the subspace Π I too quickly over time. While in the batch setting this was not a problem, in the online setting the adversary can exploit this large variance and negatively affect our performance. Therefore we prefer the slightly suboptimal but more stable KORS, where the total number Algorithm 14 PROS-N-KONS Input: Feasible parameter C, step-sizes η t , regularizer α 1: Initialize j = 0, w 0 = 0, g 0 = 0, Π 0 = 0, A 0 = αI 0 , 2: Start a KORS instance with an empty dictionary I 0 .

3: for t = {1, . . . , T } do 4:

Receive x t , feed φ t to KORS.

5:

Receive q t from KORS (point added to dictionary or not)

6:

if q t-1 = 0 then { Dictionary changed, reset.} 7: j = j + 1 8:

Build K j from I j and decompose it in U j Σ j Σ T j U T j 9:

Set A t-1 = αI j ∈ R j×j .

10:

ω t = 0 ∈ R j 11:
else { Execute a gradient-descent step.}

12:

Compute map φ t and approximate map φ t = Σ -1 j U T j Φ T j φ t ∈ R j .

13:

Compute υ t = ω t-1 -A -1 t-1 g t-1 . 14: Compute ω t = υ t - h( φ T t υt) φ T t A -1 t-1 φt A -1 t-1 φ t where h(z) = sign(z) max{|z| -C, 0} 15:
end if

16:

Predict y t = φ T t ω t . 17:

Observe g t = ∇ ωt t ( φ T t ω t ) = t ( y t ) φ t . 18: Update A t = A t-1 + σt 2 g t g T t .
19: end for of dictionary changes equals the final size of the dictionary, and we can prove that w.h.p. this number of changes is small. We start from an empty dictionary I 0 and a null predictor w 0 = 0. At each round, PROS-N-KONS (Algorithm 14) receives a new point x t and invokes KORS to decide whether it should be included in the current dictionary or not. Let t j with j ∈ [J] be the random step when KORS introduces φ t j in the dictionary. We analyze PROS-N-KONS as an epoch-based algorithm using these milestones t j . Note that the length h j = t j+1t j and total number of epochs J is random, and is decided in a data-adaptive way by KORS based on the difficulty of the problem. During epoch j, we have a fixed dictionary I j that induces a feature matrix Φ I j containing samples φ i ∈ I j , an embedding ϕ(x) : X → R j = Σ -1 j U T j Φ T j ϕ(x) based on the singular values Σ j and singular vectors U j of Φ j , with its associated approximate kernel function K and induced RKHS H j . At each round t j < t < t j+1 , we perform an exact KONS update using the approximate map ϕ. This can be computed explicitly since φ t is in R j and can be easily stored in memory. The update rules are

υ t = ω t-1 -A -1 t-1 g t-1 , ω t = Π A t-1 St (υ t ) = υ t - h( φ T t υ t ) φ T t A -1 t-1 φ t A -1 t-1 φ t , A t = A t-1 + σ t 2 g t g T t ,
where we compute the oblique projection Π A t-1 St using [START_REF] Luo | Efficient second-order online learning via sketching[END_REF]'s closed-form solution. When t = t j and a new epoch begins, we perform a reset step before taking the first gradient step in the new embedded space. We update the feature-map ϕ, but we reset A t j and ω t j to zero. While this may seem a poor choice, as information learned over time is lost, it leaves intact the dictionary. As long as (a) the dictionary, and therefore the embedded space where we perform our GD, keeps improving and (b) we do not needlessly reset too often, we can count on the fast second-order updates to quickly catch up to the best function in the current H j . The motivating reason to reset the descent procedure when we switch subspace is to guarantee that our starting point in the descent cannot be influenced by the adversary, and therefore allow us to provide regret guarantees for the overall process (Section 5.3.1).

Computational complexity. PROS-N-KONS's computation complexity is dominated by A -1 t inversion required to compute the projection and the gradient update and by the query to KORS, that internally also inverts a j × j matrix. Therefore, a naïve implementation requires O(j 3 ) per-step time, and has a space O(j 2 ) space complexity necessary to store A t . Notice that taking advantage of the fact that KORS only adds SV to the dictionary and never removes them, and that similarly, the A t matrix is constructed using rank-one updates, a careful implementation reduces the per-step cost to O(j 2 ). Overall the total runtime of PROS-N-KONS is then O(T J 2 ), which using the bound on J provided by Theorem 3.11 and neglecting logarithmic terms reduces to O(T d T eff (γ) 2 ). Compared to other exact second-order FGD methods, such as KONS or RKLS, PROS-N-KONS dramatically improves the time and space complexity from polynomial to linear. Against other approximate second-order methods, PROS-N-KONS does not add a new SV at each step. This way it removes T 2 from the O(T 2 β 2 + T d T eff (γ) 3 ) time complexity of Sketched-KONS (Calandriello et al., 2017c). Moreover, when min t τ t,t is small, Sketched-KONS needs to compensate by adding a constant probability of adding a SV to the dictionary, resulting in a larger runtime complexity, while PROS-N-KONS has no dependency on the value of the RLS. Even compared to first-order methods, which incur a larger regret, PROS-N-KONS performs favorably, improving on the O(T 2 ) runtime of exact first-order FGD. Compared to other approximate methods, the variant using rank-one updates matches the O(J 2 ) per-step cost of the more accurate first-order methods such as the Budgeted Perceptron [START_REF] Cavallanti | Tracking the best hyperplane with a simple budget perceptron[END_REF], Projectron [START_REF] Orabona | The projectron: a bounded kernel-based perceptron[END_REF], and Nyström-GD [START_REF] Lu | Large Scale Online Kernel Learning[END_REF], while improving on their regret. PROS-N-KONS also closely matches faster but less accurate O(J) methods such as the Forgetron [START_REF] Dekel | The forgetron: A kernel-based perceptron on a budget[END_REF] and Budgeted-GD [START_REF] Zhao | Fast bounded online gradient descent algorithms for scalable kernel-based online learning[END_REF].

Regret guarantees

In this section, we study the regret performance of PROS-N-KONS.

Theorem 5.8 - (Calandriello et al., 2017b). For any sequence of losses t satisfying Assumption 2 with Lipschitz constant L, let σ = min t σ t . If η t ≥ σ for all t, α ≤ √ T , γ ≤ α, and predictions are bounded by C, then the regret of PROS-N-KONS over T steps is bounded w.p. 1δ as

R T (w) ≤ J α w 2 + 4 σ d T eff α σL 2 log 2σL 2 T /α + L 2 α T γε 4(1 -ε) + 1 + 2JC, 5.3 where J ≤ 3qd T eff (γ) log(2T ) is the number of epochs. If γ = α/T the previous bound reduces to R T (w) = O α w 2 d T eff (α/T ) log(T ) + d T eff (α/T ) d T eff (α) log 2 (T ) .

5.4

Remark (bound). The bound in Eq. 5.3 is composed of three terms.

At each epoch of PROS-N-KONS, an instance of KONS is run on the embedded feature space H j obtained by using the dictionary I j constructed up to the previous epoch. As a result, we directly use the bound on the regret of KONS (Theorem 5.4) for each of the J epochs, thus leading to the first term in the regret. Since a new epoch is started whenever a new SV is added to the dictionary, the number of epochs J is at most the size of the dictionary returned by KORS up to step T , which w.h.p. is O(d T eff (γ)), making the first term scale as O(d T eff (γ)d T eff (α)) overall. Nonetheless, the comparator used in the per-epoch regret of KONS is constrained to the RKHS H j induced by the embedding used in epoch j. The second term accounts for the difference in performance between the best solutions in the RKHS in epoch j and in the original RKHS H. This error is directly controlled by the (ε, γ)-accuracy of the dictionary constructed by KORS, where the RLS regularization γ and the parameter ε contribute to the factor γε/(1ε). Since the true RKHS is never fully reconstructed, i.e., ε is constant and does not go to zero, the approximation impact on the regret is amplified by the length of the process, thus leading to an overall linear term that needs to be regularized. Finally, the last term summarizes the regret suffered every time a new epoch is started and the default prediction y = 0 is returned. Since the values y t and ŷt are constrained in S, this results in a regret of 2JC.

Remark (regret comparison).

Tuning the RLS regularization as γ = α/T leads to the bound in Eq. 5.4. Unlike Sketched-KONS, the bound finally displays only an explicit logarithmic dependency on T , but this comes at the cost of increasing the effective dimension, which now depends on the regularization α/T . While in general this could possibly compromise the overall regret, if the sequence of points φ 1 , . . . , φ T induces a kernel matrix with a rapidly decaying spectrum, the resulting regret is still competitive. For instance, if the eigenvalues of K T decrease as λ t = at -q with constants a > 0 and q > 1, then d T eff (α/T ) ≤ aqT 1/q /(q -1). This shows that for any q > 2 we obtain a regret2 o( √ T log 2 (T )) showing that KONS still improves over first-order methods. Furthermore, if the kernel is low rank or the eigenvalues decrease exponentially, the final regret is poly-logarithmic, thus preserving the full advantage of the second-order approach. Notice that this scenario is always verified when H = R d , and is also verified when the adversary draws samples from a stationary distribution and, e.g., we use the Gaussian kernel [START_REF] Sun | On the size of the online kernel sparsification dictionary[END_REF][START_REF] Wathen | On spectral distribution of kernel matrices related to radial basis functions[END_REF][START_REF] Yang | Randomized Sketches for Kernels: Fast and Optimal Non-Parametric Regression[END_REF]. This result is particularly remarkable when compared to Sketched-KONS, whose regret scales as O α w 2 + d T eff (α) log(T )/β , where β is the fraction of samples which is forced into the dictionary (when β = 1 we recover the bound for KONS). Even when the effective dimension is small (e.g., exponentially decaying eigenvalues), Sketched-KONS requires setting β to T -p for a constant p > 0 to get a sub-quadratic space complexity, at the cost of increasing the regret to O(T p log(T )). On the other hand, PROS-N-KONS achieves a poly-logarithmic regret with linear space complexity up to poly-log factors (i.e., (T d T eff (γ) 2 ), thus greatly improving both the learning and computational performance w.r.t. Sketched-KONS. Finally, notice that while γ = α/T is the best choice agnostic to the kernel, better bounds can be obtained optimizing Eq. 5.3 for γ depending on d T eff (γ). For instance, let γ = α/T s , then the optimal value of s for q-polynomially decaying spectrum is s = q/(1 + q), leading to a regret bound O(T q/(1+q) ), which is always o( √ T ) for any q > 1.

Remark (comparison in the Euclidean case). In the special case H = R d , we can make a comparison with existing approximate methods for OL. In particular, the closest algorithm is Sketched-ONS by [START_REF] Luo | Efficient second-order online learning via sketching[END_REF]. Unlike PROS-N-KONS, and similarly to Sketched-KONS, they take the approach of directly approximating A t in the exact H = R d using FD [START_REF] Ghashami | Frequent Directions : Simple and Deterministic Matrix Sketching[END_REF] to construct a k-rank approximation of A t for a fixed k.

The resulting algorithm achieve a regret that is bounded by k log(T ) + k T i=k+1 σ 2 i , where the sum T i=k+1 σ 2 i is equal to the sum of all the smallest dk eigenvalues of the covariance matrix. This quantity can vary from 0, when the data lies in a subspace of rank r ≤ k, to T d-k d when the sample lie orthogonally and in equal number along all d directions available in R d . Computationally, the algorithm requires O(T dk) time and O(dk) space. Conversely, PROS-N-KONS automatically adapt its time and space complexity to the effective dimension of the algorithm d T eff (α/T ) which is smaller than the rank for any α. As a consequence, it requires only O(T r 2 ) time and O(r 2 ) space, achieving a O(r 2 log(T )) regret independently from the spectrum of the covariance matrix. Computationally, all of these complexities are smaller than the ones of Sketched-ONS in the regime r < k, which is the only one where Sketched-ONS can guarantee logarithmic (or even sublinear) regret, and where the regrets of the two algorithms are close. Overall, while Sketched-ONS implicitly relies on the r < k assumption, but continues to operate in a d dimensional space and suffers large regret if r > k, PROS-N-KONS will adaptively convert the d dimensional problem into a simpler one with the appropriate rank, fully reaping the computational and regret benefits.

The bound in Theorem 5.8 can be refined in the specific case of squared loss as follows.

Theorem 5.9 - (Calandriello et al., 2017b). For any sequence of squared losses t = (y tŷt ) 2 , L = 4C and σ = 1/(8C 2 ), if η t ≥ σ for all t, α ≤ √ T and γ ≤ α, the regret of PROS-N-KONS over T steps is bounded w.p. 1δ as R T (w) ≤ where ε = α α -γε 1-ε -1 -1 and L * j = min w∈H t j+1 -1 t=t j φ T t wy t 2 + α w 2 2 is the best regularized cumulative loss in H within epoch j.

Let L *

T be the best regularized cumulative loss over all T steps, then L * j ≤ L * T . Furthermore, we have that d j eff ≤ d T eff and thus regret in Eq. 5.5 can be (loosely) bounded as

R T (w) = O J d T eff (α) log(T ) + +ε L * j + ε α w 2 2 .
The major difference w.r.t. the general bound in Eq. 5.3 is that we directly relate the regret of PROS-N-KONS to the performance of the best predictor in H in hindsight, which replaces the linear term γT /α. As a result, we can set γ = α (for which ε = ε/(1 -2ε)) and avoid increasing the effective dimension of the problem. Furthermore, since L * T is the regularized loss of the optimal batch solution, we expect it to be small whenever the H is well designed for the prediction task at hand. For instance, if L * T scales as O(log(T )) for a given regularization α (e.g., in the realizable case L * T is actually just α w ), then the regret of PROS-N-KONS is directly comparable with Table 5.1: Regression datasets dataset, the effective dimension is too large for PROS-N-KONS to complete and only results for B-KONS are provided. Overall, the proposed algorithm can process hundreds of thousands of points in a matter of minutes, and show that it can practically scale to large datasets.

Regression. All algorithms are trained and evaluated using the squared loss. Notice that whenever the budget J max is not exceeded, B-KONS and PROS-N-KONS are the same algorithm and obtain the same result. On regression datasets (Tab. 5.1) we set α = 1 and γ = 1, which satisfies the requirements of Thm. 5.9. On smaller datasets such as parkinson and cpusmall, where frequent restarts greatly interfere with the gradient descent, and even a small non-adaptive embedding can capture the geometry of the data, PROS-N-KONS is outperformed by simpler first-order methods. As soon as T reaches the order of tens of thousands (cadata, casp) second-order updates and data adaptivity becomes relevant and PROS-N-KONS outperform its competitors, both in the number of SVs and in the average loss. In this intermediate regime, Con-KONS outperforms PROS-N-KONS and B-KONS since it is not as much affected by restarts. Finally, when the number of samples raises to hundreds of thousands, the intrinsic effective dimension of the dataset starts playing a larger role. On slice, where the effective dimension is too large to run, B-KONS still outperforms NOGD with a comparable budget of SVs, showing the advantage of second-order updates. Binary classification All algorithms are trained using the hinge loss and they are evaluated using the average online error rate. Results are reported in Table 5.2. While for regression an arbitrary value of γ = α = 1 is sufficient to obtain good results, it fails for binary classification. Decreasing the two parameters to 0.01 resulted in a 3-fold increase in the number of SVs included and runtime, but almost a 2-fold decrease in error rate, placing PROS-N-KONS and B-KONS on par or ahead of competitors without the need of any further parameter tuning.

Recap and open questions: do we need restarts?

We introduced Sketched-KONS and PROS-N-KONS, two approximate second-order OKL algorithms. Sketched-KONS is the first approximate second-order algorithm that can provably achieve logarithmic regret with high probability, matching exact methods. Unfortunately, it only provides constant speed-ups compared to the exact method, and remains affected by the curse of kernelization, requiring O(T 2 ) overall runtime.

PROS-N-KONS breaks this quadratic barrier, and is the first second-order OKL algorithm that guarantees both logarithmic regret and avoids the curse of kernelization, taking only a near-constant O(d T eff (γ) 2 ) per-step time and space cost.

PROS-N-KONS can also be seen as an adaptive doubling strategy. In online learning, doubling strategies are common [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF] when we need to deal with parameters unknown in advance. Informally, the doubling trick corresponds to breaking up our learning process in phases, each twice as long as the previous.

In PROS-N-KONS case, we do not know which is the right RKHS H j that we should consider for our descent (using the exact H is too expensive and we do not consider it). Therefore, we break down our learning process in phases dictated by KORS.

If the adversary keeps playing points that are already in our dictionary, this phase can be indefinitely long, but our approximation is accurate and we pay logarithmic regret. If instead the adversary plays multiple points along a new direction, KORS will quickly (i.e., in logarithmic time) notice and include it in the dictionary, starting a new phase. Overall, the adversary cannot keep us playing in the wrong H j for more than a logarithmic number of rounds in each phase.

Ultimately, either the effective dimension of the problem is small, and we will quickly enter a final phase in which H j never changes, or the effective dimension is large and even exact second-order methods would have a large regret.

Nonetheless, doubling tricks are often a last resort in online learning, as they are wasteful.

And since we know that H j and H j+1 are close (they differ in a single atom), it remains an important open question if we can transfer the solution of a phase into the next one without paying a large regret. Empirically, we see that Con-KONS and B-KONS often outperform PROS-N-KONS, which suggest that this might be the case, at least in the simpler, non-adversarial setting of our experiments.

Extended proofs

Proof of Lemma 5.1. We begin by applying the definition of u t+1 and collecting A -1 t , which can always be done since, for α > 0, A t is invertible, u t+1 = w t -A -1 t g t = A -1 t (A t w tg t ).

We focus now on the last term and use the definition of A t , A t w tg t = A t-1 w t + η t g t g T t w tg t = A t-1 u t -A t-1 r t + ( √ η t g T t w t -1/ √ η t )φ t .

Looking at A t-1 r t and using the assumption ġt = 0,

A t-1 r t = h(φ T t u t ) φ T t A -1 t-1 φ t A t-1 A -1 t-1 φ t = h(φ T t u t ) φ T t A -1 t-1 φ t ġ2 t η t ġ2 t η t φ t = ġt √ η t h(φ T t u t ) φ T t A -1 t-1 φ t φ t .
Putting together all three terms, and using the fact that g T t w t = ġt φ t w t = ġt y t and denoting b t = [b t ] t we have

u t+1 = A -1 t (A t w t -g t ) = A -1 t (A t-1 u t + b t φ t ) = A -1 t (A t-1 (w t-1 -A -1 t-1 g t-1 ) + b t φ t ) = A -1 t (A t-1 w t-1 -g t-1 + b t φ t ) = A -1 t (A t-2 w t-2 -g t-2 + b t-1 φ t-1 + b t φ t ) = A -1 t (A 0 w 0 + t s=1 b s φ s ).
Proof of Lemma 5.2. Throughout this proof, we make use of the linear algebra identity from Proposition 2.12. We begin with the reformulation of [b t ] t . In particular, the only term that we need to reformulate is

φ t A -1 t-1 φ t = φ t (Φ t-1 Φ T t-1 + αΠ T ) -1 φ t = 1 α φ t (Π T -Φ t-1 (Φ T t-1 Φ t-1 + αΠ T ) -1 Φ T t-1 )φ t = 1 α (φ T t φ t -φ T t Φ t-1 (Φ T t-1 Φ t-1 + αI t-1 ) -1 Φ T t-1 φ t ) = 1 α (k t,t -k T [t-1],t (K t-1 + αI t-1 ) -1 k [t-1],t ).
For y t , we have

y t = φ T t u t = φ T t A -1 t-1 Φ t-1 b t-1 = φ T t (Φ t-1 Φ T t-1 + αΠ T ) -1 Φ t-1 b t-1 = 1 α φ T t (Π T -Φ t-1 (Φ T t-1 Φ t-1 + αΠ T ) -1 Φ T t-1 )Φ t-1 b t-1 = 1 α φ T t Φ t-1 D t-1 (b t-1 -(K t-1 + αI t-1 ) -1 K t-1 b t-1 ) = 1 α k T [t-1],t D t-1 (b t-1 -(K t-1 + αI t-1 ) -1 K t-1 b t-1 ).
Proof of Theorem 5. For R G , we reformulate

T t=1 g T t A -1 t g t = T t=1 η t η t g T t A -1 t g t = T t=1 1 η t φ T t A -1 t φ t ≤ 1 η T T t=1 φ T t A -1 t φ t = 1 η T T t=1 τ t,t = d onl (α)/η T ≤ d T eff (α) η T (1 + log( K T /α + 1)),
where d T eff (α) and K T are computed using the rescaled kernel K. Let us remind ourselves the definition D = Diag { ġt √ η t } T t=1 . Since η t = 0 and ġt = 0 for all t, D is invertible and we have λ min (D -2 ) = min T t=1 1/( ġ2 t η t ) ≥ 1/(L 2 η 1 ). For simplicity, we assume η 1 = σ, leaving the case η 1 = 1/1 = 1 as a special case. We derive ≤ log(σL 2 T /α + 1) ≤ log(2σL 2 T /α),

since Tr(K t ) = T t=1 k t,t = T t=1 φ T t φ t ≤ T t=1 1 = T .

Proof of Lemma 5.5. Through this proof, we make use of the linear algebra identity from Proposition 2.12. We begin with the reformulation of b i = [ b t ] i . In particular, the only term that we need to reformulate is

φ t A -1 t-1 φ t = φ t (Φ t-1 R t-1 R T t-1 Φ T t-1 + αΠ T ) -1 φ t = 1 α φ t (Π T -Φ t-1 R t-1 (R T t-1 Φ T t-1 Φ t-1 R t-1 + αΠ T ) -1 R T t-1 Φ T t-1 )φ t = 1 α (φ T t φ t -φ T t Φ t-1 R t-1 (R T t-1 Φ T t-1 Φ t-1 R t-1 + αI t-1 ) -1 R T t-1 Φ T t-1 φ t ) = 1 α (k t,t -k T [t-1],t R t-1 (R T t-1 K t-1 R t-1 + αI t-1 ) -1 R T t-1 k [t-1],t ).
For yt , we have

yt = φ T t u t = φ T t A -1 t-1 Φ t-1 b t-1 = φ T t (Φ t-1 R t-1 R T t-1 Φ T t-1 + αΠ T ) -1 Φ t-1 b t-1 = 1 α φ T t Π T -Φ t-1 R t-1 (R T t-1 Φ T t-1 Φ t-1 R t-1 + αΠ T ) -1 R T t-1 Φ T t-1 Φ t-1 b t-1 = 1 α φ T t Φ t-1 D t-1 b t-1 -R t-1 (R T t-1 K t-1 R t-1 + αI t-1 ) -1 R T t-1 K t-1 b t-1 = 1 α k T [t-1],t D t-1 b t-1 -R t-1 (R T t-1 K t-1 R t-1 + αI t-1 ) -1 R T t-1 K t-1 b t-1 .
Proof of Theorem 5.7. Since the only thing that changed is the formulation of the A t matrix, the bound from Proposition 5.3 still applies. In particular, we have that the regret R T of Algorithm 13 is bounded as

R(w) ≤α w 2 A 0 + T t=1 g T t A -1 t g t + T t=1
(w tw) T ( A t -A t-1σ t g t g T t )(w tw).

From Theorem 3.11, we have that KORS succeeds with high probability. In particular, using the guarantees of the (ε, γ)-accuracy (1), we can bound for the case η t = σ as

g T t A -1 t g t = η t η t g T t (Φ t R t R T t Φ T t + αΠ T ) -1 g t = 1 η t φ T t (Φ t R t R T t Φ T t + αΠ T ) -1 φ t = 1 η t p min p min φ T t (Φ t R t R T t Φ T t + αΠ T ) -1 φ t = 1 η t 1 p min φ T t 1 p min Φ t R t R T t Φ T t + αΠ T -1 φ t ≤ 1 η t 1 p min φ T t Φ t S t S T t Φ T t + αΠ T -1 φ t ≤ 1 p min η t φ T t ((1 -ε)Φ t Φ T t -εαΠ T + αΠ T ) -1 φ t = 1 (1 -ε)σ p min φ T t (Φ t Φ T t + αΠ T ) -1 φ t = τ t,t (1 -ε)σ p min ,
where in the first inequality we used the fact that the weight matrix S t contains weights such that 1/ p min ≥ 1/ p t , in the second inequality we used the (ε, γ)-accuracy, and finally, we used η t = σ and the definition of τ t,t . Therefore, (w tw) T ( A t -A t-1σ t g t g T t )(w tw)

R G = T t=1 g T t A -1 t g t ≤ 1 (1 -ε)σ p min
= T t=1
(w tw) T η t q s q g t g T tσ t g t g T t (w tw)

≤ T t=1
(σσ t )(g T t (w tw)) 2 ≤ 0.

Proof of Theorem 5.8. PROS-N-KONS predicts y t in round t. We want to bound the cumulative regret of the loss of y t with respect to an arbitrary fixed vector w ∈ H on the (mapped) points φ t . From the definition of y t in Algorithm 14, we have

T t=1 t ( y t ) -t (φ T t w) = T t=1 t φ T t ω t -t (φ T t w) = T t=1 t φ T t Φ j U j Σ -1 j ω t -t (φ T t w) = T t=1
t (φ T t w t )t (φ T t w) , with w t = Φ j U j Σ -1 j ω t . From Assumption 2 on the losses, we know that the losses t (z) satisfy t (φ T w) ≥ t (φ T u) + ∇ t (φ T u)

T (wu) + σ t 2 ∇ t (φ T u) T (wu) 2 and therefore t (φ T u)t (φ T w) ≤ ∇ t (φ T u)

T (u -w) - σ t 2 ∇ t (φ T u) T (w -u) 2 = t (φ T u) φ T t (u -w) - σ t 2 t (φ T u) φ T t (w -u) 2 .

Definition of the epochs

We define the epochs j ∈ [J] that are separated by q t = 0, which indicate the dictionary change. Formally q t j = 0, and q t = 0 for all t j < t < t j + h j where h j = t j+1t j is the length of the epoch. We also use I j for the dictionary in the j-th phase, Φ I j for the feature matrix containing samples φ i ∈ I j , and Π I j = Φ I j (Φ T I j Φ I j ) + Φ T I j for the projection matrix on the column span of Φ I j . Similarly, given I j , we define the embedding ϕ j (•) such that ϕ j (x t ) = φ t = Σ -1 j U T j Φ T I j φ t ∈ R j and φ T t φ t = φ t Π I j φ t . Given the embedding ϕ j , we introduce the restricted RKHS H j and the approximate kernel function K j (•, •). Note that although the mapping ϕ j changes across different epochs, it is unique for a fixed round t, therefore we simply write φ t instead of the more explicit φ j t .

Regret decomposition

We introduce two intermediate comparators: w t and w t . First, w j is the best fixed solution within the epoch for the dictionary of the epoch and the related H j . Second, w j is the best fixed solution within the epoch over the whole space H. Notice, neither w j nor w j change during the same epoch. Formally, for all t such that t j < t ≤ t j + h j we have w t = w j and w t = w j , defined as Fix an epoch j. We now bound A j , B j , and C j separately.

w j =
Bounding A j Since w j ∈ H j , projecting w j to H j won't change it, i.e., w j = Π I j w j and φ T t w j = φ T t Π I j w j = φ T t Φ I j U j Σ -1 j Σ -1 j U T j Φ T I j w j = φ T t ω j with ω j = Σ -1 j U T j Φ T I j w j . Remembering that w t = Φ I j U j Σ -1 j ω t for all t in the epoch t (φ T t w t )t (φ T t w j ) = t φ T t Φ I j U j Σ -1 j ω tt φ T t Π I j w j = t φ T t ω tt φ T t ω j . Now, using Assumption 2, we get t ( φ T t ω t )t ( φ T t ω j ) ≤ g T t ( ω tω j ) -σ t 2 ( g T t ( ω tω j )) 2 5.6

and due to the update rules and the contracting property of projections [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF],

ω t -ω j 2 A t-1 ≤ ω t-1 -A -1 t-1 g t-1 -ω j 2 A t-1 = ω t-1 -ω j 2 A t-1
-2 g T t-1 A -1 t-1 A t-1 ( ω t-1ω j ) + A -1 t-1 g t-1

2 A t-1 = ω t-1 -ω j 2 A t-1
-2 g T t-1 ( ω t-1ω j ) + g T t-1 A -1 t-1 g t-1

except for last round of epoch j, i.e., round t j+1 -1, where due to the reset we cannot use the update rule, and we need to treat separately. Therefore, for any round t = t j , t j + 1, . . . , t j+1 -2 of epoch j, we have that 2 g T t ( ω tω j ) ≤ ω tω j 2 Atω t+1ω j 2 At + g T t A -1 t g t .

5.7

Using the upper bound on the loss difference 5.6, we get . Second, by Algorithm 14, we know that at the beginning of each epoch, A t j = αI + g t j g T t j . This also helps us to bound the term g T t j A -1 t j g t j as g T t j A -1 t j g t j = g T t j αI + σt j 2 g t j g T t j

-1

g t j = g T t j g t j α + σt j 2 g T t j g t j ≤ g T t j g t j α = t j y t j 2 φ T t j φ t j α ≤ L 2 φ T t j φ t j α = L 2 φ T t j Π I j φ t j α ≤ L 2 φ T t j φ t j α ≤ L 2 α •
By Algorithm 14, we also know that at the beginning of each epoch ω t j = 0 which helps us to bound the two terms outside of the summation as

ω t j -ω j 2 At j
ω t jω j 2 σ t j 2 gt j g T t j

= ω t jω j 2 αI + ω t jω j 2 σ t j 2 gt j g T t j

ω t jω j 2 σ t j 2 gt j g T t j

= α ω t jω j 2 2 = α ω j 2 2 .

Extended proofs 141

Altogether, we combine the upper bounds of the terms to t j+1 -1 t=t j t (φ T t w t )t (φ T t w t )

≤ α ω j 2 + R j + L 2 α -ω t j+1 -1 -ω j 2 A t j+1 -1 + t j+1 -2 t=t j +1 g T t A -1 t g t .
Using the result of [START_REF] Hazan | Logarithmic regret algorithms for online convex optimization[END_REF] we can upper bound the sum of the quadratic forms as where G j is the j × h j matrix with σ t /2 • g t columns. Let D j be the h j × h j diagonal matrix with ġt σ t /2 on the diagonal and Φ j (resp., Φ j ) the matrix with φ t (resp., φ t ) as columns for t j ≤ t < t j+1 We can rewrite G j = Φ j D j = Σ -1 j U T j Φ T I j Φ j D j . We also have G T j G j = D j Φ T j Φ I j U T j Σ -1 j Σ -1 j U T j Φ T I j Φ j D j = D j Φ T j Π I j Φ j D j D j Φ T I j Φ I j D j , since Π I j ≤ 1 because Π I j is a projection matrix. Knowing that Det(A) ≤ Det(B) whenever A B, together with Sylvester's determinant identity, we get that Det( G j G T j /α + I) ≤ Det( D j Φ T j Φ j D j /α + I) = h j t=1 (λ t /α + 1), where λ t are the eigenvalues of D j Φ T I j Φ I j D j = D j K j D j = K j and K j is the kernel matrix between the samples in epoch j. Using Lemma 3.9 we can further bound the expression above as We will now apply the following result from [START_REF] Xu | Efficient Non-oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee[END_REF].

t
Proposition 5.10 - (Xu et al., 2017, Lemma 2). Suppose the loss functions t are L-Lipschitz continuous, and w j = Π I j w j = Φ I j U j Σ -1/2 j ω j . We have

1 h j t j+1 -1 t=t j t (φ T t w j ) + α 2 w j 2 ≤ 1 h j t j+1 -1 t=t j t (φ T t w j ) + α 2 w j 2 + L 2 2αh j Φ j -Π I j Φ j 2 2 .
First, it is important to quantify the last term in Proposition 5.10,

Φ j -Π I j Φ j 2 2 = λ max Φ j -Π I j Φ j T Φ j -Π I j Φ j = λ max Φ T j Φ j -2Φ T j Π I j Φ j + Φ T j Π I j Π I j Φ j = λ max Φ T j Φ j -Φ T j Π I j Φ j = λ max K j -K j ≤ γε 1 -ε ,
when in the last step we applied Lemma 4.3 that bounds the quality of the approximation.

In order to apply Proposition 5.10 we also need to rescale B j ,

t

 

By the definition of w j as a minimizer, we have that the difference between the summations is negative or zero. Therefore, term C j is trivially bounded as

C j ≤ α w 2 -α w j 2 .
Bounding the regret: We put all the bounds on decomposed regret together: Proof of Theorem 5.9. In the special case of squared loss, we can obtain a different kind of guarantee. We proceed in a similar way as in the proof of Theorem 5.8 and highlight the differences. Starting from the A j + B j + C j decomposition, we will bound B j differently using the following result.

T t=1 t (φ T t w t ) -t (φ T t w) = J j=1 A j + B j + C j ≤ J j=1 4 σ min d T eff α σ min L 2 log 2σ min L 2 Tr(K j )/α + α w j 2 2 + R j + L 2 α -ω t j+1 -1 -ω j 2 A t j+1 -1 + α w j 2 -α w j 2 + L 2 ε 4(1 -ε)
Chapter 5. Sequential RLS Sampling in the Online Setting Proposition 5.11 - (Zhdanov and Kalnishkan, 2010, Theorem 1). Take a kernel K on a domain X and a parameter α > 0. Let H be the RKHS for the kernel K. For any sequence {(x t , y t )} T t=1 let y T ∈ R T be the concatenation of the y t target variables. Then

L * T (H) = min f ∈H T t=1 f (x t ) -y t 2 + α f 2 H = αy T T (K T + αI) -1 y T .
Bounding B j

In our particular case, we apply Proposition 5.11 to the whole space H and all subspaces H j , one for each epoch j.

L * j (H) = = -α w j 2 2 + αy T j (Φ T j Π I j Φ j + αI) -1 y j . Now using the kernel approximation guarantees of Lemma 4.3 we have αy T j (Φ T j Π I j Φ j + αI) -1 y j ≤ αy T j Φ T j Φ j -

γε 1 -ε I + αI -1 y j = α - ε 1 -ε γ -1 α α - ε 1 -ε γ y T j Φ T j Φ j + α - ε 1 -ε γ I -1 y j = α - ε 1 -ε γ -1
α α y T j Φ T j Φ j + α I -1 y j = (1 + ε )α y T j Φ T j Φ j + α I -1 y j . Integrating this with the bound for A j obtained in the proof of Theorem 5.8 we get

  Figure 2.1: Dictionary Learning for Covariance Matrices

  e θ -1)θgd) = exp {gddgd log (g)} = e -d exp {-g (log (g) -1) -1} .

Figure 3 . 1 :

 31 Figure 3.1: Different views of the Expand-Shrink cycle.

Figure 3

 3 Figure 3.2: Merge tree for Alg. 10 with an arbitrary partitioning and merging scheme.

.

  Given two disjoint datasets D, D , and two (ε, γ)-accurate dictionaries I D , I D , let I * = I D ∪ I D and S * be the associated selection matrix. Let Φ D * be the map constructed using D * = D ∪ D , and τ D∪D ,i the RLS of φ i w.r.t. D * . Compute the approximate RLSs τ i as

Figure 3

 3 Figure 3.3: Merge trees for Algorithm 10.

Figure 3

 3 Figure 3.4: Stretching a merge tree.

3. 4

 4 Proof of main results 67 computed by SQUEAK at each node. Finally, we will also denote with d {h,l} eff (γ) = d D {h,l} eff (γ) the effective dimension of D {h,l} .

Figure 3 . 5 :

 35 Figure 3.5: The dependency graph of the considered variables. Red variables are random. Black variables are deterministically computed using their input (a function of their input), with bold lines indicating the deterministic (functional) relation. Blue variables are constants. A grey filling indicates that a random variable is observed or a function of observed variables.

  Proposition 3.21[START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF], Theorem 5.1.1. Consider a finite sequence {X k : k = 1, 2, 3, . . . } whose values are independent, random, PSD Hermitian matrices with dimension d. Assume that each term in the sequence is uniformly bounded in the sense that λ max (X k ) ≤ L almost surely for k = 1, 2, 3, . . . .

Definition 4. 3

 3 The effective resistance of an edge e in graph G is defined as r e = b T e L + G b e . The total weighted sum of effective resistances in a graph is the same for all graphs, and is equal to e a e r e = Tr(E

  10:p s,e ← min{ p s-1,e , p s,e }

Figure 4

 4 Figure 4.1: (a) The dataset of the synthetic experiment, (b) Accuracy of Stable-HFS and Sparse-HFS, (c) ratio of the number of edges |H|/|G|.

Figure 4 . 2 :

 42 Figure 4.2: Accuracy vs complexity on the TREC 2007 SPAM Corpus for different number of labels. Legend: Sparse-HFS, EigFun, SubSampling, 1-NN

5. 1

 1 Online kernel learning and the kernelized online Newton step 119 Algorithm 12 Kernelized online Newton step (KONS) Input: Feasible parameter C, stepsizes η t , regulariz. α 1: Initialize w 0 = 0, g 0 = 0, b 0 = 0, A 0 = αΠ T 2: for t = {1, . . . , T } do

  4. We need to bound R T (w * ), and we use Proposition 5.3. For R D nothing changes from the parametric case, and we use Assumption 1 and the definition of the set S to boundR D = T t=1 (η tσ t ) ġ2 t (φ T t (w tw)) 2 ≤ T t=1 (η tσ)L 2 (|φ T t w t | + |φ T t w|) 2 ≤ 4L 2 C 2 T t=1(η tσ).

  = Tr(K T (K T + αI T ) -1 ) = Tr(DK T D(DK T D + αDD -2 D) -1 ) = Tr(DK T DD -1 (K T + αD -2 ) -1 D -1 ) = Tr(K T I T (K T + αD -2 ) -1 D -1 D) = Tr(K T (K T + αD -2 ) -1 ) ≤ Tr(K T (K T + αλ min (D -2 )I T ) -1 ) ≤ Tr K T K T + α σL 2 I T -1 = d T eff α/(σL 2 ) .

  T /α + 1) ≤ log(Tr(K T )/α + 1) ≤ log(σL 2 Tr(K t )/α + 1)

  ε)σ max{τ min , γ} • To bound R D , we have T t=1

.

  L 2 log 2σ min L 2 Tr(K j )/α ≤ 2d T eff α σ min L 2 log 2σ min L 2 Tr(K t )/α .Putting it all together, and using ω j min L 2 log 2σ min L 2 Tr(K j )/α + α w j2 2 + R j + L 2 α ω t j+1 -1ω j 2 A t j+1 -1

  L 2 log 2σ min L 2 Tr (K j ) L 2 log 2σ min L 2 T /α + α w +

  the B j part of the regret we getB j ≤ -α w jChanging slightly the regularizers that we add and subtract in the bound on C j we obtain

  L 2 log 2σ min L 2 Tr(K j )/α + ε L *

  ). Let {z s } t s=1 be independent Bernoulli random variables, each with success probability p s , and denote their sum as d =

							Chapter 2. Dictionary Learning for Covariance Matrices
	t s=1 p s ≥ 1. Then,
					t
			P	s=1	z s ≥ gd ≤ e -d exp {-g (log (g) -1) -1}
	Proof of Proposition 2.13. We can bound the sum t s=1 z s with a Hoeffding-like bound
	using Markov inequality,
	P	t s=1	z s ≥ gd = inf θ>0	P e	t s=1 θzs ≥ e θgd
		≤ inf θ>0	E e	t s=1 θzs e θgd	= inf θ>0	E	t s=1 e θzs e θgd	= inf θ>0	t s=1 E e θzs e θgd
		= inf θ>0		t s=1 (p s e θ + (1 -p s )) e θgd	= inf θ>0	t s=1 (1 + p s (e θ -1)) e θgd
		≤ inf θ>0		t s=1 e ps(e θ -1) e θgd	≤ inf θ>0	e (e θ -1) t s=1 ps e θgd	= inf θ>0	e (d(e θ -1)-θgd)

  2 is ρ-accurate, with ρ = (1 + ε)/(1ε). Moreover, τ i can be computed in O(|I| 3 ) time and O(|I| 2 ) space using only φ i and atoms present in I itself.This estimator follows naturally from the reformulation of the RLS presented in Lemma 2.8. Ideally we would like to keep all ΦΦ T columns in the inverse (ΦΦ T + γΠ) -1 for maximum accuracy, but applying Lemma 2.8 would give us an R n×n matrix to invert, which requires O(n 3 ) time and space. Instead, similarly to what[START_REF] Cohen | Online Row Sampling[END_REF] did for the special Euclidean case φ i = x i , we only keep a reweighted subset (ΦSS T Φ T + γΠ) so that after applying Lemma 2.8 and eliminating the zeroed-out columns (Proposition 2.12) we are left with only a |I| × |I| matrix that can be inverted efficiently in O(|I| 3 ) time and O(|I| 2 ) space.

  p.1δ Accuracy: I and I are (ε, γ)-accurate Space: the size of the dictionaries are bounded by |I| ≤ 3q 1 and |I| ≤ 3q 2 d eff (γ)

Table 3

 3

.1: Comparison of all sampling algorithms.

Table 4 .

 4 1: Guarantees and Computational complexities. Bold text indicates unfeasible time or space complexity. Guarantees unavailable. Simple-HFS's guarantees require assumptions on the graph G.

  ONS is equivalent to vanilla projected gradient descent, which

			Chapter 5. Sequential RLS Sampling in the Online Setting
	in OL achieves O(	√	dT ) regret

  from O(d log T ) down to O(d T eff (α) log T ) ≤ O(r log T ), where r is the (unknown) rank of the dataset. Furthermore, when σ = 0, setting η t = 1/(L 2 C 2 t) gives us a regret

	O(d T eff (α) if an oracle provided us in advance with d T √ T ), which is potentially much smaller than O(r eff (α), setting η t √	T ) and O(	√	T d). Furthermore,

  Proof sketch: Given these guarantees, we need to bound R G and R D . Bounding R D Chapter 5. Sequential RLS Sampling in the Online Setting is straightforward, since by construction Sketched-KONS adds at most η t g t g T t to A t at each step. To bound R G instead, we must take into account that an unweighted A t = Φ t R t R T αΠ T can be up to p min distant from the weighted Φ t S t S T t Φ T t for which we have guarantees. Hence the max{β, τ min } term appearing at the denominator. The full proof is available in Section 5.4.

2 and the algorithm runs in O(d t eff (α) 2 + t 2 β 2 ) time and O(d t eff (α) 2 + t 2 β 2 ) space complexity for each iteration t. t Φ T t +

  1 requires an additional O(B 2 t ) time. When β ≤ d t eff (α)/t, each iteration takes O(d t eff (α) 2 ) to compute τ t,t incrementally using KORS, O(d t eff (α) 2 ) time to update A -1 t and O(d t eff (α) 2 ) time to compute [b t ] t . When β > d t eff (α)/t, each iteration still takes O(d t eff (α) 2 ) to compute τ t,t using KORS and O(t 2 β 2 ) time to update the inverse and compute [b t ] t .

  Algorithm parkinson n = 5, 875, d = 20 cpusmall n = 8, 192, d = 12 Avg. Squared Loss #SV Time Avg. Squared Loss #SV Time = 53, 500, d = 385 year n = 463, 715, d = 90 Avg. Squared Loss #SV Time Avg. Squared Loss #SV Time

	FOGD	0.04909 ± 0.00020	30	-	0.02577 ± 0.00050	30	-
	NOGD	0.04896 ± 0.00068	30	-	0.02559 ± 0.00024	30	-
	PROS-N-KONS	0.05798 ± 0.00136	18	5.16	0.02494 ± 0.00141	20	7.28
	Con-KONS	0.05696 ± 0.00129	18	5.21	0.02269 ± 0.00164	20	7.40
	B-KONS	0.05795 ± 0.00172	18	5.35	0.02496 ± 0.00177	20	7.37
	BATCH	0.04535 ± 0.00002	-	-	0.01090 ± 0.00082	-
	Algorithm	cadata n = 20, 640, d = 8 Avg. Squared Loss #SV Time Avg. Squared Loss #SV Time casp n = 45, 730, d = 9
	FOGD	0.04097 ± 0.00015	30	-	0.08021 ± 0.00031	30	-
	NOGD	0.03983 ± 0.00018	30	-	0.07844 ± 0.00008	30	-
	PROS-N-KONS	0.03095 ± 0.00110	20	18.59	0.06773 ± 0.00105	21	40.73
	Con-KONS	0.02850 ± 0.00174	19	18.45	0.06832 ± 0.00315	20	40.91
	B-KONS	0.03095 ± 0.00118	19	18.65	0.06775 ± 0.00067	21	41.13
	BATCH	0.02202 ± 0.00002	-	-	0.06100 ± 0.00003	-	-
	Algorithm slice n FOGD 0.00726 ± 0.00019	30	-	0.01427 ± 0.00004	30	-
	NOGD	0.02636 ± 0.00460	30	-	0.01427 ± 0.00004	30	-
	Dual-SGD	-	-	-	0.01440 ± 0.00000	100	-
	PROS-N-KONS	did not complete	-	-	0.01450 ± 0.00014	149	884.82
	Con-KONS	did not complete	-	-	0.01444 ± 0.00017	147	889.42
	B-KONS	0.00913 ± 0.00045	100	60	0.01302 ± 0.00006	100	505.36
	BATCH	0.00212 ± 0.00001	-	-	0.01147 ± 0.00001	-	-

Table 5

 5 Algorithmijcnn1 n = 141, 691, d = 22 cod-rna n = 271, 617, d = 8

		Accuracy	#SV	Time	Accuracy	#SV Time
	FOGD	9.06 ± 0.05	400	-	10.30 ± 0.10	400	-
	NOGD	9.55 ± 0.01	100	-	13.80 ± 2.10	100	-
	Dual-SGD	8.35 ± 0.20	100	-	4.83 ± 0.21	100	-
	PROS-N-KONS	9.70 ± 0.01	100	211.91	13.95 ± 1.19	38	270.81
	Con-KONS	9.64 ± 0.01	101	215.71	18.99 ± 9.47	38	271.85
	B-KONS	9.70 ± 0.01	98	206.53	13.99 ± 1.16	38	274.94
	BATCH	8.33 ± 0.03	-	-	3.781 ± 0.01	-	-
		α = 0.01, γ = 0.01
	Algorithm	ijcnn1 n = 141, 691, d = 22 Accuracy #SV Time	cod-rna n = 271, 617, d = 8 Accuracy #SV Time
	FOGD	9.06 ± 0.05	400	-	10.30 ± 0.10	400	-
	NOGD	9.55 ± 0.01	100	-	13.80 ± 2.10	100	-
	Dual-SGD	8.35 ± 0.20	100	-	4.83 ± 0.21	100	-
	PROS-N-KONS	10.73 ± 0.12	436	1003.82	4.91 ± 0.04	111	459.28
	Con-KONS	6.23 ± 0.18	432	987.33	5.81 ± 1.96	111	458.90
	B-KONS	4.85 ± 0.08	100	147.22	4.57 ± 0.05	100	333.57
	BATCH	5.61 ± 0.01	-	-	3.61 ± 0.01	-	-

α = 1, γ = 1 .2: Binary classification datasets

  arg minUsing the above intermediate comparators now split the regret into three parts, one per epoch,T t w t )t (φ T t w t ) + t (φ T t w t )t (φ T t w t ) + t (φ T t w t )t (φ T t w) T t w t )t (φ T t w t ) + t (φ T t w t )t (φ T t w t ) + t (φ T t w t )t (φ T t w)

					t j+1 -1			t j+1 -1
				w∈H j	t=t j	t (φ T t w) + α w 2 , and w j = arg min w∈H	t=t j	t (φ T t w) + α w 2 .
	T
	t=1	t (φ T t w t ) -t (φ T t w)
			T
	= J t (φ = t=1 tj+1-1 j=1 t=tj J tj+1-1 t (φ = t (φ T
	j=1	t=tj
		J		tj+1-1				 tj+1-1		 tj+1-1
	=			t (φ T t w t )-t (φ T t w j )	 + 	t (φ T t w j )-t (φ T t w j )  + 	t (φ T
	j=1		t=tj				t=tj	t=tj
					Aj			Bj

t w t )t (φ T t w j ) + t (φ T t w j )t (φ T t w j ) + t (φ T t w j )t (φ T t w) t w j )t (φ T t w)   Cj .

  T t w t )t (φ T t w t ) = t j+1 -1 φ T t j+1 -1 w t j+1 -1t j+1 -1 φ T t j+1 -1 w t j+1 -1 + φ T t ω tt φ T t ω j ≤ R j + j+1 -1 φ T t j+1 -1 w t j+1 -1t j+1 -1 φ T t j+1-1 w t j+1 -1 that corresponds to the regret of the last round of the epoch that we need to treat separately. Now for the all other rounds, Now we treat the terms above equation separately. First, since A t = A t-1 + σt 2 g t g T t , we have ω tω j

	t j+1 -1
	t=t j	t (φ t j+1 -2
								t=t j	t (φ T t w t ) -t (φ T t w t )
	= R j +	t j+1 -2 t=t j				t j+1 -2 t=t j	g T t ( ω t -ω j ) -	σ t 2	( g T t ( ω t -ω j )) 2 ,
	with R j = t t j+1 -2 def t=t j g T		σ t 2	( g T
								2 σ t 2 gt g T t
		= -ω t j+1 -1 -ω j	2 A t j+1 -1	+ ω t j -ω j	2 At j	+ g T t j A -1 t j g t j -ω t j -ω j	2 σ t j 2 gt j g T t j
				t j+1 -2
			+	t=t j +1	ω t -ω j	2 At -ω t -ω j	2 A t-1	+ g T t A -1 t g t -ω t -ω j	2 σ t 2 gt g T t
				2 A t-1	+ ω t -ω j	2 σ t 2 gt g T t	= ω t -ω j	2 At

t t ( ω tω j )t ( ω tω j )) 2 ≤ t j+1 -2 t=t j ω tω j 2 Atω t+1ω j 2 At + g T t A -1 t g tω tω j

  Therefore, the difference of the regularized losses for the best solution within epoch j when considering the whole space H versus subspace H j is bounded as Similarly as for B j , we add and subtract the regularizers

	5.4 Extended proofs									143
	Bounding C j
	t j+1 -1									t j+1 -1		 t j+1 -1	
		t=t j	t (φ T t w j ) -t (φ T t w) =		t=t j	t (φ T t w j )  -		t=t j	t (φ T t w)	
							 t j+1 -1					t j+1 -1
	= α w 2 -α w j	2 +		t=t j		t (φ T t w j ) + α w j	2	 -		t=t j	t (φ T t w) + α w 2
	t=t j	t (φ T t w j ) + α w j	2 = h j	  1 h j	t j+1 -1 t=t j	t (φ T t w j ) +	α 2	2 h j	w j	2	 
							≤ h j	  1 h j	t j+1 -1 t=t j	t (φ T t w j ) +	α 2	2 h j	w j	2 +	L 2 h j 4αh j	Φ j -Π I j Φ j	2 2	 
							=	t j+1 -1 t=t j	t (φ T t w j ) + α w j	2 +	L 2 h j 4α	K j -K j	2 2
							≤	t j+1 -1 t=t j	t (φ T t w j ) + α w j	2 +	L 2 ε 4(1 -ε)	h j γ α	•
		  t j+1 -1 t=t j	t (φ T t w j ) + α w j	  -	 	t j+1 -1 t=t j	t (φ T t w j ) + α w j	  ≤	L 2 ε 4(1 -ε)	h j γ α
	and therefore their unregularized counterparts are bounded as
						B j ≤ α w j	2 -α w j	2 +	L 2 ε 4(1 -ε)	h j γ α	•

j+1 -1

  Therefore, taking account for the regularization in Proposition 5.11, for any epoch j,

	t j+1 -1
	t=t j			φ T t w j -y t	2 + α w j	2 2
			t j+1 -1				
	= min w∈H			t=t j	φ T t w -y t	2 + α w 2 2	 = αy T j (Φ T
	t j+1 -1
	t=t j			φ T t w j -y t	2 + α w j	2 2
			 t j+1 -1				
	= min w∈H j		t=t j	φ T t w -y t	2 + α w 2 2	 = αy T j (Φ T
	t j+1 -1								t j+1 -1
	t=t j		φ T t w j -y t	2 = -α w j	2 2 +	t=t j	φ T t w j -y t	2 + α w j	2 2

j Φ j + αI) -1 y j , L * j (H j ) = j Π I j Φ j + αI) -1 y j .

i.e., not necessarily according to a sampling distribution.

Since the matrix XX T is PSD, non-negative weights are sufficient for reconstruction.

This approach is equivalent to learning both atoms and weights, since we can always rescale the learned atoms to include any necessary weight.

In particular, drawing q elements from a Multinomial correspond to the sequence of Binomial draws q1 ∼ B(p1, q), q2|q1 ∼ B(p2/(1 -p1), q -q1), . . . , qn-1 ∼ B(pn-1/(1 -n-2 i=1 pi), q -n-2 i=1 qi)

i.e., we do not assume the points are sampled from a distribution, but since they are part of a dataset we still refer to them as samples.

A preliminary version of the paper,(Musco and Musco,

2016), was available while[START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF] was under submission.(Calandriello et al., 2017a) and(Calandriello et al., 2017c) were developed and submitted independently from this preliminary version.

Uniform sampling (i.e., Proposition 2.3) would give us the same result, since nµ = n maxi τi(n) ≤ n 1 1+n ≤ 1.

t i=1 τ i,i = t i=1 τ t,i = d t eff (γ) coincide.But in the opposite case when all samples are identical τ 1,1 = 1 1+γ while τ t,1 = 1 t+γ , and as a result, τ 1,1 contributes to the sum n times more than it should. Nonetheless, remaining in the all-identical sample case, if we sum 3 see Definition 2.8

& & & & p n-1,t p n-1,t p n,t .Since q n,t = q j=1 z n,t,j is the sum of q Bernoullis with probabilities p n,t = τ n,t , it is by definition a Binomial B(τ n,t , q). Note that, since q t,i depends on q t-1,i , to study the dictionaries I t we would in general need to use results for random processes. Fortunately, we know that each q t,i is also

Even replacing sequential Binomial sampling with Multinomial cannot reduce this complexity, since drawing a single sample from a Multinomial over n objects still takes Ω(n) time.

Drawing a sample from a Binomial B(p, q) can be implemented in O(pq) w.h.p.. Since in Uniform we have probabilities that sum to one, the overall sampling time is O(q).

Note once again that although the sampling process is random, the merge tree is deterministic and fixed in advance.

Notice that unlike zs,i,j, ws,i,j is no longer Fs-measurable but it is F s -measurable, where F {s,i,j} = u s ,i ,j , ∀{s , i , j } ≤ {s, i, j} ∪ {ws,i,j} = F {s,i,j} ∪ {ws,i,j} .

The graph G can be either constructed from raw data (e.g., building a k-nn graph with an exponential kernel) or it can be provided directly as input (e.g., in social networks).

While the computational complexity of exact HFS is O(mn), many approximated methods can significantly reduce it. Nonetheless, any method that requires reading all the edges once has at least Ω(m) time complexity.

http://plg.uwaterloo.ca/~gvcormac/treccorpus07/

This can be easily seen as d T eff (α) = t λt/(λt + α), where λt are the eigenvalues of KT .

Here we ignore the term d T eff (α) which is a constant w.r.t. T for any constant α.

Acknowledgements

Chapter 4. Sequential RLS Sampling in the Batch Setting

Efficient learning with kernels

In this section, we show how (ε, γ)-accuracy and the guarantees of Lemma 4.3 translate into accuracy guarantees for typical kernel methods. As a consequence, the time and space guarantees of KORS or SQUEAK can be used to find provably accurate approximate solution faster than other existing methods.

In particular, we focus on regularized statistical problems: kernel ridge regression and k-rank kernel PCA. As we will see, the additive error we introduce using our kernel matrix approximation will not disturb the solution of our regularized statistical problem if the error threshold is comparable to the regularization.

For more results on kernel K-means, kernel CCA, and alternative proofs on how an (ε, γ)accurate dictionary can be used to compute a provably accurate approximate solution we refer the reader to [START_REF] Musco | Provably useful kernel matrix approximation in linear time[END_REF].

Kernel ridge regression

Consider now a regression dataset D = {(x t , y t )} n t=1 , with input x t ∈ X and output y t ∈ R. The goal of sequential kernel ridge regression is, for any time t ∈ [n], to find the vector ω t ∈ R t that minimizes the regularized quadratic loss

where λ > 0 is a regularization parameter. This objective admits the closed form solution

4.3

In batch regression, ω n is computed only once when all the samples of D are available, solving the linear system in Eq. 4.3 with K n , but in practice we might be collecting our dataset incrementally, and want to periodically construct an intermediate batch solution using the samples seen so far.

In the fixed-design kernel regression problem, the accuracy of the solution ω n is measured by the prediction error on the input set D itself. More precisely, the prediction of the estimator ω n in each point is obtained as [K n ω n ] i , while the outputs y i in the dataset are assumed to be a noisy observation of an unknown target function f * : X → R, evaluated in x i i.e., for any i ∈ [n],

where i is a zero-mean i.i.d. noise with bounded variance σ 2 . Let f * ∈ R n be the vector with components f * (x i ), then the empirical risk of ω n on dataset D is measured as

Given a dictionary I n that is (ε, γ)-accurate w.r.t. Φ n , we can use Lemma 4.3 to compute an approximate K n to replace K n in Eq. 4.3. Taking for example the regularized version of K with α = γ, and exploiting the Woodbury formula we compute the regression weights as

Chapter 4. Sequential RLS Sampling in the Batch Setting Sparse-HFS. Replacing the graph G with the sparsifier produced by Algorithm 11 (i.e., fully sequential SQUEAK), we can compute an approximate Stable-HFS solution as f = (ηlL H + I S ) + (y S -µ1), with µ = ((ηlL H + I S ) + y S ) T 1/((ηlL H + I S ) + 1) T 1. Since both G and H are connected graphs, they induce the same projection matrix Π F , and the Sparse-HFS solution can be rewritten as f = Π F (ηlL

Using the runtime guarantees of SQUEAK and of the near-linear solver by [START_REF] Koutis | A Nearly-m log n Time Solver for SDD Linear Systems[END_REF] These computational improvements have only a limited impact on the spectrum of G and all its eigenvalues are approximated up to a (1 ± ε) factor. Moreover, all of the sparsification guarantees hold w.h.p. for any graph, regardless of how it is generated, its original spectrum, and more importantly regardless of the exact order in which the edges are assigned to the blocks. Finally, we notice that the choice of the number of blocks as m/q is crucial to guarantee a logarithmic amortized time, since each iteration takes O(nq log(n)) time. This property allows to directly apply Sparse-HFS in online learning settings where edges arrive in a stream and intermediate solutions have to be computed incrementally.

Bounding the stability

In the following, we show that, unlike other heuristics, the space complexity improvements obtained with sparsification come with guarantees and do not degrade the actual learning performance of HFS. The analysis of SSL algorithms is built around the algorithm stability theory [START_REF] Bousquet | Stability and generalization[END_REF], which has been extensively used to analyse transductive learning algorithms [START_REF] Cortes | Stability of transductive regression algorithms[END_REF][START_REF] El-Yaniv | Stable transductive learning[END_REF]. We first remind the definition of algorithmic stability.

Definition 4.4 Let L be a transductive learning algorithm. We denote by f and f the functions obtained by running L on datasets X = (S, T ) and X = (S , T ) respectively. L is uniformly β-stable w.r.t. the squared loss if there exists β ≥ 0 such that for any two partitions (S, T ) and (S , T ) that differ by exactly one training (and test) point and for all x ∈ X ,

Define the empirical error as

2 and the generalization error as

Theorem 4.8 - [START_REF] Calandriello | Large-scale semisupervised learning with online spectral graph sparsification[END_REF]. Let G be a fixed (connected) graph with n nodes X and m edges E and eigenvalues 0 = λ 1 (G) < λ 2 (G) ≤ . . . ≤ λ n (G). Let y ∈ R n be the labels of the nodes in G with |y(x)| ≤ M and F be the set of centered functions such that |f (x)y(x)| ≤ c. Let S ⊂ X be a random subset of labeled nodes. If the corresponding labels y S are centered and Sparse-HFS is run with parameter ε, then w.p. at least 1δ (w.r.t. the random generation of the sparsifier H and the random Chapter 5. Sequential RLS Sampling in the Online Setting KONS up to a multiplicative factor depending on the number of epochs J and with a much smaller time and space complexity that adapt to the effective dimension of the problem (see Theorem 3.11).

Experiments on online regression and classification

We empirically validate PROS-N-KONS on several regression and binary classification problems, showing that it is competitive with state-of-the-art methods. We focused on verifying 1) the advantage of second-order vs. first-order updates, 2) the effectiveness of data-adaptive embedding w.r.t. the oblivious one, and 3) the effective dimension in real datasets. Note that our guarantees hold for more challenging (possibly adversarial) settings than what we test empirically.

Algorithms. Beside PROS-N-KONS, we introduce two heuristic variants. Con-KONS follows the same update rules as PROS-N-KONS during the descent steps, but at reset steps it does not reset the solution and instead computes w t-1 = Φ j-1 U j-1 Σ -1 j-1 ω t-1 starting from ω t-1 and sets ω t = Σ -1 j U T j Φ T j w t-1 . A similar update rule is used to map A t-1 into the new embedded space without resetting it. B-KONS is a budgeted version of PROS-N-KONS that stops updating the dictionary at a maximum budget J max and then it continues learning on the last space for the rest of the run. Finally, we also include the best BATCH solution in the final space H J returned by KORS as a best-in-hindsight comparator. We also compare to two state-of-the-art embedding-based first-order methods from [START_REF] Lu | Large Scale Online Kernel Learning[END_REF]. NOGD selects the first J points and uses them to construct an embedding and then perform exact GD in the embedded space. FOGD uses random feature expansion to construct an embedding, and then runs first-order GD in the embedded space. While oblivious embedding methods are cheaper than data-adaptive Nyström, they are usually less accurate. Finally, Dual-SGD also performs a random feature expansion embedding, but in the dual space. We do not run Sketched-KONS because the T 2 runtime is prohibitive in large datasets.

Experimental setup. We replicate the experimental setting in [START_REF] Lu | Large Scale Online Kernel Learning[END_REF] with 9 datasets for regression and 3 datasets for binary classification. We preprocess the points x t by rescaling each feature in [0, 1]. For regression the target variable y t is rescaled in [0, 1], while in binary classification the labels are {-1, 1}. For all datasets, we set β = 1, ε = 0.5 for all PROS-N-KONS variants and J max = 100 for B-KONS. We use the same Gaussian kernel with bandwidth set to 8. For each algorithm and dataset, we report average and standard deviation of the losses, the number of SVs stored in the predictor (as a proxy for space complexity) and runtime in seconds. The scores for the competitor baselines are reported as provided in the original papers [START_REF] Le | Dual Space Gradient Descent for Online Learning[END_REF][START_REF] Lu | Large Scale Online Kernel Learning[END_REF]. We only report scores for NOGD, FOGD and Dual-SGD, since they have been shown to outperform other baselines such as Budgeted Perceptron [START_REF] Cavallanti | Tracking the best hyperplane with a simple budget perceptron[END_REF], Projectron [START_REF] Orabona | The projectron: a bounded kernel-based perceptron[END_REF], Forgetron [START_REF] Dekel | The forgetron: A kernel-based perceptron on a budget[END_REF] and Budgeted-GD [START_REF] Zhao | Fast bounded online gradient descent algorithms for scalable kernel-based online learning[END_REF]. All experiments are run on a single machine with 2 Xeon E5-2630 CPUs for a total of 10 cores, and are averaged over 15 runs.

Effective dimension and runtime. We use size of the dictionary returned by KORS as a proxy for the effective dimension of the datasets. As expected, larger datasets and datasets with a larger input dimension have a larger effective dimension. Furthermore, d T eff (γ) increasing (sublinearly) when γ is reduced from 1 to 0.01 in the ijcnn1 dataset. More importantly, d T eff (γ) remains empirically small even for datasets with hundreds of thousands samples, such as year, ijcnn1 and cod-rna. On the other hand, in the slice