Keywords: analyse des séries temporelles, modèles de régression, analyse de données distribuées, données matricielles, graph learning time series analysis, regression models, distributional data analysis, matrix-variate data, graph learning

Analyse statistique de données spatio-temporelles et multidimensionnelles issues d'un réseau de capteurs Résumé Cette thèse porte sur l'analyse statistique des séries chronologiques multivariées de différentes natures de données, enregistrées par un réseau de capteurs. L'objectif général est de développer des approches permettant d'explorer et de représenter la dépendance spatiale des séries chronologiques multivariées. Nous nous intéressons tout d'abord à l'identification d'une telle structure de dépendance et à sa représentation par un graphe. En particulier, nous considérons la modélisation des données matricielles et distribuées, c'est-à-dire à chaque instant du temps un vecteur ou une distribution est observé par un des capteurs dans le réseau. À cette fin, nous proposons deux nouveaux modèles autorégressifs (AR) où les graphes gouvernent la dépendance régressive, pour caractériser respectivement les séries chronologiques matricielles et celles multivariéesdistribuées. En fittant les modèles aux données, les graphes de dépendance peuvent ensuite être inférés. Pour le modèle AR matriciel, nous nous concentrons sur online learning. Nous considérons notamment la tendance dans le modèle de données en tant que paramètres supplémentaires, puis nous proposons des algorithmes online qui peuvent mettre à jour les estimations du graphe et de la tendance simultanément lors de l'arrivée de nouvelles observations. Pour le modèle AR multivarié-distribué, nous nous appuyons sur les statistiques dans l'espace de Wasserstein pour traiter les objets non-euclidiens de données. Les modèles de régression proposés aident à la prévision de données futures, et les graphes inférés aident à la représentation des données et à l'analyse ultérieure. Par ailleurs, nous étudions les séries chronologiques vectorielles qui sont observées sur un réseau. Ceci est motivé par le fait que, dans de nombreuses applications, les observations sur un réseau présentent une forte dépendance entre les noeuds, ce qui rend les données sur un sous-ensemble de noeuds hautement prévisibles par les données sur les autres. Nous sommes donc intéressés par l'évaluation et le classement de la prévisibilité des noeuds d'un réseau. En guise d'application, les classements dérivés comme stratégies data-drivens servent à la sélection des capteurs. De ce point de vue, la prise en compte du réseau neurone comme méthode de reconstruction est innovante.

First of all, I would like to express my gratitude to my supervisor Jérémie Bigot. He gave me this great opportunity to do a thesis on statistical learning and machine learning, which have been my passion since master. His rich knowledge in this domain is the inevitable key to my transition from a classical-model user to an advancedmodel creator. In the most difficult periods where, externally, there broke out Covid pandemic, internally, my work had been stuck, it was his patience, responsibility and encouragement that helped me overcome the obstacles, and reached a higher level of research ability. In addition to the help in research, he has been willing to practice French with me. I am very grateful for this kindness, especially in the beginning, I had to stop and search the words during a conversation. Without that each time of discussion in French, I would not have turned fluent in this language during my thesis.

Secondly, I would like to thank my co-supervisor Sofian Maabout for his guidance, support and encouragement. He has been doing his best to help me even though our domains are not exactly the same. I enjoyed very much each of our discussions. Plus, the dinners I spent with his family are those of the warmest memories in Bordeaux.

Further, I would like to thank the referees Sophie Achard and Alexander Petersen for reporting my thesis. Despite their charged schedules, they agreed to be my reporters, that is a big affirmation to me. I appreciate it very much. I would also like to thank members of the jury Bernard Bercu, Fadila Bentayeb, and Nicolas Tremblay for their support with the defense.

Next, I would like to thank Professor Qing Liu. As the coordinator of the bilateral master program that I attended, he offered me the opportunity to exchange in France. Since then and during the following years in master and Ph.D, he did not guide me in an explicit way, but he pays attention to my achievements in his own way. He feels like a sensor member in my own family. Now, I would like to thank my best of best friends, Abhinandan, Yingjing, and Yulin (in alphabetical order). They are like my stars, lighthouses and harbours, whenever I feel discouraged or lost, they always raise me up. They always make up my daily happiness with our causal conversations, even now when we are not physically closed anymore.

Next I would like to thank Paul, Nicoletta. Thank you for accompanying me all these years at Bordeaux, and our differentiable tears jokes ! Big thank-you also to Fatbio! My first collaborator. We were attending SEME, but during whihch it came lockdown, the inner/outer conditions of me were very tough. But you said it was totally OK, and you supported me a lot. We are friends until today. You are 1 always very kind ! And especially thank you for bringing me to the châteaux for wine visiting, even though every time I got super carsick. I will keep in heart all the appellations in Medoc ! I would also like to thank my colleague-friends: Theo, Lara, Baptiste, Emanuele, Lan, Hui, Rolando, Gaston, Pei, Gautier, Issa, Jean, Elsa (thank you for introducing me to the IMB community in the first day, without this first sentence, the story would have not been developed), Vasileios, Marc'Anto. You help me to integrate such that I am not an isolated node. Thanks very much ! Chapter 1

Introduction (Français)

1.1 Données enregistrées sur un réseau de capteurs Les données enregistrées par un réseau de capteurs sont devenues de plus en plus populaires ces dernières années avec des applications dans de nombreux domaines tels que l'analyse du trafic [START_REF] Crovella | Graph wavelets for spatial traffic analysis[END_REF][START_REF] Yao | Deep multi-view spatial-temporal network for taxi demand prediction[END_REF][START_REF] Fang | Gstnet: Global spatial-temporal network for traffic flow prediction[END_REF], l'analyse du réseau cérébral [START_REF] Huang | A graph signal processing perspective on functional brain imaging[END_REF][START_REF] Wang | The sylvester graphical lasso (syglasso)[END_REF], météorologie [START_REF] Handcock | An approach to statistical spatial-temporal modeling of meteorological fields[END_REF][START_REF] Mei | Signal processing on graphs: Causal modeling of unstructured data[END_REF][START_REF] Xu | Spatio-temporal interpolated echo state network for meteorological series prediction[END_REF]. Les réseaux de capteurs peuvent également se situer dans un espace abstrait, qui n'est pas lié à des entités physiques, comme le réseau social [START_REF] Tabassum | Social network analysis: An overview[END_REF], le réseau de citations (Liu et al., 2019), et le réseau sémantique [START_REF] Lake | Discovering structure by learning sparse graphs[END_REF][START_REF] Sarica | Technet: Technology semantic network based on patent data[END_REF]. Ces capteurs enregistrent souvent une séquence d'observations dans le temps à une fréquence régulière, qui correspond à l'évolution du trafic ou de la météo par exemple. De telles observations sont donc dotées d'une nature à la fois spatiale et temporelle. Outre l'aspect spatio-temporel, l'observation nodale de chaque capteur peut également présenter une représentation riche en données. Trois types de données courants sont scalaire, vecteur et distribution, qui sont donc considérés par cette thèse. Dans ce qui suit, nous donnons une brève présentation de chacune de ces trois structures de données.

Scalaire est le type le plus usuel pour l'observation des noeuds en littérature [START_REF] Kolaczyk | Statistical Analysis of Network Data: Methods and Models[END_REF][START_REF] Shuman | The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains[END_REF]. Dans ce cas, pour un réseau de N noeuds, on observe une valeur x it P IR * à chaque noeud i " 1, ..., N et à instant du temps t P Z. La popularité du type scalaire vient du fait que les observations à chaque instant peuvent être représentées par le vecteur x t :" px it q N i"1 . On peut alors partir de plus d'outils existants, par exemple ceux de l'analyse multivariée des séries temporelles : [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF][START_REF] Brockwell | Time series: theory and methods[END_REF][START_REF] Neusser | Time series econometrics[END_REF], pour considérer la dépendance temporelle avec la structure spatiale, par exemple, [START_REF] Bach | Learning graphical models for stationary time series[END_REF]; [START_REF] Perraudin | Stationary signal processing on graphs[END_REF].

Deuxièmement, l'observation vectorielle fait référence au cas où un vecteur x it " px f it q F f "1 P IR F de F caractéristiques est observé à chaque noeud et in-stant. Les observations à chaque instant peuvent être représentées par la matrice X t :" px 1t , . . . , x N t q P IR N ˆF avec la ligne et la colonne correspondant à la dimension spatiale et à la dimension de la caractéristique. Dans le temps, l'observation totale est matricielle. De telles observations matricielles sont également courantes dans les applications. Parmi les exemples, citons l'observation de la mesure "epoch ĉhannel" à chaque sujet dans l'analyse de l'EEG [START_REF] Zhou | Gemini: Graph estimation with matrix variate normal instances[END_REF][START_REF] Wang | The sylvester graphical lasso (syglasso)[END_REF], et les observations de gène ˆtissu dans l'analyse des données d'expression génétique [START_REF] Yin | Model selection and estimation in the matrix normal graphical model[END_REF].

Enfin, nous considérons le cadre de l'observation distribuée. À chaque noeud i " 1, ..., N et instant t P Z, nous observons une distribution, c'est-à-dire, une mesure de probabilité µ it P PpΩq supportée sur un intervalle Ω de IR. En pratique, une distribution ne peut pas être observée directement et les données disponibles consistent plutôt en des échantillons i.i.d. qui sont générés par elle. Ainsi, pour fitter le modèle, nous devons tout d'abords extraire les distributions, plus précisément leur représentation telle que la densité, la fonction quantile ou la fonction de distribution cumulative, des échantillons par des méthodes numériques, telles que la régression linéaire locale pour les densités lisses [START_REF] Fan | Local polynomial modelling and its applications[END_REF].

Les données de distribution ont joué un rôle important dans de nombreux domaines scientifiques. Un exemple pertinent est l'analyse des distributions d'indicateurs supportés sur des intervalles d'âge, tels que la mortalité et la fertilité [START_REF] Mazzuco | Fitting age-specific fertility rates by a flexible generalized skew normal probability density function[END_REF][START_REF] Shang | Forecasting age distribution of death counts: An application to annuity pricing[END_REF], observés à travers différents pays au fil des ans dans les études démographiques. Un autre exemple important est l'analyse des distributions des rendements boursiers quotidiens dans le domaine des séries chronologiques financières [START_REF] Kokoszka | Forecasting of density functions with an application to cross-sectional and intraday returns[END_REF][START_REF] Zhang | Wasserstein autoregressive models for density time series[END_REF]. D'autres exemples incluent les distributions des corrélations entre les paires de voxels au sein des régions du cerveau [START_REF] Petersen | Functional data analysis for density functions by transformation to a hilbert space[END_REF] et les distributions des prix des maisons sur plusieurs mois [START_REF] Zhu | Autoregressive optimal transport models[END_REF]. Néanmoins, malgré l'existence déja longtemps des données distributionnelles, le développement de leurs outils adaptés n'est qu'un domaine de recherche récemment apparu.

Dans les Figures 1.1 et 1.2, nous démontrons les trois types d'observations avec des ensembles de données réels.

L'objectif général de cette thèse est de développer des approches pour les données de ces diverses formes observées sur un réseau, afin d'explorer et de représenter la dépendance spatiale des N caractéristiques indexées par les noeuds. Dans la section 1.2, nous présentons les problématiques considérés de cette thèse, ainsi que les principales contributions réalisées pour chaque problème.

Problématiques et principales contributions

Les études menées dans cette thèse s'articulent autour de deux problèmes principaux: l'identification de la structure de dépendance qui régit le processus stochastique sur un réseau, et la compréhension de la prévisibilité des noeuds dans les signaux temporels du graphe.

Le premier problème peut être rattaché au domaine de recherche de graph learning Relevés climatologiques mensuels des stations météorologiques de Californie. À gauche se trouve le réseau de stations météo de Californie. En haut à droite sont démontrée les observations scalaires sur une certaine station (capteur/noeud) i, où une valeur x it P IR de température moyenne est enregistrée à chaque instant t à i, conduisant à une série temporelle scalaire. En bas à droite sont illustrées les observations vectorielles sur une certaine station i, où un vecteur x it P IR 4 de température min/max/moy et de précipitation est enregistré par temps t à i, conduisant à 4 série temporelle scalaire. [START_REF] Dong | Learning graphs from data: A signal representation perspective[END_REF], où l'objectif central est que, étant donné les observations de multiples caractéristiques représentées par des variables ou processus stochastiques, nous souhaitons construire ou inférer la relation entre les caractéristiques qui prend la forme d'un graphe, les caractéristiques étant appelées noeuds. Dans ce cadre, lorsque l'observation du noeud est de type scalaire, de nombreuses approches ont été proposées pour apprendre de tels graphes. En particulier, pour les processus scalaires, les modèles vectoriels auto-régressifs (VAR) ont été largement adaptés pour déduire la structure d'une relation pertinente, à savoir la causalité de Granger. Le graphe résultant est appelé graphe causal. Nous nous intéressons ensuite à l'extension des modèles VAR aux modèle AR multivarié-distribué et celui matriciel pour servir l'objectif du graph learning. Les modèles dérivés répondent à la demande de développement de modèles personnalisés, étant donné que les séries temporelles matricielles et ceux distribuées deviennent plus populaires, et qu'elles n'ont pas encore été largement étudiées dans la littérature.

Le second problème est motivé par le fait que, dans de nombreuses applications, les observations enregistrées sur un réseau présentent une forte dépendance entre les noeuds, ce qui rend les données observées sur un sous-ensemble de noeuds hautement prévisibles par les données sur les autres noeuds. Ces noeuds sont à l'origine de la redondance des données du réseau, que nous pouvons donc arrêter d'enregistrer pour des raisons de stockage de données par exemple. Nous sommes alors intéressés par le classement de la prévisibilité pour les noeuds d'un réseau, étant donné un Les données brutes dans ce graphique consistent en 40 distributions annuelles. Nous les complétons avec des échantillons interpolés pour dessiner la surface. En bas à droite, nous montrons la projection de la série chronologique brute sur le plan Âge Fréquence relative. On voit que la population vieillit avec le temps.

Introduction (Français)

historique des observations du réseau, en prenant éventuellement en compte en plus la connectivité des noeuds si elle est disponible. Comme application, le classement dérivé peut être utilisé dans la sélection de capteurs, qui est un sujet important dans le domaine du traitement du signal sur graphe, voir par exemple [START_REF] Joshi | Sensor selection via convex optimization[END_REF]; Sakiyama et al. (2019b).

Dans les sous-sections 1.2.1 et 1.2.2, nous établissons les cadres mathématiques des deux problèmes avec des revues de la littérature non exhaustives, qui permettent de présenter les principales contributions de cette thèse. Tout au long de ce manuscrit, nous utilisons la notation en gras pour distinguer les objets aléatoires dans les modèles contextuels, du reste des constantes.

Structure de dépendance spatiale et modèles auto-régressifs (AR)

Cette section correspond aux préprints [START_REF] Jiang | Online graph topology learning from matrixvalued time series[END_REF]; [START_REF] Jiang | Wasserstein multivariate auto-regressive models for modeling distributional time series and its application in graph learning[END_REF].

Modèles VAR caractérisés par les graphes causaux

Dans cette section, nous rappelons les modèles VAR usuels ainsi que leurs applications au graph learning à partir des observations nodales de type scalaire. Dans des sections suivantes, nous présenterons les extensions proposées.

Dans l'analyse des séries chronologiques multivariées, le modèle auto-régressif vectoriel fait référence à l'équation différentielle stochastique:

x t " b `p ÿ l"1
A l x t´l `zt , t P Z, où z t " WNp0, Σq est un processus de bruit blanc avec la variance Σ, et b P IR N , A l P IR N ˆN sont les coefficients. Comme indiqué précédemment, les modèles VAR sont largement appliqués pour inférer la structure de causalité de Granger de N processus univariés px it q t pour tout i " 1, . . . , N . La causalité de Granger est définie par paire: px it q t est dit Granger cause px jt q t , j ‰ i si le dernier peut être prédit plus efficacement avec la connaissance du premier dans le passé et à présent prise en compte. La définition plus technique voir Définition 3.2.4. Le graphe causal fait alors référence à un tel graphe où chaque noeud représente une série temporelle univariée, et les arêtes représentent la causalité de Granger. Si les processus sont générés par un VAR(p) stationnaire, px it q t ne cause pas px jt q t si et seulement si toutes les ji-ièmes entrées des matrices de coefficients vrais A l ji ‰ 0, l " 1, . . . , p, [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF], Corollaire 2.2.1). Ainsi, nous pouvons retrouver la topologie du graphe à partir de la structure de sparsité commune dans A l . En petite dimension, cette structure peut être identifiée par le test de Wald, qui teste les contraintes linéaires des coefficients.

En grande dimension, l'inférence du graphe causal exact de Granger est principalement envisagée dans [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF]; [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF]. [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF] proposent de considérer la pénalité group lasso, λ ř i‰j }pA 1 ij , . . . , A p ij q} ℓ 2 , dans le problème usuel des moindres carrés pour les modèles VAR(p), afin d'inférer la structure de sparsité commune des matrices de coefficients A l , l " 1, . . . , p. [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF] développent la procédure online pour ce problème d'estimation. [START_REF] Mei | Signal processing on graphs: Causal modeling of unstructured data[END_REF] définissent une variante du modèle VAR, où les structures sparses des coefficients ne correspondent pas directement à la topologie du graphe, mais à la topologie des voisinages de l-arêtes. Plus précisément, ils supposent que A l " c l0 I `cl1 W `cll W l , où W est la matrice d'adjacence à inférer, et I est la matrice d'identité. De tels modèles peuvent ainsi capturer l'influence de plus de noeuds. L'estimation de la matrice d'adjacence sous-jacente s'appuie sur la pénalité Lasso pour promouvoir la sparsité.

Problématiques et principales contributions

Nous considérons tout d'abord l'extension du modèle VARp1q au processus matriciel pX t q tPZ P IR N ˆF . À cette fin, nous pouvons appliquer directement les modèles vectoriels à la représentation vectorielle vecpX t q. Cependant, cela n'est pas favorable. D'une part, traiter indifféremment les données de dimensions distinctes peut ignorer et donc gaspiller les informations de la structure intrinsèque des données. D'un point de vue technique, cela entraîne une croissance quadratique du nombre de paramètres dans le modèle, nécessitant alors un grand nombre d'échantillons pour une estimation robuste, ce qui n'est pas facile à satisfaire dans la pratique. Ainsi, la modélisation des données matricielles nécessite des techniques supplémentaires, que nous présenterons dans la section 2.2.1.2.

Somme de Kronecker et Product Cartésien de graphes

Dans la littérature, pour étendre les modèles vectoriels aux observations de variables matricielles (plus généralement de variables tensorielles), une pratique courante consiste à appliquer les modèles vectoriels aux données vectorisées, puis à imposer certaines structures aux matrices de paramètres qui encodent des informations sur les dimensions des données. Les structures les plus considérées sont la somme de Kronecker (SK) et/ou le produit de Kronecker (PK). Par exemple, pour étendre la distribution Gaussienne multivariée à la distribution gaussienne matricielle, [START_REF] Gupta | Matrix variate distributions[END_REF] vectorisent d'abord la variable aléatoire à valeur matricielle X P IR N ˆF , et suppose que vecpXq " N pµ, Ωq, puis impose la structure PK sur la matrice de variance : Ω " V b U , où U P IR N ˆN , V P IR F ˆF . Les deux matrices de sous-variance U et V sont associées séparément aux dimensions des lignes et des colonnes de la matrice de données. Par ailleurs, Kalaitzis et al. (2013) proposent un modèle Gaussien matriciel différent : vecpXq " N pµ, P ´1q, où P " Ψ ' Θ, et Ψ P IR N ˆN , Θ P IR F ˆF . C'est-à-dire, ils imposent une structure SK à la matrice de précision † . Étant donné que la structure SK d'une matrice d'adjacence correspond au produit cartésien de sous-graphes, l'intégration de la structure SK conduit à un modèle graphique Gaussien qui est interprétable pour les observations de variables matricielles, où la structure de dépendance conditionnelle totale se factorise en sous-structures distinctes des dimensions de lignes et des dimensions de colonnes.

Par conséquent, pour étendre le modèle VARp1q au processus matriciels pX t q t , nous pouvons appliquer le VARp1q usuelle au processus vectorisé pvecX t q t , puis imposer la structure SK ou PK à la matrice de coefficients A. Puisque la structure de sparsité

Introduction (Français)

de A encode la topologie du graphe causal des N F processus px f it q t , i " 1, . . . , N, f " 1, . . . , F , SK et PK impliquent tous les deux que ce graphe total peut se factoriser en deux sous-graphes, mais de manière différente. Pour expliquer cette différence, nous rappelons la définition de la somme de Kronecker. Soit A F P IR F ˆF , A N P IR N ˆN deux matrices carrées, et que I k désigne la matrice identité k ˆk. La somme de Kronecker entre A F et A N est définie comme suit

A F ' A N " A F b I N `IF b A N ,
où b est le produit de Kronecker. Comme indiqué précédemment, les structures PK et SK des matrices d'adjacence impliquent que les graphes correspondants sont les produits des graphes composants. Plus précisément, lorsque A F , A N sont les matrices d'adjacence de deux graphes G F , G N , les PK A F b A N et SK A F ' A N sont respectivement les matrices d'adjacence de leur produit tensoriel du graphe G N ˆGF et du produit Cartésien du graphe G N lG F (Sandryhaila and Moura, 2014a). Nous illustrons ces deux produits de graphes dans la figure 2.3. Pour les définitions formelles des produits cartésiens et tensoriels des graphes, nous nous référons à [START_REF] Hammack | Handbook of product graphs[END_REF]; Chen and Chen (2015); [START_REF] Imrich | Cartesian products of directed graphs with loops[END_REF].

Figure 1.3: Comparaison des produits cartésien et tensoriel des graphes. L'ensemble des noeuds de tous les deux produits de graphe est le produit Cartésien des ensemble des noeuds des composantes. Au-dessus, des différents règles d'adjacence s'appliquent. L'exemple est basé sur Sandryhaila and Moura (2014a, Figure 2). La figure 2.3 montre que les graphes de produits induits par le PK et le SK sont très différents. Par exemple, la structure en treillis du produit cartésien préserve les sousgraphes dans toutes les sections des deux dimensions. En revanche, le produit tensoriel se concentre sur la connexion inter-dimensionnelle, tout en abandonnant la dépendance intra-dimensionnelle. Cette dernière propriété fait en fait référence, dans la littérature sur les processus Gaussiens, «the cancellation of inter-task transfer», voir par exemple Bonilla et al. (2007, Section 2.3). Par conséquent, lorsque les noeuds représentent px f it q t , i " 1, . . . , N, f " 1, . . . , F , imposer la structure PK (Chen et al., 2021a) implique de supposer qu'il n'existe aucune dépendance de causalité entre px f it q t , i " 1, . . . , N pour chaque f fixé, qui représente les observations de la caractéristique f à différents noeuds dans le réseau. En revanche, les matrices de coefficients dotées de la structure SK sont capables de prendre en compte ces dépendances dans l'inférence, qui sont en fait présentes dans de nombreuses applications.

1.2.1.3 Produit causal de graphe et modèle AR(1) matriciel La première contribution consiste en l'extension du modèle AR(1) vectoriel aux séries temporelles vectorielles, X t P IR N ˆF , t P Z, en imposant ‡ la structure SK sur la matrice de coefficient A dans le modèle VAR(1) appliqué à vecpX t q:

offdpAq " A F ' A N ,

(1.2.1) où offdpAq est la matrice de même taille que A, dont la partie non diagonale est identique à celle de A, tandis que ses éléments diagonaux sont égaux à zéro. Comme indiqué dans la section 1.2.1.1, pour un VAR(1) stationnaire, la structure de sparsité du coefficient définit le graphe causal des N F composantes de vecpX t q. Par intégration du SK, nous supposons ensuite que ce graphe causal total se factorise comme le produit cartésien en le graphe spatial G N et le graphe de caractéristiques G F , où G N est préservé dans chaque caractéristique, et G F est préservé dans chaque noeud. Un travail relatif qui étend également le modèle VAR(1) au modèle AR(1) matriciel est Chen et al. (2021a), où ils proposent d'imposer la structure PK aux matrices d'adjacence. Cependant, comme nous l'avons indiqué dans la section 1.2.1.2, la PK n'est pas capable de capturer la causalité intra-dimensionnelle. Étant donné les échantillons X 0 , X 1 , . . . , X t , nous développons ensuite deux approches pour estimer A N , A F et identifier la structure de sparsité, afin d'inférer la topologie du graphe. Les approches servent respectivement pour l'apprentissage en petite et grande dimension. Nous considérons particulièrement un cadre d'apprentissage général où la sparsité partielle est poursuivie dans l'estimation de seulement A N . Ceci est motivé par un très petit nombre de caractéristiques F généralement présentes dans les applications. Ainsi, le graphe de caractéristiques peut être raisonnablement supposé entièrement connecté. D'autre part, puisque la contrainte de sparsité partielle est également un cas techniquement plus compliqué pour la méthode d'apprentissage proposée en grande dimension. Donc étant donné sa résolution, l'adaptation au cas de sparsité totale ne nécessite pas de nouvelles techniques.

En petite dimension, nous proposons l'estimateur par moindres carrés ordinaire (en équivalent moindres carrés généralisée) projeté afin de profiter des propriétés asymptotiques de l'estimateur classique. Plus précisément, nous d'abords construisons une base orthogonale pU k q k de l'espace K G , qui est un espace linéaire formé par tous les matrices de type N F ˆN F à coefficient dans IR qui ont la structure (1.2.1). En nous appuyant sur pU k q k , nous sommes en mesure d'exprimer, Proj G , la projection sur K G de manière explicite, ce qui définit en outre les estimateurs y A N,t et x A F,t par le biais du SK. Un tel y A N,t est une fonction linéaire par éléments de l'estimateur des moindres carrés ordinaire (MCO) classique de A. En appliquant le théorème de Cramer-Wold sur la TCL de l'estimateur MCO classique, nous dérivons ensuite la TCL pour y A N,t . Cette TCL permet en outre d'établir un test de Wald pour identifier la structure de sparsité de A N . ‡ Nous ne montrons que la structure clé dans cette introduction. La structure complète supposée pour A voir le chapitre 4.

Introduction (Français)

1.2.1.4 Nouveau type de Lasso: Lasso matriciel structuré et ses algorithmes d'Homotopie

Comme discuté à la fin de la section 1.2.1.1, une pratique dans la littérature pour identifier la structure de sparsité des coefficients VAR en régime de grande dimension est d'adopter des estimateurs Lasso. Celui utilisé dans [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF]; [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF] est défini comme le minimiseur du problème Lasso (1.2.2) dans le cas VAR(1).

min A 1 2t t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λt }A} ℓ 1 , (1.2.2)
où x τ est un vecteur d'échantillon, qui peut être vecpX τ q par exemple. Lasso (1.2.2) est le Lasso le plus standard dans la littérature (Hastie et al., 2009, Section 3.4.2). Une grande variété de cadres issus de l'analyse et l'optimisation convexes ont été adaptés pour calculer ses solutions pour différents scénarios, par exemple, Descente par coordonnée [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF], méthodes proximales [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], ainsi qu'une technique plus Lasso-orientée «least angle regression» [START_REF] Efron | Least angle regression[END_REF]. Cependant, le Lasso (1.2.2) n'est pas capable d'estimer A structuré et à la composante A N sparse. Par conséquent, motivés par l'estimation, nous proposons le nouveau problème de Lasso-type (1.2.3)

Apt, λ t q " arg min APK G 1 2t t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λt F }A N } ℓ 1 , (1.2.3)
La résolution ordinaire de Lasso (1.2.3) peut être effectuée en appliquant par exemple la descente de gradient proximal [START_REF] Parikh | Proximal algorithms[END_REF]. Dans le cadre de l'algorithme, la contrainte de structure et la sparsité partielle ne posent pas de difficultés supplémentaires, puisque seul le gradient par rapport à IR N F ˆN F est calculé dans la forward étape. Ensuite, la backward étape revient à un Lasso standard après projection du gradient sur K G , comme le montre l'équation (1.2.4).

A k`1 " proxpA k ´ηk ∇f pA k qq, " arg min

APK G 1 2η k › › ›A ´´A k ´ηk ∇f pA k q ¯› › › 2 ℓ 2 `λt F }A N } ℓ 1 " arg min APK G 1 2η k › › ›A ´Proj G ´Ak ´ηk ∇f pA k q ¯› › › 2 ℓ 2 `λt F }A N } ℓ 1 ðñ $ ' ' ' & ' ' ' % A k`1 N " arg min A N › › A N ´Proj G N `Ak ´ηk ∇f pA k q ˘› › 2 ℓ 2 `2η k λ t }A N } ℓ 1 , A k`1 F " Proj G F `Ak ´ηk ∇f pA k q ˘, diagpA k`1 q " Proj D `Ak ´ηk ∇f pA k q ˘,
(1.2.4) où ∇f pA k q " A k p Γ t p0q ´p Γ t p1q, nous notons A k`1 pt, λ t q par A k`1 pour éviter la notion lourde.

À ce point, nous nous concentrons sur la proposition d'algorithmes permettant de mettre à jour rapidement les solutions précédentes lors des changement de la valeur de l'hyperparamètre ou du terme des données. Pour bénéficier des solutions 1.2. Problématiques et principales contributions précédentes d'un problème, il faut considérer les méthodes spécifiques. Pour le Lasso standard (1.2.2), le cadre des méthodes de continuation par homotopie [START_REF] Osborne | A new approach to variable selection in least squares problems[END_REF] a été exploré [START_REF] Malioutov | Homotopy continuation for sparse signal representation[END_REF][START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF] pour calculer la mise à jour rapide. Puisque l'algorithme d'homotopie est dérivé de la condition d'optimalité, qui est par rapport aux matrices dans K G pour Lasso (1.2.3), nécessitant de considérer le gradient avec la structure, ainsi les algorithmes d'homotopie existants pour Lasso (1.2.2) ne sont pas applicables. Par conséquent, dans notre travail, nous calculons d'abord la condition d'optimalité de Lasso (1.2.3), basée sur l'expression de la projection sur K G . Nous dérivons ensuite les deux algorithmes d'homotopie, respectivement pour les trajets de mise à jour Apt, λ 1 q Ñ Apt, λ 2 q et Apt, λ 2 q Ñ Apt `1, λ 2 q, ainsi qu'une procédure de réglage adaptatif de l'hyperparamètre de régularisation à l'arrivée d'une nouvelle observation x t`1 . Les dérivations ne dépendent pas de la contrainte de structure spécifique, ni de la régularisation de sparsité particulière, et peuvent donc être appliquées à d'autres espaces de structure linéaire K G et à la conception de la sparsité.

Apprentissage online des graphes et des tendances

Dans les applications où le stockage des données est limité et où l'inférence rapide sur les données arrivant séquentiellement est exigée, la résolution ordinaire qui traite l'ensemble des données en une seule fois échoue. Ainsi, les algorithmes personnalisés qui remplissent ces deux conditions sont nécessaires. Cette classe d'algorithmes est appelée online. En revanche, la résolution ordinaire est appelée off-line.

Pour la résolution online de Lasso (1.2.3), les algorithmes d'Homotopie précédemment dérivés constituent la méthode pertinente, lors de l'exécution des trois étapes dans l'ordre : Étape 1 : λ t Ñ λ t`1 , Étape 2 : Apt, λq Ñ Apt, λ t`1 q, Étape 3 : Apt, λ t`1 q Ñ Apt `1, λ t`1 q,

(1.2.5)

Cependant, quand nous passons à l'inférence online, la formulation du problème (1.2.3) ne peut pas s'adapter complètement, en raison du terme de données

1 t ř t τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 .
Le terme de données implique que la série temporelle px τ q τ , τ P Z est supposée admettre une moyenne invariante en temps, qui est zéro. Néanmoins, lorsque les px τ q τ prennent des données brutes, cette hypothèse d'échantillon est très souvent fausse. Au lieu de cela, la série temporelle brute présente généralement la tendance, c'est-à-dire une fonction moyenne variant dans le temps. Ainsi, dans l'apprentissage online, une étape detrend est nécessaire, qui approche la fonction de tendance en utilisant l'ensemble des données, puis l'enlève des données brutes. Ensuite, la série chronologique traitée sera plus proche des hypothèses du modèle. Cependant, comme le principe de l'apprentissage online n'exige pas la présence de toutes les données, une telle étape de pré-traitement est interdite. Ainsi, nous devons considérer la tendance comme un paramètre explicite supplémentaire aux paramètres du graphe A N , A F dans le modèle online.

Le présent travail se concentre principalement sur un type particulier de tendance, à savoir la tendance périodique, qui est fréquemment rencontrée dans la pratique. Par exemple, une récurrence annuelle tous les 12 mois peut être trouvée dans de

Introduction (Français)

nombreux jeux de données, enregistrés mensuellement sur des années. Du point de vue de la modélisation, la tendance périodique est la fonction moyenne:

Ex τ " b 0 m , m " 0, ..., M ´1, m " τ mod M,
où M est la longueur de la période. Nous proposons donc de reformuler Lasso (1.2.3) en incorporant la tendance périodique b 0 m , m " 0, . . . , M ´1, sous la forme de l'équation (1.2.6).

Apt, λ t q, b 0 m pt, λ t q " arg min

APK G ,b 0 m 1 2t M ´1 ÿ m"0 ÿ τ PIm,t }px τ ´b0 m q´Apx τ ´1´b 0 m´1 q} 2 ℓ 2 `λt F }A N } ℓ 1 , (1.2.6) où I m,t " tτ " 1, ..., t : τ mod M " mu. Notez que b 0 ´1 désigne b 0 M ´1. Le modèle d'échantillon sous-jacent devient x t´1 ´b0 m " Apx t´1 ´b0 m´1 q `zt , A P K G , m " t mod § M,
qui est un modèle VAR(1) augmenté avec une tendance périodique. Nous adaptons ensuite les algorithmes du Pipeline (1.2.5) au Problème (1.2.6), qui permet enfin l'apprentissage online des graphes et des tendances à partir des séries temporelles matricielles.

Statistiques de Wasserstein pour les données distribuées

Deuxièmement, nous aimerions étendre le modèle VAR(1) à une collection de N mesures de probabilité dépendantes du temps pµ i t q tPZ , i " 1, . . . , N (exemple de telles données voir la section 1.1), qui peut en outre identifier et représenter les liens de dépendance significatifs entre les processus dans un graphe orienté pondéré de N noeuds. Au lieu d'un vecteur, la caractéristique du noeud devient maintenant une distribution. Pour modéliser de telles données distributives, nous devons adopter des outils plus avancés.

Puisque les distributions peuvent être caractérisées par les fonctions telles que, les densités, les fonctions quantiles, et les fonctions de distribution cumulative, alors pour analyser les séries temporelles distribuées, on peut chercher à étudier une de ses représentations fonctionnelles avec les outils de l'analyse des séries temporelles fonctionnelles [START_REF] Bosq | Linear processes in function spaces: theory and applications[END_REF]. Cependant, en raison de leurs contraintes non linéaires, telles que la monotonicité et la positivité, les fonctions de représentation des distributions ne constituent pas des espaces linéaires. Par conséquent, les notions de base des modèles VAR standard, telles que l'additivité et la multiplication scalaire, ne s'adaptent pas, de manière directe. Cela entraîne l'échec des modèles dérivés pour les éléments aléatoires d'un espace de Hilbert. Une approche existante consiste à faire correspondre les densités des distributions à des fonctions non contraintes dans l'espace de Hilbert par la transformation de «log quantile density» (LQD) [START_REF] Petersen | Functional data analysis for density functions by transformation to a hilbert space[END_REF], puis à appliquer les outils fonctionnels [START_REF] Kokoszka | Forecasting of density functions with an application to cross-sectional and intraday returns[END_REF]. Cependant, LQD ne prend pas en compte la géométrie de l'espace de distribution, § Le modulo d'un nombre entier négatif est défini par le reste positif dans ce cas, par exemple, ´1 mod M " M ´1.

Problématiques et principales contributions

et peut donc conduire à des déformations de la distance. Les approches récentes considèrent une telle géométrie en adoptant la métrique de Wasserstein [START_REF] Bigot | Geodesic pca in the wasserstein space by convex pca[END_REF][START_REF] Panaretos | Amplitude and phase variation of point processes[END_REF]Petersen and Müller, 2019b). Nous établissons donc le modèle de série temporelle dans l'espace de Wasserstein. Dans ce qui suit, nous présentons la principale contribution dans ce piste de travail. Pour l'introduction technique des statistiques dans l'espace de Wasserstein, nous nous référons à la section 3.3.

Modèle AR(1) multivarié de Wasserstein et la contrainte Nsimplexe

Il existe également d'autres travaux qui se sont penchés sur les modèles de séries temporelles distribuées. Jusqu'à présent, ils se concentrent principalement sur le cas univarié, c'est-à-dire lorsque N " 1. En général, les modèles distribués multivariés sont encore peu développés. Notez que dans le cas distribué, deux notions multivariées sont pertinentes, l'une se réfère à la dimension du support de la mesure, tandis que l'autre se réfère au nombre de mesures. Les modèles associés à ces deux notions sont peu développés. Dans ce manuscrit, le mot multivarié (multivariate) fait référence à la seconde notion.

Nous nous référons aux travaux récents Chen et al. (2021b); [START_REF] Zhang | Wasserstein autoregressive models for density time series[END_REF]; [START_REF] Zhu | Autoregressive optimal transport models[END_REF], qui développent le modèle AR distribué univarié dans l'espace de Wasserstein W 2 pIRq. Ils proposent de mapper les distributions aléatoires dans l'image logarithmique de l'espace de Wasserstein supporté sur leur moyenne de Fréchet présumée commune. Ils construisent ensuite les modèles de séries temporelles fonctionnelles univariées en termes de cartes logarithmiques. La moyenne commune de Fréchet est la distribution de référence idéale dans le sens où elle annule les espérances des cartes logarithmiques dans la formule de régression. Cependant, pour les séries temporelles multivariées, les différents processus ne peuvent pas avoir une moyenne de Fréchet commune. De plus, il est très peu probable qu'une telle mesure de référence idéale existe (cela sera expliqué dans le chapitre 5). D'autre part, une interception supplémentaire dans un tel modèle de régression fonctionnelle causera de grandes difficultés, lorsque l'on veut conserver le modèle de régression et surtout les prédictions dans l'image logarithmique de l'espace de Wasserstein. Ainsi, pour traiter les moyens de processus inégaux, nous sommes motivés par la formulation équivalente du modèle VAR(1) construit sur les série centrées qui n'inclut donc pas de terme d'interception (détails voir l'équation (3.2.5) dans la section des préliminaires)

x t ´u " Apx t´1 ´uq `zt .

Nous proposons tout d'abord un moyen de centrer toutes les distributions brutes par leurs moyennes de processus (Fréchet) de sorte que leurs moyennes deviennent la distribution uniforme, à savoir la mesure de Lebesgue. Ensuite, nous construisons le modèle sur les données centrées r µ i t , i " 1, . . . , N, t P Z. Pour chaque i, nous définissons la relation de régression clé pour la réponse r µ i t et les prédicteurs p r µ j t´1 q j , j " 1, . . . , N dans l'espace tangent de la mesure de Lebesgue, comme la combinaison linéaire des cartes logarithmiques de p r µ j t´1 q j , pondérées par les coefficients pA ij q j . La réponse r µ i t est égale à la carte exponentielle de la combinaison linéaire, poussée par une fonction de distorsion aléatoire. Par conséquent, le total des coefficients du système de régression définit une matrice A " pA ij q ij comme dans le modèle VAR(1). Pour les modèles de régression fonctionnels généraux avec réponse fonctionnelle, les coefficients peuvent être des fonctions concurrents A ij p¨q ou plus généralement des surfaces A ij p¨, ¨q, voir Wang et al. (2015, Equations (14) et (15), respectivement). Néanmoins, nous avons l'intention d'utiliser la forme matricielle pour les coefficients de régression afin d'y représenter directement la structure du graphe.

Comme l'application exponentielle n'est pas injective, le modèle proposé ci-dessus a donc un problème d'identifiabilité. Ainsi, nous conservons en outre le modèle dans l'image logarithmique en ajoutant l'hypothèse N -simplexe A1 aux coefficients (voir ci-dessous), étant donné l'injectivité de l'application exponentielle restreinte sur l'image logarithmique et la convexité de l'image logarithmique. En fait, en sortant de l'image logarithmique, le calcul dans le problème d'estimation n'est pas traçable non plus.

Assumption A1. ř N j"1 A ij ď 1 and 0 ď A ij ď 1.
Nous montrons ensuite que le système de fonctions aléatoires itérées [START_REF] Wu | Limit theorems for iterated random functions[END_REF]) associé au modèle de série temporelle proposé admet une solution unique, qui est de plus stationnaire en tant que processus fonctionnel dans l'espace de Hilbert de N -fonctions quantiles muni du produit scalaire L 2 usuel, étant donné les hypothèses supplémentaires de contraction sur l'application de régression, qui peuvent être reliées à la condition de stationnarité des modèles VAR.

Enfin, nous nous appuyons sur la méthode des moindres carrés pour dériver l'estimateur de A, qui minimise la somme des résidus au carré sous la contrainte du simplexe : A1. Nous fournissons également la garantie de consistance pour l'estimateur proposé.

Application au graph learning à partir de séries temporelles distribuées

En raison de la contrainte simplexe A1, l'estimateur dérivé porte naturellement la sparsité. Ainsi, en interprétant la matrice des coefficients comme la matrice d'adjacence de N noeuds, nous pouvons extraire un graphe orienté pondéré de son estimation, qui représente la structure de dépendance de N processus distribués pµ i t q tPZ , i " 1, . . . , N . La Figure 1.4 montre le graphe de structure d'âge inféré des pays à partir des données illustrées dans La figure 1.2. La liste complète des pays étudiés se trouve au chapitre 5.

1.2.1.9 Modèle régressif distribution-à-distribution multivarié L'approche proposée de centralisation de distributions aléatoires peut également servir au développement du modèle régressif distribution-à-distribution multivarié pour les données i.i.d., qui est également très peu développé. Les travaux existants sur le modèle régressif distribution-à-distribution considèrent principalement le cas d'un prédicteur et une réponse. Dans Chen et al. (2019), la régression ridge à noyau des fonctions de densité transformées par LQD sont construites. Même si la régression peut être étendue au cas multivarié en élargissant le domaine du noyau reproduisant, mais comme indiqué dans la section 1.2.1.6, la transformation LQD ne tient pas compte de la géométrie de l'espace de distribution. Plus récemment, les deux Chen et al. (2021b); [START_REF] Ghodrati | Distribution-on-distribution regression via optimal transport maps[END_REF] s'appuient sur la métrique de Wasserstein. Chen et al. (2021b) effectue la régression en utilisant l'opérateur de coefficient qui établit une correspondance entre l'espace tangent de la moyenne de Fréchet du prédicteur et celui de la réponse. En revanche, [START_REF] Ghodrati | Distribution-on-distribution regression via optimal transport maps[END_REF] effectue la régression directement dans l'espace de Wasserstein. Le prédicteur est pushforwardé par la carte de coefficient optimale à la moyenne de Fréchet conditionnelle de la réponse. Il n'est pas simple d'étendre les deux modèles à un plus grand nombre de prédicteurs puisque les opérations correspondantes sont toutes deux directionnelles. C'est-à-dire qu'il n'est pas évident de mapper entre plusieurs espaces tangents de prédicteurs ou de pushforwarder parmi les mesures de prédicteurs. Ainsi, la méthode de centralisation des données proposée dans ce travail peut fournir une nouvelle piste pour traiter

Introduction (Français)

plusieurs prédicteurs et réponses dans ce cadre de régression plus général.

Prévisibilité des noeud dans les signaux temporels du graphe

Pour mesurer la prévisibilité des noeuds d'un réseau, nous introduisons tout d'abord le problème de prédiction pour les observations du réseau px it q t , i P N :" t1, . . . , N u, où nous reconstruisons hypothétiquement les observations au temps t sur un sousensemble I de noeuds, x it , i P I Ă N , étant donné les enregistrements dans le passé et à présent sur le reste des noeuds px jτ q jτ , j P I c , t ´H ď τ ď t. Nous souhaitons comparer la prévisibilité évaluée par les deux méthodes de prédiction les plus populaires dans les domaines de la statistique et du traitement de signaux sur graphes : la régression ridge à noyau et le réseau de neurones. De plus, lorsque les arêtes des noeuds ne sont pas données, nous considérons la régression linéaire. Pour les deux méthodes de régression, nous obtenons le classement par l'adaptation gloutonne du problème de sélection de capteurs de I optimal, en mettant en relation les erreurs de prédiction des observations sur I avec les critères de sélection. Pour le réseau de neurones, nous mesurons la prévisibilité des noeuds de manière plus fine en attribuant un score à chaque noeud. Nous proposons d'adopter le schéma «dropout» pour faire varier les noeuds éteints dans I, qui permet en outre d'obtenir l'erreur de prédiction agrégée pour chaque noeud, en une seule formation. Dans la suite, nous présentons ces contributions dans un contexte plus technique. Cette section correspond au preprint [START_REF] Jiang | Sensor selection on graphs via data-driven node sub-sampling in network time series[END_REF].

Régression ridge à noyau des prédicteurs de temps et de noeuds

Tout d'abord, nous adaptons la régression ridge à noyau au problème de prédiction posé en régressant l'observation x iτ P IR sur les indices de noeud et de temps i P N et τ P Z. Nous proposons un design de noyau sur N ˆZ, qui prend en compte la structure du graphe et définit la similarité des points temporels. La régression ridge à noyau approche alors la fonction de régression par les fonctions dans l'espace de Hilbert à noyau reproduisant (désignés sous l'acronyme issu du titre anglais RKHS, pour Reproducing Kernel Hilbert Space) induit H k , avec l'estimateur correspondant donné par H`1ďtďT px it ṕ f t pi, tqq 2 {T , qui dépend de l'auto-covariance empirique du processus px it q t , i P N et de la matrice de Gram du noyau proposé. L'erreur dérivée donne un moyen de quantifier la prévisibilité des noeuds, qui dépend toutefois du choix de I. Nous proposons de nous appuyer sur le problème de sélection de capteurs en prenant comme critère l'erreur totale de prédiction, pour prendre en compte d'autres compositions et cardinalités possibles de I. Pour une cardinalité donnée 1 ď p ď N ´1, le problème recherche l'ensemble optimal I ˚ppq qui minimise un certain critère parmi tous les ensembles I de noeuds p. En particulier, puisque la stratégie gloutonne adaptée sélectionne les noeuds un par un, dont l'ordre forme les ensembles optimaux de cardinalité de 1 à N ´1 de manière incrémentale. Nous proposons donc de classer les noeuds selon l'ordre produit par une telle stratégie gloutonne. Le classement résultant peut être interprété comme suit : pour chaque cardinalité p, nous attribuons aux noeuds de l'ensemble optimal I ˚ppq score 1, les autres noeuds score 0. Le total des scores pour tous les p de 1 à N forme le classement.

p f t " min f PH k ÿ iPI c ,t´Hďτ ďt }x iτ ´f pi, τ q} 2 ℓ 2 `λ}f } 2 H k . ( 1

Faire varier l'ensemble éteint I par le dropout

Puisque les réseaux de neurones ont été largement adaptés aux problèmes de prédiction dans le domaine du traitement de signaux sur graphes, nous sommes particulièrement intéressés par l'évaluation de sa prédictibilité sur des noeuds{footnotePour le prédicteur de réseau, nous ne considérons que le problème de prédiction avec H " 0, car nous n'avons pas observé empiriquement l'amélioration de la reconstruction pour des H plus grands, lors de l'application du classement inféré à la sélection de capteurs.. Pour le prédicteur de réseau, nous pourrions définir le coût d'entraînement comme l'erreur de prédiction totale donnée dans la section 1.2.2.1 et le relier encore aux critères de sélection de capteurs de façon à reprendre la stratégie gloutonne. Cependant, cela conduit à OpN 2 q fois l'entraînement du réseau, ce qui est beaucoup plus coûteux que les opérations matricielles. Par conséquent, nous concevons un schéma d'entraînement pour le prédicteur de réseau, qui varie la composition de I et donne donc les scores de tous les noeuds en un seul entraînement. L'idée principale est illustrée dans la Figure 1.5. En appliquant le dropout et le dropout à l'inverse sur l'entrée et la loss de batch, le réseau entraîné agrège les modèles de prédiction liés à différents I. Dans notre expérimentations, des milliers de compositions de I ont été échantillonnées pendant l'entraînement pour N de l'ordre de la centaine. Par conséquent, l'erreur de prédiction donnée par un tel réseau représente la prévisibilité globale d'un noeud, lorsqu'il est prédit dans différentes I. Nous proposons d'utiliser le réseau entraîné pour prédire les observations de l'ensemble de validation, et de prendre l'erreur de prédiction de chaque noeud comme son score, mesuré par la métrique R 2 . La Figure 1.6 montre les scores dérivés pour les stations de vélos à Paris, calculés à partir d'un historique d'observation de 3533 heures, qui enregistre le ratio de vélos disponibles à chaque station et à chaque heure. Ces scores montrent le résultat intéressant que, les stations à haute prédictibilité forment plusieurs groupes géographiques. Nous nous concentrons sur les deux groupes. L'un suit l'itinéraire principal du Métro 1 et du RER A vers le quartier d'affaires central de La Défense (points blancs jaunes sur la rive droite de la Seine). L'autre se situe autour de la gare de Paris Montparnasse (points jaunes blancs sur la rive gauche). Ces deux groupes géographiques seront retrouvés au chapitre 5 par le modèle AR de Wasserstein proposé dans son graphe inféré sur le même jeu de données, où les formules de régression prédisant les stations à l'intérieur des groupes sont beaucoup plus significatives, avec une plus grande amplitude des coefficients non nuls.

Ces observations confirment l'efficacité des deux méthodes proposées.

Introduction (Français)

Loss (reverse dropout) Input (dropout) Output Input (no dropout)

Training Test

Prediction network

Prediction network

Variance partielle d'un processus stochastique multivarié

Enfin, nous aimerions dériver la méthode d'apprentissage pour la prévisibilité des noeuds, lorsque les informations sur la connectivité des noeuds dans un réseau ne sont pas disponibles. Nous pourrions d'abord appliquer une méthode de graph learning pour l'observation scalaire des noeuds, comme les modèles VAR, pour inférer les arêtes, puis réutiliser les méthodes proposées dans les sections 1.2.2.1 et 1.2.2.2. Cependant, puisque nous utilisons deux fois les mêmes données, le bruit des données sera amplifié et provoquera un biais plus important dans les modèles fittés. Nous avons donc l'intention de considérer les signaux temporels comme processus multivariés et de nous appuyer sur des méthodes de prédiction sans graphe.

Une notion immédiate en statistique qui mesure la prévisibilité d'une variable aléatoire x i étant donné d'autres variables x j , j ‰ i est le variance partielle de x i étant donné x j , j ‰ i, qui est essentiellement défini comme la variance du résidu de x i prédit par x j , j ‰ i par la régression linéaire. Nous nous basons ensuite sur la variance partielle de x it , i P I Ă N étant donné px jτ q jτ , j P I c , t ´H ď τ ď t pour quantifier la prévisibilité du noeud lorsque le graphe n'est pas donné. La variance partielle totale de x it , i P I étant donné px jτ q jτ , j P I c , t ´H ď τ ď t est une contrepartie de l'erreur de prédiction totale donnée dans la section 1.2.2.1. En particulier, lors de l'estimation de la variance partielle totale de x it , i P I étant donné px jτ q jτ , j P I c , t ´H ď τ ď t par l'auto-covariance empirique du processus px it q t , i P N , l'estimateur s'écrit comme un cas spécial de l'erreur de prédiction totale dérivée dans la section 1.2.2.1, avec le kernel défini par ailleurs par l'auto-covariance empirique, et λ " 0. Nous considérons alors la variance partielle totale comme un autre critère pour le problème de sélection de capteurs, et nous nous appuyons sur la stratégie gloutonne pour obtenir le classement souhaité comme dans la Section 1.2.2.1.

Application à la sélection de capteurs

Étant donné la dérivation des classements, pour toute cardinalité 1 ď p ď N ´1, les noeuds dans les top p peuvent également être considérés comme ensembles optimaux, sur lesquels les observations peuvent être prédites collectivement le mieux par les noeuds restants. Une telle stratégie de sélection prend en compte l'aspect de dépendance qui peut être estimé à partir des données, en plus de la structure du graphe. Dans les expérimentations du chapitre 6, nous montrons que la stratégie proposée présente un avantage dans la reconstruction des données manquantes par rapport aux méthodes d'échantillonnage standard dans le domaine du traitement de signaux sur graphes qui emploient uniquement la structure du graphe, comme Anis et al. (2016a); [START_REF] Puy | Random sampling of bandlimited signals on graphs[END_REF].

Organization de la thèse

Chapitre 3 Nous présentons les préliminaires qui permettront de commencer l'analyse des chapitres 4 -6.

Chapitre 4 Nous développons les approches pour les données de réseau d'observation vectorielle des noeuds. Nous proposons le modèle AR(1) matriciel et dérivons les procédures online pour inférer son graphe de caractérisation à partir des données séquentielles à tendance, en petite et grande dimension. Ce chapitre est lié au preprint [START_REF] Jiang | Online graph topology learning from matrixvalued time series[END_REF].

Chapitre 5 Nous nous concentrons sur l'analyse des observations de noeuds distribués. Nous proposons le modèle AR(1) multivarié de Wasserstein et l'estimateur 1. Introduction (Français) sparse de ses coefficients, avec les résultats théoriques. Ce chapitre est lié au preprint [START_REF] Jiang | Wasserstein multivariate auto-regressive models for modeling distributional time series and its application in graph learning[END_REF].

Chapitre 6 Ce chapitre est consacré au problème de l'apprentissage de la prévisibilité des noeuds à partir des données de réseau de type scalaire. Nous proposons des stratégies de classement pour les prédictions du modèle linéaire, du modèle RKHS et du réseau de neurones. Ce chapitre est lié au preprint [START_REF] Jiang | Sensor selection on graphs via data-driven node sub-sampling in network time series[END_REF].

Chapter 2 Introduction 2.1 Data recorded over a network of sensors Data recorded over a network of sensors have become increasingly popular in recent years with applications in many areas such as traffic analysis [START_REF] Crovella | Graph wavelets for spatial traffic analysis[END_REF][START_REF] Yao | Deep multi-view spatial-temporal network for taxi demand prediction[END_REF][START_REF] Fang | Gstnet: Global spatial-temporal network for traffic flow prediction[END_REF], brain network analysis [START_REF] Huang | A graph signal processing perspective on functional brain imaging[END_REF][START_REF] Wang | The sylvester graphical lasso (syglasso)[END_REF], and meteorology [START_REF] Handcock | An approach to statistical spatial-temporal modeling of meteorological fields[END_REF][START_REF] Mei | Signal processing on graphs: Causal modeling of unstructured data[END_REF][START_REF] Xu | Spatio-temporal interpolated echo state network for meteorological series prediction[END_REF]. The sensor networks can also represent an abstract space, which are not related to physical entities, such as a social network [START_REF] Tabassum | Social network analysis: An overview[END_REF], citation network (Liu et al., 2019), or semantic network [START_REF] Lake | Discovering structure by learning sparse graphs[END_REF][START_REF] Sarica | Technet: Technology semantic network based on patent data[END_REF]. These sensors often record a sequence of observations along time at a regular frequency, which corresponds to the evolution of traffic or weather, for example. Such observations are therefore endowed with both spatial and temporal nature. Apart from the spatio-temporal aspect, the nodal observation from each sensor can also exhibit rich data representation. Three common data types are scalar, vector, and distribution, which are therefore considered by this thesis. In the following, we give the brief presentation for each of the three data structures.

Scalar is the most usual type for node observation in the literature [START_REF] Kolaczyk | Statistical Analysis of Network Data: Methods and Models[END_REF][START_REF] Shuman | The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains[END_REF]. In this case, for a network of N nodes, we observe a value x it P IR * at each node i " 1, ..., N and time instant t P Z. The popularity of the scalar-type comes from the fact that the observations at each time can be represented by the vector x t :" px it q N i"1 . One can then start off with existing tools, for instance those from multivariate time series analysis [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF][START_REF] Brockwell | Time series: theory and methods[END_REF][START_REF] Neusser | Time series econometrics[END_REF], to consider the temporal dependency with the spatial structure, for example, [START_REF] Bach | Learning graphical models for stationary time series[END_REF]; [START_REF] Perraudin | Stationary signal processing on graphs[END_REF].

Secondly, the vectorial observation refers to the case where a vector x it " px f it q F f "1 P IR F of F features is observed at each node and time instant. Because the observations at each time can be represented by the matrix X t :" px 1t , . . . , x N t q P IR N ˆF with the row and column corresponding to the spatial dimension and the additional feature dimension. Time-wise, the total observation is matrix-variate. Such matrixvariate observations are also common in applications. Some examples include the observation of measurement epoch ˆchannel for each subject in EEG analysis [START_REF] Zhou | Gemini: Graph estimation with matrix variate normal instances[END_REF][START_REF] Wang | The sylvester graphical lasso (syglasso)[END_REF], and the observations of gene ˆtissue in gene expression data analysis [START_REF] Yin | Model selection and estimation in the matrix normal graphical model[END_REF].

Lastly, we consider the setting of distributional observation. At each node i " 1, ..., N and time instant t P Z, we observe a distribution, namely, a probability measure µ it P PpΩq supported on an interval Ω of IR. In practice, a distribution can not be directly observed and instead the available data consist of i.i.d. samples that are generated by it. Thus, for model fitting, we need to primarily retrieve the distributions, more precisely their representation such as density, quantile function, or cumulative distribution function, from the samples by numeric methods, such as the local linear regression for smooth densities [START_REF] Fan | Local polynomial modelling and its applications[END_REF].

Distributional data have been playing important roles in many scientific fields. A pertinent example is the analysis of the indicator distributions supported over age intervals, such as mortality and fertility [START_REF] Mazzuco | Fitting age-specific fertility rates by a flexible generalized skew normal probability density function[END_REF][START_REF] Shang | Forecasting age distribution of death counts: An application to annuity pricing[END_REF], observed across different countries over years in demographic studies. Another important example is the analysis of daily stock return distributions from financial time series [START_REF] Kokoszka | Forecasting of density functions with an application to cross-sectional and intraday returns[END_REF][START_REF] Zhang | Wasserstein autoregressive models for density time series[END_REF]. Other examples include the distributions of correlations between pairs of voxels within brain regions [START_REF] Petersen | Functional data analysis for density functions by transformation to a hilbert space[END_REF] and the house price distributions overs months [START_REF] Zhu | Autoregressive optimal transport models[END_REF]. Nevertheless, despite the long existence of the distributional data, the development of their tailored tools is only a recently emerging field of research.

In Figures 2.1 and 2.2, we demonstrate the three types of observations with real data sets.

The general goal of this thesis is to develop the approaches for the network data of these diverse forms, in order to explore and represent the spatial dependency of the N features indexed by the nodes. In Section 2.2, we present the main problems considered by this thesis, as well as the main contributions achieved for each problem.

Problems and main contributions

The studies in this thesis are carried on around two main problems: identification of the dependency structure which governs the stochastic process over a network, and understanding the node predictability in the time-dependent graph signals.

The first problem can be related to the research domain of graph learning [START_REF] Dong | Learning graphs from data: A signal representation perspective[END_REF], where the central purpose is that, given the observations of multiple features represented by random variables or processes, we would like to build or infer the cross-feature relationship that takes the form of a graph, with the features termed as nodes. In this framework, when the node observation is scalar-type, many approaches have been proposed to learn such graphs. Especially, for the scalar random processes, the vector auto-regressive (VAR) models have been widely adapted to infer the structure of one pertinent relationship, namely Granger causality. The resulting graph is referred to as causal graph. Our interests are then to extend the VAR models to the matrix-variate and multivariate distributional AR models to serve the purpose of graph learning. The derived models respond to the demand of developing the tailored models, as the matrix-variate and distributional time series become more popular, which have not been widely studied yet in the literature.

The second problem is motivated by the fact that, in many applications, the observations recorded over a network exhibit strong cross-node dependency, which makes the data observed on a subset of nodes highly-predictable by the data on the remaining nodes. These nodes cause the redundancy in the network data, so that we can stop recording for the sake of data storage, for instance. We are then interested in ranking the predictability for the nodes in a network, given a history of network observations, possibly taking into account additionally the connectivity of nodes if available. As an application, the derived ranking can be used in sensor selection, which is an important topic in the graph signal processing domain, see for example [START_REF] Joshi | Sensor selection via convex optimization[END_REF]; Sakiyama et al. (2019b).

In Subsections 2.2.1 and 2.2.2, we develop the mathematical frameworks of the two problems with non-exhaustive literature reviews, which allow to present the main contributions of this thesis. Throughout this manuscript, we use the bold notation to distinguish random objects in the models discussed in the context from the constants.

Spatial dependency structure and auto-regressive (AR) models

This section corresponds to the preprints [START_REF] Jiang | Online graph topology learning from matrixvalued time series[END_REF]; [START_REF] Jiang | Wasserstein multivariate auto-regressive models for modeling distributional time series and its application in graph learning[END_REF]. A distribution of age µ it P Ppr0, 1sq is recorded at each time t. On the lower left, we visualize the resulting univariate distributional time series with a surface in the coordinate system of Age ˆYear ˆRelative frequency. The raw data in this plot consist in 40 annual distributions. We complete them with interpolated samples to draw the surface. On the lower right, we show the projection of the raw time series onto the Age ˆRelative frequency plane. We can see that the population is aging along time.

VAR models characterized by the causal graphs

In this section, we recall the classical VAR models and their applications in graph learning from scalar-valued nodal observations in literature. In the following sections, we will present the proposed extensions.

In multivariate time series analysis, the vector auto-regressive model refers to the stochastic difference equation:

x t " b `p ÿ l"1 A l x t´l `zt , t P Z,
where z t " WNp0, Σq is a white noise process with variance Σ, and b P IR N , A l P IR N ˆN are the coefficients. As stated earlier, VAR models are widely applied to infer the Granger causality structure of N univariate processes px it q t for i " 1, . . . , N . The Granger causality is defined pairwise: px it q t is said to Granger cause px jt q t , j ‰ i if px jt q t can be predicted more efficiently with the knowledge of px it q t in the past and present taken into account. More technical definition see Definition 3.2.4. The causal graph then refers to such a graph where each node represents a univariate time series, and the edges represent Granger causality. If the processes generated by a stationary VAR(p) model, time series px it q t does not cause px jt q t if and only if all the ji-th entries of the true coefficient matrices A l ji " 0, l " 1, . . . , p, (Lütkepohl, 2005, Corollary 2.2.1). Thus, we can retrieve the graph topology from the common sparsity structure in A l . In low-dimensional regime, this structure can be identified through Wald test, which tests the linear constraints for the coefficients.

In the high dimensional regime, the inference of the exact Granger causal graph is mainly considered in [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF]; [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF]. [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF] propose to consider the group lasso penalty, λ ř i‰j }pA 1 ij , . . . , A p ij q} ℓ 2 , to the usual least squares problem of VAR(p) models, in order to infer the common sparsity structure of coefficient matrices A l , l " 1, . . . , p. [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF] develop the online procedure for this estimation problem. [START_REF] Mei | Signal processing on graphs: Causal modeling of unstructured data[END_REF] define a variant of VAR model, where the sparse structure of coefficients A l does not directly equal the graph topology, but the topology of l-hop neighbourhoods † . More specifically, they suppose that A l " c l0 I `cl1 W `. . . `cll W l , where W is the adjacency matrix to infer, and I is the identity matrix. Such models can thus capture the influence from more nodes. The estimation of the underlying adjacency matrix relies on the Lasso penalty to promote the sparsity.

We firstly consider the extension of the VARp1q model to the matrix-variate process pX t q tPZ P IR N ˆF . To this end, we may apply straightforwardly the vector models to the vectorized representation vecpX t q. However, this is not favorable. On one hand, dealing with the data from separate dimensions indifferently can ignore, and hence waste, the information of the intrinsic structure of data. In terms of technical complexity, it leads to the quadratical growth of the number of parameters in the model, requiring then large number of samples for robust estimation, which is not easy to satisfy in practice. Thus the modelling of the matrix-variate data requires additional techniques, which we will present in Section 2.2.1.2.

In the literature, to extend vector models to matrix-variate observations (more generally tensor-variate), a popular practice is to apply the vector models onto vectorized data, then impose certain structures on parameter matrices which encode information on data dimensions. The most considered structures are Kronecker sum (KS) or/and Kronecker product (KP). For example, to extend multivariate Gaussian distribution to matrix Gaussian distribution, [START_REF] Gupta | Matrix variate distributions[END_REF] first vectorize matrix-valued random variable X P IR N ˆF , and suppose that vecpXq " N pµ, Ωq, then impose the KP structure on the variance matrix: Ω " V b U , where U P IR N ˆN , V P IR F ˆF . The two sub-variance matrices U and V are separately associated with the row and column dimensions of the data matrix. Besides, Kalaitzis et al. (2013) propose a different matrix Gaussian model: vecpXq " N pµ, P ´1q, where P " Ψ ' Θ, and Ψ P IR N ˆN , Θ P IR F ˆF . That is they impose a KS structure on the precision matrix ‡ . Since the KS structure in an adjacency matrix corresponds to the Cartesian product of subgraphs, the embedding of KS structure leads to an interpretable Gaussian graphical model for matrix-variate observations, where the total conditional dependence structure factorizes into the separate sub-structures of the raw and column dimensions.

Therefore, to extend VARp1q model to matrix-variate process pX t q t , we can apply the classical VARp1q onto the vectorized process pvecX t q t , then impose KS or KP structure on the coefficient matrix A. Since the sparsity structure of A encodes the topology of causal graph of N F processes px f it q t , i " 1, . . . , N, f " 1, . . . , F , KS and KP both imply that, this total graph can factorize into two subgraphs, but in different ways. To explain this difference, we recall the definition of the Kronecker sum. Let A F P IR F ˆF , A N P IR N ˆN be two square matrices, and let I k denote the k ˆk identity matrix. The Kronecker sum between A F and A N is defined as

A F ' A N " A F b I N `IF b A N ,
where b is the Kronecker product. As mentioned before, the KP and KS structures in adjacency matrices imply the corresponding graphs are the product of the component graphs. More specifically, when A F , A N are the adjacency matrices of two graphs G F , G N , the KP A F b A N and KS A F ' A N are respectively the adjacency matrices of their tensor product graph G N ˆGF and Cartesian product graph G N lG F (Sandryhaila and Moura, 2014a). We illustrate these two graph products in Figure 2.3. For the formal definitions of Cartesian and tensor products of graphs, we refer to [START_REF] Hammack | Handbook of product graphs[END_REF]; Chen and Chen (2015); [START_REF] Imrich | Cartesian products of directed graphs with loops[END_REF].

Figure 2.3 shows that, the product graphs led by the KP and KS differ greatly. For example, the lattice-like structure of the Cartesian product preserves the subgraphs in all sections of both dimensions. By contrast, the tensor product focuses on the cross-dimensional connection, yet abandoning the intra-dimensional dependency. This later property actually refers to, in the Gaussian process literature, the cancellation of inter-task transfer, see for example, Bonilla et al. (2007, Section 2.3). Therefore when Figure 2.3: Comparison of the Cartesian and the tensor products of graphs. The node set of both product graphs is the Cartesian product of the components' node sets, yet follows the different adjacencies. The example is based on Sandryhaila and Moura (2014a, Figure 2). the nodes represent px f it q t , i " 1, . . . , N, f " 1, . . . , F , imposing KP structure (Chen et al., 2021a) implies assuming no causality dependencies among px f it q t , i " 1, . . . , N for each f fixed, which represent the observations of the feature f at different nodes across the network. By contrast, the coefficient matrices endowed with the KS structure is able to take such dependencies into account in the inference, which are in effect present in many applications.

Causal product graphs and matrix-variate AR(1) models

The first contribution lies in the extension of vector AR(1) model to the matrixvariate time series, X t P IR N ˆF , t P Z, by imposing § the KS structure on the coefficient matrix A of the VAR(1) model applied to vecpX t q:

offdpAq " A F ' A N , (2.2.1)
where offdpAq is the matrix of the same size as A, whose off-diagonal part is identical to that of A, while its diagonal elements are equal to zero. As stated in Section 2.2.1.1, for a stationary VAR(1), the sparsity structure of the coefficient defines the causal graph of the N F components of vecpX t q. By embedding of the KS, we then assume this total causal graph factorizes as the Cartesian product into the spatial graph G N and the feature graph G F , where G N is preserved in each feature, and G F preserved in each node. A related work which also extends the VAR(1) model to the matrix-variate AR(1) model is Chen et al. (2021a), where they propose to impose the KP structure onto the adjacency matrices. However, as we indicated in Section 2.2.1.2, the KP is not able to capture the intra-dimensional causality.

Given the samples X 0 , X 1 , . . . , X t , we then develop two approaches to estimate A N , A F and identify the sparsity structure, in order to infer the graph topology. The § We only show the key structure in this summary. The complete structure assumed for A see Chapter 4.

approaches are designed, respectively, for the low and high dimensional learning. We especially consider a general learning framework where the partial sparsity is pursued in the estimation of only A N . This is motivated by merely a very small number of features F usually present in applications. Thus, the feature graph can be reasonably assumed fully-connected. On the other hand, since the partial sparsity constraint is also a technically more complicated case for the proposed high dimensional learning method, given its corresponding resolution, the adaption to the case of fully sparsity does not require novel techniques.

In low dimension, we propose the projected OLS (equivalently GLS) estimator to benefit the asymptotic properties of the classical estimators. More specifically, we first construct an orthogonal basis pU k q k of the space K G , which is a linear space formed by all the IR N F ˆN F matrices of the structure (2.2.1). Relying on pU k q k , we are able to express, Proj G , the projection onto K G explicitly, which furthermore defines the estimators y

A N,t and x A F,t through the KS. Such y A N,t is element-wise linear function of the classical OLS estimator of A. By applying the Cramer-Wold theorem on the CLT of the classical OLS estimator, we then derive the CLT for y A N,t . This CLT moreover allows the establishment of Wald test to identify the sparsity structure of A N .

Novel Lasso type: Structured matrix-variate Lasso and its Homotopy Algorithms

As discussed at the end of Section 2.2.1.1, a common practice in the literature to identify the sparsity structure of VAR coefficients in high dimensional regime is to adopt Lasso estimators. The one used in [START_REF] Bolstad | Causal network inference via group sparse regularization[END_REF]; [START_REF] Zaman | Online topology identification from vector autoregressive time series[END_REF] is defined as the minimizer of Lasso problem (2.2.2) in the VAR(1) case.

min A 1 2t t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λt }A} ℓ 1 , (2.2.2)
where x τ is a vector of sample, which can be taken as vecpX τ q for example. Lasso (2.2.2) is the most standard Lasso in literature (Hastie et al., 2009, Section 3.4.2). A wide variety of frameworks from convex analysis and optimization have been adapted to compute its solutions for different scenarios, for example, coordinate descent [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF], proximal gradient methods [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], and a more Lasso-specific technique least angle regression [START_REF] Efron | Least angle regression[END_REF]. However, Lasso (2.2.2) is not able to estimate the structured A with the sparse component A N .

Therefore motivated by the estimation, we propose the novel Lasso type problem (2.2.3)

Apt, λ t q " arg min APK G 1 2t t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λt F }A N } ℓ 1 , (2.2.3)
The ordinary resolution of Lasso (2.2.3) can be done by applying for example the proximal gradient descent [START_REF] Parikh | Proximal algorithms[END_REF]. In the algorithm framework, the structure constraint and the partial sparsity do not pose the additional difficulties, since only the gradient with respect to IR N F ˆN F is calculated in the forward step.

Introduction

Then the backward step amounts to a standard Lasso after projecting the gradient onto K G , as shown in Equation (2.2.4).

A k`1 " proxpA k ´ηk ∇f pA k qq, " arg min

APK G 1 2η k › › ›A ´´A k ´ηk ∇f pA k q ¯› › › 2 ℓ 2 `λt F }A N } ℓ 1 " arg min APK G 1 2η k › › ›A ´Proj G ´Ak ´ηk ∇f pA k q ¯› › › 2 ℓ 2 `λt F }A N } ℓ 1 ðñ $ ' ' ' & ' ' ' % A k`1 N " arg min A N › › A N ´Proj G N `Ak ´ηk ∇f pA k q ˘› › 2 ℓ 2 `2η k λ t }A N } ℓ 1 , A k`1 F " Proj G F `Ak ´ηk ∇f pA k q ˘, diagpA k`1 q " Proj D `Ak ´ηk ∇f pA k q ˘,
(2.2.4) where ∇f pA k q " A k p Γ t p0q ´p Γ t p1q, we denote A k`1 pt, λ t q by A k`1 to avoid the heavy notation.

At this point, we focus on providing the algorithms to quickly update the previous solutions for the change in the hyperparameter value or the data term. To benefit the previous solutions of a problem, one needs to consider the specific methods. For standard Lasso (2.2.2), the framework of homotopy continuation methods [START_REF] Osborne | A new approach to variable selection in least squares problems[END_REF] has been explored [START_REF] Malioutov | Homotopy continuation for sparse signal representation[END_REF][START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF] to calculate the fast updating. Since the homotopy algorithm is derived from the optimality condition, which is with respect to the matrices in K G for Lasso (2.2.3), requiring to consider the gradient with the structure, thus the existing homotopy algorithms for Lasso (2.2.2) are not applicable. Therefore in our work, we first calculate the optimality condition of Lasso (2.2.3), based on the expression of projection onto K G . Then we derive the two Homotopy algorithms, respectively, for the updating paths Apt, λ 1 q Ñ Apt, λ 2 q and Apt, λ 2 q Ñ Apt `1, λ 2 q, together with an adaptive tuning procedure for the regularization hyperparameter at the arrival of new observation x t`1 . The derivations do not depend on the specific structure constraint, nor on the particular sparsity regularization, and thus can be applied to other linear structure space K G and the design of sparsity.

Online graph and trend learning

In the applications where the data storage is limited and fast inference upon the sequentially arriving data is demanded, the ordinary resolution which processes the entire data set at once fails. Thus the tailored algorithms which fulfill these two conditions are needed. This class of algorithm are referred to as online. By contrast, the ordinary resolution is referred to as off-line.

For the online resolution of Lasso (2.2.3), the previously derived homotopy algorithms constitute the pertinent method, when performing the three steps in the order:

Step 1 :

λ t Ñ λ t`1 , Step 2 : Apt, λ t q Ñ Apt, λ t`1 q,
Step 3 : Apt, λ t`1 q Ñ Apt `1, λ t`1 q, (2.2.5) However, when passing to the online inference, the formulation of the problem

Problems and main contributions

(2.2.3) can not fully adapt, due to the data term 1 t ř t τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 . The data term implies that the time series px τ q τ , τ P Z is supposed to admit a time-invariant mean, which is zero. Nevertheless, when the px τ q τ take raw data, this sample assumption is very often not true. Instead, the raw time series is usually present with the trend, that is a time-variant mean function. Thus, in off-line learning, a detrend step is needed, which approximates the trend function using the entire data set, then removes it from the raw data. Then the actual time series fitted to the model will be closer to the model assumptions. However, since the principle of online learning does not require the presence of all data, such pre-processing step is forbidden. Thus, we need to consider the trend as the explicit parameters additional to the graph parameters A N , A F in the online model.

The present work focuses primarily on a particular type of trend, that is the periodic trend, which is frequently encountered in practice. For example, an annual recurrence for every 12 months can be found in many data sets, recorded monthly over years. From the aspect of modelling, the periodic trend is the mean function:

Ex τ " b 0 m , m " 0, ..., M ´1, m " τ mod M,
where M is the length of period. We thus propose to reformulate Lasso (2.2.3) by incorporating the periodic trend b 0 m , m " 0, . . . , M ´1, as Equation (2.2.6).

Apt, λ t q, b 0 m pt, λ t q " arg min

APK G ,b 0 m 1 2t M ´1 ÿ m"0 ÿ τ PIm,t }px τ ´b0 m q´Apx τ ´1´b 0 m´1 q} 2 ℓ 2 `λt F }A N } ℓ 1 ,
(2.2.6) where I m,t " tτ " 1, ..., t : τ mod M " mu. Note that b 0 ´1 denotes b 0 M ´1. The underlying sample model becomes

x t´1 ´b0 m " Apx t´1 ´b0 m´1 q `zt , A P K G , m " t mod ¶ M,
which is an augmented VAR(1) model with periodic trend.

We then adapt the algorithms in Pipeline (2.2.5) to Problem (2.2.6), which finally allows the online graph and trend learning from the matrix-variate time series.

Wasserstein statistics for distributional data

Secondly, we would like to extend the VAR(1) model for a collection of N timedependent probability measures pµ i t q tPZ , i " 1, . . . , N (example of such data see Section 2.1), which can additionally identify and represent the significant dependency links among the processes in a directed weight graph of N nodes. Instead of a vector, now the node feature becomes a distribution; to model such distributional data, we need to adopt more advanced tools.

Since distributions can be characterized by certain functions, such as densities, quantile functions, and cumulative distribution functions, then to analyze the distributional time series, one may turn to study one of its functional representations with the tools from functional time series analysis [START_REF] Bosq | Linear processes in function spaces: theory and applications[END_REF]. However, due to their nonlinear constraints, such as monotonicity and positivity, the representing functions of distributions do not constitute linear spaces. Consequently, basic notions in standard VAR models, such as additivity and scalar multiplication, do not adapt, in a straightforward manner. This causes models devised for random elements of a Hilbert space to fail. One existing approach is to map the densities of distributions to unconstrained functions in the Hilbert space by the log quantile density (LQD) transformation [START_REF] Petersen | Functional data analysis for density functions by transformation to a hilbert space[END_REF], and then apply the functional tools [START_REF] Kokoszka | Forecasting of density functions with an application to cross-sectional and intraday returns[END_REF]. However LQD does not take into account the geometry of the distribution space, thus, it can lead to deformations in the distance. Recent approaches consider such geometry by adopting the Wasserstein metric [START_REF] Bigot | Geodesic pca in the wasserstein space by convex pca[END_REF][START_REF] Panaretos | Amplitude and phase variation of point processes[END_REF]Petersen and Müller, 2019b). We therefore establish the time series model in the Wasserstein space. In the following, we present the main contribution in this line of work. For the technical introduction of the statistics in Wasserstein space, we refer to Section 3.3.

Wasserstein multivariate AR(1) model and the N -simplex constraint

There are also other works which have considered the distributional time series models. They so far mainly focus on the univariate case, that is when N " 1. Generally, the multivariate distributional models are still barely developed. Note that in the distributional case, two multivariate notions are relevant, one refers to the dimension of the measure support, while the other one refers to the numbers of measures. Models associated to both notions are undeveloped. In this manuscript, the word multivariate refers to the second notion.

We refer to the recent works Chen et al. (2021b); [START_REF] Zhang | Wasserstein autoregressive models for density time series[END_REF]; [START_REF] Zhu | Autoregressive optimal transport models[END_REF], which develop the univariate distributional AR model in the Wasserstein space W 2 pIRq. They propose to map the random distributions to the Logarithmic image of Wasserstein space at their presumed common Fréchet mean. Then they construct the univariate functional time series models in terms of the logarithmic maps. The common Fréchet mean is the ideal reference distribution in the sense that it cancels out the expectations of the logarithmic maps in the regression formula. However, for multivariate time series, different processes can not have one common Fréchet mean. Moreover such ideal reference measure is very unlikely to exist (will be explained in Chapter 5). On the other hand, an extra intercept in such functional regression model will cause great difficulties, when one wants to retain the regression model especially the predictions in the logarithmic image of Wasserstein space. Thus to deal with the unequal process means, we are motivated by the equivalent formulation of VAR(1) model built on the centered series which thus does not include the intercept term (details see Equation (3.2.5) in the preliminaries section)

x t ´u " Apx t´1 ´uq `zt .

We firstly propose a way to center all the raw distributions by their process means (Fréchet) so that their means turn to be the uniform distribution namely the Lebesgue measure. Secondly we build the model on the centered data r µ i t , i " 1, . . . , N, t P Z. For each i, we define the key regression relationship for the response r µ i t and the predictors p r µ j t´1 q j , j " 1, . . . , N in the tangent space of the Lebesgue measure, as the linear combination of the logarithmic maps of p r µ j t´1 q j , weighted by coefficients pA ij q j . The response r µ i t equals the exponential map of the linear combination, pushforwarded by a random distortion function. Therefore the total coefficients of the regression system defines a matrix A " pA ij q ij as in VAR(1) model. For general functional regression models with function response, the coefficients can be concurrent A ij p¨q or the most general a surface A ij p¨, ¨q, see Wang et al. (2015, Equations ( 14) and ( 15), respectively). Nevertheless, we purpose to use matrix form for the regression coefficients in order to represent the graph structure directly.

Since the exponential map is not injective, the model proposed above has an identifiability problem. Thus we furthermore retain the model in the logarithmic image by adding the N -simplex assumption A2 to the coefficients (see below), given the injectivity of the restricted exponential map on the logarithmic image and the convexity of the logarithmic image. Actually, when falling out of the logarithmic image, the calculation in the estimation problem is not tractable.

Assumption A2. ř N j"1 A ij ď 1 and 0 ď A ij ď 1.
We then show that the iterated random function system [START_REF] Wu | Limit theorems for iterated random functions[END_REF]) associated to the proposed time series model admits a unique solution, which is moreover stationary as the functional process in the Hilbert space of N -tuple of quantile functions endowed with the L 2 inner product, given the additional contraction assumptions on the regression map, which can be related to the stationarity condition of VAR models.

Lastly, we rely on the least squares method to derive the estimator of A, which minimizes the sum of squared residuals under the simplex constraint A2. We also provide the consistency guarantee for the proposed estimator.

Application in graph learning from distributional time series

Due to the simplex constraint A2, the derived estimator naturally carries sparsity. Thus when interpreting the coefficient matrix as the adjacency matrix of N nodes, we can retrieve a weighted directed graph from its estimate, which represents the dependency structure of N distributional processes pµ i t q tPZ , i " 1, . . . , N . Figure 2.4 shows the inferred age structure graph of countries from the data illustrated in Figure 2.2. The full list of the investigated countries can be found in Chapter 5.

Multivariate distribution-to-distribution regressive model

The proposed data centeralization approach for random distribution can also serve in the development of the multivariate distribution-to-distribution regressive model for the i.i.d. data, which is also very little developed. The existing works on the distribution-to-distribution regressive model mainly consider the 1 predictor 1 response case. In [START_REF] Chen | Lqd-rkhs-based distribution-todistribution regression methodology for restoring the probability distributions of missing shm data[END_REF], the kernel ridge regression of the transformed density functions by LQD are constructed. Even though the regression can be extended to multivariate case by expanding the domain of the reproducing kernel, as stated in Section 2.2.1.6, LQD transformation does not take into account the geometry of the distributional space. More recently, both Chen et al. (2021b); [START_REF] Ghodrati | Distribution-on-distribution regression via optimal transport maps[END_REF] rely on the Wasserstein metric. Chen et al. (2021b) performs the regression using the coefficient operator which maps from the Tangent space of the predictor's Fréchet mean to that of the response's. By contrast, [START_REF] Ghodrati | Distribution-on-distribution regression via optimal transport maps[END_REF] performs the regression directly in Wasserstein space. The predictor is pushforwarded by the coefficient optimal map to the conditional Fréchet mean of the response. It is not straightforward to extend the two models to more predictors since the related operations are both directional. That is, it is not evident how to map between multiple predictor tangent spaces or push among the predictor measures. Thus, the proposed data centering method in this work can provide a new trick to handle multiple predictors and responses in this more general regression framework.

Node predictability in time-dependent graph signals

To measure the predictability of nodes in a network, we firstly introduce the prediction problem for the network observations px it q t , i P N :" t1, . . . , N u, where we hypothetically reconstruct the observations at time t over a subset I of nodes,

x it , i P I Ă N , given the past and present recordings on the rest of nodes px jτ q jτ , j P I c , t ´H ď τ ď t. We would like to compare the predictability evaluated by the two popular prediction methods in the statistic and graph signal processing domains: kernel ridge regression and neural network. Additionally, when the edges of nodes are not given, we consider linear regression. For the two regression methods, we obtain the ranking through the greedy adaptation of the sensor selection problem of optimal I, by relating the prediction errors of observations over I to the selection criteria. For the neural network, we measure the node predictability in a finer way by providing each node a score. We propose to adopt the dropout scheme to vary the turned-off nodes in I, obtaining the aggregated prediction error for each node, in one training.

In the following, we present these contributions in a more technical context. This section corresponds to the preprint [START_REF] Jiang | Sensor selection on graphs via data-driven node sub-sampling in network time series[END_REF].

Kernel ridge regression of time and node predictors

Firstly, we adapt the kernel ridge regression to the introduced prediction problem by regressing the observation x iτ P IR on the node and time indices i P N and τ P Z.

We propose a kernel design on N ˆZ that takes into account the graph structure and defines the similarity of time points. The kernel ridge regression then approximates the regression function by the functions in the induced reproducing kernel Hilbert space (RKHS) H k , with the corresponding estimator given by

p f t " min f PH k ÿ iPI c ,t´Hďτ ďt }x iτ ´f pi, τ q} 2 ℓ 2 `λ}f } 2 H k . (2.2.7)
By the RKHS-based prediction, the predictability of a node comes from the similarity between its observations and the recent observations over its neighbourhood. For the optimization as Equation (2.2.7), the representer theorem indicates the explicit form of solution. Thus, given a history of network observations, we derive the expression of the total prediction error ř iPI ř H`1ďtďT px it ´p f t pi, tqq 2 {T , which depends on the sample auto-covariance of process px it q t , i P N and Gram matrix of the proposed kernel. The derived error gives a way to quantify the predictability of nodes, which however depends on the choice of I. We propose to rely on the sensor selection problem taking the total prediction error as the criteria, to take into account other possible compositions and cardinalities of I. For a given cardinality 1 ď p ď N ´1, the problem searches the optimal set I ˚ppq which minimizes a certain criteria among all the sets I of p nodes. Especially, since the adapted greedy strategy selects the nodes one by one, their order forms the optimal sets of cardinality from 1 to N ´1 incrementally. Thus we propose to rank the nodes as the order produced by such a greedy strategy. The resulting ranking can be interpreted as follows: for every given cardinality p, we assign the nodes in the optimal set I ˚ppq score 1, the rest nodes score 0. The total scores for all p's from 1 to N forms the ranking.

Varying the prediction set I by dropout

Since the neural network has been widely adapted to prediction problems in the domain of graph signal processing, we are especially interested in evaluating its is the dropout vector masking the input over nodes in I. I is sampled again at every batch n. After the training converges, the network is tested on the validation set as illustrated on the right, whose reconstruction errors of each node gives its score. predictability over nodes ‖ . For the network predictor, we may set the training loss as the total prediction error given in Section 2.2.2.1 and then still relate it to the criteria in sensor selection so as to rely on the greedy strategy. However, this leads to OpN 2 q times network training, which is much more costly than the matrix operations. Therefore, we design a training scheme for the network predictor, which varies the composition of I hence gives the scores of all nodes within one training. The key idea is illustrated in Figure 2.5. By applying the dropout and the reverse dropout on the input and the batch loss, the trained network aggregates the prediction models related to different I's. In our experiment, thousands of compositions of I have been sampled during the training for N on the order of one hundred. Therefore, the prediction error given by such network represents the overall predictability of a node, when it is predicted within different I's. We propose to use the trained network to predict the network observations of the validation set, and take the prediction error of each node as its score, measured by the metric R 2 . Figure 2.6 shows the derived scores for the bike stations in Paris, calculated from a history of observation of 3533 hours, which records the ratio of available bike at each station and hour. These scores show the interesting result that the stations with high predictability forms several geographical groups. We focus on two groups. One goes along the main itinerary of Metro 1 and RER A to the central business district La defense (yellow white points on the right bank of the Seine river). The other is located around the train station Paris Montparnasse (yellow white points on the left bank). These two geographical groups will be found again in Chapter 5 by the proposed Wasserstein AR model in its graph inferred on the same data set, where the regression formulas predicting the stations inside the groups are much more significant, with larger magnitude of the nonzero coefficients.

These observations strongly support the effectiveness of both proposed methods.

‖ For the network predictor, we only consider the prediction problem with H " 0, because we did not observe empirically the improvement in the reconstruction for larger H, when applying the learned ranking in the sensor selection.

Partial variance of multivariate stochastic process

Lastly, we would like to derive the learning method for the node predictability, when the information on the node connectivity in a network is not available. We may firstly apply a graph learning method for the scalar-type node observation, such as VAR models, to infer the edges, then reuse the methods proposed in Sections 2.2.2.1 and 2.2.2.2. However, since this pipeline uses same data twice, the data noise will be amplified and cause larger bias in the fitted models. Thus we purpose to consider the time-dependent signals as multivariate process, and rely on the graph-free prediction methods.

An immediate notion in statistics which measures the predictability of a random variable x i given other variables x j , j ‰ i is the partial variance of x i given x j , j ‰ i, which is essentially defined as the variance of residual of x i predicted by x j , j ‰ i through the linear regression. We then rely on the partial variance of x it , i P I Ă N given px jτ q jτ , j P I c , t´H ď τ ď t to quantify the node predictability when the graph is not given. The total partial variance of x it , i P I given px jτ q jτ , j P I c , t´H ď τ ď t is a counterpart of the total prediction error given in Section 2.2.2.1. Especially, when estimating the total partial variance of x it , i P I given px jτ q jτ , j P I c , t ´H ď τ ď t by the sample auto-covariance of process px it q t , i P N , the estimator can be written as a special case of the derived total prediction error in Section 2.2.2.1, with the kernel defined otherwise by the sample auto-covariance, and λ " 0. We then regard the total partial variance as another criteria for the sensor selection problem, and rely on the greedy strategy to obtain the desired ranking as in Section 2.2.2.1.

Application in sensor selection

Given the derivation of the rankings, for any cardinality 1 ď p ď N ´1, the top p nodes can also be considered as the optimal sets over which the observations can be predicted collectively the best by the rest of the nodes. Such a selection strategy considers the aspect of dependency that can be estimated from the data, in additional to the graph structure. In the experiments of Chapter 6, we show that the proposed strategy exhibits advantages in the reconstruction of missing data when compared to standard sampling methods in graph signal processing that rely only on the graph structure, such as Anis et al. (2016a); [START_REF] Puy | Random sampling of bandlimited signals on graphs[END_REF].

Organization of the thesis

Chapter 3 We present the necessary backgrounds to start the analysis of Chapters 4 -6.

Chapter 4 We develop the approaches for the network data of vectorial node observations. We propose the matrix-variate AR(1) model and derive the online procedures to infer its characterising graph from the sequential trend data, in both low-and high-dimensions. This chapter is related to the preprint [START_REF] Jiang | Online graph topology learning from matrixvalued time series[END_REF].

Chapter 5 We focus on analysing the distributional node observations. We propose the Wasserstein multivariate AR(1) model and the sparse estimator of its coefficients, with theoretical results. This chapter is related to the preprint [START_REF] Jiang | Wasserstein multivariate auto-regressive models for modeling distributional time series and its application in graph learning[END_REF].

Chapter 6 This chapter is devoted to the problem of learning node predictability from the scalar-type network data. We propose the ranking strategies for the linear, RKHS, and neural network predictions. This chapter is related to the preprint [START_REF] Jiang | Sensor selection on graphs via data-driven node sub-sampling in network time series[END_REF].

Chapter 3

Preliminaries

Brief introduction of graphs

The most common mathematical object used to represent a network of sensors is a graph. A graph G " tN , E, Au consists of a finite set of nodes (or vertices) N " pV i q i with |N | " N , a set of edges E " pi, jq i,j , and an adjacency matrix A " pA ij q i,j which stores the connectivity information. For an undirected graph G, if there is an edge pi, jq connecting nodes i and j, the entry A ij equals the weight of the edge (equals 1 if the graph is unweighted); otherwise A ij " 0. In this case, the adjacency matrix is symmetric. By contrast, the adjacency matrix of directed graph is asymmetric. In this case, A ij represents the weight (or 1) if and only if there is an edge from node i to node j (or defined as from node j or node i in some applications). A graph signal on G is defined as a mapping h : N Ñ IR with hpV i q representing the observation at node i.

An important notion which allows the development of tools for graph signal processing is the graph Laplacian. Here we focus on the case of undirected graphs with real weights, which is the most commonly considered. For the graph Laplacian defined on directed graphs, we refer to [START_REF] Wu | On rayleigh-ritz ratios of a generalized laplacian matrix of directed graphs[END_REF]; [START_REF] Bauer | Normalized graph laplacians for directed graphs[END_REF]. The combinatorial Laplacian is defined as

L " D ´A, (3.1.1)
where A is the adjacency matrix and D P IR N ˆN is the degree matrix, that is, a diagonal matrix whose i-th diagonal element is equal to the sum of weights of all the edges incident to node i. One can normalize the combinatorial Laplacian to obtain the symmetric normalized Laplacian, which is defined as

L sym " pD `q 1 2 LpD `q 1 2 , (3.1.2)
where D `is the Moore-Penrose inverse of D. It is easy to find that the graph Laplacians defined in Equations (3.1.1) and (3.1.2) are positive semi-definite, thus allowing orthonormal eigendecomposition with non-negative eigenvalues as

L " Φ » - - - - 0 λ 1 . . . λ N ´1fi
ffi ffi ffi fl

Φ J :" ΦΛΦ J .
With real matrix and real eigenvalues, the eigenvectors can be chosen as all real. As for the classical one-dimensional Laplacian operator, the eigenvectors Φ of the graph Laplacian define the Fourier basis in the graph domain. Thus the Graph Fourier transform is given by ĥ " Φ J h, where h " hpV i q i P IR N is a graph signal.

Stationary process and vector AR models

Stationary process

A multivariate stochastic process px t q t is a collection of random vectors in IR N indexed by t P Z. We can characterize px t q t by its means and auto-covariance matrices (if these exist), defined respectively in Equations (3.2.1) and (3.2.2) as u t " Ex t , @t, P Z.

(3.2.1)

Γpt, t ´hq " Epx t ´ut qpx t´h ´ut´h q J , @t, h P Z.

(3.2.2)

The stochastic process px t q t is stationary if and only if

E}x t } 2 ℓ 2 ă 8, u t " u 0 , and Γpt, t ´hq " Γp0, ´hq, @t, h P Z.

The stationarity is in the weak-sense throughout this manuscript, otherwise stated if not. Since the mean u t and auto-covariance matrices Γpt, t ´hq for the stationary process do not depend on the time instant t, we denote them furthermore by u and Γphq, h P Z. Especially, the auto-covariance Γp0q at h " 0 will also be called variance in this manuscript. Given the time series data x 1´p , ..., x 0 , x 1 , ..., x T , p P Z and 1 ď p ď T ´1, we can then estimate the stationary mean u and stationary auto-covariances Γphq by the sample mean and sample auto-covariances, defined as

û " 1 T T ÿ τ "1
x τ , and p Γphq "

$ ' & ' % 1 T T ř τ "1
px τ ´ûqpx τ ´h ´ûq J , h " 0, ..., p, p Γp´hq J , h " ´1, ..., ´p.

(3.2.3) Note that, notation-wise, we do not distinguish between the stochastic process and a time series which is a realization of a stochastic process. When stationarity holds, these estimators are consistent under a mild condition, as stated in Theorem 3.2.1.

Theorem 3.2.1. (Neusser, 2016, Theorems 11.1, 11.3) Let px t q t , t P Z be a stationary process, that can be written as

x t " u ``8 ÿ j"´8 Ψ j z t´j ,
where z t " IIDp0, Σ z q with bounded forth moments, ř `8 j"´8 }Ψ j } F ă 8 and ř `8 j"´8 Ψ j ‰ 0. IIDp0, Σ z q denotes independent and identically distributed with mean and variance respectively 0 and Σ z . Then we have û p ÝÑ u, and p Γphq p ÝÑ Γphq.

The summability condition is quite general, and in particular, it is fulfilled by the VAR process introduced in Section 3.2.2.

Vector AR model of order 1 and estimation of coefficients

A powerful tool to analyze multiple scalar time series is VAR model. Let px t q t , t P Z be a random process in IR N . px t q t is called VAR(1) process if it is the solution of the stochastic difference equation

x t " b `Ax t´1 `zt , t P Z, (3.2.4)
where z t " WNp0, Σq is a white noise process with the variance Σ not singular, A P IR N ˆN is the coefficient matrix, and b P IR N is the intercept to cancel out the unequal expectations in the regressive formula, in other words, nonzero means Ex t and AEx t . When Ex t are known to be time-invariant, that is Ex t " u, t P Z, the VAR(1) model admits another formulation reparameterized in the unknown process mean u and the coefficient matrix A. In this case, we can find by taking expectation on both sides of Equation (3.2.4) that, the constant b is determined by u and A b " pI ´Aqu, (3.2.5)

where I is the identity matrix. Plugging this relation in the regressive formula (3.2.4), we then obtain the VAR (1) model reparametrized by u and A

x t ´u " Apx t´1 ´uq `zt , t P Z.

(3.2.6)

Therefore we can directly study the centered series x t ´u, and consider the VAR(1) model without intercept. The core of time series analysis is the study of stationary processes. Thus it is important to establish the stationarity condition for the VAR models. We present in Theorem 3.2.2 such a condition. Theorem 3.2.2. When }A} 2 ă 1, the stochastic difference equation defined in Equation (3.2.4) admits the unique stationary solution

x t " pI ´Aq ´1b `8 ÿ j"0 A j z t´j ,
where the infinite sum exists in L 2 .

A stationary VAR(1) process thus refers to such a unique solution. Since stationarity implies the time-invariant process mean, the reparameterized formulation (3.2.6) then holds for stationary VAR(1) process. Multiplying px t ´uq J on the right and taking expectation on both sides, we can obtain the following representation of the coefficient matrix

A " Γp1q rΓp0qs ´1 , (3.2.7)
where Γp0q, and Γp1q P IR N ˆN are the auto-covariance matrices defined in Equation (3.2.2) of the stationary process.

The VAR(1) model can then be extended to VAR(p) with p time lags by

x t " b `p ÿ l"1 A l x t´l `zt , t P Z.
However, VAR(p) models will not be exploited in the present work yet. For more details on VAR models, we refer to the following books: Lütkepohl (2005); Brockwell and Davis (2009); [START_REF] Neusser | Time series econometrics[END_REF].

Given the time series data x 0 , x 1 , ..., x T , we can then estimate the coefficient A by the general least squares (GLS) method or by the ordinary least squares (OLS) method, defined respectively through the optimisation problems (3.2.8) and (3.2.9) below. For the regression model (3.2.4) and (3.2.6), GLS and OLS estimators are equivalent (Lütkepohl, 2005, Equation 3.2.8), because the predictors in each equation of x it are the same. where û, p Γp¨q are the sample mean and sample auto-covariances defined in Equation (3.2.3). Analogously, when stationarity holds, these estimators are consistent, given the representation (3.2.7) and the consistency result 3.2.1. Furthermore they permit the central limit theorem (CLT), as stated in Theorem 3.2.3. Theorem 3.2.3. [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF], Proposition 3.1) Let px t q t be a stationary VAR(1) process defined as the solution of Equation (3.2.4), with additionally z t " IIDp0, Σq with bounded forth moments. Then we have

1 ? T vec ˜p b ´b p A ´A¸d ÝÑ N ˜0, ˆ1 u J u Ex t x J t ˙´1 b Σ
Applying the formula of the block matrix inversion, we can obtain the marginal distribution of p

A 1 ? T vec ´p A ´A¯d ÝÑ N `0, Γp0q ´1 b Σ ˘.

Granger causality and Wald test

The formal definition of the Granger causality is: Definition 3.2.4. [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF], Section 2.3.1) Let z t P IR, y t P IR be two univariate processes. Suppose that Ω t is the information set containing all the relevant information in the universe available up to and including period t. Let z t ph|Ω t q be the optimal (mean squared error sense) h-step predictor of z t at origin t, based on the information in Ω t . The corresponding forecast mean squared error is denoted by Σ z ph|Ω t q. The process y t is said to cause process z t in Granger's sense if

Σ z ph|Ω t q ă Σ z ph|Ω t zty τ |τ ď tuq, for at least one h P N `,
where Ω t zty τ |τ ď tu is the set containing all the relevant information except for the information in the past and present of y t .

As mentioned in Section 2.2.1.1 in the introduction, for a stationary VAR(p) model, time series px it q t causes px jt q t if and only if all the ji-th entries of the true coefficient matrices A l ji ‰ 0, l " 1, . . . , p, [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF], Corollary 2.2.1). Thus, the causal graph is defined by the sparsity structure of the coefficient matrices A l of a stationary VAR process. In low-dimensional regime, this structure can be identified by testing the zero constraints for the entries of A l via the Wald test. The Wald test considers generally the hypotheses * :

H 0 : CvecpAq " c versus H 1 : CvecpAq ‰ c,
where C P IR P ˆN 2 is of rank P with P an arbitrary integer, and c P IR P . The test statistic is given by

λ W " T αJ " C ΣW C J ı ´1 α, where α " CvecpAq´c, ΣW " " p Γp0q ı ´1 b p
Σ with p Σ some consistent estimator of the variance of noise Σ. Based on the CLT in Theorem 3.2.3, the asymptotic distribution of λ W is Corollary 3.2.5. (Lütkepohl, 2005, Proposition 3

.5) λ W d ÝÑ χ 2 pP q Under H 0 .

Statistics in Wasserstein space

In this section, we provide the required preliminaries to establish the proposed Wasserstein model in Chapter 5 for distributions on an interval of IR. For comprehensive reviews on the statistics Wasserstein in space, we refer to Bigot (2020); Panaretos and Zemel (2020); [START_REF] Petersen | Modeling Probability Density Functions as Data Objects[END_REF].

Let Ω be a (possibly unbounded) interval in IR, and BpΩq the associated σ-algebra made of Borel sets of Ω. Let µ be a probability measure (namely a distribution) over pΩ, BpΩqq with cumulative distribution function (cdf) F µ . Then the (generalized) quantile function is defined as the left continuous inverse of F µ , denoted by F ´1 µ , that is F ´1 µ ppq :" inftx P Ω : F µ pxq ě pu, p P p0, 1q.

The Wasserstein space W :" W 2 pΩq is defined as the set of probability measures over pΩ, BpΩqq with finite second moment, that is endowed with the L 2 Wasserstein distance

d W pµ, νq " ˆż 1 0 " F ´1 µ ppq ´F ´1 ν ppq ‰ 2 dp ˙1{2 , µ, ν P W 2 pΩq. (3.3.1)
It is well known that W is a complete and separable metric space.

Tangent bundle

The space W has a pseudo-Riemannian structure [START_REF] Ambrosio | Gradient flows: in metric spaces and in the space of probability measures[END_REF]. Let γ P W be an absolutely continuous measure, the tangent space at γ is defined as

Tan γ " ttpF ´1 µ ˝Fγ ´idq : µ P W, t ą 0u L 2 γ pΩq
, where id is the identity function, L 2 γ pΩq is the Hilbert space of γ square integrable functions on Ω, with inner product x¨, ¨yγ defined by xf, gy γ :" ş Ω f pxqgpxq dγpxq, f, g P L 2

γ pΩq, and the induced norm } ¨}γ . The exponential and the logarithmic maps at γ are then defined as follows.

Definition 3.3.1. The exponential map Exp γ : Tan γ Ñ W is defined as

Exp γ g " pg `idq#γ, (3.3.2)
where for any measurable function T : Ω Ñ Ω and µ P W, T #µ is the pushforward measure on Ω defined as T #µpAq " µptx P Ω : T pxq P Auq, for any set A P BpΩq.

Definition 3.3.2. The logarithmic map Log γ : W Ñ Tan γ is defined as

Log γ µ " F ´1 µ ˝Fγ ´id.
Note that the exponential map (3.3.2) is not a local homeomorphism [START_REF] Ambrosio | Gradient flows with metric and differentiable structures, and applications to the wasserstein space[END_REF]. Nevertheless, when restricted to the image of the logarithmic map, it becomes an isometry [START_REF] Bigot | Geodesic pca in the wasserstein space by convex pca[END_REF] as stated in the following proposition.

The Wasserstein distance in Equation (3.3.1) can be interpreted as } Log Leb µ Ĺog

Leb ν} Leb , which is the distance between Logarithmic maps of µ and ν in the tangent space at γ " Leb the uniform distribution namely the Lebesgue measure, over r0, 1s, if r0, 1s P Ω. For the general definition of Wasserstein distance (between probability measures supported in general metric spaces) related to the optimal transport theory, see the references [START_REF] Villani | Topics in optimal transportation[END_REF]; [START_REF] Panaretos | An invitation to statistics in Wasserstein space[END_REF]. We can also remark from the above results that the space of all quantile functions of measures in W, namely Log Leb W `id, is a complete separable metric space with respect to } ¨}Leb .

We recall below the important properties of Log γ W [START_REF] Bigot | Geodesic pca in the wasserstein space by convex pca[END_REF] that are needed in the construction of our proposed model. Proposition 3.3.4. Log γ W is a closed and convex subset of L 2 γ pΩq. Proposition 3.3.5. Let g P Tan γ , then g P Log γ W if and only if g `id is nondecreasing γ-almost everywhere.

Fréchet means in Wasserstein space

Definition 3.3.6. Let µ 1 , . . . , µ T be measures in W. The empirical Fréchet mean of µ 1 , . . . , µ T , denoted by μ, is defined as the unique minimizer of

min νPW 1 T T ÿ t"1 d 2 W pµ t , νq.
It is well known that the empirical Fréchet mean μ admits a simple expression through its quantile function that satisfies

F ´1 μ ppq " 1 T T ÿ t"1
F ´1 µ ppq, p P p0, 1q.

Definition 3.3.7. A random measure µ is any measurable map from a probability space pΩ, F, Pq to the metric space W, endowed with its Borel σ-algebra.

Definition 3.3.8. Let µ be a random measure from probability space pΩ, F, Pq to W. Assume that µ is square integrable, namely Ed 2 W pµ, νq ă 8 for some (thus for all) ν P W. Then, the population Fréchet mean of µ, denoted by µ ' , is defined as the unique minimizer of

min νPW E " d 2 W pµ, νq ‰ .
Note that µ ' also admits a simple expression through its quantile function as

F ´1 µ ' ppq " E " F ´1 µ ppq ‰ , p P p0, 1q.
In the remarks below, we point out two important facts about Fréchet mean and the Logarithmic map.

Remark 3.3.9. The random measure µ has zero expectation in the tangent space at its Fréchet mean, that is E Log µ ' µ " 0, µ ' almost everywhere.

Remark 3.3.10. The Fréchet mean can be calculated using standard expectation in the tangent space at any absolute continuous measure γ P W, that is µ ' " Exp γ pE Log γ µq.

Theory on Iterated random function system

After establishing the multivariate AR model in the Wasserstein space, we would like to analyse the existence, uniqueness and then stationarity of its solution as in Theorem 3.2.2 for the VAR(1) model. We rely on the iterated random function (IRF) system theory in [START_REF] Wu | Limit theorems for iterated random functions[END_REF], where they propose the geometric moment contraction that is a sufficiently mild condition for a general IRF system to be stable in a metric space. Since the results in [START_REF] Wu | Limit theorems for iterated random functions[END_REF] are not presented in terms of the solution of a time series model, thus we also refer to Zhu and Müller (2021, Theorem 1), where even designed for their own model, they essentially represent the stability result, hence inducing a solution for the general time series model in metric space under the geometric moment contraction condition. We then adopt the notation of Zhu and Müller (2021, Theorem 1) to show the induced solution is the unique solution with some moment finite, and finally show the stationarity of such unique solution when the metric space is moreover induced from some Hilbert space. To analyse the proposed AR model, we apply this entire theory by imposing the assumptions under which the model satisfies the geometric moment contraction condition. Additionally, we will also rely on Theorem 3 in [START_REF] Wu | Limit theorems for iterated random functions[END_REF] to derive the consistency of the proposed estimator of the model coefficient. However, we will not repeat the theorem in this section, since the presentation of the theorem allows the direct application onto the estimator of a time series parameter. We refer the readers to [START_REF] Wu | Limit theorems for iterated random functions[END_REF] for the theorem.

In Sections 3.4.1 and 3.4.2, we present the above theory in the corresponding progressive results. We consolidate the proofs of [START_REF] Wu | Limit theorems for iterated random functions[END_REF] and [START_REF] Zhu | Autoregressive optimal transport models[END_REF] to more detailed ones, which are given in Sections 3.4.3. To distinguish the contributions, we cite the original works for each result. The results without citation are developed by this work.

Time series in metric space

Let pX , dq be a complete separate metric space with Borel set X, an iterated random function (IRF) system in the state space pX , dq is defined as

X t " Φ ϵt pX t´1 q, t P Z, (3.4.1)
where ϵ t , t P Z are i.i.d. random objects taking values in a measurable space Θ, ϵ t is almost surely independent of X t´1 for all t P Z, Φ ϵ p¨q :" Φp¨, ϵq is the ϵ-section of a jointly measurable function Φ : X ˆΘ Ñ X . Note that X t , t P Z can also be seen as a X -valued nonlinear auto-regressive process.

We now define r Φ t,m :" Φ ϵt ˝Φϵ t´1 ˝. . . ˝Φϵ t´pm´1q the same way as Theorem 1 in [START_REF] Zhu | Autoregressive optimal transport models[END_REF], and we recall Conditions C1 and C2 from [START_REF] Zhu | Autoregressive optimal transport models[END_REF] as follows.

Condition C1. (Condition 1 in Wu and Shao ( 2004)) There exists Y 0 P X and α ą 0, such that Ipα, Y 0 q :" Ed α pY 0 , Φ ϵt pY 0 qq ă 8.

Condition C2. (Theorem 1 in [START_REF] Zhu | Autoregressive optimal transport models[END_REF]; Condition 2 in Wu and Shao ( 2004)) There exists X 0 P X , α ą 0, r " rpαq P p0, 1q, and C " Cpαq ą 0, such that for all t P Z, we have The random process Ă X t , t P Z is then a solution to the IRF system (3.4.1). This result is indicated in Zhu and Müller (2021, Theorem 1), however, they do not elaborate the proof. We thus provide a proof in Section 3.4.3.3. Lemma 3.4.3. (Zhu and Müller (2021, Theorem 1)) Suppose that Conditions C1 and C2 hold, then Ă X t " lim mÑ8 r Φ t,m pX 0 q, t P Z is a solution of IRF system (3.4.1), almost surely.

Ed α p r Φ t,m pX 0 q, r Φ t,m pXqq ď Cr m d α pX 0 , Xq, @X P X , m P N. ( 3
We gather all the above results in Theorem 3.4.4.

Theorem 3.4.4. (Existence; Wu and Shao (2004, Theorem 2); Zhu and Müller (2021, Theorem 1)) Suppose that Conditions C1 and C2 hold, then the IRF system (3.4.1) almost surely admits a solution Ă X t , t P Z, with the same marginal distribution π.

The uniqueness result provided in Zhu and Müller (2021, Theorem 1) is no longer valid for the general system (3.4.1), which is also not mentioned in Wu and Shao (2004, Theorem 2). Thus, we provide the uniqueness result for the general system in Theorem 3.4.5. For the proof we refer to Section 3.4.3.4.

Theorem 3.4.5. (Uniqueness) Suppose that Conditions C1 and C2 hold, then, if there is another solution S t , t P Z, such that

E " d β pS t , Z 0 q ı ă M, t P Z, (3.4.3)
for some M, β ą 0, and some Z 0 P X (thus for all), then for all t P Z Ă X t " S t , in d, almost surely.

Since Ă X t , t P Z is the unique solution of IRF system (3.4.1), we hereafter denote this solution directly by X t , t P Z. Lastly, we recall the geometric moment contraction result given in Wu and Shao (2004, Theorem 2), which will be used later on in the proof of consistency of the estimators. We reproduce their proof using our notation, in Section 3.4.3.5. Proposition 3.4.6. (Wu and Shao (2004, Theorem 2)) the IRF system (3.4.1) is geometric moment contracting in the following sense: let X " π be independent of X 1 " π, where π is the shared marginal distribution in Theorem 3.4.4, X, X 1 are independent of ϵ m , m ě 1. Let X m pXq, X m pX 1 q, m ě 1, denote the sequences generated by the model (3.4.1) starting respectively from X, X 1 . Then, for all m ě 1, there exist constants D ą 0, s P p0, 1q, such that Ed α pX m pXq, X m pX 1 qq ď Ds m .

(3.4.4)

Time series in Hilbert space

Now, we study the stationarity of the functional time series X t , t P Z. Theorem 3.4.4 has already implied that X t is stationary in the sense that X t d " π, t P Z. However, the stationarity in weak sense for time series requires additionally the auto-covariance to be time-invariant. To this end, we need furthermore a Hilbert structure in X to be able to define the notion of covariance between two random objects. Thus, we suppose pX , dq is furthermore a Hilbert space, whose inner product and the induced norm are denoted respectively by x, y and }}. To make the previous results presented in the metric space pX , dq valid as the results for the Hilbert space pX , x, yq, we assume pX , dq is indeed the induced metric space of pX , x, yq, namely, dpX, Y q " }X ´Y }, X, Y P X .

We recall the conventional definition of stationarity for process in a separable Hilbert space, see for example Zhang et al. (2021, Definition 2.2). We can now give the stationarity result in Theorem 3.4.8. The proof can be found in Section 3.4.3.6. Definition 3.4.7. A random process tV t u t in a separable Hilbert space pH, x¨, ¨yq is said to be stationary if the following properties are satisfied.

1. E }V t } 2 ă 8.

2. The Hilbert mean U :" E rV t s does not depend on t.

3. The auto-covariance operators defined as G t,t´h pV q :" E xV t ´U, V y pV t´h ´U q , V P H, do not depend on t, that is G t,t´h pV q " G 0,´h pV q for all t.

Theorem 3.4.8. Suppose that Conditions C1 and C2 hold with α ě 2, then the unique solution X t , t P Z given in Theorems 3.4.4 and 3.4.5 is stationary in pX , x, yq in the sense of Definition 3.4.7.

The condition α ě 2 is to ensure the existence of a second moment of X t , from which we can define the auto-covariance operators of the time series. Note that for any t P Z E}X t } 2 ď E}X t ´Φϵt pX 0 q} 2 `E}Φ ϵt pX 0 q ´Φϵt pY 0 q} 2 `E}Φ ϵt pY 0 q ´Y 0 } 2 `E}Y 0 } 2 " Ed 2 pX t , Φ ϵt pX 0 qq `Ed 2 pΦ ϵt pX 0 q, Φ ϵt pY 0 qq `Ed 2 pΦ ϵt pY 0 q, Y 0 q `}Y 0 } 2 paq ď `Ed α pX t , Φ ϵt pX 0 qq ˘2 α `pEd α pΦ ϵt pX 0 q, Φ ϵt pY 0 qqq

2 α `pEd α pΦ ϵt pY 0 q, Y 0 qq 2 α `}Y 0 } 2 pbq ď `Ed α pX t , Φ ϵt pX 0 qq ˘2 α `pCrd α pX 0 , Y 0 qq 2 α `Ipα, Y 0 q 2 α `}Y 0 } 2 .
Inequality (a) comes from the condition α ě 2 and Jensen's inequality. Inequality (b) comes from Conditions C1 and C2. Thus, by the upperbound (3.4.5), we have E}X t } 2 ă 8 is uniformly bounded over t P Z. We are first to show that Ipα, X 0 q :" Ed α pX 0 , Φ ϵt pX 0 qq ă 8.

Proofs

We have

Ipα, X 0 q ď Ed α pX 0 , Y 0 q `Ed α pY 0 , Φ ϵt pX 0 qq ď Ed α pX 0 , Y 0 q `Ed α pY 0 , Φ ϵt pY 0 qq `Ed α pΦ ϵt pY 0 q, Φ ϵt pX 0 qq paq ď d α pX 0 , Y 0 q `Ipα, Y 0 q `Crd α pY 0 , X 0 q ă 8. Inequality (a) comes from Condition C2 applied to Ed α pΦ ϵt pY 0 q, Φ ϵt pX 0 qq. Then, by Inequality (3.4.2), we have for all m P N E

" d α p r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ı " E ´E " d α p r Φ t,m pX 0 q, r Φ t,m pΦ ϵ t´m pX 0 qqq|ϵ t´m ıC ondition C2 ď Cr m E " d α pX 0 , Φ ϵ t´m pX 0 qq ‰ " Ipα, X 0 qCr m .
Then by the Markov inequality, we have

P " d α p r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ě r m 2 ı ď r ´m 2 E " d α p r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ı À r m 2 .
Thus

8 ÿ m"1 P " dp r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ě r m 2α ı À 8 ÿ m"1 r m 2 " r 1 2 1 ´r 1 2 ă 8.
Applying the Borel-Cantelli lemma, we have

P " dp r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ě r m 2α infinitely often ı " 0.
Thus, we have that

P " dp r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq ě r m 2α happen finitely times ı " 1.
This implies

P " dp r Φ t,m pX 0 q, r Φ t,m`1 pX 0 qq mÑ8 ÝÑ 0 ı " 1.
Thus r Φ t,m pX 0 q is a Cauchy sequence in pX , dq, almost surely. By the completeness of pX , dq, there exists a Ă X t P X , such that r Φ t,m pX 0 q mÑ8 ÝÑ Ă X t almost surely. Moreover, since for any fixed t,

t 1 P Z, r Φ t,m pX 0 q d " r Φ t 1 ,m pX 0 q, @m P N. Thus lim mÑ8 r Φ t,m pX 0 q d " lim mÑ8 r Φ t 1 ,m pX 0 q, namely Ă X t d " Ă X t 1 , almost surely. ■ 3.4.3.2 Proof of Lemma 3.4.2
Under the conditions of Lemma 3.4.1, we have dp r Φ t,m pX 0 q, Ă X t q mÑ8 ÝÑ 0 almost surely, with Ă X t P pX , dq, @m P N, t P Z. Thus Edp r Φ t,m pX 0 q, Ă X t q mÑ8 ÝÑ 0. On the other hand, Since Ed α p r Φ t,m`j pX 0 q, r Φ t,m`j`1 pX 0 qq À r m`j for any fixed m ą 0, j ě 0, then

E n ÿ j"0 d α p r Φ t,m`j pX 0 q, r Φ t,m`j`1 pX 0 qq À n ÿ j"0 r m`j À 8 ÿ j"0 r m`j " r m 1 ´r .
Thus, for any m, n P N,

Ed α p r Φ t,m pX 0 q, Ă X t q ď E n ÿ j"0 d α p r Φ t,m`j pX 0 q, r Φ t,m`j`1 pX 0 qq `Ed α p r Φ t,m`n`1 pX 0 q, Ă X t q À r m .
(3.4.5)

Then, for any fixed m P N, X P X , we have

d α p r Φ t,m pXq, Ă X t q ď d α p r Φ t,m pXq, r Φ t,m pX 0 qq `dα p r Φ t,m pX 0 q, Ă X t q.
Applying the expectation on both sides of the above inequality, we have

Ed α p r Φ t,m pXq, Ă X t q ď Ed α p r Φ t,m pXq, r Φ t,m pX 0 qq `Ed α p r Φ t,m pX 0 q, Ă X t q À Cr m d α pX, X 0 q `Ed α p r Φ t,m pX 0 q, Ă X t q À r m .
Following the same steps as in the proof of Lemma 3.4.1, we then have almost surely, dp r Φ t,m pXq, Ă X t q mÑ8 ÝÑ 0, @t P Z. We would like to show that, for any fixed t P Z, dp Ă X t , Φ ϵt p Ă X t´1 qq " 0, almost surely. Firstly, we have for any m P N

Ed α p Ă X t , Φ ϵt p Ă X t´1 qq ď Ed α p Ă X t , r Φ t,m pX 0 qq `Ed α p r Φ t,m pX 0 q, Φ ϵt p Ă X t´1 qq " Ed α p Ă X t , r Φ t,m pX 0 qq `Ed α pΦ ϵt ˝r Φ t´1,m´1 pX 0 q, Φ ϵt p Ă X t´1 qq ď Ed α p Ă X t , r Φ t,m pX 0 qq `CrEd α p r Φ t´1,m´1 pX 0 q, Ă X t´1 q.
Since dp Ă X t , r Φ t,m pX 0 qq mÑ8 ÝÑ 0 almost surely, for any t P Z. Thus, the last bound above tends to 0 as m Ñ 8. Thus Ed α p Ă X t , Φ ϵt p Ă X t´1 qq " 0, which implies dp Ă X t , Φ ϵt p Ă X t´1 qq " 0 almost surely. ■ 3.4.3.4 Proof of Theorem 3.4.5

We first show that Equation (3.4.2) holds for any α 1 P p0, αq. For any X P X , we have

Ed α 1 p r Φ t,m pX 0 q, r Φ t,m pXqq ď ´Ed α p r Φ t,m pX 0 q, r Φ t,m pXqq ¯α1 α ď C α 1 α r α 1 m α d α 1 pX 0 , Xq.
(3.4.6) Let γ " mintα, βu, then for any t, m P N, we have Ed γ p Ă X t , S t q " Ed γ p Ă X t , r Φ t,m pS t´m qq ď Ed γ p Ă X t , r Φ t,m pX 0 qq `Ed γ p r Φ t,m pX 0 q, r Φ t,m pS t´m qq

Inequality(3.4.6) ď Ed γ p Ă X t , r Φ t,m pX 0 qq `C γ α r γm α
Ed γ pX 0 , S t´m q. Since dp Ă X t , r Φ t,m pX 0 qq mÑ8 ÝÑ 0 almost surely, for any t P Z, we have Ed γ p Ă X t , r Φ t,m pX 0 qq mÑ8 ÝÑ 0 for any t P Z. On the other hand, since γ ă β, for all t P Z, we have Ed γ pS t , X 0 q ď Ed γ pS t , Z 0 q`d γ pX 0 , Z 0 q ď `Ed β pS t , Z 0 q ˘γ β `dγ pX 0 , Z 0 q ă M γ β `dγ pX 0 , Z 0 q. Therefore, Ed γ p Ă X t , S t q " 0, which implies dp Ă X t , S t q " 0, almost surely, for all t P Z. ■ 3.4.3.5 Proof of Proposition 3.4.6

We first show that given X d " X 1 d " π, we have for any t P Z, X m pXq d " X m pX 1 q d " X t , almost surely for any m P N. From Theorem 3.4.4, we have X t d " X t´m d " π, almost surely for any m P N, t P Z, and X t " r Φ t,m pX t´m q, almost surely for any m P N, t P Z. Thus, we obtain that π " r Φ t,m pπq, almost surely, which implies

X m pXq d " X m pX 1 q d " π " X t . Therefore, Ed α pX m pXq, X m pX 1 qq " d α pX m pXq, X m pX 0 qq `dα pX m pX 0 q, X m pX 1 qq " 2Ed α pX t , X m pX 0 qq ϵti.i.d. " 2Ed α pX t , r Φ t,m pX 0 qq Inequality(3.4.5) À r m .
Thus, Bound (3.4.4) is checked with s taken as r, which completes the proof. ■ 3.4.3.6 Proof of Theorem 3.4.8

We first show that the Hilbert mean ErX t s for time series X t P pX , x, yq, t P Z does not depend on time t. We are thus led to show that, for all t, t 1 P Z, ErX t s " ErX t 1 s. By the definition of Hilbert mean, this is equivalent to show that ExX t , Xy " ExX t 1 , Xy, @X P X .

Firstly, we show that @t P Z, X P X , ExX t , Xy " lim mÑ8 Ex r Φ t,m pX 0 q, Xy. We have

E|x r Φ t,m pX 0 q ´Xt , Xy| ď E} r Φ t,m pX 0 q ´Xt }}X} paq ď }X}pE} r Φ t,m pX 0 q ´Xt } α q 1 α " }X} ´Ed α p r Φ t,m pX 0 q, X t q ¯1 α mÑ8 ÝÑ 0.
Inequality (a) comes from the condition α ě 2 and Jensen inequality. Thus, for any t P Z, lim mÑ8

Ex r Φ t,m pX 0 q ´Xt , Xy " 0.

On the other hand, since ϵ t are i.i.d., there is r Φ t,m pX 0 q d " r Φ t 1 ,m pX 0 q, @t, t 1 P Z, m P N. Thus, we have for any t, t 1 P Z, X P X Ex r Φ t,m pX 0 q, Xy " Ex r Φ t 1 ,m pX 0 q, Xy, @m P N.

Then @t P Z, X P X lim mÑ8

Ex r Φ t,m pX 0 q, Xy " lim mÑ8

Ex r Φ t 1 ,m pX 0 q, Xy, @m P N, which implies @t P Z, X P X ExX t , Xy " ExX t 1 , Xy.

We denote ErX t s by U . Next, since E}X t } 2 ă 8 for all t P Z, the auto-covariance operator G t,t`h is well-defined. We are now to show G t,t`h is time-invariant, which is equivalent to show xG t,t`h pXq, Y y " xG t 1 ,t 1 `hpX q, Y y, @X, Y P X , by Definition 3.4.7, that is

ExX t ´U, XyxX t`h ´U, Y y " ExX t 1 ´U, XyxX t 1 `h ´U, Y y, @X, Y P X . (3.4.7)
Analogously, we show firstly that @t, h P Z, X, Y P X

ExX t ´U, XyxX t`h ´U, Y y " lim mÑ8 Ex r Φ t,m pX 0 q´U, Xyx r Φ t`h,m pX 0 q´U, Y y. (3.4.8)
We have

ExX t ´U, XyxX t`h ´U, Y y ´Ex r Φ t,m pX 0 q ´U, Xyx r Φ t`h,m pX 0 q ´U, Y y ďExX t ´U, XyxX t`h ´U, Y y ´Ex r Φ t,m pX 0 q ´U, XyxX t`h ´U, Y y `Ex r Φ t,m pX 0 q ´U, XyxX t`h ´U, Y y ´Ex r Φ t,m pX 0 q ´U, Xyx r Φ t`h,m pX 0 q ´U, Y y " ExX t ´r Φ t,m pX 0 q, XyxX t`h ´U, Y y `Ex r Φ t,m pX 0 q ´U, XyxX t`h ´r Φ t`h,m pX 0 q, Y y ď }X}}Y }E}X t ´r Φ t,m pX 0 q}}X t`h ´U } `}X}}Y }E} r Φ t,m pX 0 q ´U }}X t`h ´r Φ t`h,m pX 0 q} ď }X}}Y }EdpX t , r Φ t,m pX 0 qqdpX t`h , U q `}X}}Y }Edp r Φ t,m pX 0 q, U qdpX t`h , r Φ t`h,m pX 0 qq.
Since dpX t , r Φ t,m pX 0 qq mÑ8 ÝÑ 0 almost surely, for any t P Z, then dp r Φ t,m pX 0 q, U q mÑ8 ÝÑ dpX t , U q, for any t P Z. Thus, the last bound tends to 0, as m Ñ 8, which implies Equation (3.4.8).

On the other hand, since ϵ t are i.i.d., thus @t, h P Z, X, Y P X and @m P N

x r Φ t,m pX 0 q ´U, Xyx r Φ t`h,m pX 0 q ´U, Y y d " x r Φ t 1 ,m pX 0 q ´U, Xyx r Φ t 1 `h,m pX 0 q ´U, Y y, which follows Ex r Φ t,m pX 0 q´U, Xyx r Φ t`h,m pX 0 q´U, Y y " Ex r Φ t 1 ,m pX 0 q´U, Xyx r Φ t 1 `h,m pX 0 q´U, Y y.
(3.4.9) Take limit on m on both sides of Equation (3.4.9), we obtain Equation (3.4.7). Thus the auto-covariance is time-invariant. ■

Reproducing kernel Hilbert space and kernel ridge regression

In this section, we provide a concise lecture on the Reproducing kernel Hilbert space (RKHS) and RKHS-based ridge regression. We only present the RKHS as a space of real functions, which we consider as the approximation family of the regression function of x it , pi, tq P N ˆZ in Chapter 6. For the general RKHS as a space of maps, whose reproducing kernel takes values of linear continuous operator, we refer to [START_REF] Kadri | Operator-valued kernels for learning from functional response data[END_REF]. We start off by defining a kernel. Definition 3.5.1. Let X be a set. k : X ˆX Ñ IR is a kernel on X if k is symmetric: kpx, yq " kpy, xq, and k is positive definite, that is @ x 1 , x 2 , ..., x n P X , the Gram matrix K defined by K ij " kpx i , x j q is positive semi-definite (PSD).

We would like to point out the following property of a kernel. 55 3.5. Reproducing kernel Hilbert space and kernel ridge regression Remark 3.5.2. Given kernel k 1 on X 1 and kernel k 2 on X 2 , then the mapping k 1 k 2 whose value is defined by k 1 k 2 rpx 1 , y 1 q, px 2 , y 2 qs " k 1 px 1 , x 2 qk 2 py 1 , y 2 q is a also kernel on X 1 ˆX2 .

For the proof, see (Christmann and Steinwart, 2008, Lemma 4.6 p.114). This product rule allows us to define a huge variety of kernels. Next, we introduce the reproducing kernel feature map ϕ and its related feature space H k .

Given a kernel k, define the map ϕ as

† " ϕ : X Ñ IR X x Þ Ñ ϕpxq " kpx, ¨q
Consider the functional space H k H k " spanptϕpxq : x P X uq " tf p¨q "

n ÿ i"1 α i kpx i , ¨qu
, where α i P IR.

We define the inner product between f "

n ř i"1 α i kpx i , ¨q and g " m ř j"1 β j kpy j , ¨q in H k as xf, gy " ÿ i,j α i β j kpx i , y j q.
It is easy to check that this inner product is valid. The induced norm is then }f } 2 H k " xf, f y. We can rewrite the inner product and the induced norm into matrix form as xf, gy " α t Gβ, }f } 2 H k " α t Kα, where α " pα 1 , ..., α n q t , β " pβ 1 , ..., β m q t , G P IR nˆm with G ij " kpx i , x j q and K is the Gram matrix of inputs x 1 , x 2 , ..., x n .

Finally, we complete the inner product space H k to a Hilbert space, denoted by H k . H k is the RKHS uniquely defined by reproducing kernel (RK) k, meaning that k can only be the RK of H k . Conversely, if H k is a RKHS ‡ , then it has the unique RK. Note that the name reproducing comes from the reproducing property of k xf, kpz, ¨qy "

ÿ x α x kpx, zq " f pzq.
This way of building a RKHS follows the proof of Moore-Aronszajn theorem below.

Theorem 3.5.3. (Moore-Aronszajn) Suppose k is a symmetric, positive definite kernel on a set X . Then there is a unique Hilbert space of functions on X for which k is a reproducing kernel.

We can find that RKHS has many favorable attributes to work with. We point out some of them, which are shown in the following remarks.

Remark 3.5.4.

• The principle does not assume additional structure on the set X , which facilitates us to expand the input set by cartesian product later on.

• The point x in X is mapped to a high-dimensional feature space H k with the feature vector ϕpxq. The feature vector can even be infinite-dimensional, such as the one in H k rbf .

• The norm }f } 2 H k implies how fastzsmooth the function varies over X with respect to the geometry defined by the kernel § .

• The composition f pxq " ř y α y kpy, xq enables us to only set up linear models in the feature space, however we are still able to find a nonlinear function of x with the required characteristics, since the mapping can be nonlinear. This fact permits a feasible search for nonlinear predictor of the input space X .

After introducing the approximation family H k , we would like to find the best estimator of the regression function of y P IR on x P X given the observations. To accomplish this, we first set up kernel regression (3.5.1), then solve the related optimization problem with the help of the representer theorem.

Definition 3.5.5. Given the observations px 1 , y 1 q, px 2 , y 2 q, ..., px n , y n q P X ˆIR, the best fitted predictor is given by

f " arg min f PH k n ÿ i"1 Cpf px i q, y i q `gp}f } H k q (3.5.1)
where Cpx, aq is a convex cost function, }f } H k is the induced norm of inner product ă ¨, ¨ąH k , g : r0, `8q Ñ IR is a strictly monotonically increasing real-valued function.

If we choose the cost function Cpx, aq to be }x ´a} 2 ℓ 2 , gpxq to be λx 2 with nonnegative λ, then we have the kernel ridge regression (3.5.2).

f " arg min

f PH k n ÿ i"1 }y i ´f px i q} 2 ℓ 2 `λ}f } 2 H k . (3.5.2)
From the third remark of 3.5.4, we know the regularization term λ}f } 2 H k endows the potential optimizer f with the variation characteristic that we want, whose degree is quantified by λ. Therefore, the nature of using kernel ridge regression to predict is to diffuse the observed information along the input space.

Because all functions in H k has the linear combination form ř x α x kpx, ¨q. Thus once we get the optimal predictor f " ř x αx kpx, ¨q, we can predict the output of unseen input z as f pzq " ř x αx kpx, zq. Since we know the kernel value beforehand, it is only left to find the optimal value for each coefficient α x . The representer theorem, [START_REF] Schölkopf | A generalized representer theorem[END_REF], indicates that, the only nonzero coefficients are § Consider the inequality |f pxq ´f pyq| " |xf, kpx, ¨q ´kpy, ¨qy| ď }f }H k }ϕpxq ´ϕpyq}H k . 57 3.6. Dropout those corresponds to the observations, namely, essentially the solution of problem (3.5.1) has the representation

f " n ÿ i"1 α i kpx i , ¨q.
Plugging this representation into equation (3.5.2), the problem will be reduced to a common optimization problem in finite Euclidean space. We reformulate the problem using matrix notation as α " arg min

αPIR n }y ´Kn α} 2 ℓ 2 `λα t K n α,
where y " py 1 , ..., y n q t and K n is the Gram matrix of inputs x 1 , x 2 , ..., x n . Through linear algebra, we get α " pK n `λIdq ´1y. Then the resulting nonlinear predictor is f pzq "

ř n i"1 αkpx i , zq, with } f } 2 H k " y t pK n `λIdq ´1K n pK n `λIdq ´1y.

Dropout

In Chapter 6, we rely on the dropout technique to aggregate the prediction models of different missing set I's during one network training so as to evaluate the overall predictability of nodes. To understand the motivation and the principle of such training design, in this section, we review the dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] which is a regularization technique designated to prevent overfitting based on the model aggregation. For the background on netural networks, we refer to Bishop and Nasrabadi (2006, Chapter 5).

The best way to reduce overfitting is model combination, which means fitting all possible models first, and then for each test sample, to aggregate the predictions from all trained models as the final prediction. Obviously, for a class of complex models such as neural networks, independent training and explicit aggregation are too costly. Dropout is then a general training framework, purposed to enable such varying and aggregation of network models within one training. The mechanism of dropout makes a single trained neural network aggregate exponentially many network models of different architectures with respect to the number of neurons in an approximate way, and gives the ensemble prediction values on the output neurons for targets.

To be more precise, we illustrate this technique with an example. Figure 3.1 is a small neural network, which consists in one input layer, one hidden layer and one output layer. Their respective neuron values are denoted by x " px 1 , x 2 q, y " py 1 , y 2 q and z, where x " px 1 , x 2 q represents an arbitrary input sample. Without dropout, the network forward propagation of the model given on the leftmost in Figure 3.1 is z " σ 2 pv 1 y 1 `v2 y 2 q, y 1 " σ 1 pu 11 x 1 `u21 x 2 q, y 2 " σ 1 pu 12 x 1 q, where v 1 , v 2 , u 11 , u 21 , u 12 are weights, which are trainable parameters. In this example, we omit bias terms. If we apply dropout on the hidden layer, the forward propagation becomes y 1 " σ 1 pu 11 x 1 `u21 x 2 q, y 2 " σ 1 pu 12 x 1 q, r y 1 " w 1 y 1 , r y 2 " w 2 y 2 , z " σ 2 pv 1 r y 1 `v2 r y 2 q where w " pw 1 , w 2 q is the dropout vector. The components of the dropout vector are independent Bernoulli random variables, each of which has probability q of being 0, which is referred to as the dropout rate. The backpropagation is determined by the realization of w and gradient calculation. We denote the loss component of a sample as Jpzq. Then, the associated gradient components of trainable parameters are computed using the chain rule as follows:

BJpv 1 q Bv 1 " BJpzq Bz Bzpv 1 q Bv 1 " BJpzq Bz σ 1 2 pv 1 r y 1 `v2 r y 2 qw 1 y 1 , BJpv 2 q Bv 2 " BJpzq Bz Bzpv 2 q Bv 2 " BJpzq Bz σ 1 2 pv 1 r y 1 `v2 r y 2 qw 2 y 2 , BJpu i1 q Bu i1 " BJpzq Bz Bzpy 1 q By 1 By 1 pu i1 q Bu i1 " BJpzq Bz σ 1 2 pv 1 r y 1 `v2 r y 2 qv 1 w 1 By 1 pu i1 q Bu i1 , i " 1, 2, BJpu 12 q Bu 12 " BJpzq Bz
Bzpy 2 q By 2 By 2 pu 12 q Bu 12 " BJpzq Bz σ 1 2 pv 1 r y 1 `v2 r y 2 qv 2 w 2 By 2 pu 12 q Bu 12 .

Therefore if w 1 " 0 and w 2 " 1, the gradient components of v 1 , u 11 and u 21 are all 0.

The gradient components of v 2 , u 12 are then calculated by backpropagating the loss component Jpzq through the model 1 in Figure 3.1. In practice, the dropout vector is re-sampled on a regular basis, such as each sample or each batch. The gradient used to update each parameter is then calculated as usual, that is the average over its gradient components computed from all the samples t P B n used in an updating step n. For example

1 |B n | ÿ tPBn BJpv 1 ; x t , w t q Bv 1 ,
for v 1 . B n can have only one sample, but more commonly is a batch. When using the network in prediction (test time), all neurons are present with scaled-down weights, which indicates multiplying the outgoing weights of the dropout neurons by 1 ´q.

References on the involved convex optimization results

This ensures the expected neuron output in training remains the same as the actual one at test time.

Since our model re-samples the dropout vector at each batch, we present how the dropout does the model aggregation in this case with the example of Figure 3.1. B n denotes a batch in the following, and we furthermore denote its dropout vector by w pnq . Because the dropout vector is identical for all samples in a batch, one batch contributes to the training of one specific network model. For example, sampling w pnq " p0, 1q amounts to sampling and training model 1 with B n . Then, sampling w pn`1q " p1, 0q varies the architecture to model 2 in the next step with B n`1 . The parameters of a new model are initialized as the results of the preceding training phase. The ones connected to turned-on neurons are shifted towards the potential independent training results of this model. By doing this, the training needs to find the optimal weight values which are shared across models and gives the aggregated model prediction the best performance. Then when using the network in prediction (test time), no dropout leads to the aggregation of all models into the resulting trained network, which gives the overall predictions z t for the target.

References on the involved convex optimization results

The work in this thesis uses several classical convex optimization results. To avoid the content redundancy of this manuscript, we refer to the following materials for the corresponding backgrounds.

Online graph learning from matrix-variate time series

In this chapter, we develop the novel graph learning frameworks, which learns in an online fashion a Granger causal graph from matrix-variate time series px t q t P IR N ˆF . The proposed approaches extend the existing works on the inference of causal graphs, relying on the VAR models, which we have reviewed in Section 2.2.1.1. In addition to learning the causality structure between stationary processes, in the graph learning domain, there is another main line of works, that is dedicated to the inference of conditional dependency structure between random variables. This line assumes that the observations x t P IR N are independent and identically distributed from N p0, Θ ´1q. The conditional dependency structure thus is encoded in the precision matrix Θ, which is promoted to be sparse during the estimation. The resulting models are known as the Gaussian graphical models [START_REF] Meinshausen | High-dimensional graphs and variable selection with the lasso[END_REF][START_REF] Friedman | Sparse inverse covariance estimation with the graphical lasso[END_REF]. Moreover, Gaussian graphical model for the stationary process are studied in [START_REF] Bach | Learning graphical models for stationary time series[END_REF] and [START_REF] Songsiri | Topology selection in graphical models of autoregressive processes[END_REF], where they infer the conditional dependence graph from the inverse of its matrix-variate spectrum. This line of graph learning models has been firstly extended to matrix- (Kalaitzis et al., 2013) and tensor- [START_REF] Greenewald | Hero III. Tensor graphical lasso (teralasso)[END_REF][START_REF] Wang | The sylvester graphical lasso (syglasso)[END_REF] variate data. The extending works also rely on the KP/KS structure to impose a product graph structure in the precision matrix.

On the other hand, the developed graph learning frameworks are based on the novel matrix-variate AR(1) model, which is proposed by this work. At this point, we recall the other work in literature, Chen et al. (2021a), on the extension of the VAR(1) to the matrix-variate process, that we have mentioned in Section 2.2.1.2 of introduction. In contrast with our KS construction, Chen et al. (2021a) proposes to impose the KP structure in the coefficient matrix to extend VAR(1) model. The comparison of these two constructions has also been given in Section 2.2.1.2. Additional to this difference, our work is especially devised for the purpose of graph learning, thus we promote sparsity in our coefficient estimators and focus on the development of online inference, which is not considered by the work of Chen et al. (2021a).

In the rest of this chapter, we first introduce the novel matrix-variate AR model in Section 4.1. We also provide the closed form of the projection operator onto the constrained coefficient space that is a key tool in the derivation of the learning approaches. In Section 4.2, we develop the main online graph learning frameworks. In Section 4.3, we propose the augmented data model to take into account the trends, and we adapt the previous frameworks to the inference of the augmented model. Lastly, we perform the numerical experiments using both synthetic and real data in Section 4.4. All proofs are gathered in the Appendix of this chapter.

Notations We present all the notations used in this chapter in advance.

vec

Vectorized representation of a matrix.

ivec Inverse vectorized representation of a vector, such that ivec ˝vec " id.

r¨s ¨Extraction by index. The argument in rs can be a vector or a matrix. For a vector, the index argument can be a scalar or an ordered list of integers. For example, rvs k extracts the k-th entry of v, while rvs K " prvs k i q i extracts a sub-vector indexed by K " pk i q i in order. For a matrix, the index argument can be a pair of scalars or a pair of ordered lists of integers. For example, rM s k,k 1 extracts the pk, k 1 qth entry of M , while rM s K,K 1 " prM s k i ,k j q i,j extracts a sub-matrix indexed by K " pk i q i in row order, and K 1 " pk 1 j q j in column order. When K " K 1 , we denote rM s K,K 1 by rM s K .

rM s :,i

Extraction of the i-th column vector of matrix M .

rM s i,:

Extraction of the i-th row vector of matrix M .

svecpM q Vectorized representation of the upper diagonal part of matrix M , that is, ´rMs 1,2 , rMs 1,3 , ¨¨¨, rMs 2,3 , ¨¨¨¯J.

diagpMq Diagonal vector of matrix M .

Causal product graphs and matrix-variate AR(1) models

We define formally the matrix-variate AR(1) model proposed in Section 2.2.1.3. The matrix-variate stochastic process X t P IR N ˆF is said to follow a matrix-variate AR(1) process if the multivariate process vec pX t q is a VAR(1) process defined by A with the particular KS structure

K G K G " ␣ M P IR N F ˆN F : D M F P IR F ˆF , M N P IR N ˆN , such that, offdpMq " M F ' M N , with, diagpM F q " 0, diagpM N q " 0, M F " M J F , M N " M J N ( .
By constraining A P K G , we impose the KS structure into the off-diagonal part of coefficient matrix A, modelling the total causality structure by a Cartesian product graph G parameterized by the spatial graph G N and the feature graph G F . The diagonal of A is free from the structure constraint. In return, we require no self-loops in the component graphs by imposing diagpM F q " 0, diagpM N q " 0. This is to primarily address the non-identifiability problem of Kronecker sum, since AF ' AN " pAF `cI F q'pAN ´cI N q holds for any scalar c. On the other hand, a full parameterized diagonal adds the self-loops on all nodes of the total graph G, which also brings more flexibility to the model. Note that, the last constraint in K G requires the component graphs hence the product graph to be symmetric. This is because we notice that, the existing causal graphs are usually directed, which disables their further use in the methods, which require undirected graphs as prior knowledge, like kernel methods, and graph Fourier transform related methods. Therefore, we focus on learning undirected graphs. Nevertheless, we stress that the derived approaches do not depend on the specific structure of coefficient, and thus can be adapted to, for example, the relaxed constraint set without the symmetry assumption, which may suit more the applications in econometrics, as suggested by one reviewer in the recent feedback.

We then focus on the analysis of a stationary process. We recall the stationarity condition for the VAR(1) model in Theorem 3.2.2. We also need the conditions of CLT 3.2.3, which permits the consistent estimator and its Wald test. We conclude all these assumptions by data generating model (4.1.4).

We assume the samples X t P IR N ˆF are generated by the following model given the initial sample X 0 x t " Ax t´1 `zt , with A P K G , }A} 2 ă 1, t P N `, (4.1.4) where x t " vec pX t q, }A} 2 equals the largest singular value of A, z t P IR N F " IID p0, Σq is white noise with a non-singular covariance structure Σ and bounded fourth moments, and x 0 " ř 8 j"0 A j z t´j . Note that we assume the process mean is zero in this section and derive the main frameworks of Section 4.2. In Section 4.3, we will study the model with a non-zero, even time-variant, process mean, namely, a process trend, to adapt the main frameworks to the online learning.

Applying ivecp q on both sides of Model (4.1.4), we can obtain its matrix representation

X t " D ˝Xt´1 `AN X t´1 `Xt´1 A J F `Zt , (4.1.5)
where ˝is Hadamard product, D P IR N ˆF " ivecpdiagpAqq, A N and A F are the adjacency matrices such that offdpAq " A F ' A N , and Z t " ivecpz t q. In model (4.1.5), A N X t´1 describes the spatial dependency, where each column of X t´1 can be viewed as a graph signal on the same spatial graph G N . Similarly, each row of X t´1 can be seen as a graph signal on the feature graph G F .

Online Graph Learning

In this section, we detail the two online learning frameworks presented in Sections 2.2.1.3 and 2.2.1.4, which infer A N with sparsity and A F respectively in low-and Then given the orthonormal basis U k {}U k } F , k P K, the projections can be calculated explicitly as

D_1 MN_1,2 MN_1,3 MF_1,2 MN_1,2 D_2 MN_2,3 MF_1,2 MN_1,3 MN_2,3 D_3 MF_1,2 MF_1,2 D_4 MN_1,2 MN_1,3 MF_1,2 MN_1,2 D_5 MN_2,3 MF_1,2 MN_1,3 MN_2,3 D_6
Proj G pBq " ÿ kPK xU k , By 1 }U k } 2 F U k .
The projection is very straightforward to understand. To obtain a variable in diagpM q, M F and M N related to U k , we use xU k , By to calculate the average value of B as explained previously. Then this average value is repeated at the corresponding entries to construct the structure, by multiplying locator U k {}U k } 2 F . Furthermore the orthogonality of the basis implies the direct sum

K G " K D ' K F ' K N , (4.2.1)
where K D , K F , and K N are respectively spanned by pU k q kPK D , pU k q kPK F , and pU k q kPK N . Given the construction of pU k q k , Equation (4.2.1) actually reveals the product graph decomposition, note that equally we have

K D " tM P IR N F ˆN F : offdpM q " 0u, K F " tM P IR N F ˆN F : DM F P IR F ˆF , such that, M " M F b I N , with, diagpM F q " 0, M F " M J F u, K N " tM P IR N F ˆN F : DM N P IR N ˆN , such that, M " I F b M N , with, diagpM N q " 0, M N " M J N u.
The projection onto these subspaces can also be computed analogously

Proj D pBq " ÿ kPK D xU k , By 1 }U k } 2 F U k , that is the diagonal part of B. Proj F pBq " ÿ kPK F xU k , By 1 }U k } 2 F U k " « ÿ kPK F xU k , ByE k ff b I N , Proj N pBq " ÿ kPK N xU k , By 1 }U k } 2 F U k " I F b « ÿ kPK N xU k , ByE k ff .
We use Proj G F pBq and Proj G N pBq to denote the small matrices

ř kPK F xU k , ByE k and ř kPK F xU k , ByE k
, with an extra subscript G, with which we will represent the proposed estimators of AF, AN in the following sections. Finally, we have

Proj G pBq " Proj D pBq `Proj G F pBq ' Proj G N pBq. (4.2.2)

Approach 1: Projected OLS estimators and Wald test

In the low dimensional regime, VAR model (4.1.4) can be estimated by the ordinary least squares (OLS) method, defined in (3.2.9). We denote the OLS estimator of A by q

A t , and we use p A t to denote the proposed projected OLS estimator. Assume that we start receiving samples x 0 , x 1 , ..., x t from time τ " 1, we recall the OLS estimator for an intercept-free VAR(1) model

q A t " p Γ t p1q " p Γ t p0q ı ´1 ,
where

p Γ t p0q " 1 t t ÿ τ "1 x τ ´1x J τ ´1 and p Γ t p1q " 1 t t ÿ τ "1 x τ x J τ ´1
are respectively the consistent estimators of auto-covariance matrices Γp0q and Γp1q, with Γphq " E `xt x J t´h ˘, h ě 0. Moreover, the additional conditions in Model (4.1.4) permit the asymptotic properties

1. q A t p Ý Ñ A, 2. ? t vecp q A t ´Aq d Ý Ñ N p0, Σ ols q, where Σ ols " rΓp0qs ´1 b Σ.
However, due to model misspecification and limited samples, q A t will not have the same structure as A P K G . Therefore, the projection of q A t onto K G needs to be performed, which leads to the projected OLS estimator:

p A t :" Proj G p q A t q.
Given the representation (4.2.2), it is natural to define the estimators of diagpAq, A F , and A N by Proj D p q A t q, Proj G F p q A t q, and Proj G N p q A t q, respectively, denoted by y A D,t , y A F,t , and y A N,t . We now establish the Wald test with y A N,t to identify the sparsity structure of the true A N . To this end, we provide the CLT in Theorem 4.2.3. Theorem 4.2.3. Assume samples x 0 , x 1 , ..., x t satisfy the assumptions of Model (4.1.4), then the CLT holds for y A N,t , as t Ñ `8,

? t svecp y A N,t ´AN q d Ý Ñ N p0, Σ N q, where Σ N " ÿ k,k 1 PK N vecpU k q J Σ ols vecpU k 1 q `svecpE k qsvecpE k 1 q J ˘.
The proof is done through applying Cramér-Wold theorem on ? t svecp y A N,t ´AN q, given the linearity of Proj G N p q A t q and the CLT of classical OLS estimator q A t . For details, see Appendix 4.5.1, where we also derive a CLT for p A t . It is straightforward to understand the asymptotic distribution of y A N,t . The asymptotic covariance between its two entries is assigned the mean of covariance values vecpU k q J Σ ols vecpU k 1 q, following the construction of the corresponding estimators xU k , q A t y and xU k 1 , q A t y as averages as well.

Based on this large sample result, we now test the nullity of P given variables rA N s i k ,j k , k " 1, ..., P , with i k ă j k as

H 0 : α " 0 versus H 1 : α ‰ 0,
where α P IR P :" ´¨¨¨, rA N s i k ,j k , ¨¨¨¯J. The Wald test above theoretically completes the approach. In practice, we propose to test the p entries of the smallest estimate magnitudes, jointly each time, as p grows from 1 to possibly largest value |K N |. Specifically, for a given estimation y A N,t , we first sort its entries such that

" p Σ W,t ı k,k 1 " vecpU h k q J p Σ ols,t vecpU h k 1 q, such that U h k is the matrix corresponding to variable rA N s i k ,j k , p Σ ols,t " " p Γ t p0q ı ´1 b p Σ t ,
|r y A N,t s i 1 ,j 1 | ď |r y A N,t s i 2 ,j 2 | ď . . . ď |r y A N,t s i |K N | ,j |K N | |.
Then, we set up the sequence of joint tests H 0 p1q, H 0 p2q, ..., H 0 p|K N |q, where H 0 ppq : `rA N,t s i 1 ,j 1 , ¨¨¨, rA N,t s ip,jp ˘J " 0,

We perform these tests sequentially until Hpp 0 `1q is rejected for some p 0 . Lastly, we replace the entries r y A N,t s i 1 ,j 1 , ..., r y A N,t s ip 0 ,jp 0 with 0 in y A N,t as the final estimate of A N . Note that searching for p 0 resembles root-finding, since the output from each point p is binary. Thus, the search can be accelerated by using the bisection, with the maximal number of steps about log 2 p|K N |q.

The previous procedure is performed at the t-th iteration, given the OLS estimator q A t and the consistent estimator p Σ ols,t . When the new sample x t`1 comes, q A t`1 and p Σ ols,t`1 can be calculated efficiently by applying Sherman Morrison formula on r p Γ t p0qs ´1. The pseudo code is given in Algorithm 2.

Approach 2: Structured matrix-variate Lasso and homotopy Algorithms

Another estimator of A can be obtained by minimizing the following Lasso problem Apt, λq " arg min APK G L λ,t pAq, where L λ,t pAq "

1 2t t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λF }A N } ℓ 1 .
(4.2.4) We recall that only a subset of entries of A is penalized by the ℓ 1 norm. To solve this Lasso problem with structure constraint and the partial penalization, in an off-line fashion, we can adopt for example the projected gradient descent as formulated in Equation (2.2.4). Our goal is then to derive the homotopy algorithms, in order to update Apt, λ t q to Apt `1, λ t`1 q quickly upon the arrival of x t`1 .

We recall again the standard Lasso (2.2.2) in Equation (4.2.5). We formulate it differently here as in the homotopy algorithm literature. θpt, λq " arg min

θPIR d 1 2 }y ´Xθ} 2 ℓ 2 `tλ}θ} ℓ 1 , (4.2.5)
where y " py τ q t τ "1 , X " px τ q t τ "1 , and px τ , y τ q P IR d`1 is the sample at time τ . A family of homotopy algorithms are devised for Lasso (4.2.5) to quickly update the existing solution when the problem evolves. We refer to [START_REF] Malioutov | Homotopy continuation for sparse signal representation[END_REF] which updates the solution from θpt, λ 1 q to θpt, λ 2 q, Garrigues and Ghaoui (2008) which proposes an homotopy algorithm updating θpt, λ 2 q to θpt `1, λ 2 q. The adaptive λ selection rules, which update λ at the arrival of new samples are also studied in [START_REF] Monti | Adaptive regularization for lasso models in the context of nonstationary data streams[END_REF]; [START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF].

As mentioned in Section 2.2.1.4, the existing homotopy algorithms do not adapt to novel Lasso (4.2.4), due to the calculation of optimality condition. Thus, we first calculate the optimality conditions of the novel Lasso in Section 4.2.3.1.

Optimality Conditions

The key point in deriving the optimality conditions arising from the variational problem (4.2.4) is to transfer the structure of A onto the data vector x τ ´1, using an orthonormal basis of K G . We introduce the auxiliary variable A 0 , such that A " Proj G pA 0 q, and rewrite Problem (4.2.4) with respect to A 0 min

A 0 PIR N F ˆN F 1 2t t ÿ τ "1 › › › › › x τ ´ÿ kPK xU k , A 0 y 1 }U k } 2 F U k x τ ´1› › › › › 2 ℓ 2 `λ › › › › › ÿ kPK N xU k , A 0 y 1 }U k } 2 F U k › › › › › ℓ 1 . (4.2.6) Problem (4.2.6
) is weakly convex, since a minimizer of (4.2.4) can be projected from infinitely many minimizers of (4.2.6). We still use L λ,t to denote the objective function above. A minimizer A 0 of (4.2.6) satisfies the optimality conditions

0 P BL λ,t BA 0 " ÿ k,k 1 PK xU k , U k 1 p Γ t p0qy x 1 }U k 1 } 2 F U k 1 , A 0 y 1 }U k } 2 F U k ´ÿ kPK xU k , p Γ t p1qy 1 }U k } 2 F U k `λ ÿ kPK N B ˇˇxU k , A 0 y ˇˇ1 }U k } 2 F U k . (4.2.7)
Assume A 0 is a matrix which satisfies Equation (4.2.7), hence a minimizer of Problem (4.2.6). Then A " Proj G pA 0 q is a minimizer of Lasso (4.2.4). We denote its active set tk P K N : xU k , A 0 y ‰ 0u by K 1 N , that is all the non-zero variables of A N , and its non-active set by K 0 N , that is K N zK 1 N . Since tU k u kPK is an orthogonal family, Equation (4.2.7) is equivalent to

0 " ÿ kPK D Ť K F « ÿ k 1 PK xU k , U k 1 p Γ t p0qy x 1 }U k 1 } 2 F U k 1 , A 0 y ´xU k , p Γ t p1qy ff 1 }U k } 2 F U k , (4.2.8) 0 " ÿ kPK 1 N « ÿ k 1 PK xU k , p Γ t p0qy x 1 }U k 1 } 2 F U k 1 , A 0 y ´xU k , p Γ t p1qy ff 1 }U k } 2 F U k `λ ÿ kPK 1 N signxU k , A 0 y 1 }U k } 2 F U k . (4.2.9) 0 " ÿ kPK 0 N « ÿ k 1 PK xU k , U k 1 p Γ t p0qy x 1 }U k 1 } 2 F U k 1 , A 0 y ´xU k , p Γ t p1qy ff 1 }U k } 2 F U k `λ ÿ kPK 0 N B ˇˇxU k , A 0 y ˇˇ1 }U k } 2 F U k , where B ˇˇxU k , A 0 y ˇˇP r´1, 1s
(4.2.10) To furthermore derive the optimality conditions of Lasso (4.2.4) in terms of A, we introduce the projections onto sub-spaces K N 1 :" spantU k : k P K 1 N u and K N 0 :" spantU k : k P K 0 N u, denoted respectively by Proj N 1 and Proj N 0 . Note that Equation (4.2.1) in fact admits

K G " à kPK spantU k u. Thus Proj N 1 pBq " ÿ kPK 1 N xU k , By 1 }U k } 2 F U k " I F b » - ÿ kPK 1 N xU k , ByE k fi fl , and Proj N 0 pBq " ÿ kPK 0 N xU k , By 1 }U k } 2 F U k " I F b » - ÿ kPK 0 N xU k , ByE k fi fl .
Then Equations (4.2.8), (4.2.9), and (4.2.10) are equivalent respectively to Proj DF ´Ap Γ t p0q ´p Γ t p1q ¯" 0,

Proj K 1 N ´Ap Γ t p0q ´p Γ t p1q ¯`λI F b » - ÿ kPK 1 N signxE k , A N y E k fi fl " 0, Proj K 0 N ´Ap Γ t p0q ´p Γ t p1q ¯`λI F b » - ÿ kPK 0 N B|xE k , A N y| E k fi fl " 0,
where A P K G , Proj DF " Proj D `Proj F , and B|xE k , A N y| P r´1, 1s. The optimality conditions above are an extension of those for standard Lasso (4.2.5), while the former are furthermore refined to the unpenalized variables versus the penalized variables.

Homotopy from

Apt, λ 1 q to Apt, λ 2 q To develop the homotopy algorithm for the change in λ value, we need to get the formulas of the active variables indexed by K 1 N in terms of λ. To this end, we need to rely on representation (4.2.8), (4.2.9), and (4.2.10), directly in terms of each variable xU k , A 0 y. We firstly reorganize all the model variables into a vector

a s :" ˆx 1 }U k } 2 F U k , A 0 y ˙kPK " ˆx 1 }U k } 2 F U k , Ay ˙kPK .
Note that a s is in fact the scaled Lasso solution by the time the variable repeats. Then optimality conditions (4.2.8), (4.2.9), and (4.2.10) are essentially a system of linear equations of unknown a s , with λ in the coefficients. Thus we aim to firstly represent this linear system in vector form, in order to solve the unknowns. We shall introduce the following notations.

Notations of Proposition 4.2.6. Γ 0 P IR |K|ˆ|K| is a large matrix defined as

rΓ 0 s k,k 1 " xU k , U k 1 p Γ t p0qy.
γ 1 P IR |K| is a long vector defined as

rγ 1 s k " xU k , p Γ t p1qy.
w P IR |K 1 | is a long vector where rws k is defined as

$ ' & ' % " 0, k P K D Ť K F , " signra s s k , k P K 1 N , P r´1, 1s, k P K 0 N . We define K 1 :" K D Ť K F Ť K 1 N
, that are all the non-zero variables. Note that except the computational coincidence, the variables in K D Ť K F are usually non-zero. Then we denote the extractions

Γ 1 0 " rΓ 0 s K 1 , Γ 0 0 " rΓ 0 s K 0 N ,K 1 , γ 1 1 " rγ 1 s K 1 , γ 0 1 " rγ 1 s K 0 N , a s 1 " ra s s K 1 , w 1 " rws K 1 , w 0 " rws K 0 N . (4.2.11)
We can endow any orders to the elements in K 1 , K 0 N to extract the rows/columns/entries above, only if the orders are used consistently to all the extractions. With these notations, we now can retrieve a system of linear equations from Equations (4.2.8), (4.2.10), (4.2.9) of unknowns a s 1 . Each equation is obtained by equating the entries of one U k . The resulting system is given in Proposition 4.2.6. Proposition 4.2.6. A minimizer of Lasso problem (4.2.4) satisfies the linear system

# Γ 1 0 a s 1 ´γ1 1 `λw 1 " 0, Γ 0 0 a s 1 ´γ0 1 `λw 0 " 0.
The representation of the optimality conditions in Equation (4.2.6) are similar to those of classical Lasso [START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF][START_REF] Malioutov | Homotopy continuation for sparse signal representation[END_REF], where Γ 0 , γ 1 with the embedded structures correspond to X J X, X J y in the optimality conditions of standard Lasso. However in our case, the non-zero and sign pattern are only with respect to the entries of A N , thus w 1 , which is the equivalent of sign vector, has

|K D | `|K F | zeros.
Suppose that Apt, λq is the unique solution for a fixed λ of the optimization problem (4.2.4), then we invert Γ 1 0 in Proposition 4.2.6 and get the formulas of a s

1 # a s 1 " " Γ 1 0 ‰ ´1 pγ 1 1 ´λw 1 q λw 0 " γ 0 1 ´Γ0 0 a s 1 .
(4.2.12) Formula (4.2.12) is determined by the active set and the sign pattern of the optimal solution at λ. It shows that a s 1 is a piecewise linear function of λ, while w 0 is also a piecewise smooth function.

Therefore, with the assumptions that ra s s K 1 N ‰ 0 (element-wise), and | rws K 0 N | ă 1 (element-wise), due to continuity properties, there exists a range pλ l , λ r q containing λ, such that for any λ 1 P pλ l , λ r q, element-wise, ra s s K 1 N remains nonzero with the signs unchanged, and rws K 0 N remains in p´1, 1q. Hence, Formula (4.2.12) is the closed form of all the optimal solutions Apt, λ 1 q, for λ 1 P pλ l , λ r q. λ l , λ r are taken as the closest critical points to λ. Each critical point is a λ value which makes either an ra s s k , k P K 1 N become zero, or a rws k , k P K 0 N reach 1 or ´1. By letting ra s s k " 0, k P K 1 N and rws k " ˘1, k P K 0 N in Formula (4.2.12), we can compute all critical values. We now use k i to denote the orders of K 1 , K 0 N that we used in the extraction (4.2.11). The critical values are then given by

λ 0 k i " " " Γ 1 0 ‰ ´1 γ 1 1 ı i { " " Γ 1 0 ‰ ´1 w 1 ı i , k i P K 1 such that k i P K 1 N , λ ki " " γ 0 1 ´Γ0 0 " Γ 1 0 ‰ ´1 γ 1 1 ı i " 1 ´Γ0 0 " Γ 1 0 ‰ ´1 w 1 ı i , k i P K 0 N , λ ḱi " " γ 0 1 ´Γ0 0 " Γ 1 0 ‰ ´1 γ 1 1 ı i " ´1 ´Γ0 0 " Γ 1 0 ‰ ´1 w 1 ı i , k i P K 0 N .
(4.2.13) Thus, the closet critical points from both sides are

λ l :" max ␣ maxtλ 0 k , k P K 1 N : λ 0 k ă λu, maxtλ k , k P K 0 N : λ k ă λu, maxtλ ḱ , k P K 0 N : λ ḱ ă λu ( , λ r :" min ␣ mintλ 0 k , k P K 1 N : λ 0 k ą λu, mintλ k , k P K 0 N : λ k ą λu, mintλ ḱ , k P K 0 N : λ ḱ ą λu ( . (4.2.14)
If λ l " H then λ l :" 0, while if λ r " H then λ r :" `8. After λ 1 leaves the region by adding or deleting one variable to or from the active set, we update in order the corresponding entry in w, K 1 , K 0 N , and the solution formula (4.2.12) (Sherman Morrison formula for one rank update of rΓ 1 0 s ´1) to calculate the boundary of the new region as before. We proceed in this way until we reach the region covering the λ value at which we would like to calculate the Lasso solution, and use Formula (4.2.12) in this final region to compute the a s 1 with the desired λ value. Lastly, we retrieve the matrix-form optimal solution based on a s 1 and the latest K 1 . This completes the first homotopy algorithm. For detailed algorithm, see the appendix of this chapter.

Homotopy from Apt, λq to Apt `1, λq

In [START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF], a continuous variable µ is introduced in the standard Lasso formulation (4.2.5) leading to the optimization Problem (4.2.15) below (4.2.15) in order to let the problem of learning from t samples evolve to that of learning from t `1 samples, as µ goes from 0 to 1. Therefore, representing the Lasso solution as a continuous function of µ permits the development of homotopy algorithm, which computes the path θpt, λq to θpt `1, t t`1 λq. This homotopy algorithm is derived based on the fact that, the term of new sample will only result in a rank-1 update in the covariance matrix as X J X `µ2 x t`1 x J t`1 , because only 1 response variable is present. Thus, the corresponding matrix inverse in the closed form of optimal solution can be still expressed as an explicit function of µ using the Sherman Morrison formula, which furthermore allows the calculation of critical points of µ. However, for the matrix-variate Lasso (4.2.4), a new sample will cause N F rank change * in Γ 0 , that is the number of response variables in the Lasso problem † . To formally understand this change, we rewrite Γ 0 as the sum of t reorganized samples analogous to usual p Γ t p0q

min θPIR d 1 2 }y ´Xθ} 2 ℓ 2 `1 2 pµy t`1 ´µx J t`1 θq 2 `tλ}θ} ℓ 1 ,
Γ 0 " 1 t t ÿ τ "1 r X τ ´1 r X J τ ´1, where r X τ ´1 P IR |K|ˆN F with r r X τ ´1s k,i " rU k s i,: x τ ´1,
note that a new x t`1 corresponds to the change r X t r X J t in Γ 0 , which is a rank N F matrix. Thus it is impossible to express " Γ 1 0 ‰ ´1 as an explicit and simple function of one single µ. However, note that each column (rank) r r X t s :,i corresponds to introducing new sample of one response variable x t`1,i :" rx t`1 s i at node i in G, by rewriting the incremental term of Lasso (4.2.4)

}x t`1 ´Ax t } 2 ℓ 2 " › › › › › x t`1 ´ÿ kPK xU k , A 0 y 1 }U k } 2 F U k x t › › › › › 2 ℓ 2 " N F ÿ i"1 ˜xt`1,i ´ÿ kPK xU k , A 0 y 1 }U k } 2 F rU k s i,: x t ¸2
Therefore, we propose to introduce N F continuous variables µ 1 , ..., µ N F in Lasso (4.2.4), and to consider the following problem

A λ,t pµ 1 , ..., µ N F q " arg min APK G L λ,t pµ 1 , ..., µ N F q,
where L λ,t pµ 1 , ..., µ N F q " 1 2pt `1q

t ÿ τ "1 }x τ ´Ax τ ´1} 2 ℓ 2 `λF }A N } ℓ 1 `1 2pt `1q N F ÿ i"1 µ i ˜xt`1,i ´ÿ kPK xU k , A 0 y 1 }U k } 2 F rU k s i,: x t ¸2 .
(4.2.16) Given solution Apt, λq, we first apply the homotopy Algorithm of Section 4.2.3.2 on it with λ 1 " λ and λ 2 " t`1 t λ to change the constant before the old data term from 1 t to 1 t`1 . Then, we have Apt, t`1 t λq " A λ,t p0, ..., 0q and Apt `1, λq " A λ,t p1, ..., 1q. We let evolve the optimization problem (4.2.4) from time t to t `1 by sequentially varying all µ i from 0 to 1, along the paths L λ,t p0, 0, ..., 0q Ñ L λ,t p1, 0, ..., 0q Ñ L λ,t p1, 1, ..., 1q " L λ,t`1 . Proposition 4.2.7. A minimizer A λ,t p..., 1, µ i , 0, ...q of min APK G L λ,t p..., 1, µ i , 0, ...q satisfies the linear system

# Γ 1 0 pµ i qa s 1 ´γ1 1 pµ i q `p1 `1 t qλw 1 " 0 Γ 0 0 pµ i qa s 1 ´γ0 1 pµ i q `p1 `1 t qλw 0 " 0,
where a s , K 0 N , K 1 N , K 1 , w are with respect to A " A λ,t p..., 1, µ i , 0, ...q, defining furthermore the extractions through (4.2.11),

Γ 0 pµ i q " Γ 0 `1 t i´1 ÿ n"1 r r X t s :,n r r X t s J :,n `µi t r r X t s :,i r r X t s J :,i ,
and

γ 1 pµ i q " γ 1 `1 t i´1 ÿ n"1 x t`1,n r r X t s :,n `µi t x t`1,i r r X t s :,i ,
with Γ 0 , γ 1 are the same ones as in Proposition 4.2.6.

The optimal conditions given in Proposition 4.2.7 show that, each path only relates to the one rank change: µ i t r r X t s :,i r r X t s J :,i , for the latest updated Γ 0 . Thus we can apply the Sherman Morrison formula on " Γ 1 0 pµ i q ‰ ´1 to retrieve the smooth function of µ i , and express a s 1 and w 0 as smooth functions of µ i , which furthermore makes the calculation of the critical points of µ i explicit. To leverage these continuity properties, we still assume ra s s K 1 N ‰ 0 (element-wise), and | rws K 0 N | ă 1 (element-wise). For the algorithm of path A λ,t p0, ..., 0q to A λ,t p1, ..., 1q, it is sufficient to impose such assumption only on A λ,t p0, ..., 0q. By arguing as in Section 4.2.3.2, we can derive the homotopy algorithm for the whole data path. The detailed Algorithm 5 is given in the appendix of this chapter.

Update from λ t to λ t`1

Given the previous solution Apt, λ t q, one way to select the hyperparameter value λ is to introduce the empirical objective function [START_REF] Monti | Adaptive regularization for lasso models in the context of nonstationary data streams[END_REF][START_REF] Garrigues | An homotopy algorithm for the lasso with online observations[END_REF], which takes the form

f t`1 pλq " 1 2 }x t`1 ´Apt, λqx t } 2 ℓ 2 ,
and to employ the updating rule

λ t`1 " λ t ´η df t`1 pλq dλ ˇˇλ"λt ,
where η is the step size. For convenience, we write df t`1 pλq dλ ˇˇλ"λt as df t`1 pλtq dλ . Analogously, we adopt the notation dApt,λtq dλ to denote the derivative with respect to λ, taken at value λ " λ t . The objective function f t`1 can be interpreted as an one step prediction error on unseen data. Since the Lasso solution is piece-wise linear with respect to λ, it follows that when λ is not a critical point, the derivative can be calculated as

df t`1 pλ t q dλ " B G t , dApt, λ t q dλ F " B Proj G pG t q, dApt, λ t q dλ F " ´"a Gt 1 ı J " Γ 1 0 ‰ ´1 w 1 ,
first calculate its sample mean as xt " ř t τ "1 x τ {t, then use x 1 τ :" x τ ´x t , τ " 0, ..., t, in the proposed methods. As mentioned in the introduction, the detrend step is not admissible in the online fashion, however, it inspires us to consider the trend in an additional step. We focus primarily on the periodic trend, then the proposed model is

x t " b 0 m `x1
t , where x 1 t " VAR(4.1.4), (4.3.1)

where m " t mod M P t0, . . . , M ´1u, M is the length of period which is a hyperparameter to be preassigned. The observations of Model (4.3.1) is x t , while x 1 t has the similar role as the unobserved state in the state space models however the observation equation here is much simplified. Therefore the estimators are built on the series x t . Note that Model (4.3.1) admits another reparameterization with intercept

x t " b m `Ax t´1 `zt , ( 4

New OLS estimators and asymptotic distributions

For the augmented model (4.3.1), we propose a new OLS estimator of A, which is based on the new sample auto-covariances, together with the OLS estimator of b 0 m . Because two crucial properties to derive the Wald tests in Section 4.2.2 are the consistency of sample auto-covariances p Γ t p0q, p Γ t p1q, and the CLT of OLS estimator q A t , we derive the corresponding asymptotic results for the new estimators, and show that these asymptotics are exactly the same as in the stationary case. Therefore, all the results and procedures presented in Section 4.2.2 can be applied directly on the new estimators. We first define the estimator of A, still denoted as q A t , using general least squares (GLS) method

q A t , p b m,t " arg min A,bm M ´1 ÿ m"0 S m pA, b m q, (4.3.3) where S m " ÿ τ PIm,t zJ τ Σ ´1z τ , zτ " x τ ´bm ´Ax τ ´1,
with I m,t " tτ " 1, ..., t : τ mod M " mu, and Σ ´1 the true white noise covariance given in Model (4.1.4). Note that zτ represents the residual of the prediction of sample x τ . The explicit forms of q A t , p b m,t can be found through straightforward calculation, which yields new sample auto-covariances, denoted still as p Γ t p0q, p Γ t p1q, and the estimator of trend p b 0 m,t . Specifically, we have The estimators given by Formula (4.3.3) enjoy the asymptotic properties in Proposition 4.3.1.

# q A t " p Γ t p1q " p Γ t p0q ı ´1 , p b m,
Proposition 4.3.1. The following asymptotic properties hold for the estimators

p Γ t p0q, p Γ t p1q, q A t , p b 0 m,t , as t Ñ `8, 1. p Γ t p0q p Ý Ñ Γp0q, p Γ t p1q p Ý Ñ Γp1q, 2. p b 0 m,t p Ý Ñ b 0 m , q A t p Ý Ñ A, 3. ? t vecp q A t ´Aq d Ý Ñ N p0, rΓp0qs ´1 b Σq, where Γphq " E `x1 t rx 1 t´h s J ˘, h ě 0, Σ " E `zt z J t ˘.
The proofs of the above results are given in the appendix of this chapter. Thus, Theorem 4.2.3 and the bisection Wald test procedure are still valid using Proj G N p q A t q and p Γ t p0q, p Γ t p1q defined in this section. On the other hand, p Γ t p0q and p Γ t p1q satisfy the one rank update formulas: (4.3.5) where m " pt `1q mod M . Thus, when new sample comes, r p Γ t`1 p0qs ´1 can still be calculated efficiently given the matrix inverse at the previous time. The details of the extended low dimensional learning procedure see Algorithm 3 in the appendix.

p Γ t`1 p0q " t t `1 p Γ t p0q `1 t `1 " p m,t p m,t `1 px t ´x m´1,t qpx t ´x m´1,t q J ȷ p Γ t`1 p1q " t t `1 p Γ t p1q `1 t `1 " p m,t p m,t `1 px t`1 ´x m,t qpx t ´x m´1,t q J ȷ ,

Augmented structured matrix-variate Lasso and the optimality conditions

To adapt the Lasso-based approach to Model (4.3.2), the corresponding trend and graph estimators can be obtained by minimizing the augmented Matrix-variate Lasso problem Apt, λq, b m pt, λq " arg min

APK G ,bm 1 2t M ´1 ÿ m"0 ÿ τ PIm,t }x τ ´bm ´Ax τ ´1} 2 ℓ 2 `λF }A N } ℓ 1 . (4.3.6)
As in the extension of our first approach, the extra bias terms b m , m " 0, ..., M ´1, do not affect the core techniques, rather they force the methods to consider the M means in the sample autocovariances. Since b m only appear in the squares term, the minimizers b m pt, λq have the same dependency with Apt, λq as in Equation (4.3.4).

Thus the trend b 0 m can still be estimated by x m,t , and we extend the algorithms in Section 4.2.3 to update the batch solution of augmented Lasso (4.3.6) from Apt, λ t q to Apt `1, λ t`1 q, given new sample x t`1 . To compute the regularization path Apt, λ t q Ñ Apt, p1 `1 t qλ t`1 q, Proposition 4.3.2 implies that Algorithm 4 can still be used, with the adjusted definitions of Γ 0 and γ 1 .

Proposition 4.3.2. A minimizer Apt, λq of Lasso problem (4.3.6) satisfies the linear system

# Γ 1 0 a s 1 ´γ1 1 `λw 1 " 0 Γ 0 0 a s 1 ´γ0 1 `λw 0 " 0,
where a s is the vectorized scaled Lasso solution Apt, λq, w, K 1 , K 0 N are also defined analogously from Apt, λq, while p Γ t p0q and p Γ t p1q used in the definitions of Γ 0 and γ 1 are the new sample auto-covariance matrices in Equation (4.3.4).

For the data path Apt, p1`1 t qλ t`1 q Ñ Apt`1, λ t`1 q, in the same spirit of Problem (4.2.16), we introduce variables µ 1 , ..., µ N F to let evolve Lasso problem (4.3.6) from time t to t `1 through the following variational problem

A λ t`1 ,t pµ 1 , ..., µ N F q, b m,λ t`1 ,t pµ 1 , ..., µ N F q " arg min APK G ,bm L λ t`1 ,t pµ 1 , ..., µ N F q, where L λ t`1 ,t pµ 1 , ..., µ N F q " 1 2pt `1q M ´1 ÿ m"0 ÿ τ PIm,t }x τ ´bm ´Ax τ ´1} 2 ℓ 2 `λt`1 F }A N } ℓ 1 `1 2pt `1q N F ÿ i"1 µ i px t`1,i ´b m,i ´ÿ kPK xU k , A 0 y 1 }U k } 2 F rU k s i,: x t q 2 ,
where b m,i " rb ms i , m " pt `1q mod M . To extend the homotopy Algorithm of data path, we first calculate the optimality conditions of Lasso L λ t`1 ,t pµ 1 , ..., µ N F q with respect to the constraint-free A 0 in Equation (4.3.7), then extract its vector representation in terms of a s 1 . A minimizer A 0 such that A " Proj G pA 0 q of L λ t`1 ,t pµ 1 , ..., µ N F q satisfies Equation (4.3.4). Thus, by equating the quantities in xU k , U k 1 ¨y and xU k , ¨y in the corresponding subdifferential formulas respectively, update formula (4.3.5) can be induced.

0 P BL λ t`1 ,t pµ 1 , ..., µ N F q BA 0 " t t `1 » - ÿ k,k 1 PK xU k , U k 1 p Γ t p0qy x 1 }U k 1 } 2 F U k 1 , A 0 y 1 }U k } 2 F U k ´ÿ kPK xU k , p Γ t p1qy 1 }U k } 2 F U k fi fl `1 t `1 N F ÿ i"1 µ i p m,t p m,t `µi » - ÿ k,k 1 PK px t ´x m´1,t q J rU k s J i,: rU k 1 s i,: px t ´x m´1,t qx 1 }U k 1 } 2 F U k 1 , A 0 y 1 }U k } 2 F U k fi fl ´1 t `1 N F ÿ i"1 µ i p m,t p m,t `µi « ÿ kPK px t`1,i ´x m,t,i qpx t ´x m´1,t q J rU k s J i,: 1 }U k } 2 F U k ff `λt`1 ÿ kPK N B ˇˇxU k , A 0 y ˇˇ1 }U k } 2 F U k , (
We recall that we update the solution along the path

L λ t`1 ,t p0, 0, ..., 0q Ñ L λ t`1 ,t p1, 0, ..., 0q Ñ L λ t`1 ,t p1, 1, ..., 1q " L λ t`1 ,t`1 .
At each step L λ t`1 ,t p..., 1, µ i , 0, ...q, µ i P r0, 1s, the optimal solution A λ t`1 ,t p..., 1, µ i , 0, ...q is piece-wise smooth with respect to µ i , element-wise. We retrieve the linear system of a s 1 in terms of µ i for each A λ t`1 ,t p..., 1, µ i , 0, ...q in Proposition 4.3.5. Proposition 4.3.5. A minimizer A λ t`1 ,t p..., 1, µ i , 0, ...q of Lasso L λ t`1 ,t p..., 1, µ i , 0, ...q satisfies the linear system

# Γ 1 0 pµ i qa s 1 ´γ1 1 pµ i q `p1 `1 t qλ t`1 w 1 " 0 Γ 0 0 pµ i qa s 1 ´γ0 1 pµ i q `p1 `1 t qλ t`1 w 0 " 0,
where a s , w, K 1 , K 0 N are with respect to A λ t`1 ,t p..., 1, µ i , 0, ...q, defining the extractions through (4.2.11),

Γ 0 pµ i q " Γ 0 `1 t i´1 ÿ n"1 p m,t p m,t
`1 r r X t ´r X m´1,t s :,n r r X t ´r X m´1,t s J :,n `µi t p m,t p m,t `µi r r X t ´r X m´1,t s :,i r r X t ´r X m´1,t s J :,i ,

γ 1 pµ i q " γ 1 `1 t i´1 ÿ n"1 p m,t p m,t
`1 px t`1,n ´px m,t q n qr r X t ´r X m´1,t s :,n `µi t p m,t p m,t `µi px t`1,i ´px m,t q i qr r X t ´r X m´1,t s :,i , with r r X m´1,t s k,i " rU k s i,: x m´1,t , p m,t :" t mod M , Γ 0 , γ 1 the same as Proposition 4.3.2.

Therefore, the derived homotopy algorithm is essentially the previous homotopy Algorithm 5 with minor changes. The details see Algorithm 6 in the appendix.

Lastly, we derive the updating rule for the regularization parameter. We still consider the one step prediction error, which writes as the following objective function in the case of Model (4.3.2)

f t pλq " 1 2 }x t`1 ´b mpt, λq ´Apt, λqx t } 2 ℓ 2 .
Given the previous solution Apt, λ t q and b mpt, λ t q, we assume that λ t is not a critical point. Then the derivative of f t with respect to λ is calculated as

df t pλ t q dλ " B df t pλq db mpt, λq ˇˇλ "λt , db mpt, λ t q dλ F `B df t pλq dApt, λq ˇˇλ "λt , dApt, λ t q dλ F " B G b t , ´dApt, λ t q dλ x m´1,t F `BG t , dApt, λ t q dλ F " B rApt, λ t qx t ´xt`1 `b mpt, λ t qsrx t ´x m´1,t s J , dApt, λ t q dλ F , where G b t " b mpt, λ t q ´xt`1 `Apt, λ t qx t , G t " rApt, λ t qx t ´xt`1 `b mpt, λ t qsx J t . Analogous to Section 4.2.3.4, we have xG t , dApt,λtq dλ y " ´"a Gt 1 ı J " Γ 1 0 ‰ ´1 w 1 . Using
the same updating rules of the projected stochastic gradient descent presented in Section 4.2.3.4, we can compute the online solution λ t`1 . We can see that, the introduction of bias terms b m into the original model makes them center the raw data automatically during the model fitting. This enables the direct learning over raw time series, while maintaining the performance of methods comparable to the stationarity-based ones.

We summarize the complete learning procedure of this subsection in Algorithm 6 in the appendix.

Experiments

We test the two proposed approaches for the online graph and trend learning on both synthetic and real data set.

Synthetic data

Evaluation procedures

We now present the evaluation procedure for the augmented model approaches. In each simulation, we generate a true graph A with the structure indicated by K G . In particular, we impose sparsity on its spatial graph A N by randomly linking a subset of node pairs. The values of non-zero entries in A N , and the entries in A F , diagpAq are generated in a random way. Additionally, we generate a trend over a period of M time points for each node and each feature. Therefore, the true b 0 m , m " 0, ..., M ´1, consists in these N F trend vectors, each containing M elements. Then, we synthesize very few samples x t from Model (4.3.1) until time t 0 . Figure 4.2 shows an example of the synthetic time series x t , compared with its stationary source x 1 t before adding the periodic trend. The graph and trend estimators proposed in Section 4.3 only use x t . We then set up the batch Lasso problem (4.3.6) with the generated samples and use its solution Apt 0 , λ 0 q to start the high-dimensional online procedure since the next synthetic sample. The batch problem is solved via the accelerated proximal gradient descent with the backtracking line search (Parikh and Boyd, 2014, Section 3.2.2). We especially set λ 0 as a large number so as to have an over sparse initial solution. Therefore, we expect to see a decreasing λ t , together with a more accurate estimate Apt, λ t q as t grows. The updating of Γt p0q and Γt p1q starts from t " 1, using Formula (4.3.5), since they are the only inputs of the proximal gradient descent algorithm. However, we wait until there are enough samples for Γt p0q to be invertible, then we start the low-dimensional online procedure at time t with r Γt p0qs ´1. To analyse the performance of the proposed approaches, we define the average one step prediction error metric as,

t ÿ τ "1 }x τ `1 ´p b mpτ `1q,τ ´p A τ x τ } 2 t}x τ `1} 2 , mpτ `1q " pτ `1q mod M, (4.4.1)
and root mean square deviation (RMSD) as,

} p A t ´A} F }A} F , (4.4.2)
where p A τ denotes the online estimates from either approach at time τ , and

p b mpτ `1q,τ " # x mpτ `1q,τ ´q A τ x mpτ `1q´1,τ , in low-dimensional,
x mpτ `1q,τ ´Apτ, λ τ qx mpτ `1q´1,τ , in high-dimensional.

We collect the metric values along time. We perform such simulation multiple times to obtain furthermore the means and the standard deviations of error metrics at each iteration (when the estimators are available) to better demonstrate the performance. The true graph A and trends b 0 m are generated independently across these simulations.

Simulation results

We first visualize the representative estimates in heatmap with N " 10, F " 4, and M " 12 for illustration purpose. Then we plot the evolution of error metrics and regularization parameter of 30 simulations for N " 20, F " 5, and M " 12. Lastly, we report the running time. The hyperparameter settings are given in the captions of figures of corresponding results.

Figures 4.3 and 4.4 show the estimated graphs of two approaches when their corresponding online procedures start. In Figure 4.3, we can see that the batch solution which starts the high-dimensional procedure is over sparse due to the large λ 0 . We can notice from Figure 4.4 that the two initial estimations of A F are already satisfactory, especially the Lasso solution which uses only 20 samples. Actually, estimations of A F and diagpAq converge to the truth very quickly in both cases when N is significantly larger than F . Figures 4.5 and 4.6 show that the estimations of A N of both approaches tend to the true values as more samples are received. Meanwhile, Figure 4.7 shows the effectiveness of trend estimator x m,t defined in Equation (4.3.4).

We now show the numeric results of 30 simulations, with N " 20, F " 5, and M " 12. We test three different step sizes η, 5 ˆ10 ´7, 1 ˆ10 ´6, and 5 ˆ10 ´6.

With each value we perform 10 independent simulations. Figure 4.8 and 4.9 plot the evolution of error metrics (4.4.1) and (4.4.2), respectively. For better visualization effect, since the performance of the low-dimensional procedure does not depend on η, we only show one mean metric curve instead of 3, in the two figures, which is calculated from the results of these 30 simulations.

Figures 4.8 and 4.9 show the convergence of A N estimations of both procedures. Moreover, for the high-dimensional procedure, the step size η determines the convergence speed. Especially, we can see from Figure 4.8 that the RMSD of the high-dimensional procedure with η " 5 ˆ10 ´6 decreases the most quickly for the 0 1 2 3 4 5 6 7 8 A F,91 of the low-dimensional procedure (middle), and A F p20, 0.05q of the high-dimensional procedure (right). Simulation settings: N " 10, F " 4, number of model parameters = 571. first 100 iterations, after which it starts to slow down and decrease more slowly than the RMSDs of the other two step sizes. For the low-dimensional procedure, when its estimator is available, the RMSD decreases very fast, and it shows the trend to keep decreasing for larger sample size. Nevertheless, the estimator of the low-dimensional procedure performs worse than the Lasso estimators in the sense of the prediction of unseen data, as shown in Figure 4.9. This is likely linked with the fact that the selection procedure updates the regularization parameter of the Lasso estimator towards the direction that minimizes the one step prediction error (4.4.1). The larger standard deviation is due to the larger magnitude of low-dimensional estimator, contrast to the Lasso estimator which is regularized by the ℓ 1 norm. This can also be observed in the scales of the y´axis in Figures 4.3 -4.6.

For synthetic data, it is not surprising that the RMSD from the low-dimensional procedure will tend toward zero, because these data are precisely sampled from the model used in the method derivation. On the other hand, at each online iteration, the OLS estimation is calculated accurately. In contrast, for the homotopy algorithms, they still introduce small errors, possibly due to the following assumptions used in the derivation of the method: 1. the active elements of K 1 N of the algorithm inputs are not zero ‡ ; 2. the sub-derivatives of those zero elements are strictly within p´1, 1q; 3. every λ t at which we calculate the derivative as in Section 4.2.3.4 is not a critical point. Thus, for example, small non-zero entry values in the inputs may cause the numerical errors. However, in real applications, the only available metric which allows the performance comparison is the prediction error (4.4.1).

Figure 4.10 demonstrates the performance of the updating method of the regularizing parameter λ, and the impact from different step size values η. The curves emphasize the convergence of the estimation updated by the high-dimensional procedure. Moreover, we can observe that λ t are decreasing, which was expected from the experiment design. On the other hand, the results show that a larger step size will make the convergence faster, yet more affected by the noise, especially when the solution has converged. The red curves are the mean RMSD of the high-dimensional procedure, taken over 10 simulations each. The blue curve is the mean RMSD of the low-dimensional procedure, taken over the same 30 simulations. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, significance level of χ 2 test " 0.1, t 0 " 20, λ 0 " 0.03. In the first high dimensional phase, the accurate estimator of the low-dimensional procedure is not available. .9: Average one step prediction error. The red curves are the mean prediction error of the high-dimensional procedure, taken over 10 simulations each. The blue curve is the mean prediction error of the low-dimensional procedure, taken over the same 30 simulations. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, significance level of χ 2 test " 0.1, t 0 " 20, λ 0 " 0.03.

We also compare the running time of a single online update for the two methods in Figure 4.11. Firstly, it is clear that updating the Lasso solutions by the homotopy algorithms saves considerable time, which is on average 0.20 seconds for the graph size N " 20, F " 5. The running time of the accelerated proximal gradient descent performed in the beginning of these simulations costs more than 3 seconds. By contrast, an update using the low-dimensional procedure takes 25 seconds on average. We can also notice that the high-dimensional procedure with larger step size runs slower, because the updated regularization parameter is quite different from the preceding one, as evidenced by the results in Figure 4.10.

Lastly, it is worthwhile to point out that, because the true A N has a high level of sparsity, the Wald test will accept H 0 : α " 0 more easily with lower significance levels, and we can observe the Wald estimator y A N,t rejects those false non-zero entries faster. Nevertheless, since we do not know the true graph sparsity for real data, the significance level can be regarded as the hyperparameter which controls the desired sparsity as well for the first approach.

Climatology data

We use the U.S. Historical Climatology Network (USHCN) data § to test our proposed approaches. The data set contains monthly averages of four climatology features, recorded at weather stations located across the United States, over years. The four features are: minimal temperature, maximal temperature, mean temperature, and precipitation. A snippet of the data set has been given in Figure 2.1, which illustrates these feature time series observed from a certain spatial location. A clear periodic trend can be seen from each scalar time series, with period length equal to 12 months. We can also notice that some observations are missing in the 400 500 600 700 800 Iteration The red curves are the mean running time of the high-dimensional procedure, taken over 10 simulations each. The blue curve is the mean running time of the low-dimensional procedure, taken over the same 30 simulations. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, significance level of χ 2 test " 0.1, t 0 " 20, λ 0 " 0.03. data set; to focus on the evaluation of learning approaches, we do not consider the stations with incomplete time series. Geographically, we picked data only from California and Nevada for this experiment. The summary of experiment setting thus is: N " 27, F " 4, M " 12, total number of time points = 1523 months (covering the years from 1894 to 2020). We apply the approaches from Section 4.3 on the raw time series to learn the weather graph of the region. The testing procedure using the real data is identical to the evaluation procedure with the synthetic data. We use the first t 0 `1 observations to set up the corresponding batch Lasso problem, and use its solution to start the high-dimensional procedure. The low-dimensional procedure will be started once Γt p0q becomes invertible. The average one step prediction error is calculated along online iterations. t 0 and λ 0 are always set as 20 and 0.03, respectively. Their values do not affect the methods' performance much, because of the adaptive tuning procedures of the regularization parameter.

Figure 4.13 and 4.14 show the spatial graphs learned by the two proposed approaches in Section 4.3 updated at different times. Figure 4.15 plots the evolution of regularization parameter value. We can see that, for the high-dimensional procedure, when more observations are received, it finds that more location pairs actually have a Granger causal effect on each other. On the other hand, compared to the estimated graphs from the high-dimensional procedure, those from the low-dimensional procedure vary more along time, which can be caused by the following facts: 1. in the early stage, Γt p0q is still ill-conditioned, therefore its inverse brings unstable OLS solutions; 2. the low-dimensional procedure relies on large sample properties of the designed estimators. These points are also supported by the average one step prediction curve given in Figure 4.16, where it is shown that, the prediction error of the low-dimensional procedure is significantly larger than the high-dimensional procedure, especially when the sample size is around 500 to 800. Next we show the last updated feature graphs in Figure 4.17. We can see that the estimated feature relationships from the two approaches coincide in tmin and tmax, tmin and tavg, tmin and prcp. However, the relationship between tavg and prcp is very weak in the Lasso estimation, while strong in the projected OLS estimation.

In particular, Figure 4.18 reports the evolution of estimated trends from one representative spatial location along time, where we can observe the increase of temperature from the past to the present. Figure 4.17: Updated feature graph at t " 1522. the low-dimensional procedure (left), and the high-dimensional procedure (right). Experiment settings: N " 27, F " 4, M " 12, number of model parameters = 1761, significance level of χ 2 test " 0.1, η " 10 ´5, t 0 " 20, λ 0 " 0.03. Lastly, in Figure 4.19, we plot the edge overlap (considering the signs of weights) of the two last updated spatial graphs, where we also visualize this spatial graph superimposed on the actual geographical graph. We can see that the remote weather stations have less dependency with other stations, while more edges appear within the area where lots of stations are densely located together. These observations imply that the inferred graphs provide the consistent weather patterns with geographical features. Furthermore, they validate the legitimacy of Models (4.1.4) and (4.3.1), as well as the effectiveness of the proposed learning methods. Proof of Theorem 4.2.3. By Cramér-Wold theorem, ?

t vecp q A t ´Aq d Ý Ñ N p0, Σ ols q is equivalent to xΛ, ? t ´q A t ´A¯y d Ý Ñ N p0, vecpΛq J Σ ols vecpΛqq, @Λ P IR N F ˆN F .
On the other hand, we can express the entries of svec ´?t ´y A N ´AN ¯¯as a linear

function of q A t svec ´?t ´y A N ´AN ¯¯" ÿ kPK N xU k , ? t ´q A t ´A¯y svecpE k q.
Then for all λ P IR N pN ´1q 2

, we have

λ J svec ´?t ´y A N ´AN ¯¯" x ÿ kPK N λ J svecpE k qU k , ? t ´q A t ´A¯y .
Let Λ in Equation (4.5.1) be ř kPK N λ J svecpE k qU k , then we have

λ J svec ´?t ´y A N,t ´AN ¯¯y d Ý Ñ N p0, vecpΛq J Σ ols vecpΛqq.
Note that, vecpΛq " ř kPK N λ J svecpE k qvecpU k q. Thus vecpΛq J Σ ols vecpΛq " λ J Σ N λ. Use Cramér-Wold theorem again, we can get the theorem result. ■ Theorem 4.5.1. (CLT for p A t )

? t vecp p A t ´Aq d Ý Ñ N p0, Σ G q where Σ G " ř k,k 1 PK vecpU k q J Σ ols vecpU k 1 q " vecpU k qvecpU k 1 q J ‰ .
Proof: The proof is similar as before. Because, we can express any entries of p A t as a linear function of q A t :

p A t " ÿ kPK xU k , q A t yU k .
Thus, for all Λ 1 P IR Lütkepohl (2005, Section 3.6). We first construct the following matrix:

C " ¨. . . svecpE h k q J . . . ‹ ‹ ' P IR P ˆNpN´1q 2 .
Then test H 0 versus H 1 equals to

H 1 0 : CsvecpA N q " 0 versus H 1 1 : CsvecpA N q ‰ 0.
Following CLT 4.2.3, we have

? t Csvecp y A N,t ´AN q d Ý Ñ N p0, CΣ N C J q.
Hence, when

H 1 0 holds, ? t Csvecp y A N,t q d Ý Ñ N p0, CΣ N C J q.
Then by Proposition C.2 (4) in [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF], we have

? t " C p Σ N,t C J ı ´1 2 Csvecp y A N,t q d Ý Ñ N p0, I P q,
where p Σ N,t " ř k,k 1 PK N vecpU k q J p Σ ols,t vecpU k 1 q `svecpE k qsvecpE k 1 q J ˘is the consistent estimator of Σ N . Then by continuous mapping theorem: Ý Ñ N p0, ΦΣ M Φ J q, @m " 0, ..., M ´1, where Φ " `INF ´AM ˘´1 , and

t αJ t " C p Σ N,t C J ı ´1 αt d Ý Ñ χ 2 pP q.

Note that

Σ M " ř M ´1 h"0 A h ΣpA h q J . Therefore, x1 m,t p Ý Ñ 0.
Proof of Lemma 4.5.2. Because of the periodicity, px 1 τ q τ PIm,8 is also a stationary process from VAR: r X t 1 " A M r X t 1 ´1 `z t 1 , with zt 1 " IIDp0, Σ M q, for all m " 0, ..., M ´1. Thus, apply Proposition 3.3 in [START_REF] Lütkepohl | New introduction to multiple time series analysis[END_REF] Use Woodbury formula on the matrix inverse, we have

? tp q A t ´Aq " ? tp p Γ t p1q 1 " p Γ t p0q 1 ı ´1 ´Aq ´?t M ´1 ÿ m"0 p m,t t `x 1 m,t rx 1 m´1,t s J ˘"p Γ t p0q 1 ı ´1 `?t 1 ´g p Γ t p1q 1 " p Γ t p0q 1 ı ´1 M ´1 ÿ m"0 p m,t t `x 1 m´1,t rx 1 m´1,t s J ˘"p Γ t p0q 1 ı ´1 ´?t 1 ´g M ´1 ÿ m"0 p m,t t `x 1 m,t rx 1 m´1,t s J ˘"p Γ t p0q 1 ı ´1 M ´1 ÿ m"0 p m,t t `x 1 m´1,t rx 1 m´1,t s J ˘"p Γ t p0q 1 ı ´1 ,
where, g " trp

M ´1 ř m"0 pm,t t `x 1 m´1,t rx 1 m´1,t s J ˘"p Γ t p0q 1 ı ´1q.
Based on the result of (c'), to reach the same asymptotic distribution, we only need to show that, the reminder terms, namely from the second term to the last term above, all converge to 0 in probability. From Slutsky's theorem and Lemma 4.5.

´p Σ W,t ¯k,k 1 " ´p Σ W,t ¯k1 ,k " xΣ k,k 1 ii , Γ k,k 1 jj y `xΣ k,k 1 jj , Γ k,k 1 ii y `xΣ k,k 1 ij , Γ k,k 1 ji y `xΣ k,k 1 ji , Γ k,k 1 ij y, where Σ k,k 1 ii " " p Σ t ı I k ,I k 1 , Γ k,k 1 jj " " p Γ t p0q ´1ı J k ,J k 1 , Σ k,k 1 jj " " p Σ t ı J k ,J k 1 , Γ k,k 1 ii " " p Γ t p0q ´1ı I k ,I k 1 , Σ k,k 1 ij " " p Σ t ı I k ,J k 1 , Γ k,k 1 ji " " p Γ t p0q ´1ı J k ,I k 1
, and

Σ k,k 1 ji " " p Σ t ı J k ,I k 1 , Γ k,k 1 ij " " p Γ t p0q r p Γ t`1 p0qs ´1 " t`1 t r p Γ t p0qs ´1 ´t`1 t r p Γtp0qs ´1pxt´x m´1,t qpxt´x m´1,t q J r p Γtp0qs ´1 tp1`1{p m,tq`pxt´x m´1,t q J r p Γtp0qs ´1pxt´x m´1,t q , q A t`1 " p Γ t`

Characterisation of distributional time series over nodes

In this chapter, we model the multivariate distributional time series pµ i t q tPZ , i " 1, . . . , N as random processes in the Wasserstein space W defined in Section 3.3, and propose a novel auto-regressive model to describe their dependency. In Section 5.1, we introduce the proposed time series model and analyze the existence, uniqueness and stationarity of its solution. Estimators of the model coefficients are studied in Section 5.2. Lastly in Section 5.3, we implement the numerical experiments with simulated data to verify the consistency of the proposed estimator. Additionally, we fit the proposed model on two real data sets to illustrate its applications in graph learning from the multivariate distributional time series.

For the sake of reproducible research, Python code available at https://github.com/yiyej/Wasserstein_Multivariate_Autoregressive_Model implements the proposed estimator and the experiments carried out in this paper.

Wasserstein multivariate AR Models

Description of the model

As explained in Section 2.2.1.7, the proposed multivariate distributional model is inspired by the intercept-free formulation of VAR(1) in Equation (3.2.6), which is built on the centered series px t ´uq tPZ . We first rewrite the formulation element-wise to motivate the distributional construction as

r x i t " N ÿ j"1 A ij r x j t´1
`zi t , t P Z, i " 1, . . . , N, (5.1.1) where r x i t " x i t ´ui , and x i t , u i , z i t are respectively the i-th component of x t , u and z t . We now extend VAR model by replacing each scalar time series px i t q t with univariate distributional series pµ i t q t . The key point is to propose the centering notion for pµ i t q t using their common Fréchet mean. We therefore primarily assume the Fréchet mean of the series is time-invariant as below, as the assumption of formulation (5.1.1).

Assumption A3. For each fixed i " 1, . . . , N , the random probability measures µ i t , t P Z are square integrable and they have the same Fréchet mean denoted by µ i,' .

We propose the following "data centering step" for µ i t , t P Z, i " 1, . . . , N . The centered measures are denoted by r µ i t , and they are defined through their quantile functions given as

r F ´1 i,t " F ´1 i,t a F ´1 i,' :" F ´1 i,t ˝pF ´1 i,' q ´1, (5.1.2)
where F ´1 i,t is the quantile function of µ i t extended at 0 (Bobkov and Ledoux, 2014, Section A1) by F ´1 i,t p0q :" inftx P r0, 1s : F i,t pxq ą 0u, and F ´1 i,' is the quantile function of µ i,' . Note that in most cases the function pF ´1 i,' q ´1 is equal to the cdf F i,' . However, when F i,' is only right-continuous but not continuous, pF ´1 i,' q ´1 is not equal to F i,' , since the left-continuous inverse only gives a left-continuous function. Thus, we shall keep the notation pF ´1 i,' q ´1. The centering step (5.1.2) at the level of the quantile functions is thus analogous to the usual centering step in VAR models for Euclidean data. From the optimal transport point of view, the centered quantile function r F ´1 i,t is interpreted as the optimal transport map from the Fréchet mean µ i,' to the measure µ i t . The notation a in (5.1.2) as a difference operator between two increasing functions is taken from the recent work in [START_REF] Zhu | Autoregressive optimal transport models[END_REF] on auto-regressive model for univariate (that is N " 1) distributional time series. We remark that the output of this difference operator remains an increasing function.

The function r F ´1 i,t needs to be defined over r0, 1s as a valid quantile function. Furthermore, as a result of data centering, we aim to turn the Fréchet mean of r F ´1 i,t to be Lebesgue measure. Thus we impose Assumption A4 below.

Assumption A4. All µ i t , t P Z, i " 1, . . . , N are supported on the same closed and bounded interval D Ă Ω. Without loss of generality, we assume that D " r0, 1s.

Under Assumption A4, the support of µ i t , t P Z, i " 1, . . . , N is a closed and bounded interval, and thus, their cdf are strictly increasing function on this support [Proposition A7] [START_REF] Bobkov | One-dimensional empirical measures, order statistics and kantorovich transport distances[END_REF]. Consequently, all the quantile function F ´1 i,t are continuous. Thus, the Fréchet mean has a continuous quantile function F ´1 i,' . The continuity of F ´1 i,' makes Equation (5.1.3) holds (Bobkov and Ledoux, 2014, Lemma A.3 5).

E " r F ´1 i,t ppq ı " pEF ´1 i,t q " pF ´1 i,' q ´1ppq
ı " p, p P p0, 1q.

(5.1.3) Thus, all the centered distributional time series p r µ i t q tPZ have the same Fréchet mean, which equals to the Lebesgue measure. We then propose to build an auto-regressive model for multivariate distributional time series with respect to the centered data r µ i t in the tangent space of the Lebesgue measure, that takes the following expression: (5.1.4) where tϵ i,t u i,t are i.i.d. random distortion functions taking values in the space of extended quantile functions Π " tF ´1 : r0, 1s Ñ r0, 1s, such that F ´1ˇp 0,1q P Log Leb W `id, F ´1p0q :" inftx P r0, 1s : F pxq ą 0u, andF ´1p1q :" suptx P r0, 1s : F pxq ă 1uu, endowed with } ¨}Leb and the induced Borel algebra, ϵ i,t is almost surely independent of µ i t´1 , i " 1, . . . , N, for all t P Z, and E rϵ i,t pxqs " x, x P r0, 1s.

r µ i t " ϵ i,t # Exp Leb ˜N ÿ j"1 A ij Log Leb r µ j t´1 ¸, t P Z, i " 1, . . . , N,
Note that all the univariate time series of log maps are centered to 0, namely, ErLog Leb r µ i t s " 0, @t P Z, i " 1, ..., N as in Model (5.1.1). The pushforward in (5.1.4) under ϵ i,t is a valid approach to provide random distortions of probability measures as proposed in Petersen and Müller (2019a). This approach is also used in Chen et al. (2021b). An example of random distortion function satisfying the conditions in Equation (5.1.4) as well as in Assumption A6 imposed later on, can be found, for example in Chen et al. (2021b, Equation (38)). However, in these works, not many examples of valid random distortion functions which satisfy the conditions in Equation (5.1.4) are given. Thus, to demonstrate that the conditions imposed on the distortion function are not restrictive, we describe, in Section 5.3 on numerical experiments, a general mechanism to generate random distortion functions that satisfy both Equation (5.1.4) and Assumption A6.

For the purpose of graph learning from distributional time series, a N ˆN matrix of coefficient matrix A :" pA ij q i,j captures the dependency structure between N features characterized by time-dependent probability measures, and represents the structure directly in a directed and weighted graph, in contrast with using more complex nonparametric coefficient construction, for example Wang et al. (2015, Equations (14) and ( 15)).

To fit Model (5.1.4), a least squares estimator of the matrix A can be constructed by minimizing the expected squared Wasserstein distance (3.3.1) between r µ i t and its prediction Exp Leb

´řN j"1 A ij Log Leb r µ j t´1 ¯. When ř N j"1 A ij Log Leb r µ j´1
t belongs to Log Leb W, the quantile function of its Exponential map Exp Leb is simply given by ř N j"1 A ij ´r F ´1 j,t´1 ´id ¯`id. By contrast, when

ř N j"1 A ij Log Leb r µ j´1
t falls out of Log Leb W, the dependency between the quantile function and the coefficients A ij is non-tractable, see Cazelles et al. (2017, Proposition 3.1). On the other hand, retaining the model in Log Leb W will avoid the non-identifiability problem of parametric models, thanks to Proposition 3.3.3. Thus, it will be needed to assume that ř N j"1 A ij Log Leb r µ j´1 t P Log Leb W. Since r µ j´1 t can take any value in W, imposing such assumption amounts to the following N -simplex constraint on the rows of A, given the convexity of the logarithmic image. Similar assumptions are imposed in related works, see e.g. Chen et al. (2021b, Assuption (A1)) and Petersen and Müller (2019b, Assumption (A3)), to keep the regression model in the logarithmic image.

Assumption A5.

ř N j"1 A ij ď 1 and 0 ď A ij ď 1.

An additional important advantage of Assumption (A5) is that it leads to least squares estimation of the matrix A under an ℓ 1 ball constraint on its coefficients. In this manner, the estimators of the coefficients A ij will naturally be sparse, which identifies the significant dependency links among the processes.

Given Assumption A5, we can also build the auto-regressive model directly with respect to the quantile function r

F ´1 i,t as r F ´1 i,t " ϵ i,t ˝« N ÿ j"1
A ij ´r F ´1 j,t´1 ´id ¯`id ff , t P Z, i " 1, . . . , N.

(5.1.5)

When reducing to the univariate case (N " 1), Model (5.1.5) is similar to the autoregressive model proposed in Zhu and Müller (2021, Model (4)), when regression coefficient belongs to p0, 1s.

Remark 5.1.1. To deal with the unequal expectations of random measures, we may alternatively find a reference measure at whose tangent space, a proposed interceptfree regression holds in expectation. Unfortunately, such a valid reference measure is very unlikely to exist. For example, consider the following regression model of three random measures, µ, ν 1 , and

ν 2 E " Log γ pµq | ν 1 , ν 2 ‰ " λ 1 Log γ pν 1 q `λ2 Log γ pν 2 q,
at the tangent space of an unknown reference measure γ to be identified. A valid measure γ that makes the model hold in expectation, needs to satisfy

F ´1 γ " F ´1 ',µ ´λ1 F ´1 ',ν 1 ´λ2 F ´1 ',ν 2
p1 ´λ1 ´λ2 q .

(5.1.6) However, the right hand side of the above equation is not necessarily an increasing function, which implies that a valid reference measure fails to exist in all negative cases. Furthermore, when plugging relationship (5.1.6) in the regression model, we obtain its reparameterization analogous to the reparameterization (3.2.6) of VAR( 1) model

E " F ´1 µ ´F ´1 ',µ | ν 1 , ν 2 ‰ " λ 1 pF ´1 ν 1 ´F ´1 ',ν 1 q `λ2 pF ´1 ν 2
´F ´1 ',ν 2 q. However, the centering in this reparameterized model is linear given through the subtraction in Hilbert space, which does not bring well-defined new measures. Indeed, such centering can be equivalently obtained by adding an intercept term to the regression model established in the tangent space of Lebesgue measure. Therefore, by contrast to these two equivalent operations, we propose to consider nonlinear centering of quantile functions, when developing multivariate regression models in Wasserstein space.

Estimation of the regression coefficients

In this section, we develop the estimators of coefficient A, given T `1 samples µ i t , t " 0, 1, . . . , T for each feature i " 1, . . . , N . We also show the consistency result of the proposed estimator. Note that we assume that the measures are fully observed, instead of indirectly observed through their samples.

A constrained least-square estimation method

As briefly explained before the statement of Assumption (A5), we could consider the estimator based on an unconstrained least squares method, which is defined as the minimizer of the sum of squared residuals measured by the Wasserstein distance:

r A i: " arg min A i: 1 T T ÿ t"1 d 2 W « r µ i t , Exp Leb ˜N ÿ j"1 A ij Log Leb r µ j´1 t ¸ff , i " 1, . . . , N,
" arg min x r F ´1 j,t ´id , r F ´1 l,t´1 ´idy Leb .

A i: 1 T T ÿ t"1 › › › › › r F ´1 i,t ´N ÿ j"1 A ij ´r F ´1 j,t´1 ´id ¯´id › › › › › Leb , i " 1, . . . , N, (5.
Note that r A is the exact least squares estimator constructed from the stationary solution of Model (5.1.5) without any constraint. However, in practice, we do not know the population Fréchet mean F ´1 i,' , thus we can not calculate the exact centered data r µ i t as in method (5.1.2). Therefore, we propose to first estimate F ´1 i,' by the empirical Fréchet mean " arg min Finally, we add the coefficient constraints to the problem, corresponding to the simplex constraint (A5). Therefore, the estimator  that we finally propose is defined as p A i: " arg min

F ´1 μi " 1 T T ÿ t"1 F ´1 µ i,t , (5
A i: 1 T T ÿ t"1 › › › › › p F ´1 i,t ´N ÿ j"1 A ij ´p F ´1 j,t´1 ´id ¯´id › › › › › Leb , i " 1, . . . ,
A i: PB 1 `1 T T ÿ t"1 › › › › › p F ´1 i,t ´N ÿ j"1 A ij ´p F ´1 j,t´1 ´id ¯´id › › › › › Leb , i " 1, . . . , N, (5.2.5)
where B 1 `is N -dimensional simplex, that is the nonnegative orthant of the ℓ 1 unit ball B 1 in IR N . Thus, an important advantage of this constraint is to promote sparsity in p A i: , which will be illustrated in Section 5.3. The optimization problem (5.2.5) can be solved by the accelerated projected gradient descent (Parikh and Boyd, 2014, Chapter 4.3). The projection onto B 1 `is given in [START_REF] Thai | Projected sub-gradient with ℓ 1 or simplex constraints via isotonic regression[END_REF].

Consistency of the estimators

Now, we study the consistency of the proposed estimator p A. The main result of this section is Theorem 5.2.5. The details of its proof is given in the appendix of this chapter. Instead in this section, we resume the proof by the key intermediate results in its development.

The proof proceeds by firstly showing the consistency of the unconstrained least squares estimator r A (see Lemma 5. f T pβq which converges to β ˚in probability at O p pr T q. We define then the constrained estimator β as: β " arg min βPIR n f T pβq subject to: f i pβq ď 0, i " 1, ..., m h j pβq " 0, j " 1, ..., p.

(5.2.8)

If f T is strongly convex with strong convexity constant µ T satisfying 1 2µ T " O p p1q, the strong duality holds for Problem (5.2.8) and β ˚satisfies the constraints, then β is also a consistent estimator of β ˚, with β ´β o " O p pr T q.

The result is evident from the other direction, since βo p Ñ β ˚and β ˚satisfies the constraints, then βo should gradually approach the constraint set as the simple size T increases. Once βo itself satisfies the constraint, β takes βo , which can be only asymptotic though.

Many results establish conditions under which strong duality holds. For example when the objective function is convex with an open domain and the constraints f i , i " 1, ..., m, h j , j " 1, ..., p are all affine, then strong duality holds as soon as the problem is feasible (Boyd et al., 2004, Section 5.2.3), which is the case of Problem (5.2.8). Note that, the problem can be represented in vector form as p A i: " arg min

A i: PB 1 `1 2 A i: " p Γp0q ı A J i:
´"p Γp1q ı i,:

A J i: , i " 1, . . . , N.
Since Theorem 5.2.3 shows the constraint free least square estimator on the approximate series converges to A in probability, applying Theorem 5.2.4 on Theorem 5.2.3, we finally obtain the desired consistency result, given in Theorem 5.2.5.

Theorem 5.2.5. Under the conditions of Lemma 5.2.1, assume 1 4λ T " O p p1q where λ T is the smallest eigenvalue of p Γp0q, then given the true coefficient A satisfies Assumption A5, namely, A i: P B 1 `, i " 1, . . . , N , we have

p A ´A p Ñ 0.
Note that the condition on the eigenvalue of p Γp0q is very light, because of the non-linearity in the calculus p Γp0q, in practice with only a very few samples the matrix will become and stay invertible.

Numerical experiments

In Section 5.3.1, we firstly demonstrate the consistency result of the proposed estimator using synthetic data. Then, we fit the model on two real data sets in Section 5.3.2 and Section 5.3.3. For each data set, the estimated coefficient matrix p A allows us to understand the dependency structure of the multivariate distributional time series. In particular, we visualize the learned structure between features using a directed weighted graph with adjacency matrix p A.

Simulations

Generation of the synthetic data

We firstly propose a mechanism to generate the valid random distortion functions. To this end, we consider the random functions defined by

ϵ g " 1 `ξ 2 g ˝h´1 `1 ´ξ 2 h ´1, (5.3.1)
where g is a non-decreasing right-continuous constant function from r0, 1s to r0, 1s, h ´1 is the left continuous inverse of h " 1 2 pg `idq , and ξ " U p´1, 1q is a random variable. For any given function g, we can sample a family of distortion functions ϵ i,t i.i.d.

" ϵ g , when sampling ξ i,t i.i.d.

" U p´1, 1q. This construction of random distortion functions is inspired by the one proposed in Zhu and Müller (2021, Equation ( 13)), however, we have modified their construction of h and of the random coefficients. It is easy to verify that Erϵ g s " 1 2 pg ˝h´1 `h´1 q " 1 2 pg `idq ˝h´1 " id.

To make ϵ i,t satisfy additionally Model (5.1.4) and Assumption A6, we require g to be furthermore continuous and differentiable. Then on the one hand, since g is continuous and non-decreasing, any generated ϵ g is non-decreasing and left-continuous. On the other hand, note that

rh ´1s 1 " 1 h 1 ˝h´1 " 1 1 2 pg 1 `1q ˝h´1 " 2 g 1 ˝h´1 `1 .
Thus, we have

ϵ g 1 " 1 `ξ 2 `g1 ˝h´1 ˘2 g 1 ˝h´1 `1 `1 ´ξ 2 2 g 1 ˝h´1 `1 " ˆ1 `ξ 2 g 1 ˝h´1 `1 ´ξ 2 ˙2 g 1 ˝h´1 `1 " 1 `ξ ´ξ 2 g 1 ˝h´1 `1 " 1 `ξ ˆ1 ´2 g 1 ˝h´1 `1 ˙.
This implies

|ϵ g 1 | ď 1 `|ξ| ˇˇˇ1 ´2 g 1 ˝h´1 `1 ˇˇˇď 2.
The bound comes from ξ " U p´1, 1q and g 1 ě 0, which is hence tight. Thus any ϵ g generated by Formula (5.3.1) is Lipschitz continuous, with the constant uniformly bounded by 2 over ξ. Note that Assumption A6 requires the Lipschitz continuity only in expectation. Thus, the i.i.d. samples ϵ i,t of any ϵ g satisfy obviously Assumption A6 with the largest L " 2. The function g on the left is given by the natural cubic spline passing through the points p0, 0q, p0.2, 0.1q, p0.6, 0.2q, p1, 1q. On the right is one realization of 30 i.i.d. samples of the resulting ϵ g .

Secondly, we present the procedure to generate the true coefficient matrix A. We first generate a sparse matrix with the weights all positive in a random way, denoted by A 0 . We then normalize each row of A 0 by the row sum to fulfill Assumptions A5. We denote this last matrix still by A 0 . Based on the previous mechanism for the random distortion function, we take L " 2 in Assumption A7. Lastly, we scale down A 0 by p2 `αq}A 0 } 2 to obtain a valid A. We test two values α " 0.1 and α " 0.5 in our experiments.

Given a valid matrix of coefficients A and samples of ϵ i,t , we can then generate the "centered" quantile functions r F ´1 i,t from Model (5.1.5). Note that, r F ´1 i,t are only the simulations of the transformed data. Thus, we have to generate furthermore the population Fréchet mean F ´1 i,' of each univariate series in order to finally obtain the Lastly, we show in Figures 5.4 and 5.5 the complete execution time of the model fitting on the raw data with respect to the sample size T . We can see that the execution time increases linearly with respect to T , and A with the smaller ℓ 2 norm requires sightly less time (α " 0.5) than the other. The linear increase comes mainly from the loop over time t " 1, ..., T in calculating the empirical Fréchet mean (5.2.2) and in calculating the matrices p Γp0q, p Γp1q by their formulas in Equation (5.2.4). The running time of these calculations is determined by the granularity in the numerical methods to approximate the function composition, function inverse, and the inner product. The granularity applied during this simulation is 0.01, that is we input/output only the quantile function values at grid 0, 0.01, ..., 0.99 to/from each numerical approximation.

Age distribution of countries

We firstly test the proposed model with the data of distributional observations illustrated in Figure 2.2 in the introduction. These data are from the US Census Bureau's International Data Base * , which provides the population estimates and projections for countries and areas by single year of age, over years. We would like to apply the proposed model on this international age distribution data to learn about the links among the changes in the age structures of different countries. Specially, we consider the countries and the micro-states in the European Union and/or Schengen Area. Because the corresponding data used during the model fitting starts in the 1990s, we also include the former European Union member United Kingdom. Note that, Vatican City is not included since it is not available in the data base. Therefore, the list of 34 countries in this study is: Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Monaco, Netherlands, Norway, Poland, Portugal, Romania, San Marino, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom. Time-wise, we consider the 40 years between 1996 to 2035. Note that 1996 is the earliest year for which the data for all the considered countries is available.

To apply Model (5.1.4), we firstly represent the distribution of age population, of country i, at year T , by µ i t , with T " 1, ..., 40 and i " 1, ..., 34. Note that the age considered by the data base goes through 0 to 100-plus. Thus we take the 100-plus as 100, and moreover scale down the age by 100 to make the age distribution supported over r0, 1s. Then we retrieve the quantile function F ´1 i,t of µ i t from the population counts by ages of country i recorded at year T , with the numeric methods. In particular, we retrieve the quantile functions using continuous functions which take 0 on p " 0 and 1 on p " 1 so as to be consistent with Assumption A4 (for details see function generate_qt_fun defined in script age_pop.py in the code related to this paper).

We fit Model (5.1.5) on the retrieved functions F ´1 i,t , T " 1, ..., 40, i " 1, ..., 34. We use the same stopping criteria as in the simulation, while we apply the granularity of 0.002. The complete execution time of model fitting takes around 78 seconds. Figure 2.4 in the introduction comes from the inferred coefficient A in this experiment. We recall this result in Figure 5.6 with a more technical caption and the interpretation with the data set.

Firstly, we can notice that for all countries i P t1, ..., 34u, the weight of self-loop A ii dominates the weights of incoming edges A ij , j " 1, ..., 34, which are bounded by 0 ď ř 34 j"1 A ij ď 1. This is because the age structure of a country does not change much from one year to another. On the other hand, this also implies the age structure differs largely across countries. Nevertheless, there are still significant links between countries' age distribution. The first two largest weights excluding all the self-loops are respectively on the edges: Estonia Ñ Latvia, and Sweden Ñ Norway. To justify the inferred edges, we plot the evolution of age structure of these four countries in Figures 5.7 and 5.8.

We can see that within these four countries, the age structures between the linked countries are similar along time, by contrast, the structures between the unlinked countries are very different. Indeed, these two linked pairs consist both of the countries which share long distance of borders. Thus generally, the inferred edges in Figure 5.6 indicate the similarity of the age structures between countries from 1996 to 2036. Moreover, the directions of the edges imply, at the model level, that, when age structures in the outward countries (for example, Estonia, Sweden) change, it will induce relative changes in the inward countries (respectively, Latvia, Norway). These numeric findings can be furthermore explained in demography or not. On the other, we are interested in the neighbouring countries which are not linked. We verify for Since Y j and Y 1 l are increasing function from p0, 1q to r0, 1s, }Y j ´id} ď 1 2 and }Y 1 l ´id} ď 1 2 . Thus, we have furthermore

|g jl pYq ´gjl pY 1 q| ď 1 2 `}Y l ´Y1 l } `}Y j ´Y1 j } ď ? 2 2 `}Y l ´Y1 l } 2 `}Y j ´Y1 j } 2 ˘1 2 . Then sup Y,Y 1 # |g jl pYq ´gjl pY 1 q|1 ˆbř N i"1 }Y i ´Y1 i } 2 ăδ ˙+ ď ? 2 2 δ.
Thus g jl is stochastic dini-continuous. Since IRF system (5.1.8) is geometric moment contracting indicated in the proof of Theorem (5.1.2), then by Theorem 3 in [START_REF] Wu | Limit theorems for iterated random functions[END_REF],

S T pg jl q ? T :" ? T ˆ"r Γp0q ı j,l ´rΓp0qs j,l ˙d Ñ σ g jl N p0, 1q, which is followed " r Γp0q ı j,l ´rΓp0qs j,l " O p p 1 ? T q.
The proof above holds valid for any j, l " 1, . . . N . Analogously, for any j, l " 1, . . . N , we define g 1 jl : pX , x¨, ¨yq ˆpX , x¨, ¨yq Ñ IR as g jl pX, X 1 q " xX j ´id , X 1 l ´idy Leb ´E xX j ´id , X 1 l ´idy Leb .

It can be shown by the similar proof as before that g jl pX, X 1 q is stochastic dinicontinuous with respect to the product metric ρppX, X 1 q, pZ, Z 1 qq "

g f f e N ÿ i"1 }X i ´Zi } 2 `N ÿ i"1 }X 1 i ´Z1 i } 2 .
Therefore by Theorem 3 in [START_REF] Wu | Limit theorems for iterated random functions[END_REF], we have

" r Γp1q ı j,l ´rΓp1qs j,l " O p p 1 ? T q, j, l " 1, . . . N.
Since matrix inversion is a continuous mapping, thus by continuous mapping theorem " r Γp0q ´1ı j,l

´"Γp0q ´1‰ j,l " O p p 1 ? T q, j, l " 1, . . . N.

Representations of r

A and A then bring to r A j,l ´Aj,l " O p p 1 ? T q, j, l " 1, . . . N . ■ 5.4. Appendix p Γp0q, p Γp1q are given in Equation (5.2.4). Thus, we first show r p Γp0qs j,l ´rr Γp0qs j,l p Ñ 0. By calculation, we have

r p Γp0qs j,l " 1 T T ÿ t"1 x p F ´1 j,t´1 ´id , p F ´1 l,t´1 ´idy Leb paq " 1 T T ÿ t"1 x p F ´1 j,t ´id , p F ´1 l,t ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj ´id , F ´1 l,t a F ´1 μl ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl ´idy Leb ´1 T T ÿ t"1 xid , F ´1 l,t a F ´1 μl ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl y Leb ´1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl y Leb ´xid , idy Leb `Op 1 T q,
(5.4.2) and r r Γp0qs

j,l " 1 T T ÿ t"1 x r F ´1 j,t´1 ´id , r F ´1 l,t´1 ´idy Leb pbq " 1 T T ÿ t"1 x r F ´1 j,t ´id , r F ´1 l,t ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' ´id , F ´1 l,t a F ´1 j,' ´idy Leb `Op 1 T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' ´idy Leb ´1 T T ÿ t"1 xid , F ´1 l,t a F ´1 l,' ´idy Leb `Op 1 T q Lemma5.4.1 " 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' ´idy Leb `Op p 1 ? T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb ´1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , idy Leb `Op p 1 ? T q Lemma5.4.1 " 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb ´xid , idy Leb `Op p 1 ? T q.
Equations (a) and (b) come from p F ´1 i,t and r F ´1 i,t´1 are bounded between 0 and 1 for all i " 1, ..., N and t P N. Thus,

r p Γp0qs j,l ´rr Γp0qs j,l " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl y Leb ´1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb `Op p 1 ? T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj , F ´1 l,t a F ´1 μl y Leb ´1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb `1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb ´1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb `Op p 1 ? T q " 1 T T ÿ t"1 xF ´1 j,t a F ´1 μj ´F ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb `1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl ´F ´1 l,t a F ´1 l,' y Leb `Op p 1 ? T q.
(5.4.3) We are now to show 1

T ř T t"1 xF ´1 j,t a F ´1 μj ´F ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb p Ñ 0. We bound the absolute value ˇˇˇˇ1 T T ÿ t"1 xF ´1 j,t a F ´1 μj ´F ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb ˇˇˇď 1 T T ÿ t"1 ż 1 0 ˇˇpF ´1 j,t a F ´1 μj qppq ´pF ´1 j,t a F ´1 j,' qppq ˇˇˇˇpF ´1 l,t a F ´1 μl qppq ˇˇdp paq ď 1 T T ÿ t"1 ż 1 0 ˇˇpF ´1 j,t a F ´1 μj qppq ´pF ´1 j,t a F ´1 j,' qppq ˇˇdp.
Inequality (a) is because F ´1 l,t a F ´1 μl is bounded between 0 and 1. Note that for any fixed p P p0, 1q, we have

1 T T ÿ t"1 ˇˇpF ´1 j,t a F ´1 μj qppq ´pF ´1 j,t a F ´1 j,' qppq ˇˇ" 1 T T ÿ t"1 ˇˇF ´1 j,t ˝pF ´1 μj q ´1ppq ´F ´1 j,t ˝pF ´1 j,' q ´1ppq ˇ" # 1 T ř T t"1 F ´1 j,t ˝pF ´1 μj q ´1ppq ´F ´1 j,t ˝pF ´1 j,' q ´1ppq, if pF ´1 μj q ´1ppq ą pF ´1 j,' q ´1ppq, 1 T ř T t"1
´F ´1 j,t ˝pF ´1 μj q ´1ppq `F ´1 j,t ˝pF ´1 j,' q ´1ppq, otherwise, " # p ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq, if pF ´1 μj q ´1ppq ą pF ´1 j,' q ´1ppq, ´p `pF ´1 μj q ˝pF ´1 j,' q ´1ppq, otherwise, " ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇ.

Appendix

Thus

1 T T ÿ t"1 ż 1 0 ˇˇpF ´1 j,t a F ´1 μj qppq ´pF ´1 j,t a F ´1 j,' qppq ˇˇdp " ż 1 0 1 T T ÿ t"1 ˇˇpF ´1 j,t a F ´1 μj qppq ´pF ´1 j,t a F ´1 j,' qppq ˇˇdp " ż 1 0
ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp.

By Lemma 5.2.2, we have for any fixed p 0 P p0, 1q, ˇˇp 0 ´pF ´1 μj q ˝pF ´1 j,' q ´1pp 0 q ˇˇ" Op 1 ? T q. Since ˇˇp 0 ´pF ´1 μj q ˝pF ´1 j,' q ´1pp 0 q ˇˇis a bounded sequence uniformly over T , E ˇˇp 0 ´pF ´1 μj q ˝pF ´1 j,' q ´1pp 0 q ˇˇconverges to 0. Thus, E ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇč onverges pointwise to 0. On the other hand, note that for any fixed ϵ ą 0, we have lim

T Ñ8 P ˆż 1 0 ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp ą ϵ ď lim T Ñ8 E ş 1 0 ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp ϵ pbq " lim T Ñ8 ş 1 0 E ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp ϵ pcq " ş 1 0 lim T Ñ8 E ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp ϵ " 0.
Exchange (b) comes from that ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇis bounded for all p P p0, 1q, thus E ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇexists for all p P p0, 1q. Moreover, E ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇǐ s bounded uniformly over p P p0, 1q, thus we can furthermore exchange the integral and the limit by bounded convergence theorem, which brings to Equation (c). Therefore, ż 1 0 ˇˇp ´pF ´1 μj q ˝pF ´1 j,' q ´1ppq ˇˇdp p Ñ 0.

This implies

1 T ř T t"1 xF ´1 j,t a F ´1 μj ´F ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl y Leb p Ñ 0.
Since the proof above is valid for all j, l " 1, ..., N , we have immediately 1

T ř T t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 μl ´F ´1 l,t a F ´1 l,' y Leb p Ñ 0.
Combined the two convergence results in Equation (5.4.3) gives r p Γp0qs j,l ´rr Γp0qs j,l p Ñ 0, for all j, l " 1, ..., N . Analogously, we can prove r p Γp1qs j,l ´rr Γp1qs j,l p Ñ 0, for all j, l " 1, ..., N . Since matrix inversion is a continuous mapping, thus by continuous mapping theorem r p Γp0q ´1s j,l ´rr Γp0q ´1s j,l p Ñ 0, j, l " 1, ..., N.

Representations of p

A o , r A bring to p A o ´r A p Ñ 0. ■ The last term in Equation (5.4.2) implies that it is sufficient to obtain 1 T T ÿ t"1 xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb ´E xF ´1 j,t a F ´1 j,' , F ´1 l,t a F ´1 l,' y Leb Ñ 0,

Appendix

Use once again the strong convexity of f T , we obtain [START_REF] Zhou | On the fenchel duality between strong convexity and lipschitz continuous gradient[END_REF], Lemma 2 (iii))

f T p βq ´fT p βo q ě xs, β ´β o y `µT 2 › › › β ´β o › › › 2 ℓ 2
, s P Bf T p βo q.

(5.4.6)

Plug s " 0 in Equation (5.4.6), we get

2 µ T ´fT p βq ´fT p βo q ¯ě › › › β ´β o › › › 2 ℓ 2
.

The stochastic boundedness and the convergence finally leads to β ´β o " O p pr T q. ■ Chapter 6

Learning node predictability

In this chapter, we rely on the reconstruction problem over a network to measure the predictability of nodes. In Section 6.1, we develop the introduction in Section 2.2.2.1 of the adaptation of kernel ridge regression to the reconstruction problem and the devised ranking procedure. We also present the design of kernel on the domain N ˆZ. In Section 6.2, we evaluate the node predictability with neural networks and obtain a finer ranking with scores. We leverage the model aggregation effect of dropout technique to assess the overall reconstruction accuracy of each node. Additionally, in Section 6.3, we consider the linear regression for the case when the connectivity of nodes are not available. Lastly, numerical experiments are carried out in Section 6.4 on the bike-sharing data set that we have seen in the experiment of Chapter 5.

We recall the reconstruction problem. Let px it q t , i P N :" t1, . . . , N u be the observations over the network of interest N . Hypothetically, we would like to reconstruct the observations at time t over a subset I of nodes, x it , i P I Ă N , from the past and present recordings on the rest of nodes px jτ q jτ , j P I c , t ´H ď τ ď t. In this chapter, we apply the classical practice in multivariate time series domain to center or detrend the raw series px it q t , i P N in a preprocessing step. Thus we do not explicitly deal with non-zero process mean in the proposed models.

Kernel ridge regression of time and node predictor

As introduced in Section 2.2.2.1, to reconstruct a real value indexed by (thus mapped from) i P N , t P Z, we approximate the regression operator by the functions in RKHS H k induced by a spatio-temporal kernel k on N ˆZ. We furthermore require the kernel value to only depend on the time lag h rather than specific time stamps, namely k rpi, tq, pj, t ´hqs " k rpi, t 1 q, pj, t 1 ´hqs. Hence, we can denote kernel value k rpi, tq, pj, t ´hqs by kpi, j, hq, analogous to the notations of auto-covariance for stationary process. Since the rest of kernel design does not impact the development of the ranking method, we present the proposed kernel design at the end of this section. We introduce the following notations.

Kernel ridge regression of time and node predictor

Let K N phq denote the Gram matrix of N ˆthu for h P Z, that is K N phq P IR N ˆN with rK N phqs ij " kpi, j, hq, i, j P N .

We then denote the matrix extraction rK N phqs i,I c by K iI c phq. Analogously, we have K I c phq " rK N phqs I c . The extractions are defined the same way as in Chapter 4. We moreover define the block matrix We now recall the best approximation namely the estimator in kernel ridge regression is given by

p f t " min f PH k ÿ iPI c ,t´Hďτ ďt }x iτ ´f pi, τ q} 2 ℓ 2 `λ}f } 2 H k .
Representer theorem indicates that p f t has the form p f t pi, τ q " `KiI c pτ ´tq ¨¨¨K iI c pτ ´t `Hq ˘αt , i P N , τ P Z, where α t " pK H I c `λI q q ´1x H I c t and x H I c t " px I c t , x I c ,t´1 , . . . , x I c ,t´H q.

Note that x H I c t is the observations on nodes I c from time t back to time t ´H. Thus we have the prediction of x it as K H iI c α t for all i P I, where K H iI c " `KiI c p0q K iI c p1q ¨¨¨K iI c pHq.

The total prediction error given the history of network observations px it q i,t , i P N , 1 ď t ď T can then be measured by the residual sum of squares To rank the predictability of all nodes in the network, we propose to rely on the greedy adaptation of a sensor selection problem to take different I into account. We consider such sensor selection problem where for a given cardinality p, we search the optimal set I ˚ppq which minimizes the criteria ř iPI RSSpi|I c q. The greedy strategy of this selection problem makes up the first p steps of the proposed ranking algorithm 7, and it extends compatibly to p " N ´1.

ÿ iPI RSSpi|I c q " ÿ iPI 1 T T ÿ t"H`1 px it ´p Θ iλ x H I c t q 2 " ÿ iPI ´p σ i ´2 p β H iI c p Θ J iλ `p Θ iλ p α H I c p p Θ iλ q J ¯, (
Algorithm 7 Ranking of predictability evaluated by the kernel predictor.

Input: p p

Γpjqq H j"0 , K H N , λ. Initialize: n " 1, I c 0 " N . Compute pK H I c 0 `λI q q ´1. for n ă N do for i P I c n´1 do S i Ð I c n´1 zi. Compute pK H S `λI q q ´1 from pK H I c n´1 `λI q q ´1 (block matrix inversion formula). Compute RSSpi|S i q as Equation (6.1.1). end for i n Ð arg min iPI c n´1 RSSpi|S i q. I c n Ð I c n´1 zi n . Compute pK H I c n `λI q q ´1 from pK H I c n´1 `λI q q ´1 (block matrix inversion formula). end for Output: i 1 , i 2 , . . . , i N .

We recall the interpretation of the ranking derived from the greedy adaptation: for every given cardinality p, we assign the nodes in the optimal set I ˚ppq one score, the rest nodes zero score. The total scores for all p's from 1 to N forms the ranking.

Note that, in Algorithm 7, we still rely on the block matrix inversion formula to treat pK H S `λI q q ´1 when S slides over I c n to gain OpH|I c n |q speed-up with respect to the naive matrix inversion. The inner loop of Algorithm 7 will cost OpH 3 |I c n | 3 q, therefore the whole ranking algorithm at OpH 3 N 4 q. The algorithm requires to be moreover optimized to handle large scale networks in the future work.

Lastly, for the time series over a network, we propose the following kernel design to describe the similarity of data points. Assuming that we have the knowledge of the connectivity of nodes N , characterized by the edge set E, we can construct the corresponding adjacency matrix A, and the graph Laplacian L by the definitions in Section 3.1. The Laplacian defines a popular kernel on the vertex domain as below. Definition 6.1.1. Given Laplacian matrix of a graph, denoted by L. If L admits orthonormal eigendecomposition denoted as ΦΛΦ J , we can define the corresponding graph Laplacian kernel k G : N ˆN Ñ IR as k G pi, jq " K ij , where K " ΦrpΛqΦ t , with r : IR `Ñ IR `a pre-given mapping. A typical example of r is rpλq " # λ ´1, λ ‰ 0, 0, otherwise, 6.2. Dropout in predictability evaluation for neural network predictors for which the matrix K " L ´is the Moore-Penrose inverse of L.

We then propose kernel (6.1.4) for the domain X " N ˆZ.

Definition 6.1.2. The kernel k GT : X ˆX Ñ IR is defined as k GT " pi, tq, pj, t 1 q ‰ " k G pi, jq ¨kT pi, j, t ´t1 q, (6.1.4)

where k G is the graph kernel, k T pi, j, t ´t1 q " r p Γpt ´t1 qs i,j .

By Definition 3.5.1, k T is a valid kernel. Thus according to Remark 3.5.2, k GT is a valid kernel as well.

Dropout in predictability evaluation for neural network predictors

We firstly adapt a given network to the reconstruction problem introduced for the predictability evaluation. For the network predictor, we only use the present observations px jt q jPI c to reconstruct px it q iPI . Since in experiment, we do not see the gain from regressing on further history. In order to facilitate the application of dropout, we do not directly take px jt q jPI c as input of network. Instead, we compose a complete graph signal r x I c t by inserting zeros to the places of px it q iPI , that is x it , i P I c , 0, i P I. r x I c t is then an input sample. Its output is denoted by p x It " pp x it q iPI , which is compared to the target x It " px it q iPI through the loss function J. We define Jpp x It ; x It q " }p x It ´xIt } 2 ℓ 2 .

When no dropout is applied, the gradient used to update a network parameter v at the n-th updating step is then computed as the average

1 |B n | ÿ tPBn BJpp x It pvq; x It q Bv ,
where B n is the batch of the step. Now in order to vary the set I within the same training, we apply dropout on the input layer. Instead of r x I c t , we input the full graph signal x t and multiply it with the dropout vector w pnq to randomly choose a missing set I n . We recall that we re-sample the dropout at each batch. Thus the actual inputs to the prediction network from batch B n are x t d w pnq , t P B n . On the output layer, we change the dimension from |I| to N , we denote this output by p x t . We apply the reverse dropout on the batch loss as 1 |B n | ÿ tPBn › › ›p x t d p1 ´wpnq q ´xt d p1 ´wpnq q › › › 2 ℓ 2 .

Partial variance of multivariate time series

When no edge information is provided, to predict px it q iPI given px jτ q τ , j P I c , t ´H ď τ ď t, we consider the linear regression to introduce the notion of partial variance. We assume the multivariate process x t " px it q iPN is stationary under the definition of Section 3.2.1, with zero mean and auto-covariance matrices Γphq. Then the partial variance of px it q t , i P I Ă N given px iτ q τ , i P I c , t ´H ď τ ď t is defined by the variance of the regression residual as (6.3.1) where p α H I c and p β H iI c are defined in Equations (6.1.2) and (6.1.3) for the RSS formula of kernel predictor. It is easy to find sample partial variance p σ 2 i|I c is a special case of RSSpi|I c q with λ " 0 and the kernel defined totally as k T in Definition 6.1.4. Thus by the same procedure as Section 6.1, we relate ř iPI p σ 2 i|I c to the criteria of the sensor selection problem and derive Algorithm 8 from its greedy adaptation. q ´1 (block matrix inversion formula). end for Output: i 1 , i 2 , . . . , i N .

σ 2 i|I c " min Θ i PIR N
Detrend and scale The bike-sharing data set shows a significant periodic trend, thus in the preprocessing step we remove it from the raw series px it q t , i P N . We use the training set to estimate the trend, then use the estimate to detrend the whole data set. More specifically, let T denote the total hours of training set. Because the time series px it q t of data set is recorded on an hour basis, we estimate the weekly profile of data set as b m,i "

ÿ τ PIm x iτ |I m |
, m " 0, 1, ..., 167, where I m " tτ " 1, . . . , T : τ mod 168 " mu. We detrend as x it ´bm,i , t mod 168 " m. Additionally, we divide each detrended series by its standard deviation. Figure 6.2 gives an example of a comparison between the original and pre-processed series. Others H is given as 3 for the kernel and the linear methods. The regularization parameter for kernel method is set as 0.1λ max , where λ max is the largest eigenvalue of the Gram matrix K H N or p α H N .

Results

We have already shown in Section 2.2.2.2 the scores of node predictability learned through the dropout scheme. Since the scheme carries randomness, in Figure 6. We can see that the results do not change much across different runs, especially the nodes endowed the highest (in blue, yellow, and white) and lowest predictability (in dark purple) are very close. On the other hand, the curves in Figure 6.4 validate the designed training can converge. Both points support that the proposed learning method is valid and effective.

In Figure 6.5, we compare the derived node scores with the sampling probability of nodes proposed in [START_REF] Puy | Random sampling of bandlimited signals on graphs[END_REF]. In all the figures of visualization, we use the same colors to indicate the comparable concepts. The nodes with higher scores and the nodes with lower sampling probability will both be removed from the observed set (on average for the latter since it is not a deterministic method). We can see that both strategies suggest to keep the nodes from suburb in the graph. From the graph point of view, the suburb nodes have less near neighbours in the graph. For the aspect of data, the suburb stations have less chance to be visited by users, thus their observations are noisier and harder to be reconstructed. Therefore, it is reasonable to remain their observations in the reconstruction. The top 1, 10, and 27 probabilities are in blue, yellow, and white respectively. Figure 6.6 represents the rankings of predictability evaluated by the kernel predictor in Section 6.1 and the partial variance defined in Section 6.3. We can see that the two rankings show similar patterns. Especially, they both endow high predictability to stations in the transport towards La defense (upper left corner of the map). Besides, the kernel predictor indicates again the group of well-predictable stations around the train station Paris Montparnasse. Whereas, the partial variance suggests more predictable stations in the suburb areas.

Next, we evaluate the application of the predictability ranking to the sensor selection problem. We compare in Figure 6.7 the different methods their reconstruction errors of the selected set I. The reconstruction errors are measured by ℓ 2 norm for an increasing size of I. For the random sampling method, the underlying reconstruction method is given in (Puy et al., 2018, Equation (5)). We only report the results from the bandlimit k " 10 which has the best reconstruction performance. We can see that the data-driven strategies bring better reconstruction performance for all tested sampling rates. Lastly, we show in Figure 6.8 the representative reconstruction of highest rank node signals along time using the three prediction methods. The Chapter 7

Conclusion and perspectives

In this thesis, we provided the new tools to analyze the data of diverse natures recorded over a network of sensors. These tools are especially designed for a purpose of revealing the information which is not evident to the elementary data analysis, but valuable for further analysis, prediction, and decision making.

Chapter 4 and 5 focus on identifying the links between the evolution of time series observed from different nodes in a network. We proposed the novel auto-regressive models to describe the data. Therefore, by estimating the model coefficients, we were able to infer the links.

In Chapter 4, we were concerned by the more practically involved issue, which is the online inference from the raw data. We proposed the data model for the matrixvariate time series with periodic trends, and introduce two learning frameworks respectively in both low-and high-dimensional regimes. Especially in the high dimensional learning, we introduce the novel Lasso type (4.2.4) and extend the classical Homotopy algorithms. Lasso (4.2.4) differs from the classical Lasso (4.2.5) not only by replacing the coefficient vector with the coefficient matrix A, but mainly by requiring A to have the desired structure indicated by K G and the partial sparsity penalized by }A N } ℓ 1 . Thus the derivation of the Homotopy algorithms provides useful techniques to address the structure constraint. Moreover, this derivation does not rely on the specific structure, nor on the particular partial sparsity regularization. Therefore, they can be applied to other model designs. Other model extensions are possible, for example, from matrix-variate to tensor-variate time series by using the multi-way Kronecker sum notion, or considering more time lag terms in the matrix-variate AR model, and accordingly replacing Lasso penalty with group Lasso penalty in (4.2.4).

In Chapter 5, we extended the standard VAR models to the distributional multivariate AR models. The proposed model provides a way to model a collection of multiple time-dependent probability measures, and to represent their dependency structure by a directed weighted graph at the same time. Especially, the proposed data centering method for random measures allows the development of auto-regressive and regressive models with multiple predictors. Moreover, the empirical studies on the real data sets demonstrate that, the proposed approaches equipped with the distributional data representation are the efficient tools for analyzing and understanding the spatial-temporal data of distributional nature. For future research directions, this paper provides a class of multivariate AR models for distributional time series which favors the graph learning. More classes which suit different data analysis purposes are expected to explore.

Lastly, in Chapter 6, we are interested in understanding the predictability of the data observed on different nodes in a network. We proposed the approaches to evaluate moreover rank the predictability of nodes with respect to the linear, kernel, and neural network predictors. The derived rankings allow to understand furthermore the spatial distribution of such node predictability, as illustrated in the numeric experiments. Additionally, the rankings can serve as data-driven strategies for sensor selection. The presence of historical data has significantly improved the reconstruction performance. In particular, the sensor selection based on the neural networks as a reconstruction method is innovative, which is far from the existing approaches.

Mid-term: Object data analysis. Except developing extension and variants for the models proposed in this thesis, a more general direction introduced by the works in Chapters 4 and 5 is the object data analysis, which is an emerging domain gaining popularity in recent statistics. The object data analysis deals with the statistical learning of the data, where a data point is not longer a scalar or a vector, instead, is an object, such as matrix or probability measure from our works. Other examples include data on the surface of spheres [START_REF] Di Marzio | Nonparametric regression for spherical data[END_REF][START_REF] Zhu | Spherical autoregressive models, with application to distributional and compositional time series[END_REF], and phylogenetic trees [START_REF] Billera | Geometry of the space of phylogenetic trees[END_REF].

The challenge of analysing such data of complex type is that data points do not lie in a vector space, thus Euclidean methods will fail as mentioned in Chapter 5. However, the work there also demonstrates an effective treatment is view the data points as random objects in some metric space, such as Wasserstein space for distributional data. This treatment is also considered by other recent works in the domain of object data analysis. This is because of the widely availability of the notion of distance. Meanwhile, modeling in a metric space has its own advantages. More specifically, the weighted expectation and weighted integral can be easily established in a general metric space, following the idea of Fréchet mean. On noting that the weighted expectation Erwpxqys and weighted integral ş wptqf ptqdt in Euclidean setting minimize respectively the distance functionals This implies that all the classical notions which can be identified as the weighted expectation and weighted integral can be generalized immediately to metric space, leading to the extended models. Examples in literature include the extension of Nadaraya-Watson estimator [START_REF] Hein | Robust nonparametric regression with metric-space valued output[END_REF], global/local linear regression [START_REF] Petersen | Functional data analysis for density functions by transformation to a hilbert space[END_REF], and the principal components in functional PCA [START_REF] Dubey | Functional models for time-varying random objects[END_REF]. So far, many tools of common tasks for object data are still undeveloped, such as very fundamentally regression (object-to-object), auto-regressive models, that are only developed for a very few special object cases. Thus developing more object-valued tools especially with the metric-space treatments will be a very promising direction of mid-term research.

Short-term: Graph-valued data analysis. As the object closely related to the works of this thesis, graph/network itself is an important subject to work on. A critical reason is that it is adopted to represent the brain functional connectivity. This motivates statisticians to perform analysis for populations of networks, and apply the developed tools to the related neuroscience research. For example [START_REF] Ginestet | Hypothesis testing for network data in functional neuroimaging[END_REF] proposed the one-, two-, k-sample tests to test if several populations of graphs present the same mean level (compared with a prespecified value in one sample case). They then applied the test to study the impact of genders/ages on the brain functional connectivity.

Associating each observation of graph with a Laplacian matrix, graph-valued data also live in a metric space consisting of graph Laplacians, endowed with some matrix metric. Thus the metric-space treatments can apply. Additionally, as a convex subset of the Euclidean space of general matrices, the Laplacian space has its own properties to use in modelling. These set the development of graph-valued models at a good starting position.

  Figure1.1: Relevés climatologiques mensuels des stations météorologiques de Californie. À gauche se trouve le réseau de stations météo de Californie. En haut à droite sont démontrée les observations scalaires sur une certaine station (capteur/noeud) i, où une valeur x it P IR de température moyenne est enregistrée à chaque instant t à i, conduisant à une série temporelle scalaire. En bas à droite sont illustrées les observations vectorielles sur une certaine station i, où un vecteur x it P IR 4 de température min/max/moy et de précipitation est enregistré par temps t à i, conduisant à 4 série temporelle scalaire.

Figure 1 . 2 :

 12 Figure 1.2: Enregistrements annuels des distributions d'âge des pays de l'Union Européenne. En haut se trouve le réseau des 27 pays de l'Union Européenne. En bas se trouvent les observations distribuées enregistrées à i " France au cours du temps. Une distribution d'âge µ it P Ppr0, 1sq est enregistrée à chaque instant t. En bas à gauche, nous visualisons la série chronologique résultante de distribution univariée avec une surface dans le système de coordonnées Âge ˆAnnée ˆFréquence relative. Les données brutes dans ce graphique consistent en 40 distributions annuelles. Nous les complétons avec des échantillons interpolés pour dessiner la surface. En bas à droite, nous montrons la projection de la série chronologique brute sur le plan Âge Fréquence relative. On voit que la population vieillit avec le temps.

Figure 1 . 4 :

 14 Figure 1.4: Graphe de la structure d'âge inféré. Une arête implique la similarité des structures d'âge entre les deux pays de 1996 à 2036 (projeté). L'orientation peut être comprise comme la propagation des changements de la structure. Une arête plus épaisse correspond à une relation plus étroite entre les pays liés, par rapport aux autres arêtes entrantes. La taille du cercle bleu autour d'un pays correspond à la similarité de ses structures d'âge entre deux années consécutives. Un cercle plus grand représente une évolution plus lente.

Figure 1 . 5 :

 15 Figure 1.5: Application du dropout à l'apprentissage de la prévisibilité. À gauche, l'étape d'optimisation sur le nième batch, où x " px i q iPN représente l'entrée et w pnq I est le vecteur de dropout masquant l'entrée sur les noeuds de I. I est rééchantillonné à chaque batch n. Après la convergence de l'entraînement, le réseau est testé sur l'ensemble de validation comme illustré à droite, dont l'erreur de reconstruction de chaque noeud donne son score.

Figure 1 . 6 :

 16 Figure 1.6: Scores de prévisibilité des noeuds pour le réseau de velibs à Paris. Les scores les plus élevés sont indiqués en rouge et les plus bas en violet. Les noeuds de top 1, 10 et 27 sont respectivement en bleu, jaune et blanc.

  Figure 2.1: Monthly climatological records of weather stations in California. On the left is the network of weather stations in California. On the upper right are demonstrated the scalar observations on a certain station (sensor/node) i, where a value x it P IR of average temperature is recorded at each time t at i, leading to a scalar time series. On the lower right are demonstrated the vectorial observations on a certain station i, where a vector x it P IR 4 of min/max/avg temperature and precipitation is recorded per time t at i, leading to 4 scalar time series.

Figure 2 . 2 :

 22 Figure2.2: Annual records of age distributions of EU countries. On the top is the network of the 27 countries in the European union. At the bottom are the distributional observations recorded at i " France along time. A distribution of age µ it P Ppr0, 1sq is recorded at each time t. On the lower left, we visualize the resulting univariate distributional time series with a surface in the coordinate system of Age ˆYear ˆRelative frequency. The raw data in this plot consist in 40 annual distributions. We complete them with interpolated samples to draw the surface. On the lower right, we show the projection of the raw time series onto the Age ˆRelative frequency plane. We can see that the population is aging along time.

Figure 2 . 4 :

 24 Figure 2.4: Inferred age structure graph. An edge implies the similarity of the age structures between the countries from 1996 to 2036 (projected). The direction can be understood as the propagation of changes in the structure. Thicker edge corresponds closer relationship between the linked countries, with respect to other incoming edges. The size of the blue circle around a country corresponds to the similarity of its age structures between two consecutive years. Bigger circle represents slower evolution.

Figure 2 . 5 :

 25 Figure2.5: Application of the dropout in predictability learning. On the left represents the the optimization step on the n-th batch, where x " px i q iPN represents the input and w pnq I

Figure 2

 2 Figure 2.6: Scores of node predictability for bike-sharing network in Paris. The higher scores are shown in red and the lower in purple. The top 1, 10, and 27 nodes are in blue, yellow, and white respectively.

  τ ´b ´Ax τ ´1qΣ ´1px τ ´b ´Ax τ ´1q J , τ ´b ´Ax τ ´1qpx τ ´b ´Ax τ ´1q J , (3.2.9) where Σ is the true noise variance given in Model (3.2.4). Through the straightforward calculation, the explicit forms of such minimizers are found to be p A " p Γp1q " p Γp0q ı ´1 , and p b " pI ´p Aqû,

Figure 3 . 1 :

 31 Figure 3.1: Training a network with dropout can be viewed as training a collection of network models which share weights. At test time, all neurons are turned on, leading to the aggregations of models. The final trained net without dropout outputs the aggregated prediction from these models.

  Subdifferential, first order optimality condition Section B.2, Apidopoulos (2019) Proximal operator, projection operator Section 1.1 and 1.2, Parikh and Boyd (2014) Accelerated proximal gradient descent Section 4.3, Parikh and Boyd (2014) Backtracking line search Section 9.2, Boyd et al. (2004) Strong convexity Section 4, Zhou (2018) Duality Section 5, Boyd et al. (2004) Chapter 4

Figure 4 . 1 :

 41 Figure 4.1: Matrices pU k q k as entry locators, which characterise the structure of K G .

  .3.2) where b m " b 0 m ´Ab 0 m´1 , m " t mod M, extending the relation (3.2.5) of intercept and mean to the time-variant mean. Note that b 0 ´1 denotes b 0 M ´1.

Figure 4

 4 Figure 4.2: Top is the stationary time series from Model (4.1.5) at one component, bottom is the above time series with added periodic trend (M = 12).

Figure 4 . 3 :Figure 4 . 4 :

 4344 Figure 4.3: Initial spatial graph estimates which start the online procedures. True A N (left), y A N,91 of the low-dimensional procedure (middle), and A N p20, 0.05q of the high-dimensional procedure (right) are represented by heatmaps. Simulation settings: N " 10, F " 4, number of model parameters = 571, significance level of χ 2 test in Corollary 4.2.4 " 0.1.

Figure 4 . 5 :Figure 4 . 6 :

 4546 Figure 4.5: Spatial graph estimated at the arrival of the 182-nd sample. True A N (left), yA N,182 of the low-dimensional procedure (middle), and A N p182, 0.0286q of the high-dimensional procedure (right) are represented by heatmaps. Simulation settings: N " 10, F " 4, number of model parameters = 571, significance level of χ 2 test " 0.1, η " 5 ˆ10 ´6.

Figure 4 . 7 :Figure 4 . 8 :

 4748 Figure 4.7: Trend of the first node, first feature, estimated at different times. Estimation at t " 182 (left), t " 273 (middle), t " 591 (right). Simulation settings: N " 10, F " 4, M " 12, number of model parameters = 571.

Figure 4 . 10 :

 410 Figure 4.10: Regularization parameter evolution. The red curves are the mean regularization parameter values, taken over 10 simulations each. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, t 0 " 20, λ 0 " 0.03.

  Figure4.9: Average one step prediction error. The red curves are the mean prediction error of the high-dimensional procedure, taken over 10 simulations each. The blue curve is the mean prediction error of the low-dimensional procedure, taken over the same 30 simulations. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, significance level of χ 2 test " 0.1, t 0 " 20, λ 0 " 0.03.

Figure 4

 4 Figure4.11: Running time of each online update. The red curves are the mean running time of the high-dimensional procedure, taken over 10 simulations each. The blue curve is the mean running time of the low-dimensional procedure, taken over the same 30 simulations. The shaded areas represent the corresponding one standard deviations. Other simulation settings: N " 20, F " 5, M " 12, number of model parameters = 1500, significance level of χ 2 test " 0.1, t 0 " 20, λ 0 " 0.03.

  Figure 4.12: Learning set for augmented model based approaches. Monthly records of minimal temperature, maximal temperature, mean temperature, and precipitation from Station USH00040693 over a period over 20 years.

Figure 4 . 13 :

 413 Figure 4.13: Updated spatial graph by the low-dimensional procedure at different times. t " 507 (left), t " 1015 (middle), and t " 1522 (right). Experiment settings: N " 27, F " 4, M " 12, number of model parameters = 1761, significance level of χ 2 test " 0.1, η " 10 ´5, t 0 " 20, λ 0 " 0.03. The row labels are the 6-digit Cooperative Observer Identification Number of the corresponding weather stations.

Figure 4

 4 Figure 4.14: Updated spatial graph by the high-dimensional procedure at different times. t " 507 (left), t " 1015 (middle), and t " 1522 (right). Experiment settings: N " 27, F " 4, M " 12, number of model parameters = 1761, significance level of χ 2 test " 0.1, η " 10 ´5, t 0 " 20, λ 0 " 0.03. The rows and columns correspond to the weather stations whose 6-digit Cooperative Observer Identification Number are given by the row labels.

Figure 4 . 15 :

 415 Figure 4.15: Regularization parameter evolution. Experiment settings: N " 27, F " 4, M " 12, number of model parameters = 1761, significance level of χ 2 test " 0.1, η " 10 ´5, t 0 " 20, λ 0 " 0.03.

  Figure 4.16: Average one step prediction error of raw time series. the low-dimensional procedure (top), and the high-dimensional procedure (bottom).

  of results in Section 4.2.2 and the CLT for p A t

  Figure5.1: The function g on the left is given by the natural cubic spline passing through the points p0, 0q, p0.2, 0.1q, p0.6, 0.2q, p1, 1q. On the right is one realization of 30 i.i.d. samples of the resulting ϵ g .

Figure 5 . 2 :Figure 5 . 3 :

 5253 Figure 5.2: Mean (left) and standard deviation (right) of RMSD for N " 10. The mean and the variance are calculated over 100 simulations along time T , every 10 time instants.

Figure 5 . 4 :

 54 Figure 5.4: Calculation time (in seconds) of pA with respect to the sample size T for N " 10. The calculation time counting starts from the computation of the empirical Fréchet means for Data transformation (5.2.3), and ends when the accelerated projected gradient descent of Problem (5.2.5) finishes for the last row i " N .

Figure 5 . 5 :

 55 Figure 5.5: Calculation time (in seconds) of pA with respect to the sample size T for N " 100.

Figure 5 . 6 :

 56 Figure 5.6: Inferred age structure graph. The non-zero coefficients A ij are represented by the weighted directed edges from node j to node i. Thicker arrow corresponds to larger weights. The blue circles around nodes represent the weights of self-loop.

KP

  I c p´Hq K I c p´H `1q ¨¨¨K I c p0q IR qˆq .

  sample auto-covariance matrices. Likewise, the extractions in the definition above are defined the same way as in Chapter 4.

Figure 6 . 2 :

 62 Figure 6.2: Data preprocessing: detrend and scale. Top is the original time series. Bottom is the processed time series.

  3, we show the scores obtained from another run. Their training curves are given in Figure6.4.

Figure 6 . 3 :Figure 6 . 4 :

 6364 Figure 6.3: Scores of node predictability for bike-sharing network in Paris. The higher scores are shown in darker red and the lower in darker purple. The top 1, 10, and 27 nodes are in blue, yellow, and white respectively. For the nodes with lowest scores (lower than ´2), we clipped their scores to ´2 in this visualization to make the color change of other nodes visible. The same measure for Figure 2.6.

Figure 6 . 5 :

 65 Figure 6.5: Variable density sampling distribution from Puy et al. (2018) for Paris network. Lower sampling probabilities (equivalent to higher scores) are shown in red. The top 1, 10, and 27 probabilities are in blue, yellow, and white respectively.

Figure 6 . 6 :

 66 Figure 6.6: Rankings by linear (top) and kernel method (bottom) for Paris network. The ranks of nodes are represented by color, which is consistent with Figures 6.3 and 6.5.

Figure 6

 6 Figure 6.7: Root mean squared error of graph signal reconstruction (normalized by the sampling rate). Sampling rate is given by 1 ´|I|{N . The sampling rate for the method in Puy et al. (2018) is the effective sampling rate which only considers the unique nodes. For the network predictor, due to the randomness of training, we repeat the evaluation procedure (from the dropout scheme to the predictor re-training) 10 times. The orange curve represents the mean RMSE, while the shaded area around represents the 1 standard deviation.

Figure 6 . 8 :

 68 Figure 6.8: Signal reconstruction of the respective highest rank nodes, from linear (top), kernel (middle), and network predictors (bottom). The signals are predicted by all nodes after rank 13 (in a context of removing 5% nodes). The trends are added back to the predictions.

  and the weighted integral in a general metric space pX , dq can be defined by the minimizers of the analogue functionals in terms of d: arg min CPX ErwpxqdrC, Y s 2 s, and ż ' wptqF ptqdt :" arg min XPX ż wptqd rX, F ptqs 2 dt.

  Thus Condition C2 is not a uniform requirement imposed for t P Z. We recall firstly in Lemma 3.4.1 and Lemma 3.4.2 below, the stability results given inWu and Shao (2004, Theorem 2) andZhu and Müller (2021, Theorem 1).

.4.2) Note that ϵ t , t P Z are i.i.d, thus for any fixed X P X , we have r Φ t,m pXq d " r Φ t 1 ,m pXq, @t, t 1 P Z. Lemma 3.4.1. (Wu and Shao (2004, Theorem 2); Zhu and Müller (2021, Theorem 1)) Assuming that Conditions C1 and C2 hold, it follows that, for any fixed t P Z, lim mÑ8 r Φ t,m pX 0 q exists almost surely and is denoted by Ă X t . Moreover, for any fixed t, t 1 P Z, Ă X t d " Ă X t 1 . We denote this time-invariant marginal distribution as π. Lemma 3.4.2. (Wu and Shao (2004, Theorem 2); Zhu and Müller (2021, Theorem 1)) Assuming that Conditions C1 and C2 hold, then the limits in Lemma 3.4.1 do not depend on the departure point X 0 , that is, for any fixed X P X and any fixed t P Z, r Φ t,m pXq mÑ8 ÝÑ Ă X t , in d, almost surely.

  The test statistic is given by

		λ W,t " t αJ t	" Σ W,t p	ı ´1 αt ,	(4.2.3)
	where αt P IR P :" ˆ¨¨¨,	" A N,t y	ı i k ,j k

, ¨¨¨˙J, and p Σ W,t P IR P ˆP is defined as

  Note that x ´1,t denotes x M ´1,t . It is also straightforward to understand the new auto-covariance estimators. Each S m pA, b m q leads to an OLS problem of regression equation (4.3.2). Its minimization introduces two sample covariance matrices. The weighted average of all such sample auto-covariance matrices for m " 0, . . . , M ´1 is the new sample auto-covariance for Model (4.3.1). p m,t denotes the number of times that the samples from the m-th state point in the period have been predicted in the sense of Equation (4.3.3). As t grows, x m,t becomes xm,t quickly, and p m,t becomes

	with	p Γ t p0q "	M ´1 ÿ m"0	p m,t t	¨ř τ PIm,t	x τ ´1x J τ ´1 p m,t	´xm´1,t x J m´1,t ‹ ',
		p Γ t p1q "	M ´1 ÿ m"0	p m,t t	¨ř τ PIm,t p m,t x τ x J τ ´1	´x m,t x J m´1,t ‹ ',
		p m,t " |I m,t |, xm,t "	ÿ τ PIm,t	x τ p m,t	, m " 0, ..., M ´1,
		x m´1,t "	ÿ τ PIm,t	x τ ´1 p m,t	, m " 0, ..., M ´1.
								M ´1
								ÿ	ÿ
						arg min	zJ τ zτ .
						A,bm	m"0	τ PIm,t
								(4.3.4)

t " xm,t ´q A t x m´1,t ñ p b 0 m,t " x m,t por xm,t q, t M .

For the augmented model, GLS and OLS estimators are still identical, with the latter defined as

  N F ˆN F , we have N p0, vecpΛ 1 q J Σ G vecpΛ 1 qq. 92 4. Online graph learning from matrix-variate time series Use Cramér-Wold theorem again, we can get the theorem result. The distribution in this theorem is degenerate. ■ Proof of Corollary 4.2.4. The proof is an adaption of

	xΛ 1 ,	? t ´p A t	´A¯y	" x	ÿ	xΛ 1 , U k yU k ,	?	t ´q A t	´A¯y .
									kPK
	Let Λ in Equation (4.5.1) be	ř	kPK xΛ 1 , U k yU k , then
	xΛ 1 ,	? t ´p A t	´A¯y d Ý Ñ

  Csvecp y A N,t q " αt , and psvecpE k qq kPK N are orthonormal basis in IR

							4.5. Appendix
	Plug x t " b 0 t	`x1 t in the last equation above, we can get the formula only with respect
	with x 1 t
						p Γ t p0q " p Γ t p0q 1 ´M´1 ÿ m"0	p m,t t	`x1 m´1,t rx 1 m´1,t s J ˘,
	where x 1 m´1,t "	ř τ PIm,t	x 1 τ ´1 pm,t , m " 0, ..., M ´1, and p Γ t p0q 1 :"	t ř τ "1	x 1 τ ´1rx 1 τ ´1s J t	. Note
	that x 1 ´1,t " x 1 M ´1,t .
	Similarly, denote	t ř τ "1	x 1 τ rx 1 τ ´1s J t	by p Γ t p1q 1 , we have
						p Γ t p1q " p Γ t p1q 1 ´M´1 ÿ m"0	p m,t t	`x 1 m,t rx 1 m´1,t s J ˘,
	with x1 m,t "	ř τ PIm,t	x 1 τ pm,t , m " 0, ..., M ´1. Since px 1 t q t is the causal solution of VAR
	(4.1.4), we have
	(a') p Γ t p0q 1 p Ý Ñ Γp0q, p Γ t p1q 1 p Ý Ñ Γp1q,
	(b') p Γ t p1q 1	" Γ t p0q 1 p	ı ´1 p Ý Ñ Γp1q rΓp0qs ´1 " A,
	(c')	? t vec ˆp Γ t p1q 1	" Γ t p0q 1 p	ı ´1	´A˙d Ý Ñ N p0, rΓp0qs ´1 b Σ.
	Thus, to reach the results in Proposition 4.3.1, we need additionally the asymptotic
	properties of sample mean x1 m,t , which are given in Lemma 4.5.2.
	Lemma 4.5.2. (CLT of x1 m,t )
					? p m,t	m,t x1	d
							N pN ´1q 2	,
	thus we have C p Σ N,t C J " p Σ W,t .
							■
	4.5.2 Proof of Proposition 4.3.1
	From Definition (4.3.4), we have
				p Γ t p0q "	M ´1 ÿ m"0	p m,t t	¨ř τ PIm,t	x τ ´1x J τ ´1 p m,t	´xm´1,t x J m´1,t ‹ '
							t
							"	ř τ "1	x τ ´1x J τ ´1 t	´M´1 ÿ m"0	p m,t t `xm´1,t x J m´1,t	˘.
							93

  , we get the result. Since asymptotically, xm,t " x m,t , thus both means can be used to estimate b 0 m . On the other hand, based on (a), using continuous mapping theorem on the matrix inverse, we

	have q A t " p Γ t p1q	" Γ t p0q p	ı ´1 p Ý Ñ A.
	(c) When t Ñ 8, q A t equals
	« Γ t p1q 1 ´M´1 ÿ p m"0	p m,t t	`x 1 m,t rx 1 m´1,t s J ˘ff « Γ t p0q 1 ´M´1 ÿ p m"0	p m,t t	`x 1 m´1,t rx 1 m´1,t s J ˘ff´1	.
							■
	Proof of Proposition 4.3.1.
	(a) When t Ñ 8, p Γ t p0q " p Γ t p0q 1 ´M´1 ř m"0	1 M `x 1 m,t rx 1 m,t s J ˘p Ý Ñ Γp0q ´0 " Γp0q, and
	p Γ t p1q " p Γ t p1q 1 ´M´1 ř m"0	1 M `x 1 m,t rx 1 m´1,t s J ˘p Ý Ñ Γp1q, with x1 ´1,t :" x1 M ´1,t .
	(b) xm,t "	ř τ PI m,t b 0 m pm,t	`x1 τ	" b 0 m `x 1 m,t	p Ý Ñ b 0

m , @m " 0, ..., M ´1.

  2, we have the asymptotic result:

	directly from	" Γ t p0q p ı ´1 and p Σ t , to compose p Σ W,t as:
				@m,	p m,t ? t	`x 1 m,t rx 1 m´1,t s J ˘" 1 ? M	`?p m,t	x1 m,t ˘rx 1 m´1,t s J p Ý Ñ 0.
	Thus,	?	t	M ´1 ř m"0	pm,t t `x 1 m,t rx 1 m´1,t s J ˘"p Γ t p0q 1	ı ´1 p Ý Ñ 0.
	Similarly,	? t	M ´1 ř m"0	pm,t t `x 1 m´1,t rx 1 m´1,t s J ˘"p Γ t p0q 1	ı ´1 p Ý Ñ 0. Since, it is obvious that
	M ´1 ř m"0 and continuous mapping theorem, we can show the reminder terms all converge to 0 pm,t t `x 1 m´1,t rx 1 Ñ 0, then use the properties of convergence in probability m´1,t s J ˘p Ý
	in probability.
							■
							95

  Note that, since the empirical Fréchet means of all p µ i,t are also uniform, similarly to the population setting, we do not need to consider the additional terms in the estimation problem to cancel out the unequal empirical Fréchet means. Analogously,

			p A o " p Γp1q	" Γp0q p ı ´1 ,	(5.2.4)
	where	r p Γp0qs j,l "	1 T	T t"1 ÿ x p F ´1 j,t´1 ´id , p F ´1 l,t´1 ´idy Leb
	and	r p Γp1qs j,l "	1 T	T t"1 ÿ	x p F ´1 j,t ´id , p F ´1 l,t´1 ´idy Leb .

N,

  Let β ˚P IR n be some constant of interest. Assume βo is an estimator, defined as:βo " arg min

	Lemma 5.2.2. Under the conditions of Lemma 5.2.1, we have
	1 T	T ÿ t"1	r F i,t´1 ppq ´p " O p p	1 ? T	q, @p P p0, 1q, i " 1, . . . N.	(5.2.6)
	Since r F i,t " F ´1 i,t a F ´1 i,' , we have equivalently,
	F ´1 μi ppq ´F ´1 i,' ppq " O p p	1 ? T	q, @p P p0, 1q, i " 1, . . . N.	(5.2.7)
	Theorem 5.2.3. Under the conditions of Lemma 5.2.1
			p A o ´r A	p Ñ 0,
	which implies		p A o	´A p Ñ 0.
	Theorem 5.2.4. βPIR n
			2.1) that uses the knowledge of the population
	Fréchet mean F ´1 i,' . Secondly, we show F ´1 μi	p Ñ F ´1 i,' element-wise (see Lemma 5.2.2),
	and aim to rely on this result to prove that p A o ´r A	p Ñ 0 (see Theorem 5.2.3). Thirdly,
	we show a general result (see Theorem 5.2.4 below) on the consistency of constrained
	estimators that is not restricted to the proposed estimator in this work. Finally, we
	apply this result to p A o to derive the consistency of the proposed estimator p A.
	Lemma 5.2.1. Assume that µ i t , i " 1, . . . , N satisfy Assumption A3 for t " 0, 1, . . . , T . Assume also that the transformed sequence r F ´1 t , t " 0, 1, . . . , T sat-
	isfies Model (5.1.8) with Assumption A5 true. Suppose additionally that r F ´1 0	d " π
	with π the stationary distribution defined in Theorem 5.1.2. Given Assumptions A6
	and A7 hold true, we obtain
			r A ´A " O p p	1 ? T	q.

  6.1.1) where pΘ iλ denotes K H iI c pK H I c `λI q q ´1,

  ´p E}x it ´Θi x H I c t } 2 l 2 " σ i ´βH iI c pα H I c q ´1pβ H iI c q J I c p´Hq Γ I c p´H `1q ¨¨¨Γ I c p0q IR qˆq , Γ I c phq " rΓphqs I c , β H iI c " `ΓiI c p0q Γ iI c p1q ¨¨¨Γ iI c pHq ˘P IR pˆq , Γ iI c phq " rΓphqs iI c . In practice, σ 2i|I c can be estimated by the sample auto-covariance matrices asp σ 2 i|I c " p σ i ´p β H iI c p p α H I c q ´1p p β H iI c q J ,

	where
	α H I c "	¨ΓI c p0q Γ I c p´1q . . .	Γ I c p1q Γ I c p0q . . .	¨¨¨Γ I c pHq ¨¨¨Γ I c pH ´1q . . . . . .	' ‹ ‹ ‹	P

Γ

  Algorithm 8 Ranking of predictability measured by partial variance.

	Input: p p Γpjqq H j"0 .
	Initialize: n " 1, I c 0 " N . Compute p p α H I c 0 for n ă N do	q ´1.
	for i P I c n´1 do S i Ð I c n´1 zi. Compute p p α H S q ´1 from p p α H I c n´1	q ´1 (block matrix inversion formula).
	Compute p σ 2 i|S i as Equation (6.3.1). end for
	i n Ð arg min iPI c n´1 p σ 2 i|S i . I c n Ð I c n´1 zi n . Compute p p α H I c n q ´1 from p p α H I c n´1

* The value can also be complex. However, we focus on real observations for all the cases throughout this thesis.

† For a node, its neighbours are in the 1-hop neighbourhood of the node. All the neighbours of its neighbours are in the 2-hop neighbourhood of the node, so far and so forth.

¶ The modulo of a negative integer is defined by the positive reminder in this case, for example, ´1 mod M " M ´1.

* We only demonstrate Wald test with VAR(1) model, as the preliminaries of the present work. For the test of VAR(p) model, see the cited book.

Proposition 3.3.3. Let γ P W be any absolutely continuous measure. Then Exp γ | Log γ W is an isometric homeomorphism from Log γ W to W, with the inverse map Log γ , satisfying d W pµ, νq " } Log γ µ ´Log γ ν} γ .

† This part of the thesis is from the lecture note[START_REF] Bartlett | Lecture notes in statistical learning theory[END_REF]. ‡ There is an equivalent definition of RKHS, which deos not rely on k: a RKHS is a Hilbert space of maps from X to Hilbert space Y if its evaluation operator is linear continuous for every x P X .

* On the other hand, this implies that Γ0 will quickly become non-singular from the initial time, as new samples xτ come in.† More general, a new sample will cause a rank-N F update in the corresponding matrix INF b p Γtp0q in Lasso (2.2.2).

‡ This hypothesis means that, some zero pa s 1 q ipkq , k P K 1 N should not satisfy the first equations of the optimality condition (4.2.6) and (4.2.7), due to the computation coincidence.

§ The data set is available at https://www.ncdc.noaa.gov/ushcn/data-access

´1ıI k ,J k 1, with order indices I k :" ti k , i k `F, ..., i k `pN ´1qF u,I k 1 :" ti k 1 , i k 1 `F, ..., i k 1 `pN ´1qF u, J k :" tj k , j k `F, ..., j k `pN ´1qF u, J k 1 :" tj k 1 , j k 1 `F, ..., j k 1 `pN ´1qF u.4.5.4 Extended algorithm 2 for the augmented modelAlgorithm 3Input:x t`1 , x t , p Γ t p0q, p Γ t p1q, r p Γ t p0qs ´1, m, t, tp m,t u M ´1 m"0 , tx m,t u M ´1 m"0 . Update p Γ t`1p0q, p Γ t`1 p1q from Equation (4.3.5).

p m,t p m,t`1 x m´1,t `1 p m,t`1 x t , and x m,t`1 Ð x m,t , @m ‰ m ´1. p m,t`1 Ð p m,t `1, and p m,t`1 Ð p m,t , @m ‰ m,t Ð t `1. Output: p A t`1 , p Γ t`1 p0q, p Γ t`1 p1q, r p Γ t`1 p0qs ´1, t, tp m,t u M ´1 m"0 , tx m,t u M ´1 m"0 .

* The data base in open access through https://www.census.gov/data/developers/data-sets/ international-database.html.

P IR qˆq , p Γ I c phq " r p Γphqs I c , (6.1.2) p β H iI c " ´p Γ iI c p0q p Γ iI c p1q ¨¨¨p Γ II c pHq ¯P IR pˆq , p Γ iI c phq " r p Γphqs iI c , (6.1.3)

Acknowledgements

high-dimensional regimes. In the following section, we firstly introduce the tools on constraint set K G , which is crucial to derive the proposed frameworks.

4.2.1 Orthonormal basis and projection operator of K G K G defined as Equation (4.1.1) is a linear space of dimension N F `1 2 F pF 1q `1 2 N pN ´1q. We now endow K G with the Frobenius matrix inner product, that is xB, Cy F " trpB J Cq. An orthogonal basis of K G is then given in the following Lemma.

Lemma 4.2.1. The set of matrices U k , k P K :" t1, . . . , N F `1 2 F pF ´1q `1 2 N pN 1qu, defined below form an orthogonal basis of

where, when k P K D , E k P IR N F ˆN F , with rE k s i,j " 1, if i " j " k, otherwise 0,

when k P K N , E k P IR N ˆN is almost a zero matrix except

In Figure 4.1, we give an example of this orthogonal basis of K G for N " 3, F " 2, where U k are visualized with respect to their non-zero entries. We can find that each U k relates to one variable of diagpM q, M F and M N , and characterises how it contributes to the structure of M by repeating at multiple entries. Thus, taking the inner product with U k actually calculates the average value of an arbitrary matrix over these entries. This is important to understand how to project an arbitrary matrix onto K G .

It is easy to verify that xU k , U k 1 y F " 0 for any k ‰ k 1 in K, and pU k q k spans K G . Thus the normalized matrices U k {}U k } F , k P K forms an orthonormal basis of K G . We introduce the orthogonal projection onto K G and provide an explicit formula to calculate it using pU k {}U k } F q k in Proposition 4.2.2. Proposition 4.2.2. For a matrix A P IR N F ˆN F , its orthogonal projection onto K G is defined by Proj G pBq " arg min

where a Gt 1 P IR |K 1 | is defined as ´aGt 1 ¯i " xU k , G t y, k i P K 1 , with K 1 , w 1 ,

associated with Apt, λ t q, and G t " pApt, λ t qx t ´xt`1 q x J t .

The derivatives of the entries of Apt, λq indexed by K 1 at λ t can be calculated through the formula (4.2.12) of a s 1 . By contrast, the derivatives of the entries of Apt, λq indexed by K 0 N all equal zero. To obtain the non-negative parameter value, we project λ t`1 onto interval r0, `8q by taking maxtλ t`1 , 0u, whenever the result from Equation (4.2.3.4) is negative.

Note that λ t`1 defined in Equation (4.2.3.4) can be interpreted as the online solution from the projected stochastic gradient descent derived for the batch problem

Therefore, the sublinear regret property of projected stochastic gradient descent implies that, when η is given as Op 1

? n q, we have

Equation (4.2.3.4) implies that in the sense of average one step prediction error defined as Equation (4.2.3.4), the adaptive hyperparameter sequence tλ t u t will perform almost as well as the best parameter λ n, for a large number of online updates, with sufficiently small step size η. This completes the online procedure in the high dimensional domain, which we conclude in Algorithm 1.

Algorithm 1 Online Structured matrix-variate Lasso

‰ ´1 are associated with Apt, λ t q, and w 1 N " rws K 1 N . Select λ t`1 according to Section 4.2.3.4. Update Apt, λ t q Ñ Apt, t`1 t λ t`1 q using Algorithm 4. Update Apt, t`1 t λ t`1 q Ñ Apt `1, λ t`1 q using Algorithm 5.

Augmented model for periodic trends

The online methods derived previously are based on the data process (4.1.4), which assumes the samples px τ q τ PN have the time-invariant mean zero. In this section, we propose a more realistic data model which considers the trends, and adapt the online methods for stationary data to this augmented model.

A common detrend strategy in offline learning is subtracting the sample mean from observations of all time points, that is, given a raw time series x 0 , x 1 , ..., x t , we

Note that, since multiplication with vecpU h k q amounts to extracting elements in the matrix from the corresponding locations, in practice, we take the elements 4.5.5 Homotopy algorithm for regularization path Apt, λ 1 q to Apt, λ 2 q Algorithm 4

N , where `is the ordered append of two lists. #Computing the regularization path (the steps in parentheses are the modifications for the case λ 1 ą λ 2 ):

Remove the i-th row together with the i-th column from Γ 1 0 , and use Sherman Morrison formula to update

else if rw 0 s i reaches 1 for some

Append 1 to the end of sign vector w 1 N . Append row rΓ 0 s k,K 1 , column rΓ 0 s K 1 ,k after the last row and last column Γ 1 0 , respectively, and use Sherman Morrison formula to update

else if rw 0 s k reaches ´1 for some

Append ´1 to the end of sign vector w 1 N . Append row rΓ 0 s k,K 1 , column rΓ 0 s K 1 ,k after the last row and last column Γ 1 0 , respectively, and use Sherman Morrison formula to update

end if else λ " λ 2 . end if end while Compute a s 1 , using Equation (4.2.12) and the last updated

4.5.6 Homotopy algorithm for data path Apt, t`1 t λq to Apt `1, λq Algorithm 5

‰ ´1 are associated with Apt, t`1 t λq, and

N , where `is the ordered append of two lists. Generate Γ 0 0 , γ 1 1 , γ 0 1 , w 1 , based on Proposition 4.2.6.

7:

a s 1 "

e " x t`1,i ´pr r

12:

if µ 1 " H, µ 1 Ð `8.

13:

if µ 1 ă 1 then 14:

µ " µ 1 .

15:

if µ 1 is some µ 0 k then 16:

17:

Remove rw 1 N s i´|K D |´|K F | from w 1 N .

18:

Remove the i-th row, the i-th column from Γ 1 0 , use Sherman Morrison formula to update " Γ 1 0 ‰ ´1.

19:

else if µ 1 is some µ k (or µ ḱ ) then 20:

21:

Append 1 (or ´1) to the end of sign vector w 1 N .

22:

Append row rΓ 0 s k,K 1 , column rΓ 0 s K 1 ,k after the last row and last column Γ 1 0 , respectively, and use Sherman Morrison formula to update

23:

end if

24:

else 25:

µ " 1.

26:

end if

27:

end while 28:

4.5. Appendix

Online graph and trend learning from matrix-variate time series in high-dimensional regime

Algorithm 6

Input:

´1 are associated with Apt, λ t q. Select λ t`1 according to the end of Section 4.3.2. Update Apt, λ t q Ñ Apt, t`1 t λ t`1 q using algorithm 4.

x m´1,t , and center r X t Ð r X t ´r X m´1,t . Update Apt, t`1 t λ t`1 q Ñ Apt `1, λ t`1 q using algorithm 5, with modifications: Line 8 change to α " r r X t s J K 1 ,i u `p m,t , Line 28, 29 change respectively to:

Existence, uniqueness and stationarity

To study the legitimacy of the iterated random functions (IRF) system defined by Model (5.1.5), we shall consider the product metric space pX , dq :" pT , } ¨}Leb q bN , where T :" Log Leb W `id is the space of all quantile functions of W, equipped with the norm } ¨}Leb of the tangent space at Lebesgue measure. Thus, we have dpX, Yq :"

The auto-regressive model (5.1.5) can be interpreted as an iterated random functions (IRF) system operating on the state space pX , dq, written as

where X t " pX i,t q N i"1 , ϵ t " pϵ i,t q N i"1 , and Φ ϵt pX t´1 q " pΦ i ϵt pX t´1 qq N i"1 with

We first study the existence and the uniqueness of the solution to the IRF system in the metric space pX , dq.

For time series models in a Hilbert space, two standard assumptions that ensure the existence and the uniqueness of the solutions are the boundedness of the L p norm of random additive noise and the contraction of the regression operator. For Model (5.1.8), the random noise ϵ i,t is bounded between 0 and 1, and thus E rd p pX, ϵqs is bounded for all X P X , which is the L p norm equivalent in the metric space setting. Then, to have a contractive map Φ ϵt , we shall rely on an interplay between properties of the matrix A of coefficients and the random noise distortion since it is applied in a nonlinear way. More specifically, we impose Assumptions A6 and A7 below on Model (5.1.8). Assumption A6. E rϵ i,t pxq ´ϵi,t pyqs 2 ď L 2 px ´yq 2 , @x, y P r0, 1s, t P Z, i " 1, . . . , N , Assumption A7. }A} 2 ă 1 L . Note that, Assumption A6 implies that ϵ i,t is L-Lipschitz in expectation. For increasing functions from r0, 1s to r0, 1s, the smallest L is 1 that is attained by the identity function. Therefore, Assumption A7 implies that }A} 2 ă 1, which is the contraction assumption for VAR(1) model. We now state the existence and uniqueness results. The proofs for this section is given in the appendix.

Theorem 5.1.2. Under Assumptions A5, A6 and A7, the IRF system (5.1.8) almost surely admits a solution X t , t P Z, with the same marginal distribution π, namely, X t d " π, @ t P Z, where the notation d " means equality in distribution. Moreover, if there exists another solution S t , t P Z, then for all t P Z X t d " S t , almost surely.

Theorem 5.1.2 states that under Assumptions A6 and A7, a well-defined IRF system (5.1.8) (namely when Assumption A5 is satisfied) permits a unique solution in pX , dq almost surely. Next, we show that this solution is furthermore stationary as a functional time series in a Hilbert space. To this end, we need to assume that there is an underlying Hilbert space associated to pX , dq, with its inner product inducing d as the norm. Such an Hilbert space exists, with corresponding inner product given by xX, Y y "

Then, Theorem 5.1.3 below gives the stationarity result.

Theorem 5.1.3. The unique solution given in Theorem 5.1.2 is stationary as a random process in pX , x¨, ¨yq in the sense of Definition 3.4.7.

Besides, Proposition 5.1.4 below states that the stationary solution of the IRF system (5.1.8) satisfies the property (5.1.3) of the transformed series r F ´1 i,t , t P Z. Thus, it is consistent to propose the IRF system (5.1.8) as the process that generated the data r F ´1 i,t , which completes the building of Model (5.1.5) as valid approach to analyze multivariate distributional time series. Proposition 5.1.4. The stationary solution X t of the IRF system (5.1.8) satisfies:

1. X i,t ppq P r0, 1s, @p P p0, 1q, 2. E rX i,t ppqs " p, @p P p0, 1q.

Finally, we point out in Proposition 5.1.5 additional properties of the IRF system (5.1.8) that will serve in the following section of the estimation of coefficients in Model (5.1.5).

Proposition 5.1.5. The matrix A of coefficients in the stationary IRF system (5.1.8) admits the representation A " Γ H p1q rΓ H p0qs ´1 , (5.1.9)

where Γ H p0q, and Γ H p1q P IR N ˆN are defined as

Note that representation (5.1.9) of coefficient in the proposed model is analogous to the one for VAR(1) models given in Equation (3.2.7), with matrices Γ H p0q, Γ H p1q carrying out the information on the correlation. However, compared to the autocovariance operators in Definition 3.4.7, matrices Γ H p0q and Γ H p1q rather reflect the average auto-covariance taking into account additionally the correlated level along the function domain. To avoid the heavy notation in the estimators of matrices Γ H p0q and Γ H p1q, in the rest of this chapter, we abuse the notation by omitting the subscript H in these matrices.

synthesized "raw" data, as the inverse of transformation (5.1.2):

We set F ´1 i,' as the natural cubic spline of the points: p0, 0q, p0.2, 0.1q, p0.6, 0.2 0.2i{N q, p1, 1q, i " 1, ..., N . The empirical Fréchet mean F ´1 μi and the proposed estimator p

A are calculated on the synthesized "raw" data F ´1 i,t . In Section 5.3.1.2, we aim to demonstrate the consistency result given in Theorem 5.2.5 with the synthetic data.

Experiment settings and results

In this experiment, we demonstrate the consistency of the proposed estimator p A for two different values N " 10 and N " 100. For each N , we generate two true matrices A for α " 0.1 and 0.5 respectively, according to the procedure presented in Section 5.3.1.1. With each A, we calculate the root mean square deviation (RMSD) successively

with the synthetic data that it generates along time. To furthermore study the mean and the variance of the RMSD (5.3.2), we run 100 independent simulations for the same A.

Note that the value of p A we use in Equation (5.3.2) is the approximation obtained by the projected gradient descent applied to Problem (5.2.5). Thus the corresponding approximation error also accounts for the deviation which is on the order of the threshold we set in the stopping criteria. For all values of N , we use the same error threshold. We stop the algorithm as soon as the difference between the previous and the current updates in ℓ 2 norm reaches 0.0001, for the resolution of each row p A i: . We firstly show the evolution of RMSD for N " 10, 100 in Figures 5.2 and 5.3, respectively. We can see that all means and variances of the RMSD decrease towards zero as the sample size T increases, for each N and α value. This demonstrates empirically that, when the model assumptions A5, A6 and A7 hold true for the data, the proposed estimator p A converges to A in probability, which is implied actually by the convergence of p A to A in L 2 . Additionally, we can notice that, the RMSD for α " 0.1 which corresponds to larger ℓ 2 norm of A has a smaller mean in both cases, and also a smaller variance for most of the sample sizes T investigated.

Also we would like to remark that, during the first few T values, the samples are insufficient for a meaningful estimation. Thus the projected gradient descent will terminate rapidly, as shown in the very beginning of Figures 5.4 and 5.5. The output p A will be a zero matrix, since we initialize p A as zero. This results in the low RMSD values during the early phase, since the true A is generated as a sparse matrix with small weights, as shown in Figures 5.2 and 5.3. As T increases, more entries of p A become non-zero, which brings to the growth of RMSD. Upon the arrival of more new samples, p A starts to converges, meanwhile the RMSD starts to decrease accordingly. example the age structures of France, Italy, in Figure 5.9. We can see that the age structures are as expected very different. All these observations strongly support the usefulness of our model. Lastly, in Table 5.1, we provide the first 5 edges of the largest weights.

Bike-sharing network in Paris

Secondly, we test our model on the bike-sharing data set of Paris. The original data set is available at: https://maxhalford.github.io/blog/a-short-introduction-andconclusion-to-the-openbikes-2016-challenge/. It contains the coordinates of all bike stations and their usage recordings. After data cleaning, we extract the variable unused bike ratio at the recording time for each station. To have the time series at the common time for all stations, we perform the linear interpolation with hourly observation frequency. The clean data set consists in the observations of unused bike ratio of 274 stations over 4417 consecutive hours. We are interested in the temporal evolution of the bike availability of stations, and would like to identify how the stations relate mutually in their evolution using the proposed method. To this end, we firstly represent the data by taking into account the distribution aspect. We From To 1 Estonia Latvia 2 Sweden Norway 3 Belgium Germany 4 Finland Netherlands 5 France Greece Table 5.1: Top 5 edges with the largest weights excluding all the self-loops introduce the temporal variable T which represents hours in a day. Accordingly, the distribution µ i t considered by Model (5.1.4) represents the distribution of the bike availability of the station i, at hour T in a day, with T " 1, ..., 24 and i " 1, ..., 274. Thus the quantile function F ´1 i,t of distribution µ i t can be retrieved from all the data points of station i recorded at hour T, by the numeric methods. We use the similar method as for the age distribution data to retrieve the valid quantile functions, however the implementation is different due to the availability of different quantity types (for details see function generate_qt_fun defined in script bike_net.py in the code related to this paper).

Note that since these observations are distanced from each other in time, they can be considered approximately as being independent samples. For more comments on this point, we refer to Remark 5.3.1. We then fit Model (5.1.5) on data F ´1 i,t , T " 1, ..., 24, i " 1, ..., 274. We use the same stopping criteria as previously, and we apply the granularity of 0.002. The complete execution time of model fitting takes around 20 minutes. We now demonstrate the visualization of the inferred matrix of coefficients A on the map of Paris, represented by the directed weighted graph. Since the edges of the complete graph will be densely located in the plot when fitting the graph to the paper size, for better visual effects, we show two subgraphs each of 50 nodes, in Figure 5.10 and Figure 5.11 respectively. The complete graph and the subgraphs are available in the interactive form at https://github.com/yiyej/Wasserstein_ Multivariate_Autoregressive_Model.

These figures show some interesting links between the bike utilisation in different areas of Paris. For example, we can notice in Figure 5.10 that the regression relationship between the station 41604-lagny-saint-mande and the stations along the flow from Saint-Mandé in southeast to Neuilly-sur-Seine in northwest are very significant. We recall the result of node predictability shown in Figure 2.6 and Chapter 6, which is also learned from this data set. This flow going through the main itinerary of Metro 1 and RER A in Paris corresponds to the first group of stations with high predictability. On the other hand, the second group is located at the bottom of Figure 5.10 around the train station Paris Montparnasse. We can see that, the result from this section also shows that the stations in this area have significant predictors, especially the bike stations around the other train station Gare du Nord, which is situated at the top of Figure 5.10. Note that, since Figure 5.10 shows only a subset of edges between the 50 randomly chosen nodes, the complete graph can have also re-found other patterns of the derived node ranks in Chapter 6.

Appendix

Additionally, we can see in Figure 5.11 that the stations near Charles de gaulleétoile (upper left) need to be predicted by numerous stations together, among which is the station 02012-quatre-septembre. By contrast, station 02012-quatre-septembre can only be predicted by itself with no edges pointed in. These observations all support the effectiveness of the proposed AR model and its estimation method.

Remark 5.3.1. Note that another way to represent the data set by distributions is to consider every K consecutive observations as the i.i.d. samples of a distribution. Thus T " 1, 2, ..., 185{K represents the ongoing time in hours. However, the consecutive observations are highly correlated. Moreover, the distribution retrieved in this manner does not clearly represent a random variable. Thus we do not go further with this data representation method. With pX , dq given in Definition (5.1.7), Φ ϵt defined in Equation (5.1.8), and Θ defined as the product space of N pΠ, }} Leb q with pΠ, }} Leb q defined in Equation (5.1.4), we show the resulting system (5.1.8) satisfies Conditions C1, C2 in Section 3.4.1.

Appendix

The verification of Condition C1 does not require the additional assumptions. We consider any X P pX , dq, α " 2, then

Since pX , dq is a space of bounded functions from p0, 1q to r0, 1s. Thus Epd 2 pX, Φ ϵ 1 pXqqq is bounded.

We are now to show that under Assumptions A6 and A7, IRF System (5.1.8) satisfies Condition C2. We first examine the case of m " 1, we consider any X, X 1 P X , then

Exchange (b) of the integral and the expectation comes from that the integrand is bounded. Then for any fixed p P p0, 1q,

By Assumption A7, L 2 }A} 2 2 ă 1, thus by taking r " L 2 }A} 2 2 , C " 1, Equation (3.4.2) is checked for m " 1. Now, suppose for any fixed m, we have

(5.4.1)

Then for m `1, we have

Inequality (c) comes from that r Φ t,m pXq d " r Φ t´1,m pXq, @t P Z and the hypothesis (5.4.1). Thus, by induction, Equation (3.4.2) is verified for all m ě 1, thus Condition C2 is checked by IRF System (5.1.8).

Appendix

By Theorems 3.4.4, 3.4.5, we have proved Theorem 5.1.2. Note that since X is the set of quantile functions from p0, 1q to r0, 1s, all the elements in X are bounded uniformly in the sense of Inequality (3.4.3) with d defined as (5.1.7). Thus Theorem 3.4.5 applies to all solution S t in pX, dq. Moreover, since α is taken as 2, by Theorem 3.4.8, we obtain furthermore the stationarity of the unique solution X t in the underlying Hilbert space pX, x, yq with xX, Y y " ř N i xX i , Y i y Leb . Lastly, applying Proposition 3.4.6, we have additionally that IRF system (5.1.8) is geometric moment contracting. ■ 5.4.2 Proof of Proposition 5.1.4

Since X t is a solution of IRF system (5.1.8) defined in T N , d, where T is a set of quantile functions over r0, 1s. Thus point 1 is checked.

For any p P p0, 1q, from the definition of system (5.1.8), we have

Take expectation on both sides, gives

A ij pEX j,t´1 ppq ´pq `p.

Equation (a) comes from that ϵ i,t is independent of X t´1 . Thus we have EX i,t ppq´p " ř N j"1 A ij pEX j,t´1 ppq ´pq , i " 1, . . . , N, p P p0, 1q. When X t , t P Z is a stationary solution, the equation becomes EX i,t ppq ´p " ř N j"1 A ij pEX j,t ppq ´pq , i " 1, . . . , N, p P p0, 1q. Let ⃗ x p " pEX 1,t ppq ´p, . . . , EX N,t ppq ´pq J , we then have ⃗ x p " A⃗ x p ðñ pI ´Aq⃗ x p " 0.

Since }A} 2 ă 1, I ´A is invertible, which implies ⃗ x p " 0 for any p P p0, 1q. Thus EX 1,t ppq " p, i " 1, . . . , N, p P p0, 1q. ■ 5.4.3 Proof of Proposition 5.1.5

Process (5.1.8) writes in terms of components X i,t as

Subtract id from the both sides, we obtain

Pass the both sides to the inner product with X l,t´1 ´id, then take the expectation, we get

A ij E xX j,t´1 ´id , X l,t´1 ´idy .

Compare the definitions of Γp0q, Γp1q, we can then retrieve Representation (5.1.9). ■

Proof of Lemma 5.2.1

Since r F ´1 0 follows the stationary distribution π defined in Theorem 3.4.4, the sequence r F ´1 t , t " 1, . . . T generated successively by the stationary model (5.1.8) follow π, furthermore, their auto-covariance is time-invariant and equal to the stationary solution X t , t P Z. Thus we can use the data r F ´1 t in Representation 5.1.5, which writes as A " Γp0q rΓp1qs ´1 , where rΓp0qs j,l " E x r F j,t´1 ´id , r F l,t´1 ´idy Leb and rΓp1qs j,l " E x r F j,t ´id , r F l,t´1 ídy Leb . Since r A " r Γp0q " r Γp1q ı ´1. We first show r r Γp0qs j,l ´rΓp0qs j,l " Op 1 ? T q by applying Theorem 3 in [START_REF] Wu | Limit theorems for iterated random functions[END_REF].

To this end, we define g jl : pX , x¨, ¨yq Ñ IR as g jl pXq " xX j ´id , X l ´idy Leb ´E xX j ´id , X l ´idy Leb .

The construction implies that for any random object Y " pY i q N i"1 in pX , x¨, ¨yq, Eg jl pYq " 0. Moreover, since all Y i are bounded function from p0, 1q to r0, 1s, |g jl pYq| p ă 8 for all p ą 2, which leads to E|g jl pYq| p ă 8 for all p ą 2. It is then left to show that g jl is stochastic dini-continuous (Wu and Shao, 2004, Equation (9)).

For any Y and Y 1 identically distributed in pX , x¨, ¨yq, we have

Proof of Lemma 5.2.2

To prove Equation (5.2.6), we define, for any fixed p P p0, 1q, i " 1, . . . N , g p i : pX , x¨, ¨yq Ñ IR defined as g p i pXq " X i ppq ´E X i ppq.

Follow the similar steps in the proof of Lemma 5.2.1, we have

Then plug the transformation relation in r F i,t´1 , we have

where pF ´1 i,' q ´1 is the left continuous inverse of the quantile function F ´1 i,' . By the statement under Assumption A3, F ´1 i,' is continuous, thus pF ´1 i,' q ´1 is strictly increasing. We therefore substitute q for pF ´1 i,' q ´1ppq, it follows p " F ´1 i,' pqq, which brings to Equation (5.2.7). ■

Proof of Theorem 5.2.3

We first show a convergence result in Lemma 5.4.1 which will be required by the demonstration of the theorem.

Lemma 5.4.1. Under the conditions of Lemma 5.2.1, we have

The proof of Lemma 5.2.2 follows the same steps in the one of Lemma 5.2.1, by considering g j : pX , x¨, ¨yq Ñ IR defined as g j pXq " xid , X j ´idy Leb ´E xid , X j ´idy Leb .

Then use E r F i,t ppq " p, @p P p0, 1q, we obtain the result. The Lagrangian of Problem (5.2.8) is given by

Let Λ ˚be a dual solution, then the strong duality implies the primal solution β minimizes Lpβ, Λ ˚q, with ř m i"1 λ i f i p βq `řp j"1 ν j h j p βq " 0, see Boyd et al. (2004, Section 5.5.2). Therefore we have

We subtract f T p βo q from both sides of the inequality above, which gives 0 ď f T p βq ´fT p βo q ď f T pβ ˚q ´fT p βo q `m ÿ i"1 λ i f i pβ ˚q `p ÿ j"1 ν j h j pβ ˚q. (5.4.4)

Note that the non-negativity comes from that βo is the minimizer of f T .

On the other hand, because f T is strongly convex with the constant µ T , thus for any s P Bf T p βo q, there is (Zhou, 2018, Lemma 3 (iii)),

(5.4.5)

Because βo is the minimizer of f T , thus 0 ď f T pβ ˚q ´fT p βo q. Moreover, 0 P f T p βo q, thus, we pick s " 0, Equation (5.4.5) becomes

Since }β ˚´β o } 2 ℓ 2 " O p pr T q, and 1 2µ T is stochastically bounded, thus f T pβ ˚q fT p βo q " O p pr T q. Furthermore, β ˚satisfies the constraints, thus ř m i"1 λ i f i pβ ˚q`ř p j"1 ν j h j pβ ˚q ď 0, combined with the previous convergence result in Inequality (5.4.4), we have f T p βq ´fT p βo q " O p pr T q.

Thus the computation of the gradient does not take into account the prediction errors of observations on I c . Because the missing set I n remains unchanged for all samples in batch B n , at each optimization step, the network is trained as a specific prediction model with input r x I c n t and target x Int . Sampling the dropout vectors then varies missing set I across the optimization steps in a training.

Therefore, by the same spirit of model aggregation, we propose to test the trained model on the validation set V without dropout. We input the complete graph signal x t to the network and obtain the output p x t . For each node i, we calculate the total prediction error on the validation set as R 2 ppp x it q tPV , px it q tPV q :" 1 ´řtPV px it ´x it q 2 ř tPV px it ´x i q 2 , (6.2.1)

where xi " ř tPV x it . Note that, we use R 2 score to measure the prediction error instead of the ℓ 2 norm as before. Because we observe in the experiment that the selected nodes by R 2 have better reconstruction performance. The prediction error (6.2.1) from the aggregated network represents the overall predictability of node i when it is predicted within different I's. We take this prediction error as the score of node, which derives furthermore the rank. The higher scores are endowed with the higher ranks. Figure 6.1 reviews the predictability learning by a network. We stop the training when the validation loss does not decrease with respect to the previous epoch. The key is the convergence of the validation loss. In the experiments, it shows when a small dropout rate (ď 0.2, |B n | " 100 for our data sets) is given, the validation loss will converge. By contrast, when the dropout rate is large, the loss is likely to become infinity in training, since the missing rate is relatively high for the reconstruction. Additionally, we use gradient descent as the optimization algorithm, because it does not use the previously computed gradients to update network parameters, so that each batch represents an independent model training.

Experiments

In this section, we evaluate the proposed methods on the bike-sharing data set in Chapter 5. Especially, we apply the inferred rankings to the sensor selection problem and compare the reconstruction performance of the selected sensors with the random sampling method in [START_REF] Puy | Random sampling of bandlimited signals on graphs[END_REF].

Settings

Training test split We split the whole data set into training and test sets by the ratio of 0.8 : 0.2. The predictability learning for all three proposed methods use the training set. When comparing the reconstruction performance of the selected set with the literature method, we construct the corresponding predictors still with the training set data. We only use the test set for reconstructing the unseen observations on the selected nodes in order to compare the method performance. For the network method, we take furthermore the last 0.1 history of training set as validation set.

Network and training For the network predictor, we take ChebNet [START_REF] Defferrard | Convolutional neural networks on graphs with fast localized spectral filtering[END_REF] as an example to illustrate the proposed method in Section 6.2. The ChebNet used in the experiments consists of the input layer, 1 convolutional layer (K 1 " 50, F 1 " 16) with no pooling, followed by 3 fully-connected layers (respectively, 128, 500, and 64 neurons, all with bias terms) and the output layer. When learning the predictability, the input and output dimensions are both 274. The hyper-parameters in its training stage are: dropout rate 0.05, batch size 100, and learning rate 0.05. When comparing the reconstruction performance, we re-train the network with the output dimension reduced to |I|. The training samples are constructed as explained in the beginning of Section 6.2. The batch size and the learning rate are reset as 1000 and 0.001, respectively. We use early stopping in all network training.

Graph construction For kernel ridge regression, the ChebNet, and the random sampling method of [START_REF] Puy | Random sampling of bandlimited signals on graphs[END_REF], we build the graph by connecting each station with its k-nearest neighbors using the geographical coordinates. More specifically, we first compute the Euclidean distance between nodes (sensors) using the latitude and longitude, denoted as dp¨, ¨q. Then, for each node i, we take its k 0 nearest neighbors, denoted as i 1 , i 2 , ..., i k 0 , with the associated distances dpi, i 1 q, dpi, i 2 q, ..., dpi, i k 0 q. We add edges pi, i s q, i " 1, ..., N, s " 1, ..., k 0 into E, with weights calculated as a iis " exp ˆ´d 2 pi, i s q σ i σ is ẇhere σ j is the local scale of node j, that is chosen as σ j " dpj, j k 1 q, with j k 1 the k 1 -th nearest neighbour of j. This self-tuning approach is proposed in [START_REF] Zelnik-Manor | Self-tuning spectral clustering[END_REF]. In our experiments, we took k 0 " 20 and k 1 " 7. For the pairs pi, jq R E, a ij is defined as 0. Thus, we have built the adjacency matrix A " pa ij q i,j . We use the combinatorial Laplacian for all the graph methods.