Olivier Coulaud

Alena Shilova

Esragul Korkmaz

Yanfei Xiang Thank

Jean-François David

Xunyi Zhao

Thank

Abdou Guermouche

Alycia Lisito

Aurélien Esnard

Florent Pruvost

Guillaume Sylvand

Lionel Eyraud-Dubois

Olivier Beaumont

Pierre Estérie

Pierre Ramet

O Agullo

L Coulaud

G Giraud

Marait

Keywords: calcul tensoriel, compression de rang faible, algèbre linéaire numérique, analyse de données. Numerical linear algebra tensor calculus, low rank compression, numerical linear algebra, data analysis

Algèbre linéaire numérique et analyse de données en grande dimensions utilisant le format tenseur

Résumé : L'objectif de ce travail est d'établir quelles propriétés théoriques des techniques d'algèbre linéaire classique développées dans deux contextes différents, à savoir l'algèbre linéaire numérique et l'analyse de données, sont préservées et lesquelles sont perdues, lorsqu'elles sont étendues aux tenseurs de rang faible. En outre, ce manuscrit vise à mettre en évidence les avantages et les inconvénients d'une approche tensorielle par rapport à son homologue matricielle classique dans les deux domaines considérés, avec une attention particulière aux aspects computationnels. Dans la partie d'algèbre linéaire numérique, nous étudions expérimentalement les effets des erreurs d'arrondi sur un solveur itératif et plusieurs méthodes d'orthogonalisation, lorsqu'ils sont étendus aux tenseurs par le formalisme du Train Tensoriel (TT). Dans tous les algorithmes considérés, nous introduisons des étapes d'arrondi supplémentaires, avec l'algorithme de compression TT-rounding, pour faire face aux contraintes de mémoire, toujours cruciales lorsqu'on traite des tenseurs. Nos tests suggèrent que pour ces algorithmes, les limites classiques dues à la propagation des erreurs d'arrondi restent valables, en remplaçant la précision de l'arithmétique par celle de l'algorithme TT-rounding. Le solveur itératif considéré est le Generalised Minimal RESidual (GMRES). Nous comparons notre version de TT-GMRES avec une réalisation précédente, en montrant numériquement sa grande robustesse. De plus, nous abordons le problème de la résolution simultanée par TT-GMRES de nombreux systèmes linéaires au format TT et établissons des bornes qui garantissent la qualité numérique de la solution individuelle extraite. Les schémas classiques d'orthogonalisation généralisés aux tenseurs sont CGS, CGS2, MGS, MGS2, Householder et Gram. Pour compléter leur étude, nous étudions comment ils affectent les performances du solveur de problèmes aux valeurs propres basé sur des itérations de sous-espaces étendu aux tenseurs avec le format TT.

Dans la partie analyse de données, nous étudions deux techniques d'analyse de données, l'une destinée aux données de variables catégorielles et l'autre aux données climatiques, généralisées aux tenseurs par le biais du format Tucker, en soulignant les avantages et les inconvénients de ce choix par rapport à l'approche matricielle correspondante. L'Analyse des Correspondances (AC) est un outil bien connu pour visualiser et interpréter des tableaux catégoriels à deux variables. Nous étudions géométriquement la généralisation de l'AC aux tableaux multivoies par la technique de décomposition tensorielle de Tucker, contribuant ainsi à la compréhension de l'Analyse des Correspondances MultiVoies (ACMV). Les résultats théoriques sont complétés par des exemples de ACMV appliqués à des ensembles de données. En particulier, nous réalisons l'ACMV sur le jeu de données écologique original mis à notre disposition dans le cadre du projet Malabar. Pour les données climatiques, nous considérons l'analyse de la Fonction Orthogonale Empirique (EOF). En particulier, nous montrons comment récupérer le résultat final de l'EOF en s'appuyant sur le format compressé de Tucker. Cette approche peut être avantageuse sur le plan du calcul si les données sont disponibles directement au format Tucker. Pour être complet, nous étudions numériquement l'effet de l'approximation des données par le modèle de Tucker sur le résultat EOF final.

unfortunately), Giulia Vighi, Marco Langella, Maria Laura Rossi, Rocco Mora, Sara Martani and Stefano Ardizzoni. You all together made the three years at the University of Parma unforgettable. Gathering every year during Christmas or summer times feels like getting back to the old, but gold times. My deep gratitude goes to all the friends I made during my high-school time in Cremona, namely Alexei Farina, Anna Ginestri, Carla Faroni, Carolina Carrera, Lara Fusar Poli, Irene Forzoni, Marco Luvié, Maria Politi, Mattia Pollenghi, Nicole Chiozzi, Valentina Gualazzi, Valentina Negro, Valeria Lodigiani. I am well aware I am not spending enough time with you, but I am proud to say that even if the quantiy of time has reduced, the quality has not changed over the years. A thought of recognition for the people who helped me settle in Bordeaux goes to Masataka Sawayama, Cristina Vaghi, Michele Giuliano Carlino, Luca Cirrottola, and Silvia Pagliarini. Talking from time to time with them, in my mother tongue for some, has been of great help to my mood. I am grateful to my future advisor Lieven De Lathauwer, who helped me enlarge my knowledge during the postdoc interview and who I am sure will help me deepen my research. I am looking forward to joining the Stadius team in Belgium.

Even if I can already hear Olivier, Esra, and Yanfei murmuring about the excessive length of this section2 , there are still people I need to thank. To begin with, I am deeply thankful to my parents, Amelia and Alfonso, who since the very first moment have lovely fed me and my curiosity, inspiring me to do my best. I am aware sometimes it is difficult to be with me, but I love you so much for all you ever gave and still give me. Then, I have to thank Daniela, my younger sister, who went through hot and cold with me, but who never forgot to show me her love. I am deeply grateful to my family members, my cousins Benedetto and Valentino, aunts Antonella, Lucia, and Tiziana, uncle Gianfranco, grandmother Gina, and to Aldo, Danilo, Francesco, Giovanni, and Rosa, who unfortunately can not share this moment with us, but who certainly would be proudly joyful. My real last thought goes to you, Rocco, who spent so many happy years close to me, always knowing how to wring a smile from me, even in my worst moments. Thank you. And thank you passing reader, who has come this far and still has much to read.

Résumé étendu

Tôt ou tard, presque toutes les disciplines, des mathématiques pures à l'ingénierie, de la physique aux sciences naturelles, de l'économie aux sciences sociales, se tournent vers l'algèbre linéaire et ses résultats numériques pour énoncer mathématiquement leurs problèmes et pour espérer les résoudre. Les améliorations technologiques récentes, rapides et en croissance continue, ouvrent de nouvelles questions mathématiques et informatiques, ce qui rend nécessaire le développement de l'algèbre linéaire et de ses branches appliquées. En particulier, la popularité croissante des données volumineuses et des problèmes de haute dimension stimule les progrès de l'algèbre multilinéaire et de ses aspects numériques.

Les principaux objets d'étude de l'algèbre multilinéaire sont les tenseurs et les fonctions multilinéaires. Les tenseurs en tant qu'outils mathématiques sont apparus dès le XIXème et le XXème siècle dans les études de géométrie de Riemann, Christoffel, Ricci-Curbastro et Levi-Civita et dans les travaux de physique de Voigt et Hamilton [START_REF] Favier | Historical Elements of Matrices and Tensors[END_REF]. Cependant, la première étude mathématique sur leur nature est apparue plus tard en 1927 par Hitchcock [START_REF] Favier | Historical Elements of Matrices and Tensors[END_REF][START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF]. A partir de ce moment, de nombreux travaux issus de domaines très différents ont contribué à la théorie des tenseurs, en proposant des algorithmes de factorisation des tenseurs, par exemple, le Tucker, le FACteur PARAllel (PARAFAC) la DECOMposition CANonique (CANDECOM), les techniques Matrix-Product State (MPS) étudiées respectivement par Cattel [START_REF] Cattell | Parallel proportional profiles" and other principles for determining the choice of factors by rotation[END_REF], Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF], Harshman [START_REF] Harshman | Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis[END_REF], Carroll et Chang [START_REF] Carroll | Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition[END_REF], White [START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF]. Ces résultats ont été redécouverts et développés pour aboutir aux techniques de factorisation et d'approximation tensorielles les plus populaires actuellement, à savoir la décomposition polyadique canonique (CP) [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF], la décomposition de Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF] calculée avec l'algorithme HOSVD (High Order Singular Value Decomposition), (récemment renommé Multilinear Singular Value Decomposition [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF]), la décomposition de Tucker hiérarchique (H-Tucker) [START_REF] Grasedyck | Hierarchical Singular Value Decomposition of Tensors[END_REF] et la décomposition TT (Tensor Train) [START_REF] Oseledets | Tensor-Train Decomposition[END_REF]. Nous renvoyons le lecteur à [START_REF] Kolda | Tensor Decompositions and Applications[END_REF] pour plus de détails sur ces techniques. Ces méthodes et les algorithmes qui leur sont associés, avec leurs avantages et leurs inconvénients, ont des domaines d'application spécifiques, ce qui pousse à approfondir leurs propriétés mathématiques et à trouver de nouvelles possibilités d'application.

En s'appuyant sur les différentes méthodes de décomposition tensorielle, l'objectif de ce travail est d'établir quelles propriétés théoriques des techniques classiques d'algèbre linéaire développées dans deux contextes différents, à savoir l'algèbre linéaire numérique et l'analyse de données, sont conservées et lesquelles sont perdues, une fois qu'elles sont étendues aux tenseurs. Une attention particulière est consacrée aux aspects computationnels. En outre, ce manuscrit vise à mettre en évidence les avantages et les inconvénients I II algèbre linéaire numérique d'une approche tensorielle par rapport à son homologue matricielle classique dans les deux cadres considérés.

Algèbre linéaire numérique

La façon dont un algorithme propage les erreurs d'arrondi est un sujet clé en algèbre linéaire numérique. Classiquement, l'étude est effectuée avec le modèle IEEE de l'arithmétique à virgule flottante, c'est-à-dire en affirmant que la représentation informatique de x ∈ R est x = x + δ avec |δ| ≤ u, où u est le unité d'arrondi et il limite la précision de la représentation de x. De même, le calcul arithmétique en précision finie induit des perturbations relatives similaires dans tous les calculs élémentaires.

L'objectif de la Partie I est d'étudier numériquement les effets de l'erreur d'arrondi pour un solveur itératif et six schémas d'orthogonalisation classiques lorsqu'ils sont étendus au cadre tensoriel par le formalisme TT. Dans tous les algorithmes considérés, nous introduisons des étapes d'arrondi supplémentaires, par le biais de l'algorithme d'arrondi TT [START_REF] Oseledets | Tensor-Train Decomposition[END_REF] pour faire face aux contraintes de mémoire, toujours cruciales lorsqu'on traite des tenseurs. Nos expériences suggèrent que pour ces algorithmes, les limites classiques dues à la propagation des erreurs d'arrondi sont valables, en remplaçant l'arrondi unitaire u de l'arithmétique de précision finie par la précision δ de l'algorithme d'arrondi TT.

Generalized Minimal RESidual

III et N. Schenkels. Le but de notre travail est d'étudier numériquement la stabilité arrière de GMRES lorsque la précision de la représentation des données et celle des calculs sont différentes. En effet, dans certaines situations, une partie des données doit être compressée avec une précision différente (généralement inférieure) de celle du calcul en raison de contraintes de mémoire de stockage. La procédure de compression introduit des perturbations, qui sont numériquement décrites par une erreur de représentation limitée par la précision de la compression. Ainsi, nous avons deux erreurs de représentation, l'une due à la compression et l'autre liée à l'arithmétique à précision finie. Si la compression perturbe les données par composant, l'analyse théorique de stabilité arrière de GMRES [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] s'applique toujours. Cependant, il existe des cas où les erreurs de compression affectent les données de manière normalisée, de sorte que les hypothèses mathématiques de l'analyse [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] ne s'appliquent pas facilement. Nos exemples numériques suggèrent que même avec des perturbations de compression normales, les deux variantes stables en arrière de GMRES restent stables en arrière, c'est-à-dire que l'erreur arrière normale atteignable est du même ordre que le maximum entre la compression et la précision de calcul. Tous les résultats de ce travail sont également présentés dans [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF].

La deuxième partie du Chapitre 2 présente la deuxième contribution liée à la GMRES : l'étude numérique d'un algorithme GMRES robuste adapté au cas tensoriel avec le formalisme TT, brièvement appelé TT-GMRES. Ce solveur itératif est destiné à résoudre des systèmes linéaires, dont l'opérateur agit parmi des espaces tensoriels d'ordre d. Pour simplifier, nous appellerons ces systèmes linéaires des systèmes linéaires tensoriels d'ordre d. Nous présentons l'algorithme TT-GMRES avec l'erreur à rebours comme critère d'arrêt et quelques exemples numériques, qui soutiennent sa stabilité à rebours, obtenus à partir de la discrétisation par des grilles cartésiennes de certaines EDP classiques. En effet, GMRES dans le format TT représente une étude de cas des perturbations normales, décrites dans la première partie du Chapitre 2 et dans [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF]. Pour être complet, notre algorithme TT-GMRES est comparé à une précédente adaptation de GMRES au cadre tensoriel présentée dans [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. Nous montrons que la réalisation proposée dans [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF] n'est pas stable en arrière et ne garantit pas la précision de la solution calculée. De plus, nous nous concentrons sur la résolution avec TT-GMRES des systèmes linéaires tensoriels dont les opérateurs ou les côtés droits dépendent d'un paramètre. Sous une certaine hypothèse mathématique, lorsque l'opérateur ou le côté droit dépend d'un paramètre, la solution de p systèmes linéaires tensoriels d'ordre d peut être calculée en résolvant un unique système linéaire tensoriel d'ordre (d + 1) avec une taille p le long du (d + 1)-ième mode. Nous prouvons théoriquement que dans ce cadre, il existe des bornes d'erreur arrière reliant la solution du système tensoriel unique d'ordre d à sa solution correspondante extraite de la solution du système linéaire tensoriel d'ordre (d + 1). La qualité de ces bornes est également étudiée numériquement sur divers exemples. Pour être aussi complet que possible, nous étudions numériquement la qualité du préconditionneur. Ces résultats sont rassemblés dans [START_REF] Coulaud | A note on GMRES in TT-format[END_REF].

IV algèbre linéaire numérique

Schémas d'orthogonalisation

Nous considérons six algorithmes pour calculer une base orthonormale de l'espace couvert par un ensemble de vecteurs linéairement indépendants : Gram-Schmidt Classique (CGS) et sa version avec réorthogonalisation (CGS2), Gram-Schmidt Modifié (MGS) et sa mise en oeuvre avec réorthogonalisation (MGS2), la transformation de Householder et l'algorithme de Gram. L'estimation de la perte d'orthogonalité de la base calculée est importante pour évaluer la qualité d'un schéma d'orthogonalisation. La perte d'orthogonalité pour Householder est prouvée dans [START_REF] Wilkinson | Modern Error Analysis[END_REF] comme étant O(u) ; la même limite est valable pour MGS2 et CGS2 comme montré dans [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF][START_REF] Smoktunowicz | A note on the error analysis of classical Gram-Schmidt[END_REF] sous l'hypothèse que κ(A)u < 1 et κ 2 (A)u < 1 respectivement, où κ(A) est le nombre de condition de l'ensemble des vecteurs d'entrée à orthogonaliser. Dans [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF], l'auteur indique que si κ(A)u < 1, alors la perte d'orthogonalité du MGS est O(κ(A)u), tandis que celle du CGS est O(κ 2 (A)u). Enfin, dans [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF], il est prouvé que O(κ 2 (A)u) limite également la perte d'orthogonalité de la technique d'orthogonalisation de Gram.

Contribution

Dans le Chapitre 3, nous étudions d'un point de vue numérique la perte d'orthogonalité de ces six algorithmes d'orthogonalisation classiques et populaires, correctement adaptés pour fonctionner dans le format tensoriel. Plus précisément, nous reformulons les algorithmes de Gram, CGS et MGS, avec et sans réorthogonalisation, en remplaçant l'arithmétique vectorielle classique par l'arithmétique TT. Alors que ces méthodes lisent directement les tenseurs, l'algorithme de transformation de Householder nécessite un plan minutieux et bien pensé, que nous décrivons, puisque les entrées des tenseurs en format compressé ne sont pas directement accessibles. De plus, dans les six schémas d'orthogonalisation, nous introduisons une ou plusieurs étapes d'arrondi à une précision donnée, basée sur l'arrondi TT [START_REF] Oseledets | Tensor-Train Decomposition[END_REF]. Les étapes d'arrondi jouent un rôle crucial : d'une part, elles maintiennent l'exigence de mémoire à un niveau abordable, d'autre part, leur précision prend le rôle de l'arrondi unitaire de l'analyse classique de stabilité à rebours. En effet, nos expériences numériques suggèrent que les limites théoriques prouvées pour les schémas d'orthogonalisation classiques dans le contexte IEEE s'appliquent toujours dans le cadre TT, où la précision de l'arrondi remplace l'arrondi unitaire. Ces résultats, également disponibles dans [START_REF] Coulaud | On some orthogonalization schemes in TT-format[END_REF], n'ont jamais été rapportés auparavant, à notre connaissance.

L'analyse des besoins en mémoire et l'application de ces noyaux dans un eigensolveur classique, adapté correctement au cadre tensoriel, complètent l'étude numérique des schémas d'orthogonalisation au format TT. Nous considérons la méthode d'itération subspatiale [START_REF] Bauer | Das Verfahren der Treppeniteration und verwandte Verfahren zur Lösung algebraischer Eigenwertprobleme[END_REF] pour travailler avec des tenseurs dans le format TT, et nous étudions les performances des différents algorithmes d'orthogonalisation. Enfin, nous comparons la qualité et les propriétés des paires propres en fonction des différents noyaux d'orthogonalisation pour un cas test académique, à savoir l'opérateur Laplacien exprimé au format TT [START_REF] Kazeev | Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse[END_REF].

résumé étendu

V

Analyse des données

L'analyse des données est la discipline de la science des données qui explore les méthodes d'extraction, d'interprétation et de visualisation des caractéristiques importantes des données. Plusieurs techniques développées en analyse de données reposent sur des outils d'algèbre linéaire, par exemple, l'Analyse en Composantes Principales (ACP) et toutes ses variantes adaptées aux différents contextes, qui sont basées sur la SVD. Depuis l'arrivée d'ensembles de données de grande taille provenant d'entreprises technologiques publiques et privées, les modèles tensoriels ont gagné en popularité, offrant de nouvelles directions de recherche dans différents domaines.

La Partie II a pour but d'étudier certaines techniques d'analyse de données pour les tenseurs exprimés au format Tucker, en soulignant les avantages et les défauts de ce choix par rapport à l'approche matricielle correspondante.

Analyse des correspondances

L'Analyse des Correspondances (AC) est l'outil couramment utilisé pour analyser et interpréter les tableaux de contingence, c'est-à-dire les ensembles de données dont les entrées sont des comptes ou des fréquences de deux catégories de deux variables différentes [START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF]. L'AC est basée sur l'ACP et, par conséquent, elle est à la fois une technique d'analyse et de visualisation, fournissant simultanément un point de vue algébrique, géométrique et statistique sur les données [START_REF] Benzécri | Statistical analysis as a tool to make patterns emerge from data[END_REF][START_REF] Benzécri | Sur l'analyse des tableaux binaires associés à une correspondance multiple[END_REF][START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF][START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF].

Lorsque plus de deux variables catégorielles sont présentes dans les ensembles de données, généralement appelés tableaux à voies multiples, la technique la plus utilisée pour l'analyse est l'Analyse des Correspondances Multiples (ACM). L'ACM procède en associant un tableau d'indicateurs ou de Burt, qui est mathématiquement une matrice de zéros et de uns, au tableau multivoie [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. En raison de ce choix, toutes les interactions entre plus de deux variables sont perdues et ne peuvent être récupérées [START_REF] Beh | Correspondence Analysis[END_REF]. Cependant, il existe un outil alternatif appelé Analyse des Correspondances MultiVoies (ACMV) [START_REF] Beh | Multiple and multiway correspondence analysis[END_REF], basé sur la CP [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF] et sur le modèle de Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF]. Les formulations algébriques et statistiques de l'ACMV ont été présentées, au moins pour trois variables, dès les années 90, voir par exemple [START_REF] Franc | Etude Algébrique des multitableaux: apports de l'algèbre tensorielle[END_REF][START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF], mais ses potentialités ne sont pas pleinement explorées, comme l'indiquent [START_REF] Beh | Multiple and multiway correspondence analysis[END_REF][START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF].

Extended summary

Sooner or later, almost every discipline, from pure mathematics to engineering, from physics to natural science, from economics to social science, turns to linear algebra and its numerical results for mathematically stating their problems and for hopefully solving them. The recent fast and continuously growing technological improvements open new mathematical and computational questions, making necessary further development in linear algebra and its applied branch. In particular, the increased popularity of big data and high dimensional problems is driving the progress of multilinear algebra and its numerical aspects.

Multilinear algebra main objects of investigation are tensors and multilinear functions. Tensors as mathematical tools appeared already between the XIX and the XX century in the geometry studies of Riemann, Christoffel, Ricci-Curbastro, and Levi-Civita and in the physics works of Voigt and Hamilton [START_REF] Favier | Historical Elements of Matrices and Tensors[END_REF]. However, the first mathematical study on their nature appeared later in 1927 by Hitchcock [START_REF] Favier | Historical Elements of Matrices and Tensors[END_REF][START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF]. From that moment, many works from very different domains contributed to the tensor theory, proposing tensor factorization algorithms, for example, the Tucker, the PARAllel FACtor (PARAFAC) the CANonical DECOMposition (CANDECOM), Matrix-Product State (MPS) techniques investigated by Cattel [START_REF] Cattell | Parallel proportional profiles" and other principles for determining the choice of factors by rotation[END_REF], Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF], Harshman [START_REF] Harshman | Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis[END_REF], Carroll and Chang [START_REF] Carroll | Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition[END_REF], White [START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF] respectively. Those results were rediscovered and developed further leading to the currently most popular tensor factorization and approximation techniques, which are the Canonical Polyadic decomposition (CP) [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF], the Tucker decomposition [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF] computed with High Order Singular Value Decomposition (HOSVD) algorithm, (recently renamed as Multilinear Singular Value Decomposition [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF]), the Hierarchical Tucker decomposition (H-Tucker) [START_REF] Grasedyck | Hierarchical Singular Value Decomposition of Tensors[END_REF] and the Tensor Train (TT) decomposition [START_REF] Oseledets | Tensor-Train Decomposition[END_REF]. We refer the reader to [START_REF] Kolda | Tensor Decompositions and Applications[END_REF] for further details about these techniques. These methods and their related algorithms, with advantages and drawbacks, have specific areas of applications, which in turn push for further investigation of their mathematical properties and new application possibilities.

Relying on the different tensor decomposition methods, the purpose of this work is to establish which theoretical properties of classical linear algebra techniques developed in two different contexts, that are numerical linear algebra and data analysis, are saved and which are lost, once they are extended to tensors. Particular attention is dedicated to the computational aspects. Moreover, this manuscript aims to highlight the benefits and the flaws of a tensor approach compared to its classical matrix counterpart in the two considered frameworks.

VII VIII numerical linear algebra

Numerical linear algebra

How an algorithm propagates rounding errors is a key topic in numerical linear algebra. Classically, the investigation is performed with floating point arithmetic IEEE model, i.e., stating that the computer representation of x ∈ R is x = x + δ with |δ| ≤ u, where u is the unit round-off and it bounds the accuracy of the representation of x. Likewise, the finite precision arithmetic calculation induces similar relative perturbations in all elementary computations.

The purpose of Part I is to study numerically the rounding error effects for an iterative solver and six classical orthogonalization schemes when they are extended to the tensor framework through the TT-formalism. In all the considered algorithms we introduce additional rounding steps, through the TT-rounding algorithm [START_REF] Oseledets | Tensor-Train Decomposition[END_REF] to face memory constraints, always crucial when dealing with tensors. Our experiments suggest that for those algorithms classical bounds due to rounding error propagation hold, replacing the unit round-off u of the finite precision arithmetic by the precision δ of the TT-rounding algorithm.

Generalized Minimal RESidual

The Generalized Minimal RESidual is a robust iterative method to solve linear systems. GMRES relies on the Krylov subspace technique for approximating the solution. The approximated solution is computed by minimizing the norm of the linear system residual on the associated Krylov space, whose dimension increases iteratively. The minimization problem is simplified by constructing an orthogonal basis of the Krylov space through the MGS and the Householder scheme, leading to the MGS-GMRES and Householder-GMRES implementation presented in [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] and [START_REF] Walker | Implementation of the GMRES method using Householder transformations[END_REF] respectively. A meaningful and frequently used stopping criterion is based on the backward error associated with the matrix and the right-hand side of the linear system. This choice is motivated by the backward stability of both GMRES variants in the classical floating point arithmetic proved in [START_REF] Paige | Solution of Sparse Indefinite Systems of Linear Equations[END_REF] and [START_REF] Drkosova | Numerical stability of GMRES[END_REF] respectively. This backward stability property ensures that the solution computed by GMRES can be interpreted as the solution of the same linear system with relative norm perturbations on the matrix and the right-hand side that are bounded by the unit round-off of the working precision.

Contribution

The first contribution to the study of the GMRES solver, presented at the beginning of Chapter 2, comes from a joint work with few scientists of the Concace (previously HiePACS) Inria team, namely E. Agullo, O. Coulaud, L. Giraud, G. Marait and N. Schenkels. The purpose of our work is to study numerically the backward stability of GMRES when the accuracy of the data representation and the computations are different. Indeed, in some situations, part of the data needs to be compressed with an accuracy different (usually lower) than the computational one due to storing memory constraints.

extended summary

IX

The compression procedure introduces perturbations, which are numerically described by a representation error bounded by the compression accuracy. Thus, we have two representation errors, one due to the compression and one linked to the finite precision arithmetic. If the compression perturbs the data componentwise, the theoretical backward stability analysis of GMRES [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] still applies. However, there are cases when the compression errors affect the data normwise so that the mathematical assumptions of the analysis [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] do not readily apply. Our numerical examples suggest that even with normwise compression perturbations, the two backward stable variants of GMRES remain backward stable, that is the attainable normwise backward error is of the same order as the maximum between the compression and the computational accuracy. All the results of this work are also presented in [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF].

The second part of Chapter 2 presents the second contribution related to GMRES: the numerical study of a robust GMRES algorithm adapted to the tensor case with the TT-formalism, shortly called TT-GMRES. This iterative solver is meant for solving linear systems, whose operator acts among tensor spaces of order d. We will refer for simplicity to those linear systems as tensor linear systems of order d. We present the TT-GMRES algorithm with the backward error as a stopping criterion and some numerical examples, that support its backward stability, obtained from the discretization through Cartesian grids of some classical PDEs. Indeed, GMRES in TT-format represents a case study of normwise perturbations, described in the first part of Chapter 2 and in [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF]. For completeness, our TT-GMRES algorithm is compared with a previous adaptation of GMRES to the tensor framework presented in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. We show that the realization proposed in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF] is not backward stable and does not guarantee the accuracy of the computed solution. Moreover, we focus on solving with TT-GMRES tensor linear systems whose operators or right-hand sides depend on a parameter. Under some mathematical hypothesis, when the operator or the right-hand side is parameter dependent, the solution of p tensor linear systems of order d can be computed by solving a unique tensor linear system of order (d + 1) with size p along the (d + 1)-th mode. We prove theoretically that in this framework there exist backward error bounds linking the solution of the single tensor system of order d with its corresponding one extracted from the solution of the tensor linear system of order (d + 1). The quality of those bounds is also studied numerically on various examples. To be as comprehensive as possible, we numerically investigate the quality of the preconditioner. These results are collected in [START_REF] Coulaud | A note on GMRES in TT-format[END_REF].

Orthogonalization schemes

We consider six algorithms to compute an orthonormal basis of the space spanned by a set of linearly independent vectors: Classical Gram-Schmidt (CGS) and its version with re-orthogonalization (CGS2), Modified Gram-Schmidt (MGS) and its implementation with re-orthogonalization (MGS2), Householder transformation and the Gram algorithm. Estimating the loss of orthogonality of the computed basis is important to evaluate the quality of an orthogonalization scheme. The loss of orthogonality for Householder is proved in [START_REF] Wilkinson | Modern Error Analysis[END_REF] to be O(u); the same bound holds for MGS2 and CGS2 as showed in [51, X data analysis [START_REF] Smoktunowicz | A note on the error analysis of classical Gram-Schmidt[END_REF] under the hypothesis that κ(A)u < 1 and κ 2 (A)u < 1 respectively, where κ(A) is the condition number of the input set of vectors to orthogonalize. In [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF], the author states that if κ(A)u < 1, than the loss of orthogonality of MGS is O(κ(A)u), while the CGS one is O(κ 2 (A)u). Finally, in [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF], it is proved that O(κ 2 (A)u) bounds also the loss of orthogonality of the Gram orthogonalization technique.

Contribution

In Chapter 3, we study from a numerical point of view the loss of orthogonality of these six classical and popular orthogonalization algorithms, properly adapted to work in the tensor format. More precisely, we reformulate the Gram, CGS and MGS algorithm, with and without re-orthogonalization, replacing the classical vector arithmetic by the TT-arithmetic. While those methods directly read for tensors, the Householder transformation algorithm requires a careful and well-thought-out outline, that we describe, since entries of compressed-format tensors are not directly accessible. Moreover, in all the six orthogonalization schemes, we introduce one or more rounding steps at a given accuracy, based on the TT-rounding [START_REF] Oseledets | Tensor-Train Decomposition[END_REF]. The rounding steps play a crucial role: on one side, they keep the memory requirement affordable, on the other, their accuracy takes over the role of the unit round-off of the classical backward stability analysis. Indeed, our numerical experiments suggest that the theoretical bounds proved for the classical orthogonalization schemes in the IEEE context still apply in the TT-framework, where the rounding accuracy replaces the unit round-off. Those results, available also in [START_REF] Coulaud | On some orthogonalization schemes in TT-format[END_REF], were never reported before, to the best of our knowledge.

The analysis of the memory requirement and the application of these kernels into a classical eigensolver, adapted properly to the tensor framework, complete the numerical study of the orthogonalization schemes in TT-format. We consider the subspace iteration method [START_REF] Bauer | Das Verfahren der Treppeniteration und verwandte Verfahren zur Lösung algebraischer Eigenwertprobleme[END_REF] to work with tensors in TT-format, and investigate the performance of the different orthogonalization algorithms. Finally, we compare the quality and the properties of the eigenpairs depending on the different orthogonalization kernels for an academic test case, i.e., the Laplacian operator expressed in TT-format [START_REF] Kazeev | Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse[END_REF].

Data analysis

Data analysis is the data science discipline that explores methods to extract, interpret and visualize important features of the data. Several techniques developed in data analysis rely on linear algebra tools, for example, Principal Component Analysis (PCA) and all its variations tuned to the different contexts, which are based on SVD. Since the advent of large-size datasets coming from public and private tech companies, tensor models have gained popularity providing new research directions in different fields.

The Part II aim is to investigate some data analysis techniques for tensors expressed in Tucker format, highlighting the benefits and the flaws of this choice compared to the corresponding matrix approach.

extended summary

XI

Correspondence analysis

Correspondence Analysis (CA) is the commonly used tool for analysing and interpreting contingency tables, i.e., datasets whose entries are counts or frequencies of two categories of two different variables [START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF]. CA is based on PCA and consequently, it is both an analysis and a visualization technique, providing simultaneously an algebraic, a geometric and a statistical point of view on the data [START_REF] Benzécri | Statistical analysis as a tool to make patterns emerge from data[END_REF][START_REF] Benzécri | Sur l'analyse des tableaux binaires associés à une correspondance multiple[END_REF][START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF][START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF].

When more than two categorical variables are present in the datasets, usually referred to as multiway tables, the most used technique for the analysis is Multiple Correspondence Analysis (MCA). MCA proceeds by associating an indicator or Burt table, which mathematically is a matrix of zeros and ones, to the multiway table [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. Because of this choice, all the interactions among more than two variables are lost and can not be retrieved [START_REF] Beh | Correspondence Analysis[END_REF]. However, there exists an alternative tool called MultiWay Correspondence Analysis (MWCA) [START_REF] Beh | Multiple and multiway correspondence analysis[END_REF], based on the CP [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF] ad on the Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF] model. The MWCA algebraic and statistical formulations were presented, at least for three variables, already in the 90s, see for example [START_REF] Franc | Etude Algébrique des multitableaux: apports de l'algèbre tensorielle[END_REF][START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF], but its potentialities are not fully explored, as stated in [START_REF] Beh | Multiple and multiway correspondence analysis[END_REF][START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF].

Contribution

In Chapter 4, we study MWCA mainly on the geometric and visualization side, to fulfil the existing gap for this technique generalized to tensors [START_REF] Beh | Multiple and multiway correspondence analysis[END_REF][START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF]. Our theoretical and practical contributions to MWCA derive from joint work with O. Coulaud and A. Franc, Inrae research scientist in the BioGeCo Inrae team, also a member of the Pleiade Inria project team. This collaboration aims to investigate the role and the benefits of using the tensor theory for studying ecological multiway contingency tables. Firstly, we contribute to the description of the geometrical properties of MWCA. After developing its algebraic structure based on the Tucker model for a generic number of variables, we focus on MWCA visualization and on the geometrical interpretations that are inferred. To highlight the benefit and the flaws of a tensor approach in practice, two study cases show the difference between the MWCA and the CA visualization and interpretation.

The multiway contingency table provided by the Malabar project (IFREMER, CNRS, INRAE, Labex COTE) [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF] allows us to test our theoretical results on a real ecology study case. Our practical contribution is the analysis of this Malabar dataset. We apply the previously developed MWCA results to geometrically interpret the relations among the data variables. For completeness, we compare the MWCA results with the CA ones, transforming conveniently the data.

Empirical Orthogonal Function analysis

In climatology, a popular method for the study and the visualization of data depending on space and time variables is Empirical Orthogonal Functions (EOF), based on PCA. The EOF technique is regarded both as a dimensional reduction and as a pattern extraction technique [START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF], leading to the independent visualization of the space and of the time XII data analysis information carried by the analyzed data. To the best of our knowledge, no description of the EOF method in the tensor framework has been presented so far, even if studies of climate data based on tensor models are available, for example [START_REF] Zhang | A Parallel Nonnegative Tensor Factorization Algorithm for Mining Global Climate Data[END_REF].

Contribution

During the collaboration with O. Coulaud and L. Terray, a research scientist in Climate, Environment, Coupling, and Uncertainties (CECI), a joint research team between Cerfacs and CNRS, we investigate the advantages and the drawbacks of introducing the tensor theory in the EOF method. In Chapter 5, we conclude that from the computational cost point of view a tensor approach is not beneficial in general. However, we show that the computational cost of the EOF technique can be reduced, under the strong assumption that the climate data are made available or approximated in Tucker format. Indeed, we theoretically retrieve the EOF results from the compressed data expressed in Tucker format. To complete the study, we approximate a dataset through the HOSVD technique and study the errors between the EOF results generated through classical and tensor techniques.

List of Symbols

Objects a k

k-th Tensor Train core of a A k (i k )

i k -th slice along mode 2 of k-th Tensor Train core a k V * dual of a linear subspace F generic algebraic field I n identity matrix of size n V linear subspace A (i) matricization of tensor a along mode i A matrix A linear space endowed with a metric N integer number set A linear or multilinear operator O(m × n) set of orthogonal matrices of m rows and n columns Q rational number field R real number field R + real positive number set A set a [i k ,k] i k -th slice along mode k a multidimensional array as element of a tensor space A multidimensional array representing a multilinear operator a vector

Operations

•

H L H R contraction of modes in H L and H R for left and right tensor respectively

• , • inner product ⊗ K Kronecker product • 2 L2-norm •
Frobenious norm ⊗ tensor product × k matrix-tensor product along mode k XIII Chapter 1

Notation and preliminary results

Introduction and notation

A tensor is a mathematical object carrying multiple meanings; we present briefly the main three: tensors as multidimensional arrays, as elements of a tensor space, and as representations of multilinear operators. From a very practical viewpoint, often used in computer science, a tensor is simply a multidimensional array. Exactly as a vector of R n is identified with the one-dimensional array which stores its n real coordinates with respect to the canonical basis, a tensor of R n 1 ו••×n d can be seen as an array of d dimensions storing all its entries. Definition 1.1.1. The object a ∈ R n 1 ו••×n d is a tensor, the integer d defines its order, the integer n k its k-th mode size.

A visual representation of tensors as multidimensional arrays of order d ∈ {1, 2, 3} is given in Figure 1.1.

Henceforth tensors are denoted by lowercase bold letters, matrices by uppercase letters, and vectors by lowercase ones. The expression a(i 1 , . . . , i d ) stands for the (i 1 , . . . , i d )-th entry of a ∈ R n 1 ו••×n d . The analogous expression is used for matrix and vector components. To further clarify the chosen notation, we present an example.

Example 1.1.1. The multidimensional array a in Figure 1.1C is a tensor of order 3 with From a more theoretical point of view typical of some disciplines, as multilinear algebra for example, tensors of order d are elements belonging to the tensor product, denoted by ⊗, of d linear subspaces V k defined over the field F for k ∈ {1, . . . , d}. Let {v

1 4 7             (A) vector or order-1 tensor 1 8 4 9 2 2 7 1 6             (B) matrix or order-2 tensor 1 8 4 9 2 2 7 1 6             5 3 1 2 4 9 6 8 7             (C) order-3 tensor
(k) 1 , . . . , v (k) r k } be a basis of V k the linear subspace of R n k of dimension r k , then V 1 ⊗ • • • ⊗ V k is a linear space generated by the elements v (1) i 1 ⊗ • • • ⊗ v (d) i d for every i k ∈ {1, . . . , r k } and k ∈ {1, . . . , d}. Consequently every tensor a ∈ V 1 ⊗• • •⊗V k can be
written as a linear combination of the basis elements, by the definition of linear space, i.e.,

a = r 1 ,...,r d i 1 ,...,i d =1 λ i 1 ,...,i d v (1) i 1 ⊗ • • • ⊗ v (d) i d . The linear space V 1 ⊗ • • • ⊗ V k is the tensor product space, in which we highlight a special set of elements. Definition 1.1.2. The tensor v ∈ V 1 ⊗ • • • ⊗ V d is an elementary tensor if it can be expressed as v = v 1 ⊗ • • • ⊗ v d with v k ∈ V k for every k ∈ {1, . . . , d}.
The (i 1 , . . . , i d )-th element of the elementary tensor v is equal to the product of the

i k component of v k ∈ V k for all k ∈ {1, . . . , d}, that is v(i 1 , . . . , i d ) = v 1 (i 1 ) • • • v d (i d ).
It is convenient to present an example, which will clarify this last statement.

Example 1.1.2. Let V k be a subspace of dimension 1 generated by v k ∈ R n k for k ∈ {1, 2, 3} with v 1 = 1 2 , v 2 =    9 1 1    and v 3 =    2 8 0    .
Then the tensor product space

V 1 ⊗ V 2 ⊗ V 3 is generated by the order 3 tensor v = v 1 ⊗ v 2 ⊗ v 3 , that is all the elements of V 1 ⊗ V 2 ⊗ V 3 can be expressed as (λv) with λ ∈ R.
Tensors as multidimensional arrays and as elements of the tensor product space are linked. Indeed, a vector of a linear space is uniquely identified by its coordinates once the linear space basis is fixed, similarly, a tensor as an element of the tensor product space is uniquely identified by a multidimensional array once a basis of the tensor product is set. We illustrate this link using the previous example.

CHAPTER 1. NOTATION AND PRELIMINARY RESULTS

3

Example 1.1.3. Considering the vectors v h defined in Example 1.1.2, we express their tensor product in the canonical basis of the tensor space. Each vector v h can be expressed as a linear combination of the canonical basis vectors {e

(h) i h } of V h for h ∈ {1, 2, 3}
where the coefficients are its entries, i.e.,

v 1 = 2 i=1 v 1 (i)e (1) i = e (1) 1 + 2e (1) 2 v 2 = 3 j=1 v 2 (j)e (2) j = 9e (2) 1 + e (2)
2 + e

(2) 3

v 3 = 3 k=1 v 3 (k)e (3) k = 2e (3) 1 + 8e (3) 2
where e

(1) 1 = [1, 0] , e (1) 1 = [0, 1] , e (2) 1 = e (3) 1 = [1, 0, 0] , e (2) 2 = e (3) 2 = [0, 1, 0] and e (2) 3 = e (3) 3 = [0, 0, 1] . As consequence, the tensor v gets v = v 1 ⊗ v 2 ⊗ v 3 = 2 i=1 v 1 (i)e (1) i ⊗ 3 j=1 v 2 (j)e (2) j ⊗ 3 k=1 v 3 (k)e (3) k
and thanks to the multilinearity of the tensor product, this last equation becomes

v = 2 i=1 v 1 (i)e (1) i ⊗ 3 j=1 v 2 (j)e (2) j ⊗ 3 k=1 v 3 (k)e (3) k = 2 i=1 3 j=1 3 k=1 v 1 (i)v 2 (j)v 3 (k) e (1) i ⊗ e (2) j ⊗ e (3) k = 2 i=1 3 j=1 3 k=1 v (i, j, k)e (1) i ⊗ e (2) j ⊗ e (3) k
where v is a 3-dimensional array of modes [START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF][START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF][START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF], whose (i 1 , i 2 , i 3 )-th entry is the product

of the i h -th entry of v h for h ∈ {1, 2, 3}, that is v (i 1 , i 2 , i 3 ) = v 1 (i 1 )v 2 (i 2 )v 3 (i 3 ). In this example, once the canonical basis is fixed for R 2 and R 3 , the (1, 1, 1) element of v is 1 × 2 × 9 = 18. Henceforth a d-dimensional array will represent the coordinates of a R n 1 ⊗ • • • ⊗ R n d tensor with respect to its canonical basis {e (1) i 1 ⊗ • • • ⊗ e (d) i d } where {e (k) i k } is the canonical basis of R n k for every k ∈ {1, .
. . , d}. This perspective will not be addressed further in this work; we refer the reader to [START_REF] Landsberg | Tensors: Geometry and Applications: Geometry and Applications[END_REF] for more details.

Another classical interpretation arises in the context of multilinear algebra where a tensor is regarded as a multilinear operator among the tensor product of linear spaces, in analogy with the matrix case. Indeed a matrix A ∈ R n 1 ×n 2 is both a real value array of order 2, it is an element of the linear space R n 1 ×n 2 and it describes the action of a linear operator from R n 2 to R n 1 , once a basis of R n 1 and R n 2 has been fixed. In this manuscript, we decide to denote by uppercase bold calligraphic letters the multilinear

TENSOR BASIC OPERATIONS operator

A : R n 1 ⊗ • • • ⊗ R n d → R m 1 ⊗ • • • ⊗ R m d ,
while the corresponding letter in uppercase bold, in this case A ∈ (n 1 ×m 1 )ו••×(n d ×m d ) , stands for the multidimensional array representing the multilinear operator A once the canonical basis of

R n 1 ⊗ • • • ⊗ R n d and R m 1 ⊗ • • • ⊗ R m d are fixed.
From now on, a tensor representing a multilinear operator among a tensor product of spaces will be referred to as tensor operator. Remark that the tensor operator A is a tensor of order 2d according to our definition, which acts on tensors of order d. This choice is aligned with the canonical linear algebra case, where a matrix is an array of dimension 2 representing the action of a linear operator over a vector expressed as an array of dimension 1.

Tensor basic operations

This section presents the basic tensor calculus operations used in the entire manuscript and the most common object manipulations. In the first part, we establish an order for performing tensor manipulations. The second part presents the operations among tensors, regarded as multidimensional arrays, by analogy with the corresponding ones defined for matrices or vectors. Remark that tensor operations and manipulations can be performed even when the tensor is expressed in a compact form. We will provide further details on how to perform certain operations or manipulations in compressed format, once the decompositions are presented.

Tensor reshaping

Many algorithms benefit or rely on manipulations of the multidimensional array representing tensors. In the classical matrix context, it is common to select from a given matrix A ∈ R n 1 ×n 2 a column c j ∈ R n 1 or a row r i ∈ R n 2 , that is r i = A(i, :) and c j = A(:, j), where the symbol ':' stands for taking all the entries along the row or column mode, using the python or MATLAB notation. This operation of extracting from a 2-dimensional array a 1-dimensional array is generalized to the tensors by slices and fibres. k] is a (d -1) order tensor representing the i k -th slice along mode k whose generic element writes

Definition 1.2.1. Let a ∈ R n 1 ו••×n d be a d order tensor, then a [i k ,
a [i k ,k] (i 1 , . . . , i k-1 , i k+1 , . . . , i d ) = a(i 1 , . . . , i k-1 , i k , i k+1 , . . . , i d )
for every i j ∈ {1, . . . , n j } and j ∈ {1, . . . , d}. This concept is easily generalized to multiple mode slices. In particular the

(i 1 , . . . , i k-1 , i k+1 , . . . , i d ) slice along modes (1, . . . , k -1, k + 1, . . . , d) is called mode k fibre.
If the order is d = 3, then the (j, k) slice along modes (2, 3) is said column fibre, while the (i, k) slice along modes [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF] is the row fibre and the (i, j) slice along modes [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF] is the tube fibre.

Since multimode slices will not appear in this document, we do not introduce a specific notation, but we propose an example to present practically the idea of fibres and slices for order 3 tensors. The following example stresses that fibres can be seen as an analogue of matrix rows and columns Example 1.2.1. Consider the tensor a ∈ R 3×3×2 depicted in Figure 1.1C. The first and second slices with respect to mode 3 are the (3 × 3) matrices a [1,3]

=    5 3 1 2 4 9 6 8 7   
and a [2,3]

=    1 8 4 9 2 2 7 1 6    .
The (1, 1) column fibre corresponds to the vector [START_REF] Atchley | Multivariate Statistical Methods: Among-Groups Covariation[END_REF][START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF][START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF] , which is also the first column of the slice a [1,3] , the (1, 1) row fibre corresponds to the vector [5, 3, 1] , i.e., the first row of a [1,3] , and the (1, 1) tube fibre to the vector [START_REF] Atchley | Multivariate Statistical Methods: Among-Groups Covariation[END_REF][START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF] .

Another common practice in the tensor framework is reorganizing the tensors into vectors or matrices. To do so, the fibres along a specific mode are arranged into a vector, defining the vectorization of a tensor, or as columns of a matrix, constructing the tensor matricization or unfolding. For both the vectorization and the matricization, we follow the order proposed in [START_REF] Kolda | Tensor Decompositions and Applications[END_REF].

Definition 1.2.2. Let N k = {1, . . . , n k } be a subset of N for every k ∈ {1, . . . , d}, define the function φ : N 1 × • • • × N d → N such that φ(i 1 , . . . , i d ) = 1 + d =1 (i -1)m with m = -1 h=1 n h
The image of the function φ is equal to the set N = {1, . . . , n} where n is equal to the mode size product, that is n = d h=1 n h . Henceforth the expression i 1 , . . . , i d denotes the image of (i 1 , . . . , i d ) through φ, i.e., i 1 , . . . ,

i d = φ(i 1 , . . . , i d ). Let a ∈ R n 1 ו••×n d be an order-d, its vectorization vec(a) ∈ R n with n is such that vec(a)(j) = a(i 1 , . . . , i d ) with j = φ(i 1 , . . . , i d ).
The vectorization is also used to reshape a tensor into a matrix, applying it to tensor slices. (k) denotes the mode k matricization or k unfolding whose (i, j) element writes

Definition 1.2.3. Let a ∈ R n 1 ו••×n d be a d order tensor and let n =k = (n 1 • • • n d )/n k , then the (n k , n =k ) matrix A
A (k) (i, j) = a(j 1 , . . . , j k-1 , i, j k+1 , . . . , j d )
for every i ∈ {1, . . . , n k } and j = j 1 , . . . , j k-1 , j k+1 , . . . , j d . This definition is extended to multimode matricization, using the index linear combination for computing the row and column index consistently.

From now on, we denote the matricization of a tensor along a certain mode by the same letter in uppercase with the mode index between round brackets as superscript, as proposed in [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF]. Notice that, generally speaking, the fibres arrangement order is not significant as long as it is consistent throughout the entire work. We use the previous examples to illustrate concretely the tensor vectorization and matricization.

TENSOR BASIC OPERATIONS

Example 1.2.2. Consider the tensor a ∈ R 3×3×2 depicted in Figure 1.1C. The vectorization of a stacks vertically all the row fibres fixing (i 2 , i 3 ) for increasing values of i 2 and i 3 in the given order. The vectorization of a writes with the row fibres as

vec(a) =       r (1,1) r (2,1)
. . .

r (3,2)      
where The matricization of a with respect to mode 1 is A (1) ∈ R 3×6 , obtained stacking one next to the other all the row fibres, i.e.,

r (i 2 ,i 3 ) ∈ R 3 represents the (i 2 , i 3 ) row fibre, that is r (i 2 ,i 3 ) = a(:, i 2 , i 3 ) for i 2 ∈ {1,
A (1) = [r (1,1) , r (2,1) , • • • , r (3,2) ]
with r (i 2 ,i 3 ) ∈ R 3 represents the (i 2 , i 3 ) row fibre. The final result of the matricization with respect to mode 1 is The tensor matricization is connected with the Kronecker product, another relevant operation in the tensor framework.

A (1) =    5

Definition 1.2.4. The Kronecker product between matrices

A ∈ R n 1 ×n 2 and B ∈ R m 1 ×m 2 is A ⊗ K B a (n 1 m 1 ) × (n 2 m 2 ) matrix such that A ⊗ K B =     A(1, 1)B • • • A(1, n 2 )B . . . . . . . . . A(n 1 , 1)B • • • A(n 1 , n 2 )B     .
With an abuse of notation, the Kronecker product of two vectors

a ∈ R n and b ∈ R m is the vector a ⊗ K b ∈ R nm defined as a ⊗ K b =     a(1)b . . . a(n)b     . By construction the k-th element of a ⊗ K b is equal to a(i)b(j) with k = j + m(i -1) = ij.
From this last remark, it follows this tiny but useful lemma.

Lemma 1.2.3. Given an elementary tensor

a ∈ R n 1 ו••×n d , expressed as a = v 1 ⊗ • • • ⊗ v d
with v ∈ R n , then its matricization with respect to mode writes

A ( ) = v ⊗ v d ⊗ K • • • ⊗ K v +1 ⊗ K v -1 ⊗ K • • • ⊗ K v 1 for every ∈ {1, . . . , d}
Proof. Since this is a classical result in the tensor theory, we prove it just for the mode 1 matricization under the hypothesis d = 3. By construction the (i 1 , i 2 i 3 )-th element of A (1) is

A (1) (i 1 , i 2 i 3 ) = v 1 (i 1 )v 2 (i 2 )v 3 (i 3 ). By construction v 2 (i 2 )v 3 (i 3 ) is the (i 2 i 3 )-th element of v 3 ⊗ K v 2 , i.e., A (1) (i 1 , i 2 i 3 ) = v 1 (i 1 ) v 3 ⊗ K v 2 (i 3 i 2 ).
and thanks to the definition of the tensor product, it follows

A (1) (i 1 , i 2 i 3 ) = v 1 (i 1 ) v 3 ⊗ K v 2 (i 3 i 2 ) = v 1 ⊗ (v 3 ⊗ K v 2 ) (i 1 , i 3 i 2 ),
that is the thesis.

Another property of the Kronecker product we use in the following sections is called mixed-product property. Let A i ∈ R n i ×m i , B i ∈ R m i ×p i for i ∈ {1, 2}, then the Kronecker product of (A 1 B 1 ) and (A 2 B 2 ) is expressed as the matrix product of the Kronecker product of A i with B i , that is

(A 1 B 1 ) ⊗ K (A 2 B 2 ) = (A 1 ⊗ K A 2 )(B 1 ⊗ K B 2 ).
(1.1)

Tensor calculus

As previously mentioned, tensors in this document are thought of as multidimensional arrays, belonging to the space R n 1 ו••×n d , which is a linear space. Consequently, the tensor sum and the product with real scalars are linearly defined component-wise.

The first operations presented involves tensors and matrices. Let

M k ∈ R m×n k and a ∈ R n 1 ו••×n k ו••
×n d be a matrix and a tensor respectively, then the matrix-tensor product between M k and a along the mode k is denoted by

(M k × k a) ∈ R n 1 ו••×mו••×n d , whose (i 1 , . . . , j k , . . . , i d )-th element is M k × k a (i 1 , . . . , j k , . . . , i d ) = n k i k =1 M k (j k , i k )a(i 1 , . . . , i k , . . . , i d ).

TENSOR BASIC OPERATIONS

Given d matrices M k ∈ R m k ×n k , then (M 1 , . . . , M d )a denotes the product of the tuple of matrices (M 1 , . . . , M d ) and the tensor a, whose general element writes

(M 1 , . . . , M d )a (j 1 , . . . , j d ) = M 1 × 1 . . . M d × d a (j 1 , . . . , j d )) = n 1 ,...,n d i 1 ,...,i d =1 M 1 (j 1 , i 1 ) . . . M d (j d , i d )a(i 1 , . . . , i k , . . . , i d )
for j k ∈ {1, . . . , n k } and k ∈ {1, . . . , d}. Remark that this operation is order-independent. The matricization of a tensor expressed as a matrix-tensor product satisfies the following lemma, which will be widely used in Chapter 4. Lemma 1.2.4. [START_REF] Kolda | Multilinear Operators for Higher-order Decompositions[END_REF]Proposition 3.7] Let a ∈ R n 1 ו••×n d be a tensor and let M k ∈ R m k ×n k be a matrix for every k ∈ {1, . . . , d}, then the matricization of b = (M 1 , . . . , M d )a with respect to mode is

B ( ) = M A ( ) (M d ⊗ K • • • ⊗ K M +1 ⊗ K M -1 ⊗ K • • • ⊗ K M 1 )
for every ∈ {1, . . . , d}.

In the tensor context, the tensor contraction generalizes the matrix-vector product.

Let A ∈ R (n 1 ×m 1 )ו••×(n d ×m d ) and b ∈ R m 1 ו••×m d be a tensor operator and a tensor, the tensor contraction of A and b is (A • b) ∈ R n 1 ו••×n d defined as (A • b)(i 1 , . . . i d ) = m 1 ,...,m d j 1 ,...,j d =1 A(i 1 , j 1 , . . . , i d , j d )b(j 1 , . . . , j d ).
The tensor contraction is defined for any tensor since any tensor can be regarded as a tensor operator. Therefore given two tensors a ∈ R n

c ∈ R n 1 ו••×n d 1 ×m 1 ו••×m d 2 is the order-(d 1 + d 2 -2) tensor from the contraction (a • h k b), then its (i 1 , . . . , i h-1 , i h+1 , . . . , i d 1 , j 1 . . . , j h-1 , j h+1 , . . . , j d 2 )-th element is c such that c = (a • h k b)(i 1 , . . . , i h-1 , i h+1 , . . . , i d 1 , j 1 . . . , j k-1 , j k+1 , . . . , j d 2 ) = p =1 a(i 1 , . . . , , . . . , i d 1 )b(j 1 , . . . , , . . . , j d 2 ). (1.2)
The contraction between tensors can be extended to more modes, replacing in Equation (1.2) the single index h and k by sets of indices. To shorten the notation, we omit the bullet symbol and the mode indices, when the modes to contract are clear from the context, for example given a ∈ R n 1 ×p×m 1 and b ∈ R p×n 2 ×m 2 , then the contraction a • b is among the second mode of a with the first one of b, since they are the only modes having the same dimension. The tensor contraction enables us to compute mulitilinear endomorphism powers, exactly as in the matrix framework the matrix-vector product is used to compute the matrix powers representing the composition of the associated linear endomorphism. Let A : R n

A ∈ R (n 1 ×n 1 )ו••×(n d ×n d ) be its representation with respect to the canonical basis of R n 1 ו••×n d . Then B ∈ R (n 1 ×n 1 )ו••×(n d ×n d )
is the tensor representing with respect to the canonical basis A 2 , the composition of A with itself, whose element B(i 1 , j 1 , . . . , i d , j d ) writes

B(i 1 , j 1 , . . . , i d , j d ) = A • H L H R A (i 1 , j 1 , . . . , i d , j d ) = n 1 ,...,n d k 1 ,...,k d =1 A(i 1 , k 1 , . . . , i d , k d )A(k 1 , j 1 , . . . , k d , j d )
with H L = {2, 4, . . . , 2d} and H R = {1, 3, . . . , 2d -1}. From this, we recursively obtain the tensor associated with A h for h ∈ N. Remark that the contraction enables us to express also the matrix-tensor product since matrices are in fact order-2 tensors. Given

M k ∈ R m×n k and a ∈ R n 1 ו••×n k ו••×n d , then (M k × k a) the matrix-tensor product along mode k is expressed as a contraction of M k and a as (M k × k a) = M k • 2 k a
by its definition. As already stated in Section 1.1, the tensor space R n 1 ו••×n d is a linear space and it is usually endowed by an inner product, which extends the inner product defined for the simple linear space R n . Let a, b ∈ R n 1 ו••×n d be two order-d tensors, then a, b ∈ R is their inner product defined as

a, b = n 1 ,...,n d i 1 ,...,i d =1 a(i 1 , . . . , i d )b(i 1 , . . . , i d ).
Remark that by construction the inner product is also expressed as the tensor contraction over all modes, that is as a, b = a • b. As in the classical linear algebra framework, the tensor inner product on R n 1 ו••×n d induces a norm on the same space defined as a = a, a for every a ∈ R n 1 ו••×n d . By construction, this norm extends the idea of the vector L2norm and of the Frobenius matrix to tensors.

Since tensors represent also multilinear operators, it is worthwhile defining the 2norm induced by the vector L2-norm. Let A ∈ R (n 1 ×m 1 )ו••×(n d ×m d ) be an order-2d tensor representing a multilinear operator

A from R m 1 ו••×m d to R n 1 ו••×n d with respect to the canonical basis, then A 2 is the 2-norm of A defined as A 2 = max Ax x : x ∈ R m 1 ו••×m d \ {0} .

TENSOR DECOMPOSITION

Tensor decomposition

As pointed out in Section 1.1, after fixing a basis of R n 1 ⊗ • • • ⊗ R n d , we can identify an element of the tensor linear space with the multidimensional array representing it. Let {e (1)

i 1 ⊗ • • • ⊗ e (d) i d } be the canonical basis of R n 1 ⊗ • • • ⊗ R n d , with an abuse of notation every tensor a ∈ R n 1 ⊗ • • • ⊗ R n d is expressed as a = n 1 ,...,n d i 1 ,...,i d =1 a(i 1 , . . . , i d )e (1)
i 1 ⊗ • • • ⊗ e (d) i d (1.3)
where a ∈ R n 1 ו••×n d is the multidimensional array storing its entries. The expression of tensors given in Equation (1.3) is usually referred to as dense or canonical format. This representation is affected by the so-called 'curse of dimensionality'. Indeed if storing one entry of an array costs one unit of memory, then storing a requests O(n d ) units of memory with n = max{n 1 , . . . , n d }, which scales exponentially with the order of the tensor. To face this significant practical limit, throughout the years, different compression techniques have been proposed. Each method can be regarded firstly as an exact way of decomposing a tensor and secondly as a procedure to obtain an approximation of it, which satisfies a certain criterion.

In the following sections, we describe two main decomposition and approximation techniques, the Tucker and the Tensor-Train one. For both methods, we present the theoretical and computational aspects. It is convenient to denote by O(n × m) the set of matrices of size (n, m) with orthogonal columns, briefly referred by orthogonal matrix set.

Tucker decomposition

As clearly reconstructed in [START_REF] Kolda | Tensor Decompositions and Applications[END_REF], the history of the Tucker decomposition dates back to 1963 when Tucker proposed it in [START_REF] Tucker | Implications of factor analysis of three way matrices for measurements of change[END_REF], even if it was already in Hitchcock paper of 1927 [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF]. It was further improved and more detailed described in [START_REF] Levin | Three-Mode Factor Analysis[END_REF][START_REF] Tucker | The extension of factor analysis to three-dimensional matrices[END_REF][START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF]. It has been renamed many times over the years, as Three-mode factor analysis, Three-mode factor PCA, n-mode PCA, n-mode SVD, see [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF][START_REF] Kapteyn | An approach to n-mode components analysis[END_REF][START_REF] Kroonenberg | Principal Component Analysis of three modes data by means of Alternating Least Square algorithms[END_REF][START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF][START_REF] Vasilescu | Multilinear Analysis of Image Ensembles: TensorFaces[END_REF].

Definition 1.3.1. Let a ∈ R n 1 ו••×n d be a d order tensor, its Tucker decomposition writes a = r 1 i 1 =1 . . . r d i d =1 c(i 1 , . . . , i d ) u (1) i 1 ⊗ • • • ⊗ u (d) i d
where the multidimensional array c is the core tensor, {u

( ) i 1 , . . . , u ( )
r } is an orthonormal basis of a R n subspace of size r , which is equal the rank of A ( ) , the matricization along mode of a for every ∈ {1, . . . , d}. We refer to the tuple (r 1 , . . . , r d ) as the multilinear rank of a, as stated in [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF]. Compactly, the orthogonal Tucker decomposition is expressed as a = (U 1 , . . . , U d )c

where the orthogonal matrix U ∈ O(n × r ) has u

( ) i
as -th column. The matrix U is called -th factor matrix for every ∈ {1, . . . , d}.

The idea of the Tucker decomposition is expressing a tensor a of multilinear rank (r 1 , . . . , r d ) in an orthonormal basis {u (1)

i 1 ⊗ • • • ⊗ u (d) i d } for i k ∈ {1, . . . , r k } and k ∈ {1, . . . , d}. Consequently, if we denote by U k the subspace of dimension r k of R n k spanned by {u (k) 1 , . . . , u (k) r k }, then tensor a belongs to U 1 ⊗ • • • ⊗ U d .
Starting from this Tucker decomposition, we formulate an approximation problem as follows. Given a tensor a ∈ R n

Computational aspects

While PCA approximation problem has an optimal solution, guaranteed by the Eckart-Young theorem [START_REF] Eckart | The approximation of one matrix by another of lower rank[END_REF] and provided by the Singular Value Decomposition(SVD), the Tucker approximation problem has not a known closed formula for the solution. In this section, we describe the most popular algorithms for computing the Tucker decomposition and approximation Nowadays the most common algorithm for computing the Tucker decomposition is the High Order Singular Value Decomposition (HOSVD), proposed in [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF] and displayed in Algorithm 1. Given an input tensor a, of which the Tucker decomposition is requested, iteratively HOSVD matricizes it along mode i and computes its left singular vectors arranged in matrix U i ∈ O(n i × r i ) where r i = rank(A (i) ), for i ∈ {1, . . . , d}, as in Algorithm 1 line 3. Thanks to the orthonormality of U i , that is U i U i = I r i , from the definition of Tucker decomposition, it follows that the core tensor is actually given by c = (U 1 , . . . , U d )a, as computed in Algorithm 1 line 5. The factor matrices are the left singular matrices U k for every k ∈ {1, . . . , d}.

To address the Tucker approximation problem, the authors of [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF] proposed the Truncated HOSVD (T-HOSVD) method, given in Algorithm 2, a slightly different version of HOSVD. T-HOSVD takes the multilinear rank (r 1 , . . . , r d ) as input parameter together with the tensor a to approximate. In the truncated version of HOSVD, the full SVD is replaced by a Truncated SVD (T-SVD). Namely, at line 3 of Algorithm 2, we compute

1.3. TENSOR DECOMPOSITION Algorithm 1 (U 1 , . . . , U d ), c = HOSVD(a) 1: input: a ∈ R n 1 ו••×n d 2: for i = 1, . . . , d do 3: U i , Σ i , V i = SVD A (i)
SVD of each matricization 4: end for 5: c = (U 1 , . . . , U d )a compute the core tensor 6: return: (U 1 , . . . , U d ), c just the first r i left singular vectors of A (i) , instead of the full decomposition returned in line 3 of Algorithm 1. All the remaining steps of Algorithm 1 and 2 are exactly equal. The approximation provided by T-HOSVD is not the optimal one. In fact, if we denote by â the approximation provided by T-HOSVD of a at multilinear rank (r 1 , . . . , r d ), then the approximation error is bounded by

a -â 2 ≤ d i=1 Σi 2 where Σi (j, j) =    0 j ≤ r i Σ i (j, j) j > r i
where Σ i is the diagonal singular values matrix of A (i) for every i ∈ {1, . . . , d}.

Algorithm 2 ( Û1 , . . . , Ûd ), ĉ = T-HOSVD(a, r)

1: input: a ∈ R n 1 ו••×n d , r ∈ N d 2: for i = 1, . . . , d do 3:
Ûi , Σi , Vi = T-SVD A (i) , r(i) truncated SVD of each matricization 4: end for 5: ĉ = ( Û 1 , . . . , Û d )a compute the core tensor 6: return: ( Û1 , . . . , Ûd ), ĉ

We want to mention, without entering in details, the Sequentially Truncated HOSVD (ST-HOSVD), proposed in [START_REF] Vannieuwenhoven | A New Truncation Strategy for the Higher-Order Singular Value Decomposition[END_REF] to compute the Tucker approximation given a multilinear rank. Differently from T-HOSVD, ST-HOSVD initializes the tensor core by the input one. Then it iteratively updates the tensor core ĉ multiplying it by the transposed truncated left singular vector matrix Û i along mode i, that is, ĉ is updated by Û i × i ĉ. This choice reduces iteratively the size of the matrix decomposed by the truncated SVD, decreasing the computational costs. However not always the approximation error of ST-HOSVD is lower than the T-HOSVD. Moreover, the ST-HOSVD approximation error depends on the matricization order, while the T-HOSVD one is independent. We refer the reader to [START_REF] Vannieuwenhoven | A New Truncation Strategy for the Higher-Order Singular Value Decomposition[END_REF] for a complete ST-HOSVD analysis and comparison with other approximation techniques.

The last method for computing the Tucker approximation is called High Order Orthogonal Iterations (HOOI), proposed in [START_REF] Lathauwer | On the Best Rank-1 and Rank-(R 1 , R 2 , . . . , R N ) approximation of high order tensors[END_REF]. The HOOI is an Alternating Least Squares (ALS) bases algorithm, which outcompeted the other ALS algorithms proposed in [START_REF] Kroonenberg | Principal Component Analysis of three modes data by means of Alternating Least Square algorithms[END_REF] Algorithm 3 ( Û1 , . . . , Ûd ), ĉ = HOOI(a, r, δ) Ûi , Σi , Vi = T-SVD C (i) , r(i) and in [START_REF] Kapteyn | An approach to n-mode components analysis[END_REF]. The ALS part aims to minimize the projection error along each tensor mode. Indeed, as previously highlighted, finding the best Tucker approximation of a at multilinear rank (r 1 , . . . , r d ) can be reformulated as finding the subspaces U k of R n k of dimension r k whose tensor product minimizes the projection error, that is a -( Û1 , . . . , Ûd )ĉ where Ûk ∈ O(n k × r k ) columns form an orthogonal basis of U k . Elaborating the square of the projection error, we get that

1: input: a ∈ R n 1 ו••×n d , r ∈ N d , δ ∈ R + 2: ( Û1 , . . . ,
a -( Û1 , . . . , Ûd )ĉ 2 = a 2 -( Û 1 , . . . , Û d )a 2 ,
we refer the reader to [START_REF] Lathauwer | On the Best Rank-1 and Rank-(R 1 , R 2 , . . . , R N ) approximation of high order tensors[END_REF][START_REF] Kolda | Tensor Decompositions and Applications[END_REF] for a complete sequence of computations. Since the norm of a is fixed, to reduce the projection error we have to maximize the term ( Û 1 , . . . , Û d )a . In particular, for each mode i the HOOI algorithm solves the maximization problem Ûi = argmax

V i ∈O(n i ×r i ) ( Û 1 , . . . , Û i-1 , V i , Û i+1 , . . . , Û d )a .
Thanks to Corollary 1.2.4, this last equation cab also be written as Ûi = argmax

V i ∈O(n i ×r i ) V i A (i) ( Ûd ⊗ K • • • ⊗ K Ûi+1 ⊗ K Ûi-1 ⊗ K • • • ⊗ K Û1 ) . (1.4)
and it is solved by the first r i left singular vectors from the SVD of A (i) for every i ∈ {1, . . . , d}. The local maximization problem stated in Equation (1.4) is repeatedly updated and solved for every mode i, until the projection error a -( Û1 , . . . , Ûd )ĉ gets smaller than the given tolerance threshold δ ∈ R + . For simplicity in Algorithm 3, we sketch the HOOI version proposed in [START_REF] Kolda | Tensor Decompositions and Applications[END_REF].

TENSOR DECOMPOSITION

Memory requirement

At the beginning of Section 1.3, we pointed out that storing an order-d tensor in dense format requests O(n d ) of memory units where n is the maximal mode dimension. If the same tensor is expressed in the Tucker format by a tuple of d matrices and by a core tensor of order d, then the storing cost is O(dnr + r d ) units of memory where r is the maximal multilinear rank component. Indeed storing the entire core tensor needs O(r d ) units of memory, while storing one of the d factor matrix costs O(nr) units of memory. Remark that the Tucker approximation does not really overcome the curse of dimensionality, since the storing costs of the core tensor still grow exponentially with the order of the tensor. Therefore other compression techniques have been presented as the Hierarchical Tucker [START_REF] Grasedyck | Hierarchical Singular Value Decomposition of Tensors[END_REF] or the more promising Tensor-Train [START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF], presented in details in the following section.

Tensor Train decomposition

The Tensor-Train (TT) decomposition is a representation that overcomes the curse of dimensionality, since the memory requirement to store a tensor in TT-format grows linearly with the order d. It was introduced in [START_REF] Oseledets | Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions[END_REF], almost at the same time of the Hierarchical Tucker decomposition [START_REF] Grasedyck | Hierarchical Singular Value Decomposition of Tensors[END_REF], as a mathematical formulation of the matrix product states [START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF][START_REF] Gelß | The Tensor-Train Format and Its Applications[END_REF] concept used in quantum physics. The TT-decomposition was described in detail in [START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF]. The key idea of Tensor Train is expressing a tensor of order d as the contraction of d tensors of order 3, which leads to a significant saving in memory, as illustrated after.

Definition 1.3.2. Let a ∈ R n 1 ו••×n d be an order-d tensor, its Tensor-Train decomposition writes a = a 1 a 2 • • • a d ,
where

a k ∈ R r k-1 ×n k ×r k is called k-th TT-core for k ∈ {1, . . . , d}, with r 0 = r d = 1. The value r k is the k-th TT-rank.
Notice that a 1 ∈ R r 0 ×n 1 ×r 1 and a d ∈ R r d-1 ×n d ×r d reduce essentially to matrices, but for the notation consistency we represent them as tensor. Henceforth the k-th TT-core of a tensor is denoted by the same bold letter underlined with a subscript k. If a is expressed in TT-format, then its (i 1 , . . . , i d )-th element writes

a(i 1 , . . . , i d ) = r 0 ,...,r d j 0 ,...,j d =1 a 1 (j 0 , i 1 , j 1 )a 2 (j 1 , i 2 , j 2 ) . . . a d-1 (j d-2 , i d-1 , j d-1 )a d (j d-1 , i d , j d ).
Remark that the index j 0 and j d play no role since they are always equal to 1; they are added for completeness. Given an index i k ∈ {1, . . . , n k }, we denote the i k -th matrix slice of a k with respect to mode 2 by

A k (i k ), i.e., A k (i k ) = a [i k ,2] k
. Then each element of tensor a expressed in TT-format is given by the product of d matrices, i.e.,

a(i 1 , . . . , i d ) = A 1 (i 1 ) • • • A d (i d ) with A k (i k ) ∈ R r k-1 ×r k for every i k ∈ {1, . . . , n k } and k ∈ {2, . . . , d-1}, while A 1 (i 1 ) ∈ R 1×r 1 and A d (i d ) ∈ R r d-1 ×1
. Remark that A 1 (i 1 ) and A d (i d ) are vectors, but as before to have a homogeneous notation they can be written as matrices with a single row or column.

The arithmetic operations among tensors in TT-format are performed in a particular way, benefiting from the chosen formalism. Indeed, given two tensors in TT-format

a, b ∈ R n 1 ו••×n d , let a k ∈ R r k-1 ×n k ×r k and b k ∈ R s k-1 ×n k ×s k be their k-th TT-core respec- tively, the k-th TT-core of c = (a + b) is c k ∈ R (r k-1 +s k-1 )×n k ×(r k +s k ) defined block-wise as C k (i k ) = A k (i k ) 0 0 B k (i k ) (1.5)
where 0 denotes the matrix filled with zeros of the appropriate size, for every i k ∈ {1, . . . , n k } and k ∈ {2, . . . , d -1}. The first and last TT-core of c = (a + b) are c 1 ∈ R 1×n 1 ×(r 1 +s 1 ) and c d ∈ R (r d-1 +s d-1 )×n d ×1 , that can be written as

C 1 (i 1 ) = A 1 (i 1 ) B 1 (i 1 ) and C d (i d ) = A d (i d ) B d (i d )
with i 1 ∈ {1, . . . , n 1 } and i d ∈ {1, . . . , n d }. Finally, we verify that by construction the (i 1 , . . . , i d )-th element of c is equal to the sum of the corresponding element of (a and b), that is

c(i 1 , . . . , i d ) = A 1 (i 1 ) B 1 (i 1 ) A 2 (i 2 ) 0 0 B 2 (i 2 ) • • • • • • A d (i d ) B d (i d ) = A 1 (i 1 ) • • • A d (i d ) + B 1 (i 1 ) • • • B d (i d ) = a(i 1 , . . . , i d ) + b(i 1 , . . . , i d ) = a + b (i 1 , . . . , i d ).
Similarly let λ ∈ R be a scalar and a ∈ R n 1 ו••×n d be a tensor in TT-format, then the tensor (λa) has the k-th TT-core equal to the corresponding one of a for k ∈ {2, . . . , d}, while the 1-st TT-core is equal to the first TT-core of a multiplied by λ. Thus, the (i 1 , . . . , i d )-th element of (λa) gets

λa (i 1 , . . . , i d ) = λA 1 (i 1 ) A 2 (i 2 ) • • • A d (i d ) = λ A 1 (i 1 ) • • • A d (i d ) = λa(i 1 , . . . , i d ).
We report now the formulas provided in [START_REF] Oseledets | Tensor-Train Decomposition[END_REF] for other operations, that not necessary correspond to the optimal algorithmic implementation. Also, the computation of the inner product benefits from the TT-formalism. Given a, b ∈ R n 1 ו••×n d two tensors of order d in TT-format, their inner product writes

a, b = n 1 ,...,n d i 1 ,...,i d =1 a(i 1 , . . . , i d )b(i 1 , . . . , i d ) = n 1 ,...,n d i 1 ,...,i d =1 A 1 (i 1 ) • • • A d (i d ) B 1 (i 1 ) • • • B d (i d ) .

TENSOR DECOMPOSITION

If the product of the TT-cores of a and b of this last equation is regarded as a degenerated case of Kronecker product among matrices of size [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF], that are scalars, then we get

a, b = n 1 ,...,n d i 1 ,...,i d =1 A 1 (i 1 ) • • • A d (i d ) B 1 (i 1 ) • • • B d (i d ) = n 1 ,...,n d i 1 ,...,i d =1 A 1 (i 1 ) • • • A d (i d ) ⊗ K B 1 (i 1 ) • • • B d (i d ) .
Applying the mixed-product property of the Kronecker product, stated in Equation (1.1) in the last equation, it becomes

a, b = n 1 ,...,n d i 1 ,...,i d =1 A 1 (i 1 ) • • • A d (i d ) ⊗ K B 1 (i 1 ) • • • B d (i d ) = n 1 ,...,n d i 1 ,...,i d =1 A 1 (i 1 ) ⊗ K B 1 (i 1 ) • • • A d (i d ) ⊗ K B d (i d ) .
To compute the scalar product of two tensors in TT-format we compute the Kronecker product of their TT-cores, multiply them and sum over all the indexes of each mode.

The TT-formalism enables us to express compactly also tensor operators between tensor spaces.

Definition 1.3.3. Let A ∈ R (n 1 ×m 1 )ו••×(n d ×m d ) be a

tensor operator, its Tensor-Train decomposition writes

A = A 1 • • • A d ,
where A k ∈ R r k-1 ×n k ×m k ×r k , is its k-th TT-core, with r 0 = r d = 1. Henceforth we call TTmatrix a tensor in TT-format representing a tensor operator; while we refer by TT-vector to a tensor regarded as an element of a linear tensor space.

The (i 1 , j 1 , . . . , i d , j d )-th element of A a TT-matrix writes

A(i 1 , j 1 , . . . , i d , j d ) = r 0 ,...,r d h 0 ,...,h d =1 A 1 (h 0 , i 1 , j 1 , h 1 ) • • • A d (h d-1 , i d , j d , h d ). Let A k (i k , j k ) ∈ R r k-1 ×r k be the (i k , j k )-th slice with respect to mode (2, 3) of A k for every i k , ∈ {1, . . . , n k }, j k ∈ {1, . . . , m k } and k ∈ {1, . . . , d}.
Then the last equation is equivalently expressed as

A(i 1 , j 1 , . . . , i d , j d ) = A 1 (i 1 , j 1 ) • • • A d (i d , j d ).
As remarked in Section 1.2, the inner product is the contraction over all the indexes of every mode of two tensors. Thus, as the inner product can be computed smartly if the tensors are expressed in TT-format, also the contraction among a TT-matrix and a TTvector, that is the action of a multilinear operator over a tensor in the domain, benefits from the TT-format. Let A ∈ R (n 1 ×m 1 )ו••×(n d ×m d ) be a TT-matrix and b ∈ R m 1 ו••×m d be a TT-vector, the (i 1 , . . . , i d )-th element of their contraction writes

Ab (i 1 , . . . , i d ) = m 1 ,...,m d j 1 ,...,j d =1 A(i 1 , j 1 , . . . , i d , j d )b(j 1 , . . . , j d ) = m 1 ,••• ,m d j 1 ,...,j d =1 A 1 (i 1 , j 1 ) • • • A d (i d , j d ) B 1 (j 1 ) • • • B d (j d ) .
As in the case of the inner product, if this last product is regarded as a degenerate Kronecker product, then the mixed-product property stated in (1.1) applies and leads to

Ab (i 1 , . . . , i d ) = m 1 ,...,m d j 1 ,...,j d =1 A 1 (i 1 , j 1 ) • • • A d (i d , j d ) B 1 (j 1 ) • • • B d (j d ) = m 1 ,...,m d j 1 ,...,j d =1 A 1 (i 1 , j 1 ) • • • A d (i d , j d ) ⊗ K B 1 (j 1 ) • • • B d (j d ) = m 1 ,...,m d j 1 ,...,j d =1 A 1 (i 1 , j 1 ) ⊗ K B 1 (j 1 ) • • • A d (i d , j d ) ⊗ K B d (j d ) = m 1 j 1 =1 A 1 (i 1 , j 1 ) ⊗ K B 1 (j 1 ) • • • m d j d =1 A d (i d , j d ) ⊗ K B d (j d ) . Said differently, the k-th TT-core of c = (Ab) ∈ R n 1 ו••×n d is c k ∈ R (r k-1 s k-1 )×n k ×(r k s k ) such that for every i k ∈ {1, . . . , n k } C k (i k ) = m k j k =1 A k (i k , j k ) ⊗ K B k (j k ) (1.6)
for every k ∈ {1, . . . , d}.

Memory requirement

The purpose of the TT-format declared by their authors in [START_REF] Oseledets | Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions[END_REF][START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF] is getting rid of the curse of dimensionality, which affects other decomposition, for example, the Tucker one as underlined in Section 1.3.1.2. Given a tensor a ∈ R n 1 ו••×n d , the storage cost in TT-format is O(dnr 2 ), since we need to store d TT-cores requesting O(r 2 n) units of memory, under the assumptions that n and r as the maximum of the mode size and of the TT-ranks respectively. Similarly storing a tensor operator requests O(dmnr 2 ) memory units, where n and m are the maxima of the mode size of the domain and the image tensor space, while r is the maximum of the TT-ranks. In this case the memory footprint growths linearly with the tensor order, but to keep this formalism advantageous the value r has to stay bounded and small. Since the TT-rank determines the benefit in storing a tensor in TT-format, usually knowing the maximal TT-rank is sufficient to get an idea of the benefit. However, to be more accurate, we introduce the compression ratio measure.

TENSOR DECOMPOSITION

Let a ∈ R n 1 ו••×n d be a tensor in TT-format, then the compression ratio is the ratio among the storage cost of a in TT-format over the dense format storage cost, i.e.,

d i=1 r i-1 n i r i d j=1 n j (1.7)
where r i is the i-th TT-rank of a.

A first drawback of the TT-format appears with the addition of two TT-tensors. Indeed given two TT-tensors a and b with k-th TT-rank r k and s k respectively, then the k-th TT-rank of (a + b) is equal to r k + s k , as shown in Equation (1.5). Consequently, when two tensors in TT-format are summed, the TT-ranks of their sum grow as the sum of the TT-ranks. In particular if (2a) is computed as (a + a), then its TT-ranks are the double of the TT-ranks of the tensor obtained multiplying a by 2. Similarly when a TT-matrix A is contracted with a TT-vector b, then their TT-ranks are multiplied. If r k and s k are the k-th TT-rank of A and b, then the k-th TT-rank of (Ab) is (r k s k ), as highlighted in Equation (1.6). In the following section dedicated to computation aspects, we discuss a solution to address this issue.

Tensor-Train compression

The TT-rank growth is a crucial point in the implementation of algorithms using TTtensors. It may lead to memory deficiencies and prevent the calculation to complete. To address this issue, a rounding algorithm to reduce the TT-rank was introduced in [START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF], together with the algorithm that converts a dense format tensor into TT-format. We present firstly this latter one, sketched in Algorithm 4. To convert a dense tensor a in TT-format we prescribe a representation accuracy threshold δ ∈ R + which bounds the relative error, i.e., a -ã a ≤ δ where ã is the representation of a in TT-format. To ensure the relative error accuracy, we define δ scaling δ by the norm of the input tensor and the problem order, as

δ = δ ||a|| √ d -1
As first step, the input tensor a is matricized along the first mode, then A (1) of size (n 1 , n

1: input: a ∈ R n 1 ו••×n d in dense format, δ ∈ R + 2: δ = δ
a i = reshape( Ûi , [r i-1 , n i , r i ]
) where r i is the number of columns of Ûi

C i+1 = reshape C, [r i n i+1 , p] 11: end for 12: a d = C
define the last TT-core 13: return: a 1 , . . . , a d As mentioned in Section 1.3.2.1, many basic operations among tensors in TT-format induce growth in the TT-ranks, reducing the storing benefit of this formalism. To address this flaw, the TT-rounding algorithm in [START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF] it was introduced a , which takes in input a tensor a in TT-format and an accuracy threshold δ ∈ R + and returns â a tensor in TT-format such that a -â a ≤ δ.

As in Algorithm 4, the first step is scaling the rounding accuracy by the square root of the order minus one and multiplying it by the norm of the input tensor. Then, from the last to the second, each TT-core is matricized along mode 1, transposed and factorized through the QR algorithm, see Algorithm 5, line 4. The row orthogonal factor Q i is reshaped to define the i-th TT-core a i , while the R factor is used to update the (i -1)-th TT-core a i-1 , see line 7 of Algorithm 5. Indeed, from the QR-factorization theory, the columns of Q i span the same space, that is, carry the same information, as the columns of a i matricized along mode 1 and transposed. Once completed this loop, usually referred to as the right-to-left orthogonalization loop, we proceed with the compression of the TT-cores. From the first to the second-last, each TT-core is matricized along mode 3, transposed to have (r i-1 n i ) rows and decomposed with the truncated SVD given accuracy δ , as depicted in line 11 and 12 of Algorithm 5. Finally the left orthogonal matrix Ûi from the truncated SVD is reshaped to define the new truncated i-th TT-core, while the singular value diagonal matrix times the right orthogonal one ( Σi V i ) are used to update the successive (i + 1)-th TT-core, see lines 13 and 14 of Algorithm 5. As stated in [START_REF] Oseledets | DMRG Approach to Fast Linear Algebra in the TT-Format[END_REF], given an input tensor a ∈ R n 1 ו••×n d , the computational cost, in terms of floating point operations, of the TT-rounding method, described in Algorithm 5 is O(dnr 3 ), where n 1.4. CONCLUDING REMARKS and r are the maximal mode size and TT-rank respectively. Indeed, the cost of each of the d QR-factorization and each truncated SVD is estimated as O(nr 3 ) float point operations, since both are performed over a matrix of size (nr, r) (or its transposed).

To estimate the benefit of applying the TT-rounding over an input TT-vector, we introduce the compression gain measure. Let a ∈ R n 1 ו••×n d be a TT-vector with i-th TT-rank equal to r i , if â is its approximation obtained from the TT-rounding at accuracy δ ∈ R + with s i the i-th TT-rank, then the compression gain of the TT-rounding is equal to the ratio of the compression ratio, i.e.,

q = d i=1 r i-1 n i r i d j=1 s j-1 n j s j .
(1.8)

To better understand the meaning of the compression gain, we consider the following example. Let a ∈ R n 1 ו••×n d be a TT-vector and â its approximation at accuracy δ from the TT-rounding, then if the compression gain is equal to q ∈ Q, it means that after the compression, storing â needs 1/q of the memory we would need to store a.

Concluding remarks

The purpose of this chapter was to give an overview presenting tensors, their main operations, and the decomposition techniques, on which the following chapters will rely. Tensors carry multiple meanings, which together with the sometimes tangled notation, may easily confuse the reader.

In particular Section 1.1 of this chapter aims to clarify three different natures of tensors, i.e., tensors as multidimensional arrays, as an abstract element of tensor product of linear spaces, as multilinear operator among linear spaces. We emphasized that also in the classical linear algebra framework matrices are regarded from these three different viewpoints, depending on the purpose. Section 1.2 presents tensor slices and fibres, introducing a specific notation, which we will simplify in other chapters of this manuscript. The tensor matricization with its performing order, and its connection with the Kronecker product are described as well. In this same section, tensor operations, from the most common, such as the tensor sum or the product of tensor and scalars, to more peculiar ones, such as the tensor product or the tensor contraction, are presented in detail. We underline as much as possible that many tensor operations, such as the matrix-times-tensor one or the inner product, are actually contractions, to which we assign a name because of their importance in the tensor theory.

The second part illustrates two tensor decomposition techniques, used in the chapters that follow. We describe firstly the Tucker model and the Tucker approximation problem, relating it with the classical matrix best rank r approximation problem. In Section 1.3.1, we address the computational key aspects related to the Tucker approximation problem. The best-known algorithms, i.e., HOSVD, T-HOSVD, and HOOI are entirely illustrated and analysed. Moreover, we present the benefit of approximating a tensor by the Tucker Algorithm 5 â1 , . . . , âd = TT-rounding (a 1 , . . . , a d ), δ

1: input: a i ∈ R r i-1 ×n i ×r i for every i ∈ {1, . . . , d}, δ ∈ R + 2: δ = δ √ d-1 a
scale the rounding accuracy δ 3: for i = d, . . . , 2 do 4:

A i = matricize(a i , mode = 1)
matricize each core compute the QR-factorization of the transposed matricization 5:

Q i , R = QR(A i ) set the i-th core equal to its orthogonal factor Q 6: a i = reshape(Q i , [r i-1 , n i , r i ])
move the upper triangular factor R to the previous core 7: A i = matricize(a i , mode = 3) matricize the i-th core compute the truncated SVD of the transposed matricization 12:

a i-1 = a i-1 × 3
Ûi , Σi , Vi = T-SVD(A i , δ ) define the approximated i-th TT-core 13: âi = reshape( Ûi , [s i-1 , n i , s i ]) with s i equal to the number of columns of Ûi move the singular values and the right orthogonal basis to the next TT-core 14:

a i+1 = a i+1 × 1 ( Σi V
i ) 15: end for 16: âd = a d define the last TT-core 17: return: â1 , . . . , âd 1.4. CONCLUDING REMARKS model from the memory storage requirement viewpoint. The following Section 1.3.2 focuses on the Tensor-Train model. After stating its definition, we describe how to benefit from the TT-formalism, when performing some basic tensor operations. It is highlighted that this formalism is suitable also for representing tensors as multilinear operators, becoming consequently one of the most popular tensor decomposition techniques in the numerical linear algebra community. As for the Tucker case, the computational aspects, i.e., the fundamental algorithms for representing in TT-format and approximating in TTformat a tensor, are analysed. Moreover, the memory benefits and flaws in storing a tensor in TT-format are presented to complete the Tensor Train section.

I.I Introduction

As stated in 1947 in [START_REF] Neumann | Numerical inverting of matrices of high order[END_REF], numerical linear algebra studies, among other topics, the errors that appear when linear algebra operations are computationally performed. Generally speaking, numbers are represented with round-off errors inside computers. To those representation errors, computational ones are added when arithmetic operations are performed. Potentially the computational result of a sequence of linear algebra operations, for example, the computed solution of a linear system, could be extremely far from the theoretically expected one. Therefore, mathematicians and computer scientists studied and still do how rounding errors propagate when a numerical linear algebra algorithm is used, developing the so-called numerical stability analysis. A very common approach to studying the error propagation in numerical linear algebra is the backward stability analysis, popularized by Wilkinson during the 1960s in [START_REF] Wilkinson | Modern Error Analysis[END_REF][START_REF] Wilkinson | Rounding Errors in Algebraic Processes[END_REF]. We say that the computational solution of a problem is backward stable if it is the exact solution of the same problem with a perturbation. The algorithm producing this computed result is consequently said backward stable. In floating point arithmetic, the perturbation is usually expressed in terms of unit round-off with which a representation and a computation accuracy are usually associated; we present a brief overview of the floating point system in the following sections.

I.I.I Solving linear system: the Generalized Minimal RESidual

The backward stability analysis has been carried out over algorithms often meant to solve linear systems. Indeed, among the many research topics addressed in numerical linear algebra, solving linear systems of equations has always been a key one. The techniques for computing a solution of the linear systems are usually classified into two groups: direct and iterative methods. The former computes the solution of the given linear system in a finite number of steps, while the latter ones iteratively converge toward the exact solution from an initial guess. Among the direct solvers, it is worthwhile mentioning back substitutions, Gauss elimination, LU factorization, QR factorization, and Cholesky factorization. Examples of classical iterative methods are the conjugate gradient, the biconjugate gradient, the MINimal RESidual (MINRES) [START_REF] Paige | Solution of Sparse Indefinite Systems of Linear Equations[END_REF] and the Generalised Minimal RESidual (GMRES) [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF].

The GMRES solver was presented in 1986 by Saad and Schultz as a generalization of the 1975 solver MINRES. They are both based on the Krylov subspace technique, previously described by Krylov in 1931 [START_REF] Krylov | On the Numerical Solution of Equation by Which are Determined in Technical Problems the Frequencies of Small Vibrations of Material Systems[END_REF]. The key idea of GMRES is approximating the solution of the input linear system by minimizing the residual norm in the considered Krylov space, whose dimension increases iteratively. To achieve this purpose, GMRES relies on the Arnoldi relation, firstly presented in 1951 [START_REF] Arnoldi | The principle of minimized iterations in the solution of the matrix eigenvalue problem[END_REF], which simplifies the minimization problem through a projection. Moreover, an orthogonalization kernel is usually employed in this solver to construct a basis of the Krylov subspace; the most common choices are Modified Gram-Schmidt (MGS), from 1986 original work [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] and Householder from 1988 variant [START_REF] Walker | Implementation of the GMRES method using Householder transformations[END_REF], because of the numerical quality of their basis. As a

I.II. FINITE PRECISION ARITHMETIC

stopping criterion for GMRES, the choice suggested in the literature [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Greenbaum | Iterative methods for solving linear systems[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] is the backward error based on the linear system matrix and right-hand side associated with the iterative solution. When the backward error is lower than a prescribed threshold, the iterations stop, implying that the current iterative solution can be considered as the exact solution of a perturbed problem, where the relative norm of the perturbation is lower than a prescribed threshold. The backward stability of both MGS and Householder GMRES implementations was theoretically proved in [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] and [START_REF] Drkosova | Numerical stability of GMRES[END_REF] respectively, showing that the attainable normwise backward error of GMRES is of the same order as the unit round-off.

I.I.II Orthogonalization schemes

It is quite common for numerical linear algebra algorithms to require the orthogonalization of a set of vectors, which have different computational properties. In a theoretical context, an orthogonal basis is generated from a set of linear independent vectors through the Gram-Schmidt process, proposed by Schmidt in 1907 in [START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF], who credited also Gram's previous work of 1883 [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF]. The link between the two techniques was highlighted in 1935 by Wong [START_REF] Wong | An Application of Orthogonalization Process to the Theory of Least Squares[END_REF], but it is worthwhile mentioning that similar procedures appeared in previous works of Gauss, Legendre, Laplace, and Cauchy [START_REF] Leon | Gram-Schmidt orthogonalization: 100 years and more[END_REF]. As remarked in [START_REF] Leon | Gram-Schmidt orthogonalization: 100 years and more[END_REF], from an algorithmic point of view the methods proposed by Gram and Schmidt differ, even though in a theoretical framework they lead to the same results. Their tiny difference went unnoticed for many years, as mentioned in [START_REF] Wilkinson | Modern Error Analysis[END_REF]. Nowadays, we know that the Modified Gram-Schmidt (MGS), closer to the algorithm proposed in [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF], is more stable than the Classical Gram-Schmidt (CGS), which is similar to the method described in [START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF], comparing the results of [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF] and [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF].

Essentially, we can affirm that the closer the orthogonal basis is to the expected theoretical one, the more reliable the orthogonalization kernel will be inside a more articulated algorithm. The loss of orthogonality is the preferred tool to estimate how much the computed orthogonal basis differs from the expected theoretical one. Therefore over the years, mathematicians and computer scientists proposed new orthogonalization schemes, trying to reduce the loss of orthogonality of MGS, improving its quality as an orthogonalization kernel. In 1958, Householder transformation made its first appearance in [START_REF] Householder | Unitary Triangularization of a Nonsymmetric Matrix[END_REF] and it became a new alternative and reliable tool for orthogonalizing vector sets. The loss of orthogonality of the Householder orthogonalization algorithm, proved in [START_REF] Wilkinson | The algebraic eigenvalue problem[END_REF], outcompetes the CGS and MGS one, even if this kernel is computationally more expensive than the other two. In more recent years, other orthogonalization methods have been proposed as the Gram-Schmidt algorithm with re-orthogonalization loops, which firstly appeared in 1966 and was further studied [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Daniel | Reorthogonalization and Stable Algorithms for Updating the Gram-Schmidt QR Factorization[END_REF][START_REF] Parlett | The Symmetric Eigenvalue Problem[END_REF], or the Gram algorithm proposed in 2002 [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF].

I.II Finite precision arithmetic

In this section, we present the fundamental concepts around finite precision arithmetic, that will play a key role in Chapter 2 and 3.

Because of their finiteness, computers store a finite set of numbers from the infinite real set, approximating them with a finite number of digits. The floating-point system is a representation model for describing the storing possibilities of computers. Definition I.II.i. Given the natural numbers β, t ∈ N, the floating point number system is a set F ⊂ R whose generic element x ∈ F writes x = ±m × β e-t (I.ii)

where the significand m ∈ N 0 is such that 0 ≤ m ≤ β t -
R(F ) = {y ∈ F β e min -1 ≤ |y| ≤ β emax (1 -β -t )}.
We briefly present as an example of a floating point system the: IEEE standard arithmetic, which appeared as IEEE standard 754 in 1985 [START_REF]IEEE Standard for Binary Floating-Point Arithmetic[END_REF]. It is one of the most popular floating point systems [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF] and the one used for all the numerical experiments presented in the following chapters.

off u ∈ R + is u = 1 2 β 1-t .
In the IEEE standard arithmetic floating point system, the single precision unit roundoff is u = 10 -8 , while for the double precision format it is u = 10 -16 . The unit round-off and the rounding are strictly connected, as shown in the following classical theorem.

I.III. ROUNDING ERROR ANALYSIS

Theorem I.II.ii. [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF]Theorem 2.2] Let F be a floating point system, if x ∈ R lies in R(F ) the range of F, then there exists δ ∈ R such that

f l(x) = x(1 + δ) with |δ| ≤ u.
This theorem states that the rounding of x in the considered floating point system is equal to the mathematical value of x times a factor very close to 1.

I.III Rounding error analysis

The representation of real numbers through a floating point system introduces necessarily some errors, called rounding errors. These rounding errors may propagate or they may cancel out [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF] through a sequence of operations, that is an algorithm. Thus, in numerical linear algebra, the rounding error analysis is a fundamental step to evaluate the quality of the considered algorithms.

Once the floating point system F is chosen, the rounding error analysis is performed under the standard model assumption that if x, y ∈ F then there exists δ ∈ R such that

f l(x op y) = (x op y)(1 + δ) with |δ| ≤ u
with u the unit round-off of F and op being a basic arithmetic operation among sum, difference, multiplication, or division. Sometimes among the basic operations satisfying the previous equation we include also the square root one. This hypothesis states that the computed value of (x op y) is equal to the value of theoretical operation (x op y) times a factor that is extremely close to 1. This standard model assumption holds in particular for the IEEE standard arithmetic system.

The rounding error analysis aims to establish bounds that express the effects of rounding errors on the outcome of an operation sequence. There exist different approaches for studying rounding errors. For example, let g : R → R be function such that y = g(x), when the function g is implemented on the machine, we get ŷ = g(x) for every x ∈ R. A first possibility is investigating how close y and its computed value ŷ are, bounding the relative forward error written as

e f (ŷ) = |y -ŷ| |y| .
Another possibility is seeking how many and how large are the values ∆x such that ŷ = g(x + ∆x), bounding the backward error expressed as

e b (x) = |x -x| |x| = |∆x| |x| with ŷ = g(x) = g(x + ∆x).
Thus, investigating the quality of the backward error means studying how large the perturbation ∆x of the value of x can be, that is how close x and x = x + ∆x can be, to part I. numerical linear algebra 29 lead to the same final value of ŷ. The two main benefits of this backward error analysis approach are the follows. First, the rounding error is interpreted as a perturbation of the input data, so the solution we get is the best up to the quality of the available data [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF]. Second, the forward error can be addressed separately through the perturbation theory [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF]. Consequently, the backward error analysis purpose is to establish when an algorithm is backward stable in the considered floating point system.

Definition I.III.i. The method to compute y = g(x) is backward stable in the considered floating point system F if denoted by ŷ the computed value of y, for every x ∈ R we have

ŷ = g(x + ∆x) with |∆x| |x| ≤ cu
with c ∈ R + small and u the unit round-off for F.

Frequently, in the backward error analysis of algorithms working with matrices, the matrix condition number, defined next, has a central role.

Definition I.III.ii. Let

A ∈ R n×n be a matrix, the condition number κ 2 (A) ∈ R + with respect to the L2 norm is κ 2 (A) = ||A|| 2 ||A -1 || 2 = σ 1
σ r where σ 1 and σ r are the maximum and minimum non-zero singular values of A respectively, with r = rank(A).

We refer the reader to [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF] for a complete presentation of the floating point system and the backward stability analysis of many numerical linear algebra algorithms.

Chapter 2

A robust GMRES in TT-format

Introduction

In many domains of sciences and engineering, the problems to be solved can naturally be modelled mathematically as linear systems of equations, i.e., as

Ax = b
where A ∈ R n×n represents a linear endomorphism of R n , b ∈ R n is the right-hand side and x ∈ R n is the searched solution. Two main approaches have emerged in the search for techniques to solve linear systems: direct and iterative methods. Direct solvers compute the solution in a finite number of steps, usually transforming equivalently the matrix A, as LU or QR-factorization methods, while iterative solvers approximate iteratively the solution starting from an initial guess, as conjugate gradient or minimal residual method. The Generalized Minimal RESidual (GMRES) method is an iterative solver based on the Krylov subspace technique [START_REF] Krylov | On the Numerical Solution of Equation by Which are Determined in Technical Problems the Frequencies of Small Vibrations of Material Systems[END_REF], which approximates iteratively the solution minimizing the linear system residual in the associated Krylov space. Two main variants of GMRES exist: the original one [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] that constructs an orthogonal basis of the Krylov space through the Modified Gram-Schmidt kernel (MGS-GMRES) and an alternative one [START_REF] Wilkinson | Rounding Errors in Algebraic Processes[END_REF] relying on Householder transformations, namely Householder-GMRES (H-GMRES). In [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF], the authors prove the norm-wise backward stability of both these algorithms under the classical IEEE model assumption, i.e., with the unit rounding error of floating point operations and the data storage both bounded by the same unit round-off u, which depends on the selected working arithmetic. This chapter aims at investigating the backward stability of GMRES when the leading part of the rounding error is related to the selected data representation. The use of low-rank tensor approximation, employed to tackle the 'curse of dimensionality' when solving tensor linear systems, represents a practical study case.

When storing all the data in classical IEEE floating point format is not affordable, part of the data has to be compressed, introducing relative perturbations on them that will later on be involved in further computations. If the data compression introduces

INTRODUCTION

relative component-wise perturbations, the existing backward stability analyses for MGS-GMRES [START_REF] Drkosova | Numerical stability of GMRES[END_REF] and H-GMRES [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] still apply. However, if only the norm-wise relative perturbations of the data storage can be controlled, technical details of the theoretical backward stability results do not readily apply. In the first half of this chapter, we investigate numerically the backward stability of both GMRES variants in a synthetic context, imposing perturbations that target a certain accuracy component-wise or normwise, and in a practical framework, using an agnostic lossy compressor. The key idea of our study is to compress vectors all at once, into which particular scalars are therefore not constrained by any memory alignment. In particular, they do not need to stick with an available (hardware) precision such as fp32 or fp16. Scalars do not live in "two or more precisions chosen from a small number of available precisions" [START_REF] Higham | Accuracy and Stability of Numerical Algorithms: Second Edition[END_REF] but on a continuum of possible accuracy. We call such as scheme variable accuracy data storage, or, for short, variable accuracy [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF].

Different attempts [START_REF] Ballani | A projection method to solve linear systems in tensor format[END_REF][START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF][START_REF] Kressner | Low-Rank Tensor Krylov Subspace Methods for Parametrized Linear Systems[END_REF] have been made over the years to extend iterative solvers to the tensor framework to compute the solution of high order tensor linear systems, that is i.e., as Ax = b

where A represents multilinear endomorphism on

R n 1 ו••×n d , b ∈ R n 1 ו••×n d is the right- hand side and x ∈ R n 1 ו••×n d is the searched solution.
For sake of simplicity, henceforth these linear systems will be referred to as order d tensor linear systems, where d is the order of the tensor space where they are defined. They all rely on compression techniques, as HOSVD [START_REF] Lathauwer | A Multilinear Singular Value Decomposition[END_REF], Hierarchical Tucker [START_REF] Grasedyck | Hierarchical Singular Value Decomposition of Tensors[END_REF] or TT [START_REF] Oseledets | Tensor-Train Decomposition[END_REF], meant to tackle the 'curse of dimensionality', i.e., the exponential growth of the variables number in function of the order of the linear system. Since these compression techniques introduce perturbations of a prescribed accuracy to avoid memory deficiencies, their application inside iterative solvers fits in the variable accuracy scheme. The second half of this chapter studies numerically the backward stability of GMRES extended to the tensor framework with the TT-formalism (TT-GMRES), comparing our results with another GMRES realization in TT-format presented in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. Moreover, we highlight how the tensor structure enables us to solve simultaneously many tensor linear systems of order d depending on a parameter by simply reformulating the problem in a space of order (d + 1). The remainder of this Chapter is structured as follows. Section 2.2 collects the results related to the variable accuracy of GMRES in the matrix framework. After summarizing in Section 2.2.1 the backward stability theoretical results for GMRES, we present the experiments with component-wise and norm-wise perturbations, both from a synthetic compression and an agnostic lossy compressor, in Section 2.2.2 and 2.2.3 respectively. The variable accuracy study of GMRES extended to the tensor framework is found in Section 2.3. The TT-GMRES algorithm is described in Section 2.3.1, while Section 2.3.2 collects all the theoretical results related to the simultaneous solution of many parameter dependent tensor linear systems. We report about our numerical experiments with TT-GMRES in Section 2.3.3. In particular, in Section 2.3.3.1 we compare our TT-GMRES version with the one proposed in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF] from the backward stability viewpoint, proving that our method is more robust. The experiments with parameter dependent multilinear operator or right-hand side are analysed in Section 2.3.3.2, illustrating the tightness of the bounds constructed in Section 2.3.2.

GMRES in matrix computation framework

In this section we present our variable accuracy approach applied to classical linear systems. After recalling briefly the theoretical backward stability results for GMRES, we describe the variable accuracy approach. Finally, we report about the numerical experiments we perform introducing component-wise and norm-wise perturbation, coming either from a synthetic compression or from an agnostic lossy compressor.

Background on GMRES

Starting from the zero initial guess, GMRES [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] constructs a series of approximations x k in Krylov subspaces of increasing dimension k so that the residual norm of the sequence of iterates is decreasing over these nested spaces. More specifically:

x k = argmin x∈K k (A,b) b -Ax , with K k (A, b) = span{b, Ab, . . . , A k-1 b}
the k-dimensional Krylov subspace spanned by A and b. In practice, an orthogonal matrix

V k = [v 1 , . . . , v k ] ∈ R n×k
and an upper Hessenberg matrix Hk ∈ R (k+1)×k are iteratively constructed using the Arnoldi procedure such that span{V k } = K k (A, b) and

AV k = V k+1 Hk , with V T k+1 V k+1 = I k+1 .
This is often referred to as the Arnoldi relation. Consequently,

x k = V k y k with y k = argmin y∈R k βe 1 -Hk y ,
where β = b and e 1 = (1, 0, . . . , 0) T ∈ R k+1 so that in exact arithmetic the following equality holds between the least square residual and the true residual

rk = βe 1 -Hk y = b -Ax k . (2.1)
In finite precision calculation, this equality no longer holds. Another matrix equality can be formed based on the Arnoldi relation that reads

(v 1 , AV k ) = V k+1 R k+1 , (2.2)
where R k+1 is a (k + 1) × (k + 1) upper triangular matrix that writes

R k+1 = e 1 , Hk . Equation (2.
2) shows that the Arnoldi algorithm actually computes a QR factorization of (v 1 , AV k ). Such a factorization could also be computed using Householder transformations, this is the core idea of the H-GMRES variant proposed by Walker [START_REF] Walker | Implementation of the GMRES method using Householder transformations[END_REF] and depicted in Algorithm 7. The original GMRES by Saad and Schultz [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] is based on a more classic Arnoldi algorithm that uses Modified Gram-Schmidt procedure to orthogonalize the Krylov basis, often referred to as MGS-GMRES and depicted in algorithm 6. We refer to [START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF][START_REF] Saad | Iterative Methods for Sparse Linear Systems[END_REF] for a more detailed presentation.

Algorithm 6 x, hasConverged = MGS-GMRES(A, b, m, ε)

1: input: A, x 0 , b, ε. 2: r 0 = b, β = r 0 and v 1 = r 0 /β 3: for k = 1, . . . do 4:
w k = av k 5:

for i = 1, . . . , k do 6: H(i, k) = v i , w 7: w k = w k -H(i, k)v i 8:
end for 9: end if 17: end for 18: return: x = x k , hasConverged Because the orthonormal basis V k has to be stored, a restart parameter defining the maximal dimension of the search Krylov space is used to control the memory footprint of the solver in both the implementations. If the maximum dimension of the search space is reached without converging, the algorithm is restarted using the final iterate as the initial guess for a new cycle of GMRES. Furthermore, it is often needed to consider a preconditioner to speed-up the convergence. Using right-preconditioned GMRES consists of considering a non singular matrix M , the so-called preconditioner that approximates the inverse of A in some sense. In that case, GMRES is applied to the preconditioned system AM t = b. Once the solution t has been computed the solution of the original system is recovered as x = M t. The right-preconditioned GMRES is sketched in Algorithm 8 for a restart parameter m and a convergence threshold ε .

H(k + 1, k) = w k 10: v k+1 = 1/ H(k + 1, k) w k 11: y k = argmin y∈R k βe 1 -Hk y with Hk = H(: k + 1, : k) 12: x k = x 0 + v k y k 13: if η A,b (x k ) < ε
The backward error analysis, popularized by J.H. Wilkinson who contributed significantly to its development [START_REF] Wilkinson | Modern Error Analysis[END_REF][START_REF] Wilkinson | Rounding Errors in Algebraic Processes[END_REF], gives powerful stopping criteria for iterative solvers, as GMRES, based indeed on the backward error [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Greenbaum | Iterative methods for solving linear systems[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF]. In particular, Rigal and

Algorithm 7 x, hasConverged = H-GMRES(A, b, m, ε) 1: input: A, x 0 , b, ε. 2: r 0 = b, β = r 0 3: compute u 1 to define the Householder reflector P 1 = I -2u 1 u T 1 such that P 1 r 0 = βe 1 4: for k = 1, . . . , m do 5: v k = 1 j=k P j e k 6: w = k j=1 P j Av k 7:
compute u k+1 to define the Householder reflector

P k+1 = I -2u k+1 u T k+1 s.t. P k+1 w(k + 1 : n) = w(k + 1 : n) e k+1 (k + 1 : n) and p k+1 w(1 : k) = w(1 : k) 8: H(:, k) = (P k+1 w)(1 : k + 1)
z = P j (y k (j)e j + z) 13:
end for 14: end if 19: end for 20: return:

x k = x 0 + z 15: if η A,b (x k ) < ε then 16: hasConverged = True
x = x k , hasConverged Algorithm 8 x, hasConverged = Right-GMRES(A, M , b, x 0 , m, ε) 1: input: A, M , b, m, ε. 2: hasConverged = False 3: x = x 0 4: while not(hasConverged) do 5: r = b -Ax Iterative refinement step with at most m GMRES iterations on AM 6: t k , hasConverged = GMRES(AM , r, m, ε) 7: x = x + M t k
Update the unpreconditionned with the computed correction 8: end while 9: return: x, hasConverged

GMRES IN MATRIX COMPUTATION FRAMEWORK

Gache [START_REF] Rigal | On the Compatibility of a Given Solution With the Data of a Linear System[END_REF] showed that the approximate solution x k at iteration k can be interpreted as the exact solution of a perturbed linear system where the relative norms of the perturbations can be easily computed. We denote η A,b (x k ) this norm-wise backward error for linear systems

η A,b (x k ) = min ∆A,∆b {τ > 0 : ∆A ≤ τ A , ∆b ≤ τ b and (A + ∆A)x k = b + ∆b} = Ax k -b A x k + b . (2.3)
In some circumstances, a simpler backward error criterion based on perturbations only in the right-hand side can also be considered, that leads to the second possible choice

η b (x k ) = min ∆b {τ > 0 : ∆b ≤ τ b and Ax k = b + ∆b} = Ax k -b b . ( 2
QT Q = I + E with E 2 ≈ u, ( 2.5)
while MGS produces [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF] a Q factor such that

QT Q = I + E with E 2 ≈ κ(S) u, ( 2.6)
where κ(S) is the 2-norm condition number of the matrix S whose i-th column is s i ∈ S.

Thanks to the backward stability of Householder QR given by (2.5) for computing the Householder vectors, the backward stability of H-GMRES was established in [40, Theorem 4.1 and Corollary 4.2] assuming that the matrix A is not close to singularity. Remark 2.2.1. Under technical assumptions [START_REF] Drkosova | Numerical stability of GMRES[END_REF]Corollary 4.2], it is shown that the normwise backward error at the last iterate

x n of H-GMRES is such that η A,b (x n ) = O(u),
where u is the unit round-off error of the working precision.

In MGS-GMRES, it is known that the Arnoldi basis progressively departs from orthogonality as indicated by (2.6). However, the following result has been shown in [START_REF] Paige | Residual and Backward Error Bounds in Minimum Residual Krylov Subspace Methods[END_REF]

η A,b (x k ) • I -Ṽ T k Ṽk F = O(u), (2.7)
where the columns of Ṽk ∈ R n×k , computed by MGS-Arnoldi, form a basis for the Krylov space K k (A, b). This result implies that it is impossible to have a significant loss of orthogonality as long as the norm-wise relative backward error is very small. Later, the backward stability of MGS-GMRES was proved in [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF], that reads: Theorem 2.2.2 ([113]). Assuming that A is not close to singularity, that is,

σ min (A) n 2 ||A||u,
where u is the unit round-off error of the working precision; let k ∈ N be the first integer such that κ( Ṽk+1 ) > 4 3 , (2.8)

then x k satisfies η A,b (x k ) = O(u).
We refer to [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] for the details of the tedious and technical proof and to [104, p. 227] for a short exposure. We also note that for technical reason, the proof is established using the Frobenius norm of A to define η A,b so that the norm-wise perturbations on A are measured in Frobenius norm and not in Euclidean norm. In practice, the backward stability is observed using the 2-norm and we only consider it in the sequel.

Numerical experiments with component-wise perturbations

For the numerical experiments of this and the following section, the norm of the linear operator is estimated using a randomized SVD [START_REF] Martinsson | A randomized algorithm for the decomposition of matrices[END_REF][START_REF] Rokhlin | A Randomized Algorithm for Principal Component Analysis[END_REF]. In addition, for all numerical experiments right preconditioned GMRES is considered to enable a fast convergence, so that GMRES does effectively solve

AM z = b
where M is a preconditioner operator. For calculation performed in regular matrix case, we consider a preconditioner based on an ILU (t) [START_REF] Saad | ILUT: A dual threshold incomplete ILU factorization[END_REF] factorization. In the preconditioned context, the backward error η AM,b (z k ) is the one that GMRES can drive down to O(u).

Variable accuracy approach

We are interested in the numerical behaviour of GMRES where the processed data may be numerically altered for two reasons. The first possible source is the employed finite precision arithmetic discussed above. We still denote u the associated unit round-off error. In this work, we further assume that the data may be compressed leading to another possible source of inaccuracy due to the corresponding compressed data representation. More precisely, we denote δ-storage(•) the function that enables to store a given data in a format, referred to as δ-component-wise data format, that induces an unit round-off δ; that is:

δ-storage(a) = a(1 + ξ) with |ξ| ≤ δ.
When a basic calculation is performed and the result stored in a δ-representation we have

f l δ (a op b) = δ-storage(f l(a op b)) = (a op b)(1 + ε)(1 + ξ),

GMRES IN MATRIX COMPUTATION FRAMEWORK

with |ε| ≤ u and |ξ| ≤ δ. Neglecting the second order term εξ, we obtain

f l δ (a op b) = (a op b)(1 + ε + ξ),
with |ε + ξ| ≤ u + δ, so that all the theoretical results, presented in the previous section, that were established with the same unit round-off u for the basic operations and the data representation still hold, but with unit round-off u + δ. In can be noted that in the particular case u δ, it further reduces to f l δ (a op b) = (a op b)(1 + ξ), with |ξ| ≤ δ and those same results then hold with unit round-off δ. For a matter of readability, we will make this particular assumption (results hold with unit round-off δ) in rest of this section, but they can be more generally interpreted without this assumption, in which case a u + δ shall be considered instead.

δ-component-wise data storage

We numerically investigate the behaviour of the MGS-GMRES and H-GMRES where the storage function δ-storage(•) is used to have a δ-representation of all the vectors of size n. Regular IEEE calculation is used and data associated with small dimension matrices and vectors are stored in regular IEEE format. That are all the data involved in the least squares problem solution.

In Figure 2.1, we plot the convergence history of the norm-wise backward error for both MGS-GMRES and H-GMRES for various values of δ for a test matrix from the Florida test collection [START_REF] Davis | The university of Florida sparse matrix collection[END_REF]. The dashed vertical blue line indicates the iteration where κ( Ṽk ) > 4 3 for MGS-GMRES. It can be seen that the backward stability property does still hold for δ round-off unit due to data representation. Although this is not predicted by any theoretical argument, the convergence of the two GMRES variants is identical up to the proximity of δ. On this example, the convergence of MGS-GMRES and H-GMRES perfectly overlap. Furthermore the relation given by (2.7) between the loss of orthogonality and the backward error for MGS-GMRES, represented by the dashed green curve, does also hold for δ unit round-off. To illustrate that both data representation and finite precision calculation play a role in the attainable accuracy, we present numerical experiments in Figure 2.2 where the calculation are performed either in fp32 (left plots) or fp64 (right plots). On the first row, where the component-wise perturbation is δ = 10 -4 , which is much larger than u 32 and u 64 , the various GMRES implementations do exhibit the same convergence and identical attainable accuracy, i.e., O(δ). On the second row, the fp32 implementation has an attainable accuracy that is O(u 32 ) because the δ componentwise perturbation is hidden by the fp32 format. The fp64 calculation reveals the attainable accuracy already observed in

Solution techniques using SZ compressed format

In this section we present a first practical application of the GMRES algorithm in variable accuracy. We consider an implementation where the Arnoldi basis in MGS-GMRES or the reflector vectors in H-GMRES are compressed to alleviate their memory footprint. More precisely, in MGS-GMRES we compress w k in line 10 before normalizing it to define v k+1 . In H-GMRES, we compress the vectors u k that defines the Householder reflectors P k in lines 3 and 7 of Algorithm 7. For those experiments, we used the SZ [START_REF] Di | Fast error-bounded lossy HPC data compression with SZ[END_REF] lossy compressor, an agnostic compressor that does not attempt to exploit underlying numerical properties of the vectors but operates on their binary representation. For the experiments in this section, we use the capability of SZ to ensure a prescribed componentwise relative error between the original data z ∈ R n and the decompressed data z

∈ R n , that is, max i=1,...,n |z(i) -z(i)| |z(i)| ≤ δ, with z(i) the ith component of z.
Similarly to the previous section, we display the convergence histories of MGS-GMRES and H-GMRES in Figure 2.3, for a few values of the compression control parameter δ. Although the δ component-wise perturbations only occur when storing the Arnoldi basis or reflector vectors, the general trend of the convergence histories of η AM,b remains qualitatively the same. The attainable accuracy is slightly, but not significantly, better. The memory saving enabled by SZ is reported in Table 2.1. A general expected rule of thumb is: "the larger δ, the larger the memory gain". For some matrices, e.g., gre_343, more than 20% of memory saving is observed for a 10 -14 accuracy. It illustrates the possible significant benefit of these GMRES implementations based on a compressor like SZ for the solution of large problems.

Numerical experiments with norm-wise perturbations

In many contexts, having a control on the relative component-wise error on the vectors is not feasible and only a norm-wise monitoring is possible. Although the existing backward stability analyses of both considered GMRES variants do not readily apply, because some of the technical arguments of the proof are no longer valid, we show through numerical experiments that the property still holds in practice. In Section 2.2.3.1, we report on experiments where the results of all calculations performed on length n vec-

GMRES IN MATRIX COMPUTATION FRAMEWORK

δ 10 -4 10 -6 10 -8 10 -10 10 -12 10 -14 gre_115 with ILU(10 tor is artificially perturbed by a relative norm-wise perturbation. Next we consider two practical computational contexts where such norm-wise perturbations are encountered in some steps of the algorithms.

δ-norm-wise data storage

In this section we consider the situation where all the length n vectors involved in the algorithms are stored in a norm-wise δ-representation. That is, any vector z ∈ R n is replaced by z so that z -z z ≤ δ.

(2.9)

We display the convergence history of the norm-wise backward error for both considered GMRES variants, as well as the product of the backward error times the loss of orthogonality for MGS-GMRES, in Figure 2.4. It can be seen that the trends are very similar to those that can be observed in Figures 2.1. The same observations can be made: both the attainable backward error accuracy and the product η AM,b I -V T V F reach values close to δ.

Solution techniques using SZ compressed format

We report the convergence histories of MGS-GMRES and H-GMRES for a few values of the norm-wise compression control parameter δ in Figure 2.5. These results are the norm-wise counterparts of those displayed in Figure 2.3 for component-wise compression. It can be observed that the general trends are similar and that the attainable value of η AM,b always becomes close to δ. Because ensuring norm-wise relative error is easier for the SZ-compressor than ensuring component-wise relative error, the memory saving gains reported in Table 2.2 are larger than those in Table 2.1.

Tensor Train GMRES

The second half of this chapter focuses on generalizing GMRES to tensor linear systems through the TT-formalism, see Section 1.3.2. To prevent memory issues, we introduce in the classical GMRES outline compression steps through the TT-rounding algorithm [START_REF] Oseledets | Tensor-Train Decomposition[END_REF]. Under this choice, our TT-GMRES represents a perfect study case for our variable accuracy approach of GMRES with norm-wise perturbation.

After describing our TT-GMRES algorithm, we study how to solve simultaneously many tensor linear systems when the operator or the right-hand side depend on a parameter. In particular, we present some theoretical bounds that guarantee the numerical quality of the solution for a single parameter tensor linear system when it is extracted from the simultaneous solution. Finally, experiments highlighting the robustness of TT-GMRES and the tightness of these bounds are analysed.

Preconditioned GMRES in Tensor Train format

Assume A ∈ R (n 1 ×n 1 )ו••×(n d ×n d ) to be a tensor operator and b ∈ R n 1 ו••×n d a tensor, then the general tensor linear system is Ax = b (2.10) with x ∈ R n 1 ו••×n d .
Notice that by setting d = 1 we have the standard linear system from classical matrix computation. A possible way for solving (2.10) is using a tensor-extended version of GMRES. Since all the operations appearing in this iterative solver are feasible with the TT-formalism, we assume that all the objects are expressed in TT-format. The main drawback of this approach is the repetition of sums and contractions in the different loops, which leads to TT-rank growth and possible memory over-consumption. Therefore introducing compression steps in TT-GMRES is essential but particular attention should be paid to the choice of the rounding parameter to ensure that the prescribed GMRES tolerance ε can be reached. Our TT-GMRES algorithm is fully presented in Algorithm 9.

In Algorithm 9 and 10 there is an additional input parameter δ, i.e., the rounding accuracy. The TT-rounding algorithm at accuracy δ is applied to the result of the contraction between A and the last Krylov basis vector computed in Line 4, to the new Krylov basis vector after orthogonalization in Line 9 and to the updated iterative solution, Line 13. The purpose is to balance the rank growth due to the tensor contraction or sum, that occurred in the immediately previous step, with the rounding. As it will be observed in the numerical experiments of Section 2.3.3, the rounding accuracy δ has to be chosen lower or equal than the GMRES target accuracy ε.

Solution of parametric problems in Tensor Train format

In this section, we investigate the situation where either the tensor representation of the linear operator or the right-hand side has a mode related to a parameter that is discretized. In the case of the parametric linear operator, we are interested in the numerical for i = 1, . . . , k do 6:

H(i, k) = v i , w 7: w = w -Hi,k v i 8:
end for 9:

w = TT-round(w, δ)

10:

H(k + 1, k) = w 11: v k+1 = (1/ H(k + 1, k))w 12:
y k = argmin y∈R k βe 1 -Hk y with Hk = H(: k + 1, : k)

13:

x k = TT-rounding k+1 j=1 y k (j)v j , δ 14: if η A,b (x k ) < ε then 15:
hasConverged = True

r = TT-round(b -Ax, δ) 6: t k , hasConverged = GMRES(AM, r, m, ε, δ)
Update the unpreconditionned with the computed correction 7:

x = TT-round(x + Mt k , δ) 8: end while 9: return: x, hasConverged quality of the computed solutions when we solve for all the parameters at once compared to the solution computed when the parametric systems are treated independently. In the case of the right-hand sides depending on a parameter, we investigate the links between the search space of TT-GMRES enabling the solution of all the right-hand sides at once and the spaces built by the GMRES solver on each right-hand side considered independently.

Parameter dependent linear operators

This subsection focuses on a specific type of parametric tensor operators expressed as

A α = B 0 + αB 1 with α ∈ R and B 0 , B 1 two tensor operators of R (n 1 ×n 1 )ו••×(n d ×n d ) . Assuming that α takes p different real values in the interval [a, b], we define p linear systems of the form A y = b (2.11)
where

A = B 0 + α B 1 , b ∈ R n 1 ו••×n d , α ∈ [a, b]
for every ∈ {1, . . . , p}. At this level, it is possible to choose between classically solving each system independently or solving them simultaneously in a higher order space defining the so-called "all-in-one" system. This latter system writes Ax = b (2.12)

where

A ∈ R (p×p)×(n 1 ×n 1 )ו••×(n d ×n d ) such that A(h, , i 1 , j 1 , . . . , i d , j d ) =    A (i 1 , j 1 , . . . , i d , j d ) if h = , 0 if h = , (2.13) and the right-hand side is b ∈ R p×n 1 ו••×n d defined as b( , i 1 , . . . , i d ) = b (i 1 , . . . , i d ) (2.14) for i k , j k ∈ {1, . . . , n k }, k ∈ {1, .
. . , d} and , h ∈ {1, . . . , p}. The tensor operator A writes in a compact format as

A = I p ⊗ B 0 + diag(α 1 , . . . , α p ) ⊗ B 1 .
The ( , )-th slice of A with respect to modes (1, 2) is denoted

A [ ] = B 0 + α B 1 = A (2.15)
and similarly the -th slice of b with respect to the first mode is b [ ] = b by construction. So that Equation (2.11) also writes

A [ ] x [ ] = b [ ]
with x [ ] = y . It shows that once the "all-in-one" system, Equation (2.12), has been solved, the solution related to a specific parameter can be extracted as a slice of the "all-in-one" solution, obtaining an extracted individual solution. In other words, given the 50

TENSOR TRAIN GMRES

k-th iterate x k of the "all-in-one" system, the extracted individual solution for the -th problem is x [ ] k , i.e., the -th slice with respect to the first mode defined as

x [ ] k = x k ( , i 1 , . . . , i d ).
In the following propositions, we investigate the relation between the backward error of the "all-in-one" system solution and the extracted individual one. The equalities given for the "all-in-one" system are clearly true if the tensor and the tensor operators are given in full format, but they hold also in TT-format.

In TT-format

As stated in Section 1.2, given a tensor

a ∈ R n 1 ו••×n d in TT-format with TT-cores a k ∈ R r k-1 ×n k ×r k , we denote by a [k,i k ] the i k -th slice with respect to mode k, which in TT-format is expressed as a [k,i k ] = a 1 • • • a k-1 A k (i k )a k+1 • • • a d with A k (i k ) ∈ R r k-1 ×r k .
Since henceforth we will take slices only with respect to the first mode, instead of writing a [1,i 1 ] for the i 1 -th slice on the first mode we will simply write

a [i 1 ] . Similarly A [ ] denotes the ( , )-th slice of A ∈ R (p×p)×(n 1 ×n 1 )ו••×(n d ×n d )
with respect to the first two modes. We start constructing the "all-in-one" system tensor linear operator. Let C, G ∈ R (n 1 ×n 1 )ו••×(n d ×n d ) be two TT-matrices with k-th TT-core c k ∈ R r k ×n k ×n k ×r k+1 and g k ∈ R q k ×n k ×n k ×q k+1 for k ∈ {1, . . . , d} with q 1 = r 1 = r d+1 = q d+1 = 1, whose TT-expression is

C = c 1 • • • c d and G = g 1 • • • g d . (2.16) Given a diagonal matrix D = diag(α 1 , . . . , α p ), we define A ∈ R (p×p)×(n 1 ×n 1 )ו••×(n d ×n d ) as A = I p ⊗ C + D ⊗ G. (2.17)
Then the expression of

a k ∈ R (r k +q k )×n k ×n k ×(r k+1 +q k+1 ) the k-th TT-core of A is A k (i k , j k ) = C k (i k , j k ) 0 0 G k (i k , j k ) and A d (i d , j d ) = C d (i d , j d ) G d (i d , j d ) (2.18) for every i k , j k ∈ {1, . . . , n k } and k ∈ {1, . . . , d}. The first TT-core a 0 ∈ R 1×p×p×2 writes A 0 ( , m) = δ ,m a with a = 1 α (2.19)
with δ ,m the Kronecker delta, for , m ∈ {1, . . . , p}. The final TT-expression of A is

A = a 0 a 1 • • • a d .
Remark now that A ( ,m) the ( , m)-th slice with respect to mode 1 of A is

A [ ,m] = A 0 ( , m)a 1 • • • a d = δ ,m a a 1 • • • a d . If = m, then A [ ,m] = 0.
On the other side, if and m are equal, then

A [ , ] = I( , )a a 1 • • • a d = c 1 • • • c d + α g 1 • • • g d = C + α G.
We illustrate now the construction of the "all-in-one" system right-hand side from the p individual right-hand sides. Let b ∈ R n 1 ו••×n d be a TT-vector for every ∈ {1, . . . , p} with TT-cores b ,k ∈ R s ,k ×n k ×s ,k+1 for every k ∈ {1, . . . , d} with

s 1 = s d+1 = 1, i.e., b = b ,1 • • • b ,d (2.20)
and its (i 1 , . . . , i d ) element writes

b (i 1 , . . . , i d ) = B ,1 (i 1 ) • • • B ,d (i d ) with B ,k (i k ) ∈ R s ,k ×s ,k+1 , B ,1 (i 1 ) ∈ R 1×s ,2 and B ,d (i d ) ∈ R s ,d ×1
. For simplicity we impose s k = s ,k for every ∈ {1, . . . , p} and k ∈ {1, . . . , d}.

Remark 2.3.1. This assumption on the TT-rank of b is not binding. Indeed setting

s k = max h∈{1,...,p} s h,k , then each core tensor b h,k of mode sizes (s h,k , n h , s h,k+1 ) can be extended with zeros to (s k , n h , s k+1 ) We want to construct a tensor b ∈ R p×n 1 ו••×n d such that its -th slice with respect to the first mode of b is b , i.e., b [ ] = b . As consequence, the k-th TT-core of b is b k ∈ R ps k ×n k ×ps k+1 such that B k (i k ) =     B 1,k (i k ) . . . B p,k (i k )     ∈ R ps k ×ps k+1 for k ∈ {1, . . . , d -1}, while b d ∈ R ps d-1 ×n d ×1 and b 0 ∈ R 1×p×ps 1 are B d (i d ) =     B 1,d (i d ) . . . B p,d (i d )     ∈ R ps d ×1 and B 0 ( ) = 0 • • • 1 • • • 0 ∈ R 1×ps 1 with the -th component of B 0 ( ) being the only non-zero element. The TT-expression of b is b = b 0 b 1 • • • b d . (2.21)
By construction, we have that the -th slice of b with respect to mode 1 is

b [ ] = B 0 ( )b 1 • • • b d = b ,1 • • • b ,d = b .
Let consider A and b as defined in Equation (2.17

TENSOR TRAIN GMRES

We want to prove that r [ ] the -th slice with respect to the first mode of r is equal to the difference of the -th slices, i.e.,

r [ ] = A [ , ] x [ ] -b = (C + α G)x [ ] -b (2.22)
since the ( , )-th slice of A is C + α G for every ∈ {1, . . . , p}. Remark that the -th slice of b is b by construction. As consequence, the Equation (2.22) is true if we show that the -th slice of the contraction between A and x is equal to the contraction of their -th slices, i.e., (Ax

) [ ] = A [ , ] x [ ] .
Lemma 2.3.2. Given A, C, G as in Equations (2.16) and (2.17), let x ∈ R p×n 1 ו••×n d be a (d + 1)-order tensor. Then the -th slice of y = Ax is equal to the product of their -th slices, i.e.

(Ax) [ ] = A [ , ] x [ ] = (C + α G)x [ ] . Define w = (C + α G)x [ ] ,
y [ ] (i 1 , . . . , i d ) = n 1 ,...,n d j 1 ,...,j d =1 A( , , i 1 , j 1 , . . . , i d , j d )x( , j 1 , . . . , j d ) = n 1 ,...,n d j 1 ,...,j d =1 A [ ] (i 1 , j 1 , . . . , i d , j d ) x [ ] (j 1 , . . . , j d ).
by the definition of first mode slides. As already observed the ( , )-th slice of A with respect to the first mode is equal to C + α G. Thus, the previous equation becomes

y [ ] (i 1 , . . . , i d ) = n 1 ,...,n d j 1 ,...,j d =1 (C + α G)(i 1 , j 1 , . . . , i d , j d ) x [ ] (j 1 , . . . , j d ),
i.e., the thesis.

By the result of Lemma 2.3.2, we get that the -th slice of Ax writes

(Ax) [ ] = Cx [ ] .
Therefore the -th slice of r is

r [ ] = Cx [ ] -b [ ] .
As a conclusive result of this construction, we want to show that

||r|| 2 = p =1 ||r [ ] || 2 .
Lemma 2.3.3. Given s ∈ R n 0 ×n 1 ו••×n d and its i 0 -th slice with respect to the first mode

s (i 0 ) then ||s|| 2 = n 0 i 0 =1 ||s [i 0 ] || 2 .
Proof. By definition, the squared Frobenious norm of s is

s 2 = n 0 ,n 1 ,...,n d i 0 ,i 1 ,...,i d =1 s 2 (i 0 , i 1 , . . . , i d ) = n 0 i 0 =1 n 1 ,...,n d i 1 ,...,i d =1 s 2 (i 0 , i 1 , . . . , i d ) .
By the definition of slice, the last equation can be written as

s 2 = n 0 i 0 =1 n 1 ,...,n d i 1 ,...,i d =1 s 2 (i 0 , i 1 , . . . , i d ) = n 0 i 0 =1 n 1 ,...,n d i 1 ,...,i d =1 s [i 0 ] 2 (i 1 , . . . , i d ) ,
but by the definition of squared Frobenious norm of s [i 0 ] , we obtain

s 2 = n 0 i 0 =1 n 1 ,...,n d i 1 ,...,i d =1 s [i 0 ] 2 (i 1 , . . . , i d ) = n 0 i 0 =1 ||s [i 0 ] || 2 ,
i.e., the thesis.

By the result of Lemma 2.3.3, we have

||r|| 2 = p =1 ||r [ ] || 2 .

Theoretical bounds

Once the "all-in-one" system has been completely described in its construction in TTformat, we present some bounds that enable us to tune the convergence threshold when solving for multiple parameters while guaranteeing a prescribed quality for the individual extracted solutions. In particular, the bound given by Equation (2.23) in Proposition 2. 3.4 shows that if a certain accuracy ε is expected for the extracted individual solution in terms of the backward error in (2.4), a more stringent convergence threshold should be used for the "all-in-one" system solution that should be set to ε/ √ p.

η b (x k ) √ p ≥ η b (x [ ] k ) (2.23)
for ∈ {1, . . . , p}.

Proof. For the sake of simplicity we use η b and η b squared throughout the proof and discard the subscript of the k-th "all-in-one" iterate. The quantity

η 2 b (x [ ] ) explicitly gets η 2 b (x [ ] ) = A x [ ] -b 2 b 2 while η 2 b (x) is η 2 b (x) = Ax -b 2 b 2 . (2.24)
Thanks to the diagonal structure of A and the Frobenius norm definition, Equation (2.24) writes

η 2 b (x) = n =1 Ax -b [ ] 2 p j=1 b [j] 2 = n =1 A x [ ] -b 2 p j=1 b j 2 = p =1 η 2 b (x [ ] ) p (2.25) since ||b|| 2 = n j=1 ||b j || 2 = p.
From the square root of both sides of this last equation, the result follows.

For the backward error based on perturbation of both the linear operator and the right-hand side defined by (2.3), a similar result can be derived. While informative this result has a lower practical interest as the term ρ (x k ) in (2.26) depends on the solution; so defining the convergence threshold for the 'all-in-one' solution to guarantee the individual backward error requires some a priori information on the solution norms.

η A,b (x k ) ρ (x k ) ≥ η A ,b (x [ ] k ) where ρ (x k ) = A 2 x k + √ p A x [ ] k + 1 (2.26)
with x k the k-th "all-in-one" iterate and x [ ] k its -th slice with respect to mode 1.

CHAPTER 2. A ROBUST GMRES IN TT-FORMAT 55

Proof. For the sake of simplicity, as previously, the subscript of the k-th "all-in-one" iterate is dropped. The quantity η A,b (x) explicitly writes

η A,b (x) = Ax -b A 2 x + b .
If the previous equation is multiplied equivalently by η b (x), it gets

η A,b (x) = Ax -b A 2 x + b η b (x) η b (x) = b A 2 x + b η b (x) = √ p A 2 x + √ p η b (x) (2.27) by the definition of η b (x) and b = √ p. Similarly η A ,b (x [ ] ) is expressed in function of η b (x [ ] ) as η A ,b (x [ ] ) = b A 2 x [ ] + b η b (x [ ] ) = 1 A 2 x [ ] + 1 η b (x [ ] ) (2.28) since b = 1. Multiplying each side of Equation (2.27) by ( A 2 x + √ p), it follows A 2 x + √ p η A,b = η b √ p.
Thanks to the result of Proposition 2.3.4, we have

A 2 x + √ p η A,b (x) = η b (x) √ p ≥ η b (x [ ] ) = A 2 x [ ] + 1 η A ,b (x [ ] ) (2.29)
from Equation (2.28). Dividing both sides of Equation (2.29) by A 2 x [ ] +1, it becomes

A 2 x + √ p A x [ ] + 1 η A,b (x) ≥ η A ,b (x [ ] ) (2.30) since A 2 x [ ] ≥ A x [ ]
by the definition of the L2 norm.

Parameter dependent right-hand sides

We consider a particular case of this "all-in-one" approach. We intend to solve p tensor linear systems with the same multilinear operator and different right-hand sides. Given a linear tensor operator

A 0 ∈ R (n 1 ×n 1 )ו••×(n d ×n d )
, we define the -th tensor linear system as

A 0 y = b (2.31) with b ∈ R n 1 ו••×n d for every ∈ {1, . . . , p}.
To solve simultaneously all the righthand sides expressed in Equation (2.31), we repeat the construction introduced in Subsection 2.3.2, except that A 0 is repeated on the 'diagonal' of tensor linear operator A defined in Equation (2.13). Thanks to the tensor properties, the tensor operator i.e., we seek a tensor

A ∈ R (p×p)×(n 1 ×n 1 )ו••×(n d ×n d ) writes A = I p ⊗ A 0 so that A [ ] = A
x k ∈ K k (A, b) such that x k = argmin x∈K k (A,b) Ax -b .
Due to the diagonal structure of A, the Frobenius norm of r k = Ax kb is naturally written as follows

||r k || 2 = p =1 ||b -A 0 x [ ] k || 2 with x [ ]
k being the -th slice with respect to the first mode of x k , as in the previous sections. Thanks to the diagonal structure of A, we have that the -th slice of the Krylov basis vector A h b with respect to the first mode is

-A 0 x [ ] k ≥ b -A 0 y ,k .
Remark 2.3.6. We notice that a block TT-GMRES method could also be defined for the solution of such multiple right-hand side problems. In that situation, each individual residual norm would be minimized over the same space spanned by the sum of the individual Krylov space. This would be somehow the dual approach to the one described above, where we minimize the sum of the residual norms on each individual Krylov space.

Regarding the numerical quality of the extracted solution compared to the individually computed solution, the bound stated in Proposition 2.3.4 is still true. As in the previous section an informative, but with lower practical interest, bound similar Proposition 2.3.5 can be derived.

. , p} the following inequality holds

η A,b (x k ) ψ (x k ) ≥ η A ,b (x [ ] k ) where ψ (x k ) = x k + √ p/ A 0 2 x [ ] k + 1/ A 0 2 . ( 2
k || -1| ≤ ν for every k ≥ k * , then η A,b (x k ) ρ * (x k ) ≥ η A ,b (x [ ] k ) where ρ * (x k ) = A 2 x k + √ p 2 -ν (2.
† ∈ N such that x [ ] k ≤ A -1 2 √ p for every k ≥ k † , then η A,v (x k ) ρ † ≥ η A ,v (x [ ] k ) where ρ † = √ p 2 -ν (1 + κ 2 (A)) (2.34) with κ 2 (A) = ||A|| 2 ||A -

Numerical experiments

In this section, we investigate the numerical behaviour of the TT-GMRES solver for linear problems with increasing order as it naturally arises in some partial differential equation (PDE) studies. We start by illustrating how the TT-operators of our numerical examples are directly constructed in TT-format, thanks to their peculiarity. For all the examples, we illustrate numerical concerns related to the algorithm convergence and computational costs, with a focus on memory growth and memory saving.

The linear operators of the main problems, addressed in the following sections, are Laplace-like operators. The Laplace-like tensor operator A ∈ R (n 1 ×m 1 )•••×(n d ×m d ) is the sum of operators written as

A =M 1 ⊗ R 2 ⊗ R 3 ⊗ • • • ⊗ R d-2 ⊗ R d-1 ⊗ R d + L 1 ⊗ M 2 ⊗ R 3 ⊗ • • • ⊗ R d-2 ⊗ R d-1 ⊗ R d + • • • + L 1 ⊗ L 2 ⊗ L 3 ⊗ • • • ⊗ L d-2 ⊗ M d-1 ⊗ R d + L 1 ⊗ L 2 ⊗ L 3 ⊗ • • • ⊗ L d-2 ⊗ L d-1 ⊗ M d (2.35) with L k , M k , R k ∈ R n k ×m k for every k ∈ {1, . . . , d}.
As relevant property, these linear operators are expressed in TT-format with TT-rank 2, i.e.,

A = L 1 M 1 ⊗ L 2 M 2 0 R 2 ⊗ • • • ⊗ L d-1 M d-1 0 R d-1 ⊗ M d R d (2.36)
as proved in [81, Lemma 5.1]. Remarking that the general expression of the discrete d-order Laplacian on a uniform grid of n points in each direction is

∆ d = ∆ 1 ⊗ I n ⊗ • • • ⊗ I n + • • • + I n ⊗ I n ⊗ • • • ⊗ ∆ 1
where I n is the identity matrix of size n and ∆ 1 ∈ R n×n is the discrete matrix Laplacian using the central-point finite difference scheme with discretization step h = 1 n+1 , i.e.,

∆ 1 = 1 h 2          -2 1 0 . . . 0 1 -2 1 . . . 0 . . . . . . . . . . . . . . . 0 . . . 1 -2 1 0 0 . . . 1 -2         
.

Then the TT-expression of ∆ d is

∆ d = I n ∆ 1 ⊗ I n ∆ 1 0 I n ⊗ • • • ⊗ I n ∆ 1 0 I n ⊗ ∆ 1 I n . (2.37)
To solve linear systems efficiently, we consider an approximation of the inverse of the discrete Laplacian operator, M, as a preconditioner [START_REF] Hackbusch | Low-rank Kronecker-product Approximation to Multi-dimensional Nonlocal Operators. Part I. Separable Approximation of Multi-variate Functions[END_REF][START_REF] Hackbusch | Low-rank Kronecker-product Approximation to Multi-dimensional Nonlocal Operators. Part II. HKT Representation of Certain Operators[END_REF]. This operator writes

M = q k=-q c k exp(-t k ∆ 1 ) ⊗ • • • ⊗ exp(-t k ∆ 1 ) (2.38)
where c k = ξt k , t k = exp(kξ) and ξ = π q . Thanks to the previously stated property of the sum of TT-vectors, we conclude that the TT-ranks of M will be at most 2q + 1. In Section 2.3.3 we consider the linear system Ax = b and to speed up its convergence we apply the preconditioner TT-matrix M, effectively solving AMt = b. The preconditioner TT-matrix M is always computed by q addends equal to a quarter of the grid step size. To keep the TT-rank of the preconditioner small, we choose to round it to 10 -2 . The choice of the number of addends and the rounding compression is further discussed in Section 2.3.3.1.

To evaluate the converge of the TT-GMRES at the k-th iteration, we display in Section 2.3.3 the stopping criterion η AM,b , that is

η AM,b (t k ) = AMt k -b AM 2 t k + b (2.39)
with t k the preconditioned approximated solution at the k-th iteration. We compute exactly the norm of residual, of the right-hand side, and the iterative preconditioned approximated solution. The L2-norm of the preconditioner operator AM is instead computed by the following sampling approximation. Let W be a set of normalized TT-vectors generated randomly from a normal distribution, then AM 2 is approximated by the maximum of the norm of the image of the elements of W through AM, i.e.,

AM 2 ≈ max w∈W AMw .
Similarly, the L2-norm of A is also approximated by max Aw s.t.w ∈ W . Because we are interested in the magnitude of these norms, we keep this norm estimation process simple and only compute 10 random TT-vectors of W.

In order to investigate the main numerical features of the GMRES implementation described in the previous section we consider two classical PDEs, i.e., the Poisson and convection-diffusion equations.

The Poisson problem writes

   -∆u = f in Ω = [0, 1] 3 , u = 0 in ∂Ω, ( 2.40)
where

f : R 3 → R is such that the analytical solution of this Poisson problem is u : [0, 1] 3 → R defined as u(x, y, z) = (1 -x 2 )(1 -y 2 )(1 -z 2 ).
Let set a grid of n points per mode over Ω, the discretization of the Laplacian over the Cartesian grid is the linear operator -∆ d defined in Equation (2.37) with d = 3. Let b ∈ R n×n×n be the discrete right-hand side in TT-format such that b(i

1 , i 2 , i 3 ) = f (x i 1 , y i 2 , z i 3 ).
The convection-diffusion problem, identical to the one considered in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF], writes

       -∆u + 2y(1 -x 2 ) ∂u ∂x -2x(1 -y 2 ) ∂u ∂y = 0 in Ω = [-1, 1] 3 , u {y=1} = 1 and u ∂Ω\{y=1} = 0 . (2.41)
Setting a grid of n points per mode over [-1, 1] 3 , the Laplacian is discretized as in Equation (2.37) with d = 3. Let ∇ x be discretization of the first derivative of u with respect to mode 1 defined as ∇ x = ∇ 1 ⊗ I n ⊗ I n , similarly ∇ y is the discrete first derivative with respect to mode 2 written as ∇ y = I n ⊗ ∇ 1 ⊗ I n , where ∇ 1 is the order-2 central finite difference matrix, i.e.,

∇ 1 = 1 2h          0 1 0 . . . 0 -1 0 1 . . . 0 . . . . . . . . . . . . . . . 0 . . . -1 0 1 0 0 . . . -1 0          . Let v : [-1, 1] 3 → R 2 be a function such that v(x, y, z) = 2y(1 -x 2 ), -2x(1 -y 2 ) , the two components of v are discretized over the Cartesian grid set on [-1, 1] 3 defining two tensors V 1 , V 2 ∈ R (n×n)×(n×n)×(n×n) such that V 1 = diag(1 -x 2 ) ⊗ diag(2y) ⊗ I n and V 2 = diag(-2x) ⊗ diag(1 -y 2 ) ⊗ I n . Then the discrete diffusion term D writes D = V 1 • ∇ x + V 2 • ∇ y = diag(1 -x 2 )∇ 1 ⊗ diag(2y) ⊗ I n + diag(-2x) ⊗ diag(1 -y 2 )∇ 1 ⊗ I n . (2.42)
The final operator passed to the TT-GMRES algorithm is A = -∆ 3 + D, the right-hand side is the TT-vector b ∈ R n×n×n and the initial guess is the zero TT-vector x 0 . To ensure a fast convergence, similarly to [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF], we consider a right preconditioner M from Equation (2.38) for this test example.

TENSOR TRAIN GMRES

Main features and robustness properties

In this section, we first illustrate in Section 2.3.3.1 the major differences between our GMRES implementation and the one proposed in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF] that mostly highlights the robustness of our variant. We motivate the need for effective preconditioners in Section 2.3.3.1 and illustrate the performance and the main features of preconditioned GMRES in Section 2.3.3.1. All the experiments were performed using python 3.6.9 and with the tensor toolbox ttpy 1.2.0 [109].

Comparison with previous tensor GMRES algorithm

In this section we describe the TT-GMRES introduced in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF], which we refer to as relaxed TT-GMRES, that attempts to use advanced features enabled by the inexact GMRES theory [START_REF] Bouras | Inexact matrix-vector products in Krylov methods for solving linear systems: a relaxation strategy[END_REF][START_REF] Giraud | Convergence in Backward Error of Relaxed GMRES[END_REF][START_REF] Simoncini | Theory of Inexact Krylov Subspace Methods and Applications to Scientific Computing[END_REF][START_REF] Van Den Eshof | Inexact Krylov subspace methods for linear systems[END_REF]. In particular, these inexact GMRES theoretical results show that some perturbations can be introduced in the linear operator when enlarging the Krylov space so that the magnitude of these perturbations can grow essentially as the inverse of the true residual norm of the current iterate. In that context, the accuracy of computation of the linear operator can be relaxed, which motivated the use of this terminology in [START_REF] Bouras | Inexact matrix-vector products in Krylov methods for solving linear systems: a relaxation strategy[END_REF][START_REF] Giraud | Convergence in Backward Error of Relaxed GMRES[END_REF]. The inexact GMRES theory assumes exact arithmetic so that Equation (2.1) holds. In practice, this equality becomes invalid as soon as some loss of orthogonality appears in the Arnoldi basis so that

rk = βe 1 -Hk y = r k = b -Ax k ; (2.43)
that is, the norms of the least squares residual and the true residual differ. In a TT-computational context, these inexact Krylov results motivated the heuristic presented in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF], which consists in transferring the perturbation policy from the matrix to the output of the matrix-vector product. More precisely, the variable perturbation magnitude is implemented by varying the rounding threshold δ applied to the tensor resulting from the matrix-vector product along the iterations. Furthermore, the magnitude of the rounding δ is computed using the least squares residual norm rather than the true residual norm for practical computational reasons. A possible consequence of this choice is that δ is somehow artificially increased.

Although the rounding is performed exactly at the same step in the two algorithms, there are two differences between our TT-GMRES and the relaxed TT-GMRES [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. The first difference is related to the rounding threshold policy that is variable (or relaxed to use the terminology of the pioneering paper on inexact GMRES [START_REF] Bouras | Inexact matrix-vector products in Krylov methods for solving linear systems: a relaxation strategy[END_REF]) and constant in our case. We simply define the value of δ essentially to the value of the target accuracy in terms of backward error (2.3) ((2.39) when a preconditioner is used). The second difference is related to the stopping criterion that is defined in terms of backward error (2.3) in our case ((2.39) when a preconditioner is used) while it is based on a scaled least squares residual defined by Equation (2.44) in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]:

ηb (x k ) = rk b . (2.44)
Because in practice the true residual differs from the least squares residual, this latter is monotonically decreasing towards zero, such a stopping criterion can lead to an earlier stop.

We choose this stopping criterion based on backward error because it is the one for which, in the matrix framework, GMRES is backward stable in finite precision [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF]. Through intensive numerical experiments [START_REF] Agullo | The backward stable variants of GMRES in variable accuracy[END_REF], we observed that our TT-GMRES inherits the same backward stability property. Indeed if δ is the rounding accuracy and x k the GMRES solution at iteration k, then η A,b (x k ) is O(δ) as δ is the dominating part of the rounding error occurring during the numerical calculation. Consequently assuming δ ≤ ε, our GMRES variant is able to ensure a ε-backward stable solution. This property is well illustrated in Figure 2.6 in the case of preconditioned GMRES. The 3d convection-diffusion problem with 63 discretization points is solved using 3 different rounding accuracies, i.e., δ ∈ {10 -3 , 10 -5 , 10 -8 }, and a maximum of 100 iterations. For each value of δ, the backward error η AM,b (t k ) decreases and stagnates around δ.

The second significant difference between the two GMRES variants is the choice of the rounding threshold along the iterations that is constant for us and varies as the inverse rk in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. This variation of the rounding is illustrated in Figure 2.7. We solve with the two different algorithms the same convection-diffusion problem with 63 discretization points in each spatial dimension. We select three different rounding accuracies δ ∈ {10 -3 , 10 -5 , 10 -8 } and perform 100 iterations of full GMRES (i.e., no restart). In Figure 2.7F we see the extreme growth of the rounding threshold, when it is scaled by the norm of rk , the least-squares residual norm that becomes smaller and smaller. When the rounding accuracy becomes significantly large, the TT-ranks in relaxed TT-GMRES are cut to 1, losing almost all the information carried in the tensor. Figure 2.7A shows the scaled residual used as stopping criterion in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. We observe that if δ is not relaxed along the iterations, the value of ηb decreases extremely quickly, reaching 10 -10 for δ = 10 -3 and at least 10 -14 for the other rounding accuracies. On the other hand, if the rounding accuracy is relaxed during the iterations, we see that in all the cases ηb reaches at least 10 -6 . However, the comparison of Figure 2.7A and Figure 2.7B illustrates the numerical difference of the least squares residual norm and the true residual norms given by Equation (2.43). This comparison reveals that ηb (x k ) with the relaxed δ converges, but η b (x k ), that is also a backward error as defined in (2.4), does not. It means that the solutions computed using the relaxed δ are meaningless in terms of backward error accuracy. Similar conclusions can be drawn from Figure 2.7C that presents the history of η AM,b for the two algorithms. When the rounding accuracy is kept constant, we recover a backward stable behaviour similar to the one proved for finite precision calculation in classical linear system solution in matrix format. Indeed η AM,b always reaches and stagnates around the selected constant value of δ. On the contrary, when δ is relaxed at each iteration, the quantity η AM,b stagnates quickly slightly above 10 -3 , whatever the starting value of δ. From these two figures, we conclude that relaxing the rounding accuracy and using ηb as a stopping criterion, together or independently, do not provide any insight into the quality of the computed solution.

Obviously, the choice of relaxing the rounding accuracy has a powerful effect on the rank of the last Krylov basis vector and the solution, as illustrated by Figure 2.7D and 2.7E. Indeed in the case of the last Krylov basis vector, its TT-rank oscillates around 1 for all the iterations, after the 15-th one approximately. Similarly, the solution TT-rank stays equal to 1, after increasing at the very first steps. Unfortunately, the computed solutions are numerically meaningless. In the following, we consider calculation with convergence threshold and rounding accuracy equal to 10 -5 , that is, δ = ε = 10 -5 , with a maximum of 500 iterations and restart m = 25.

Poisson problem

We consider the restarted TT-GMRES for the solution of the 3-d Poisson problem with n ∈ {63, 127, 255}. Figure 2.8A shows that the algorithm is able to converge to the prescribed tolerance ε = 10 -5 with the number of iterations that increases with the number of discretization points. This high number of steps to solve a quite simple PDE motivates the need for a preconditioner. Indeed in general the larger the number of TT-GMRES iterations, the larger the TT-rank growth for the Krylov basis vectors; consequently, the higher the computational cost per iteration. For that example, it can be seen in Figure 2.8B that the rank of the current iterate grows significantly during the first iterations (first 100 iterations for n = 63 and the first 200 steps for n ∈ {127, 255}) before decreasing in a non monotonic way. We infer that this particular behavior is related to the separable nature of the analytical solution, which is

u(x, y, z) = diag(1 -x 2 ) ⊗ diag(1 -y 2 ) ⊗ diag(1 -z 2 )
with rank 1 and as consequence its TT-rank is also bounded by 1. After some iterations TT-GMRES seems to capture the main structure of the solution, being able to almost halve the TT-ranks, as it is visible in Figure2.8B. Another quantity monitored during the iterations is the growth of the last Krylov vector TT-ranks. In Figure 2.8C the maximum TT-rank of the last Krylov vector presents a steep increase during the first phase, followed by a slightly decreasing phase. The behavior of the maximum TT-rank establishes the trend in the compression ratio of the last vector and the entire basis. Indeed the curves of Figures 2.8D and 2.8E are the same of Figure 2.8C scaled by a constant, equal to n 3 for the first and kn 3 for the second where k is equal to the current iteration in the restart. Lastly, in Figure 2.8C mainly during the second phase, there are many consecutive drops in the maximum TT-ranks which appear with a specific frequency. They are due to the restart after every other 25-th iteration. In fact, at restart the new Krylov vector is equal to the normalized rounded residual, whose basic starting TT-ranks is the one of x, equal to 21 at maximum. Lastly notice that in the worst case storing the last Krylov vector and the entire Krylov basis request approximately 80% for n = 63 of the memory that would be used for storing entirely them. Furthermore, this ratio decreases when the number of points per mode increases (i.e., n ∈ {63, 127, 255}), which is an appealing feature of the TT-format that allows the solution of larger problems for a given memory budget compared to the situation where the full tensors would have to be stored.

Preconditioner parameter study

In this section, the preconditioner firstly introduced in (2.38) is further investigated. In particular, we focus on the effect on the convergence of the number of addends and on the compression accuracy chosen to compute it. As in Equation (2.38), let M ∈ R (n×n)ו••×(n×n) be the d-order TT-matrix that approximates the inverse of the discrete Laplacian ∆ d , cf. [START_REF] Hackbusch | Low-rank Kronecker-product Approximation to Multi-dimensional Nonlocal Operators. Part I. Separable Approximation of Multi-variate Functions[END_REF][START_REF] Hackbusch | Low-rank Kronecker-product Approximation to Multi-dimensional Nonlocal Operators. Part II. HKT Representation of Certain Operators[END_REF], defined as

M = q k=-q c k exp(-t k ∆ 1 ) ⊗ • • • ⊗ exp(-t k ∆ 1 )
where c k = ηt k , t k = exp(kη) and η = π/q. As already mentioned, since M is a sum of tensors, its TT-rank is greater or equal than 2q + 1. The magnitude of TT-ranks of M conditions the TT-ranks of the Krylov basis vectors and the final solution. Therefore it is convenient to keep M TT-rank significantly small, either by reducing the number of addends, i.e., choosing a low value for q, or by compressing M with an accuracy τ . With the help of a Poisson problem, we illustrate the trade-off between the number of addends and compression accuracy which leads to the optimal convergence. We consider the Poisson problem, written as

   -∆u = 1 in Ω = [0, 1] 3 u = 0 in ∂Ω
so that the effect of the preconditioner is as evident as possible. Let -∆ 3 be the discretization of the Laplacian operator over a grid of n = 63 points per mode. Similarly let b ∈ R n×n×n be a tensor with all the entries equal to 1. Then, setting A = -∆ 3 , TT-GMRES solves the tensor linear system Ax = b, preconditioning it on the right as AM q,τ t = b for q ∈ {2, 8, 16, 32, 64} and τ ∈ {10 -2 , 10 -8 }. The parameters of TT-GMRES are tolerance ε = 10 -16 , rounding accuracy δ = 10 -5 , dimension of the Krylov space m = 25 and a maximum of 2 restart. We set TT-GMRES tolerance equal to the machine precision so that the algorithm performs all the 50 iterations. In Table 2.3, we report the maximal TT-rank of M q,τ and the AM q,τ 2 rounded at the third digits for all the combinations of q and τ . Remark that fixed a value for q the L2 norm of the preconditioned linear system is the same up to the third digits for both the values of τ . This seems to suggest that the number of addends plays a key role in determining the quality of the preconditioner, while the rounding accuracy τ affects more significantly the TT-rank, removing kind of unnecessary information. Indeed for τ = 10 -2 and q ≥ 8, the maximal value of the TT-rank is always 5, but depending on an increasing number of addends, the L2 norm gets closer to 1. Similarly for τ = 10 -8 and q ≥ 32, the maximal TT-rank is 15 and the rounded L2 norm is equal to 1.

τ = 10 -2 τ = 10 -8 q 2 8

TENSOR TRAIN GMRES

Looking at the convergence history in Figures 2.9A and 2.9B, a value of q ≥ 16 is already sufficient to reach in a very low number of iterations the bound 10 -5 , due to the TT-GMRES rounding value δ. Figure 2.9B shows clearly that by keeping more information in the preconditioner, TT-GMRES may reach very low levels. However in Figure 2.9D we observe the side effect of more information. The TT-rank of the last Krylov vector increases significantly for a very accurate preconditioner. More precisely, comparing Figures 2.9C and 2.9D, the TT-rank of the last Krylov vector doubles if the preconditioner is more accurately rounded. Notice also that in Figure 2.9C, the TT-rank for q ∈ {16, 32, 64} is almost the same. For the solution viewpoint, the rounding accuracy chosen for the preconditioner has not a big impact on its TT-rank. Indeed, as plotted in Figure 2.9E and 2.9F, for both the values of τ and for all q ≥ 8, the TT-rank of the solution is equal to 5, while only for q = 2 it increases, meaning that only 5 addends are not sufficient to speed up the discrete Laplacian convergence.

Convection-diffusion

We test three different grid sizes, i.e., n ∈ {63, 127, 255}, with preconditioner M from Equation (2.38) with q ∈ {16, 32}. Indeed without it, even with the smallest size, TT-GMRES does not converge to the prescribed tolerance ε = 10 -5 in a reasonable number of iterations. However using the preconditioner defined in Equation (2.38), an approximated solution is found in 5 or less iterations, as displayed in Figure 2.10A. The preconditioner in this case has an extremely strong effect, from which the TT-rank growth and the memory consumption benefit. In Figure 2.10B the maximum TT-rank exceeds in the worst case the value of 35, but to fully interpret this information the compression ratio must be taken into consideration. Figure 2.10C shows that in the worst case to store the last Krylov maximal TT rank q = 2 q = 8 q = 16 q = 32 q = 64 (D) Maximal TT-rank of the last Krylov vector for τ = 10 -8 maximal TT rank q = 2 q = 8 q = 16 q = 32 q = 64 (F) Maximal TT-rank of the iterative solution for τ = 10 -8 vector in TT-format we use approximately 12% of the memory we would need to store the full tensor. Similarly, in Figure 2.10D, we see that storing in TT-format the entire Krylov basis request in the worst case only 7% of the memory that would be used to store the full tensors basis. Although not reported in this document, a more stringent accuracy would require a smaller rounding threshold and consequently a larger memory to store the TT-vectors.

Solution of parameter-dependent linear operators

This section focuses on 4-d PDEs, namely parametric convection-diffusion and stationary heat equations. The domain of both problems is obtained as a Cartesian product of a 3-d space domain and a further parameter space. The common idea for these PDEs is solving for all discrete parameter values simultaneously, getting an "all-in-one" solu-tion. The structure of the operators enables us to check numerically the quality of the theoretical bounds stated in Section2.3.2.

Parametric convection-diffusion

The parametric convection-diffusion problem is a variation of Problem (2.41), defined as

       -α∆u + 2y(1 -x 2 ) ∂u ∂x -2x(1 -y 2 ) ∂u ∂y = 0 in Ω = [-1, 1] 3 , u {y=1} = 1 and u ∂Ω\{y=1} = 0 . (2.45)
As in Section 2. discrete "all-in-one" problem fits into the hypothesis of Proposition 2.3.5 and 2.3.7. Remark that the "all-in-one" linear operator is directly constructed as a TT-matrix from the TT-matrix of the single linear system, while the "all-in-one" right-hand side is constructed as a full tensor and then converted into a TT-vector. TT-GMRES is used for solving the "all-in-one" linear system for n ∈ {63, 127, 255} and p = 20, with the order-d preconditioner M defined in Equation (2.38) with q ∈ {16, 32} tensorized with the identity

M = I p ⊗ M. (2.46)
Figure 2.11A shows that the algorithm converges in less than 20 iterations for the first two values of n and in less than 25 for n = 255; that is, no restart is needed. For the computational side, Figure 2.11B displays the maximal TT-rank of the last Krylov vector, which in the worst case is lower than 100. This result translates in terms of memory by a need of slightly more than 4% of the memory that would be required to store the full Krylov vector in the worst case, as highlighted by Figure 2.11C. Looking at the cost of storing the entire Krylov basis in Figure 2.11D, we see that TT-format requires around 2% of the memory necessary to store the entire Krylov basis in full tensor format. We now investigate the tightness of the bound given in Proposition 2.3.5 and

TENSOR TRAIN GMRES

2.3.7. Figure 2.13 shows the quality of the bound for η b for ∈ {1, . . . , p}. For all the values of n, the η b 1 curve dominates the other during the first half of the iterations. In the optimal case, the difference between η b and η b is lower than one order of magnitude. The three scaled curves overlap from the third iterations for all the grid sizes, meaning that the approximation of the scaling coefficient given by ρ * is extremely valid in this example. We see that the orange curve corresponding to η A 5 M,b 5 and the blue one for η A 20 M,b 20 intersect frequently, with a difference of one order at most. Moreover the difference between η A 5 M,b 5 and η AM,b scaled by ρ 5 is lower than one order of magnitude in the optimal case, while in the worst case it is not larger than two orders. Therefore we conclude that for this PDE the bound of the "all-in-one" for the individual solution is quite tight. Notice that to estimate ρ * no extra computation is required, while the norm of A Mt [ ] k has to be computed to get the value of ρ (t k ).

Heat equation with parametrized diffusion coefficient

We consider the heat equation with parametrized diffusion coefficient studied in [START_REF] Kressner | Low-Rank Tensor Krylov Subspace Methods for Parametrized Linear Systems[END_REF] and defined as

-∇ • (σ θ (x, y, z)∇u(x, y, z)) = 1 in Ω = [-1, 1] 3 , u = 0 in ∂Ω. (2.48)
where the coefficient σ θ is a piece-wise constant function such that The problem stated in Equation (2.48) writes equivalently

σ θ (x, y, z) = 1 + θ in [-0.5, 0.5]
     -∆u(x, y, z) -θ∇ • 1 Ξx (x)1 Ξy (y)1 Ξz (z)∇u(x, y, z) = 1 in Ω = [-1, 1] 3 , u = 0 in ∂Ω.
(2.49) After setting a grid on n points along each direction on Ω, the first term B 0 of the operator in (2.49) is discretized by the 3-d Laplacian ∆ 3 . For the second term B 1 , notice that the indicator function 1 Ξ is trivially not differentiable on Ξ boundaries. So it is approximated on the grid points, paying attention to not set them on ∂Ξ. The final expression of B 1 is for different values of θ, we tensorize B 0 and B 1 by a diagonal matrices, adding a fourth dimension. The tensor discrete operator A ∈ R (p×p)×(n×n)×(n×n)×(n×n) of the "all-in-one" problem writes

B 1 = D x ∆ 1 ⊗ D y ⊗ D z + D x ⊗ D y ∆ 1 ⊗ D z + D x ⊗ D y ⊗ D z ∆ 1 where ∆ 1 is the 1-d discrete Laplacian, D x = diag(1 Ξx i ) ∈ R
A = I p ⊗ B 0 + Θ ⊗ B 1
where Θ = diag(θ 1 , . . . , θ p ) for θ i ∈ [0, 10] uniformly distributed for i ∈ {1, . . . , p}. The right-hand side of the "all-in-one" problem is

b = I p ⊗ c.
Remark that since c = 1 by construction, then b = √ p. We perform experiments with full TT-GMRES (i.e., no restart) for n ∈ {63, 127} and p = 20, with the preconditioner defined in Equation (2.46) with q ∈ {16, 32}. Figure 2.14A shows that TT-GMRES converges to the prescribed tolerance in approximately 20 iterations. From the point of view of the memory consumption, in Figure 2.14B we see that for n = 63 the maximum TT-rank is lower than 200, while for n = 127 it is lower than 250. In terms of memory saving, Figure 2.14C shows that in the worst case we are using only 10% and less than 5% of the memory necessary to store one full tensor of the Krylov basis and the entire full basis respectively. In Figure 2.15 we have the relation of η b and η b for ∈ {1, . . . , p}. All the curves present the same shape, with the one associated with θ 1 = 0 being the most peculiar one. We see that in the optimal case the distance between the "all-in-one" curve and the individual ones is lower than one order of magnitude, while in the worst case, realized by θ 1 = 0, the difference is approximately of two orders. A similar argument holds for η AM,b bound. As in Section 2.3.3.2, we compute m and M , as defined in Equation (2.47), which are equal to m = 20 and M = 1 respectively. In Figure 2.16 we see that the two curves η A M,b have a starting and ending overlapping part, while in the internal part they differ by less than one order of magnitude. The three scaled curves for η AM,b overlap from the third iteration. As in the previously studied case, ρ * from Corollary 2.3.8 provides a good approximation of the scaling coefficient. In the optimal case, the distance is of one order of magnitude approximately, while in the worst one a little more than one order.

Solution of parameter dependent right-hand sides

This section aims to investigate the numerical properties of some examples in the context of the multiple right-hand side solution, following the tensorized approach described in Section 2.3.2.

Poisson problem

In this section, we solve simultaneously multiple Poisson problems stated in Equation ( where e [ ] ∈ R n×n×n is the -th slice with respect to the first mode of e ∈ R p×n×n×n a realization of the normal distribution N (0, 1). Since the aim is solving simultaneously the p problems, as in Section 2.3.2, we define the "all-in-one" tensor linear operator A ∈ R (p×p)×(n×n)×(n×n)×(n×n)

A = I p ⊗ (-∆ 3 )
while the "all-in-one" right-hand side is c ∈ R p×n×n×n such that

c = I p ⊗ b + e.
We consider the solution of the problem with n ∈ {63, 127, 255} and p = 20. To speed up the convergence we introduce the preconditioner defined in (2.46) with q ∈ {16, 32}. Notice that theoretically, the TT-rank of c may become extremely large, leading to memory over-consumption and higher computational costs. To face this drawback, we impose a small TT-rank to e [ ] , so that the TT-rank of c ends up being 11 at maximum. To study the bounds stated in Section 2.3.2, we need to comply with the hypothesis so that we scale each individual right-hand side by its norm, so that c = √ p.

As we can see in Figure 2.17A, TT-GMRES converges in 5 iterations for n = 63, in 7 for n = 127 and in 9 for n = 255. Figure 2.17B shows that the TT-rank of the last Krylov vector becomes quickly large, with maximum values ranging from 200 to 300. However looking at Figures 2.17C and 2.17D, the compression ratio for a single basis vector and for the entire basis remains extremely small, from 0.05 to 0.2 for the first one and from 0.02 and 0.14 for the entire basis, meaning that the TT approach is still effective from the memory point of view. As in the parametric operator case, we study the bounds expressed in Propositions 2.3.4 and 2.3.7. In Figure 2.18, we see that the bound for η b is always quite tight, around 1 order of magnitude approximately. To use the result of Proposition 2.3.7, we set w equal to the number of iterations to converge and for every ∈ {1, . . . , p}, we define the vector γ ∈ R w such that

γ (i) = ψ (t k ) for every k ∈ {1, . . . , w}.
We define m and M as the indexes which realize the minimum and the maximum of γ norm, i.e., In this specific case for each grid point step, the value of m and M is reported in Figure 2.19. The same Figure shows that the bound in this specific case is quite good, with approximately less than 1 order of magnitude of difference, in the optimal and the worst case. Moreover, the three scaled "all-in-one" curves overlap from the second iteration, suggesting again that ρ * from Corollary 2.3.8 is a good approximation of the scaling factors.

Convection-diffusion problem

As previously, this section aims to illustrate the solution of multiple convection-diffusion problem (2.41), with different right-hand sides. Let A 0 be the discretization of (2.41) operator over a Cartesian grid of n points per mode for the domain Ω = [0, 1] 3 . Let b ∈ R n×n×n be the right-hand side discretization defined in Section 2.3.3.1. We define the individual linear system as

A 0 y = b + e
where e ∈ R n×n×n is a realization of the normal distribution N (0, 1) for every ∈ {1, . . . , p}.

Since the aim is solving simultaneously the p problems, as in Section 2.3.2, we define the "all-in-one" tensor linear operator A ∈ R (p×p)×(n×n)×(n×n)×(n×n) while the "all-in-one" right-hand side is c ∈ R p×n×n×n such that

A = I p ⊗ (-∆ 3 ) 1
c( , i 1 , i 2 , i 3 ) = b(i 1 , i 2 , i 3 ) + e (i 1 , i 2 , i 3 ).
for every i k ∈ {1, . . . , n k }, ∈ {1, . . . , p} for k ∈ {1, . . . , 3}. The problem is solved for n ∈ {63, 127} and p = 20. As in all the previous cases of study, we use the preconditioner stated in (2.46) with q ∈ {16, 32} and we impose a small TT-rank to e so that the TT-rank of c ends up being 11 at maximum. Figure 2.20A illustrates the convergence history in 5 iterations for both the grid sizes. If we compare Figure 2.20A with 2.10A, we observe that the curves are very similar. Generally speaking, the number of iterations for GMRES to converge, neglecting the effect of the rounding, is equal to the number of eigenvectors that span the subspace where the right-hand side lives. This implies that if all the right-hand sides belong to the same linear subspace, the number of iterations necessary to converge is the same, implying that under this hypothesis solving for 1 or p right-hand sides requires the same number of iterations. This point is further discussed in Section 2.3.3.3. From the point of view of the memory consumption, the comparison of Figure 2.10B and 2.20B shows that the solution for 20 right-hand sides leads to TT-rank significantly larger, from 25 to 30 in the single right-hand side solution versus more than 200 for the 20 right-hand side "all-in-one" system. However, if we had solved 20 systems independently, summing all the TT-ranks, we could have reached a maximum of 500 up to 700. This becomes more interesting if we compare the compression ratios for the last Krylov vector, looking at Figure 2.10C and 2.20C. We have a ratio from 0.02 up to 0.12 for a single right-hand side solution versus 0.1 up to 0.17 for the simultaneous one, which shows that these ratios are extremely closed, considering that in the second case we are solving in a higher dimension. A similar argument holds for the ratio of compression of the entire Krylov basis. In Figure 2.20D, the ratio is between 0.06 and 0.12, while in Figure 2.10D it is between 0.01 and 0.07.

In Figure 2.21, we present the bound for η b stated in Proposition 2.3.4. We see that it is quite tight during the first iterations and gets looser at the end, setting at more than 1 order of difference. As in the previous subsection, we compute m and M according to Equation (2.50), deciding which curves are plotted in Figure 2.22. The resulting bound, displayed in Figure 2.22, is quite tight, being of slightly less than 1 order of magnitude approximately, with the three scaled curves overlapping from the second iteration.

Multiple right-hand sides: a focus on eigenvectors

In this appendix, we study further the convergence of a multiple right-hand side problem. Indeed comparing the convergence history of the convection-diffusion problem, see Subsection 2.3.3.1 and of the multiple right-hand side convection-diffusion problem discussed in 2.3.3.3, notice that the number of iterations necessary to converge is equal, 5 in both cases. This appendix explains the causes of the phenomenon.

As we already explained, given a (tensor) linear system Ax = b and the null tensor as initial guess, at the k-th iteration GMRES minimizes with respect to x the norm of the residual Axb on the Krylov space of dimension k defined as

K k (A, b) = span b, Ab, . . . , A k-1 b
where A h is obtained from h contractions over the indexes (1, 3, . . . , 2d -1, 2d + 1) of tensor operator A. If b is equal to e i an eigenvector of the tensor operator A, then the Krylov space writes

K k (A, b) = span b, Ab, . . . , A k-1 b = span e i , λ i e i , . . . , λ k-1 i e i = span e i }
where λ i is the i-th eigenvalue of A. The Krylov space dimension is equal to 1, i.e., the number of eigenvectors e i necessary to express the right-hand side. Theoretically, the number of iterations necessary to converge, i.e., the dimension of the Krylov space where the exact solution lives, is by linearity equal to the number of eigenvectors necessary to express the right-hand side as their linear combination. In the problems presented in 2.3.3.3, we add a random generated tensor to the chosen right-hand side. Comparing the results for a single right-hand side and multiple ones in the convection-diffusion problem, we may conclude that the introduced error has not increased the number of eigenvectors, for the tolerance chosen.

Let now consider a more peculiar problem with two right-hand sides, living in subspaces generated by different eigenvectors. More in detail, one right-hand side belongs to the subspace generated by a single eigenvector, while the other to the subspace generated by other j different eigenvectors. Thanks to our previous argument, theoretically the two systems converge independently with a different number of iterations, one for the first and j for the second. When we solve the two systems together, we expect the "all-in-on" system to converge as the slowest one, i.e., as the slowest converging one. Let e 1 , . . . , e j+1

TENSOR TRAIN GMRES

be the first (j + 1) different eigenvectors of the 3-dimensional discrete Laplacian -∆ 3 . We consider the two following linear systems

-∆ 3 y 1 = e 1
(2.51)

-∆ 3 y 2 = j+1 =2
e .

(2.52)

As described in Section 2.3.2, we define the "all-in-one" linear system with the 'diagonal' tensor operator A and the "all-in-one" right-hand side b. TT-GMRES is used to solve this "all-in-one" system for j = 10, for a grid step size equal to n ∈ {63, 127, 255}, without preconditioner, with tolerance ε and rounding accuracy δ equal to 10 -5 , no restart and a maximum of 50 iterations. Figure 2.23A shows the convergence history of the problem with the three different grid sizes. Since there is no preconditioner the convergence is kind of slow if compared with Figure 2.17A. At the same time, the TT-ranks grow not too quickly, because both there is not a randomly generated error and there are just two right-hand sides. The residual curve associated with Equation 2.51 in blue is almost flattened for all the grid sizes, suggesting that GMRES almost immediately minimized it completely. On the other side the orange curve of problem (2.52) residual decreases slowly, meaning that minimizing its norm requires more steps. Lastly in all the plots, the "all-inone" residual curve follows exactly the eigenvector sum problem residual, confirming that the general convergence of the "all-in-one" system is decided by the slowest converging system, that is our intuition, confirmed in for all the grid sizes in Figure 2.24. As last remark, the "all-in-one" doesn't converge in 10 iterations because of the rounding effect, which slows down the convergence.

Conclusive remarks

The focus of this chapter is set on the backward stability of the iterative solver GMRES when the data are perturbed. We study numerically the GMRES backward stability in the framework, referred to as variable accuracy, where the storage and the computational accuracy are decoupled. This choice reflects and models the need of compressing data due to memory constraints, leading to perturbations in their representation. In particular, the storage perturbations are either synthetic in the matrix case or due to practical implementation choices, for example when an agnostic lossy compressor or the TT-formalism are applied in the matrix and tensor context respectively. Our experiments show that the backward stability of GMRES is maintained in the variable accuracy approach, whether they are component-wise, consistently with the theoretical results of [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] either they are norm-wise. Our realization of GMRES in TT-format is further compared numerically with the previous relaxed version of TT-GMRES, presented in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF], concluding that our is significantly more robust than the older one. Moreover, we develop some theoretical backward stable bounds for the simultaneous solution of many order d tensor linear systems through a unique order (d + 1) tensor linear systems. The tightness of these bounds is numerically analysed too.

This chapter is formed by two main sections: the first, i.e., Section 2.2 dedicated to the matrix results, and the second, that is Section 2.3, to the tensor ones. After recalling the main linear algebra theorems related to the backward stability of GMRES in Section 2.2.1, we describe our variable accuracy approach in Section 2.2.2.1. We report about the numerical stability of GMRES when the perturbations due to compression simulations or to an agnostic lossy compressor are component-wise in Section 2.2.2 and in Section 2.2.3 when they are norm-wise.

The second half of this chapter, namely Section 2.3, aims to investigate the GMRES solver extended to the tensor framework through the TT-formalism, which is a particular case of the norm-wise perturbation in the variable accuracy approach. The TT-GMRES

CONCLUSIVE REMARKS

algorithm structure is presented in Section 2.3.1. In Section 2.3.2, we focus on the solution of many tensor linear systems of order d whose operator or right-hand side depends on a parameter. After stating the assumptions, we illustrate how to mode many tensor linear systems of order d into a unique tensor linear system of order (d + 1). Moreover, we theoretically prove some bounds linking the solution of a single tensor linear system of order d with the corresponding slice extracted from the solution of the order (d + 1) tensor linear system, presenting the construction with dense format tensors and TT-format ones, in Section 2.3.2.1. These bounds are meant to guarantee a certain numerical quality when solving many tensor linear systems at one. All the numerical experiments are collected in Section 2.3.3. In particular, in Section 2.3.3.1, the numerical robustness of our TT-GMRES algorithm is shown in comparison with the relaxed implementation of GM-RES in TT-format, proposed in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. Further experiments and outcomes related to the backward stability of our TT-GMRES solver are reported in Section 2.3.3.1 and 2.3.3.1. Except for the results in Section 2.3.3.1, all the others are obtained with the help of a tensor right preconditioner for the multilinear operator. The chosen preconditioner depends on several parameters, whose effects are numerically analysed in Section 2.3.3.1. The outcomes related to parameter-dependent multilinear operators are collected in Section 2.3.3.2, while the numerical outputs of parameter-dependent right-hand sides are found in Section 2.3.3.3.

Chapter 3

Orthogonalization schemes in TT-format

Introduction

Many numerical methods require to generate from any set of m independent vectors of R n an orthogonal basis of the subspace of R n spanned by these vectors. Different orthogonalization algorithms have been proposed during the years to accomplish this task. From another viewpoint, they allow also the computation of the matrix QR factorization. Indeed, if the input m vectors of R n are organized into a matrix A ∈ R n×m , then some orthogonalization schemes are able to factorize A into the product of an orthogonal matrix Q ∈ R n×m and an upper triangular one R ∈ R m×m . Among the most important numerical algorithms, we consider the Classical Gram-Schmidt (CGS) [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF][START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF], the Modified Gram-Schmidt (MGS) [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF][START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF], their versions with re-orthogonalization, named CGS2 and MGS2 [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Daniel | Reorthogonalization and Stable Algorithms for Updating the Gram-Schmidt QR Factorization[END_REF][START_REF] Parlett | The Symmetric Eigenvalue Problem[END_REF], the Gram approach [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF] and the Householder transformation [START_REF] Householder | Unitary Triangularization of a Nonsymmetric Matrix[END_REF]. CGS and MGS are algorithms realizing the Gram-Schmidt method whose fundamental idea is sequentially removing from an input vector its projection along the previously computed orthonormal vectors and normalizing it after. The CGS2 and MGS2 procedures are meant to enhance the CGS and MGS qualities orthogonalizing once more in the same way the basis computed with CGS and the MGS respectively. The Gram method computes the orthogonal basis taking advantage of the Cholesky factorization of the Gram matrix, defined by the scalar product of input vector, while the Householder transformation relies on orthogonal reflections constructed from the input vector set and used to reflect the canonical basis.

A key point for these orthogonalization algorithms is their numerical stability in finite precision calculation, estimated by the loss of orthogonality that expresses how much the computational rounding errors affect the computed basis orthogonality. This aspect has been deeply studied throughout the years, leading to many interesting theoretical results, relating the loss of orthogonality to the linearly dependency of the input vectors. The authors of [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF][START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF] establish some theoretical bounds for CGS and MGS loss of orthogonality, showing that the basis produced by MGS is better in terms of orthogonality 86

ORTHOGONALIZATION SCHEMES

than the CGS one. The bounds proved in [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF] for CGS2 and MGS2, confirms that this reorthogonalization effectively improves the orthogonality of the computed basis. Bounds for the loss of orthogonality of the Householder transformation of the Gram method are proved in [START_REF] Wilkinson | Modern Error Analysis[END_REF] and [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF] respectively.

All the cited algorithms naturally translate in the tensor world, i.e., starting from a set of m tensors of R n 1 ו••×n d , they produce an orthogonal basis for the relative subspace of dimension m of R n 1 ו••×n d . Clearly, these orthogonalization schemes can work with dense tensors, see Equation (1.3), but as already stated, the storage and operation cost for full format tensor grow exponentially with the tensor order. Thus, it is convenient to work with compressed tensor format.

As inevitably happens with TT-algorithms, the sequence of operations defining the orthogonalization kernels in TT-format, described in Section 1.3.2, leads to a growth in the TT-ranks, making necessary some rounding steps that may affect the orthogonality quality of the basis. The objective of this chapter is to investigate both the orthogonality of the computed TT-vector basis and to relate the results with the theoretical results of classical numerical matrix computation. To complete the analysis, we study also a possible application of these kernels: the subspace iteration eigensolver [START_REF] Saad | Numerical Methods for Large Eigenvalue Problems[END_REF], an eigensolver requiring orthogonalization schemes. In particular, we extend it to the tensor framework with the TT-formalism, defining the the SUBSPace ITeration in TT-format (TT-SUBSPIT), and we test it with the six orthogonalization kernels in TT-format, previously studied. Thus, the TT-SUBSPIT numerical performances, i.e., the quality of the eigenvalues, are investigated in relation with the orthogonalization kernels plugged into.

The remainder of this chapter is organized as follows. Section 3.2 begins with the description of the six orthogonalization schemes extended to the tensor context through the TT-formalism. We address also the complexity in terms of TT-rounding operations, representing the computationally most expensive operation. In Section 3.2.4 we recall briefly the classical numerical linear algebra theoretical bounds related the loss of orthogonality of these schemes. The theoretical results are linked with the numerical experiments, collected in Section 3.2.5, of the same orthogonalization schemes extended with the TT-format. The similarities between the classical orthgonalization kernels and their TT-versions are summarized in Section 3.2.6. The second Section 3.3 presents the TT-SUBSPIT algorithm, the TT-version of the subspace iteration eigensolver, as a case of study for these orthogonalization techniques. In Section 3.3.1 we describe the TT-SUBSPIT algorithm and in Section 3.3.2 we present the output of the numerical examples. In particular, we analyse the quality of the eigenvalue approximations, of the convergence and the memory requirements depending on the selected orthogonalization kernel.

Orthogonalization schemes

In the following sections, we describe the classical orthogonalization kernels, we propose their extensions to TT-vectors and we discuss their numerical stability comparing it with the theoretical results of classical matrix computation.

Classical and Modified Gram-Schmidt

In theoretical linear algebra the Gram-Schmidt process [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF][START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF] is a tool to produce an orthonormal basis from a given set of vectors. Let A = {a 1 , . . . , a m } be a set of m linearly independent vectors of R n , then the key idea of the Gram-Schmidt process is to incrementally build an orthonormal basis of the space spanned by the elements of A. At the i-th step, the i-th element a i is made orthogonal to the previously computed (i -1) orthonormal vectors {q 1 , . . . , q i-1 }, subtracting from a i its projection along q j , given by the scalar product of a i and q j for j ∈ {1, . . . , i -1}. After normalization the new vector is q i , the i-th vector of the final orthonormal basis. This mechanism is easily transported in the tensor framework. Consequently instead of stating the theory of Gram-Schmidt procedure with the tensor notation, we illustrate the two different realizations of this theoretical tool only in the TT-format. We carefully underline the differences with the classical matrix implementations.

Classical schemes without reorthogonalization

The direct implementation of the Gram-Schmidt process is known as Classical Gram-Schmidt (CGS) and its TT-version is sketched in Algorithm 11. Iteratively TT-CGS initializes p i to a i for every i ∈ {1, . . . , m}, see line 3. Then in the core loop, the algorithm subtracts from p i the projection of a i along the (i -1) previously computed tensors q j of the new orthogonal basis, as described in lines 5 and 6. As last step, p i is normalized and added in the new orthonormal basis Q = {q 1 , . . . , q i }. The projections of a i along q j for j ∈ {1, . . . , i -1} define the i-th column of R, which is consequently upper triangular by construction. The i-th diagonal entry of R is the norm of p i computed in line 9. These steps appear in both the tensor and in the matrix version of the CGS algorithm. However when dealing with compressed format tensors, we must ensure that different algorithm steps do not reduce significantly the compression quality. Since we are dealing with low-rank TT-vectors, we have to ensure that the TT-ranks stay small. Indeed assuming that p k and q j have maximum TT-rank r for every j ∈ {1, . . . , k -1}, then after (k -1) subtractions, i.e., after (k -1) repetitions of line 6, the TT-rank of p will be bounded by kr. To limit the TT-rank growth, we introduce in line 8 the compression of p i through the TT-rounding algorithm with accuracy δ, which is the most computationally expensive operation in orthogonalization algorithms. Consequently the TT-CGS complexity depends on the number of TT-rounding calls and their complexity, which is known to be O(dnr 3 ) where d is the order of the TT-vector rounded, n and r are the maximum of the mode size and of the TT-rank respectively. However in TT-CGS and in the other orthogonalization methods studied, the TT-rank is not always known, since we are prescribing a rounding accuracy δ. Therefore here and after, we estimate the complexity of the orthogonalization algorithm in terms of the number of TT-rounding operations. The complexity of TT-CGS is equal to m TT-rounding operations. for j = 1, . . . , i -1 do compute the projection of a i along q j 5:

R(i, j) = a i , q j remove the projection of a i along q j 6:

p = p -R(i, j)q j 7:
end for 8: p = TT-round(p, δ)

9:
R(i, i) = ||p|| 10:

q i = 1/R(i, i) p normalize p 11: end for 12: return: Q = {q 1 , . . . , q m }, R Algorithm 12 Q , R = TT-MGS(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + 2: for i = 1, . . . , m do 3: p = a i 4:
for j = 1, . . . , i -1 do compute the projection of p along q j 5:

R(i, j) = p, q j remove the projection of p along q j 6:

p = p -R(i, j)q j 7:
end for 8: p = TT-round(p, δ)

9:
R(i, i) = ||p|| 10:

q i = 1/R(i, i) p normalize p 11: end for 12: return: Q = {q 1 , . . . , q m }, R
In the classical matrix framework Classical Gram-Schmidt is known for suffering from loss of orthogonality in the computed basis, as discussed later on, cf. [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF]. The Modified Gram-Schmidt (MGS) algorithm introduces in the Classical Gram-Schmidt a tiny algorithmic change, which is sufficient to guarantee a generally better numerical orthogonality. Indeed, in TT-MGS we remove the projection along q j of p i and not of a i for every j ∈ {1, . . . , i -1}, as we see comparing line 5 of Algorithm 11 and 12 respectively. This modification reduces the error propagation, improving the general stability of the algorithm both in the classical matrix and in the tensor case, as we discuss in Section 3.2.4. For the remaining steps, the two algorithms are exactly identical. Under the same assumptions stated previously to estimate the complexity, we conclude that TT-MGS computational complexity in TT-format is equal to TT-CGS one, given by m TT-rounding calls.

Classical schemes with reorthogonalization

CGS and MGS are know to have stability issues, described in details in Section 3.2.4, i.e., the close the input vectors are to linear dependency, the more the algorithms propagate the rounding errors, spoiling the final orthogonality of the new basis. As reported in [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF], over the years, different articles, as for example [START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Daniel | Reorthogonalization and Stable Algorithms for Updating the Gram-Schmidt QR Factorization[END_REF][START_REF] Hoffmann | Iterative algorithms for Gram-Schmidt orthogonalization[END_REF], addressed this flaw introducing re-orthogonalization steps, that is, orthogonalizing repeatedly through the same approach the basis produced by the algorithm itself. In [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF], the authors showed theoretically that one re-orthogonalization step is sufficient to improve significantly the orthogonality of the new basis generated by CGS and MGS. We present briefly the idea of CGS and MGS with re-orthogonalization, called CGS2 and MGS2, in the tensor case, highlighting those steps performed only in the TT-version.

In CGS2, sketched in Algorithm 13, as in CGS, the input TT-vector a i is orthogonalized with respect to the (i -1) previously computed orthogonal TT-vector {q 1 , . . . , q i-1 }, subtracting from a i its projection along q j . The projection of a i along q j defines the (j, i) element of the first matrix R 1 . With these first (i -1) iterations, given in line 6 of Algorithm 13, we define the TT-vector p 1 , which is then rounded in the TT-GCS2, see Algorithm 13 line 10. Up to this point, TT-CGS2 acts exactly as TT-CGS. However in TT-CGS2, after the rounding, the TT-vector p 1 is orthogonalized again against {q 1 , . . . , q i-1 }, defining p 2 . The projections along q j of p 1 determine the (j, i) component of the second matrix R 2 . Once p 2 is completely defined, it is rounded and normalized, defining the i-th orthogonal TT-vector q i , as stated in line 12 and 13 of Algorithm 13. The norm of p 2 at the i-th iteration defines the (i, i)-th diagonal component of R 2 . The R factor from the QR decomposition computed by CGS2 is obtained summing R 1 and R 2 . The difference of MGS2 from CGS2 appears in line 7 of Algorithm 14 and 13 respectively. During the first orthogonalization loop, that is for k = 1 in line 4 of both methods, in the classical Gram-Schmidt version a i is projected along q j defining p 1 , while in the modified Gram-Schmidt one p 1 is projected along q j updating itself. In the second orthogonalization loop, i.e., when k = 2 in line 4 of both algorithms, in TT-CGS2 it is removed from p 1 its projection along q j defining p 2 , while in the TT-MGS2 version, it is p 2 the TT-vector projected along q j and updated. All the remaining steps, including the TT-rounding and the construction of R 1 , R 2 and their sum R, are identical between TT-CGS2 and TT-MGS2. Remark that the rounding steps are performed only in the TT-version of MGS2 and CGS2. for k = 1, 2 do 5:

p k = p k-1 6:
for j = 1, . . . , i -1 do compute the projection of p k-1 along q j 7:

R k (i, j) = p k-1 , q j subtract the projection of p k-1 along q j 8:

p k = p k -R k (i, j)q j 9:
end for 10:

p k = TT-round(p k , δ) 11:
end for 12: R 2 (i, i) = ||p 2 || 13: for k = 1, 2 do 5:

q i = 1/R 2 (i, i)
p k = p k-1 6:
for j = 1, . . . , i -1 do compute the projection of p k along q j 7:

R k (i, j) = p k , q j subtract the projection of p k along q j 8:

p k = p k -R k (i, j)q j 9:
end for 10:

p k = TT-round(p k , δ) 11:
end for 12: R 2 (i, i) = ||p 2 || 13:

q i = 1/R 2 (i, i)p 2
normalize p 2 14: end for compute the R factor from the repeated orthogonalization loop 15:

R = R 1 + R 2 16: return: Q = {q 1 , . . . , q m }, R
On the basis of the previously used hypothesis, the computational complexity of TT-CGS2 and TT-MGS2 is estimated as 2m TT-rounding operations, since during the m 3.2. ORTHOGONALIZATION SCHEMES iterations we have two temporary TT-vectors p 1 and p 2 rounded.

Gram approach

In [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF], the authors propose an algorithm, that we refer to as Gram's algorithm, to generate orthogonal basis starting from a set m linearly independent vectors of R n with m n. This scheme is based on the Gram matrix, which under the hypothesis m n is significantly small. The key idea is decomposing the small Gram matrix through its Cholesky factorization and benefiting from it to generate the orthogonal basis. We briefly describe the main ideas of this orthogonalization scheme in the classical matrix framework and we describe in details the implementation of the Gram algorithm in the TT-format. Indeed the tensor realization of this procedure is extremely close to the matrix one, so that describing the tensor case and its differences with the classical matrix one is sufficient to ensure a good comprehension.

Given a set of vectors A = {a 1 , . . . , a m } with a i ∈ R n , the Gram matrix G ∈ R m×m is defined by the their scalar product as G(i, j) = a i , a j for every i, j ∈ {1, . . . , m}. Equivalently, let a i be i-th column of the matrix A ∈ R n×m , then in the matrix computation the Gram matrix is

G = A A. ( 3.1)
If the elements of A are linearly independent, then G is symmetric positive definite.

As consequence, its Cholesky factorization exists and is expressed as G = LL with L ∈ R m×m a lower triangular matrix. Let now denote the transpose of L by R, then the Gram matrix is

G = R R. (3.2)
Comparing Equation (3.1) and (3.2), we conclude that R is the R factor from the QR decomposition of A, i.e., it expresses the same information of A in a different basis, up to the action of an invertible matrix. The matrix Q from the QR decomposition of A is written as Q = AR -1 where R = L . The columns of Q form an orthogonal basis Q = {q 1 , . . . , q m }, whose j-th element strictly speaking is a linear combination of the first j elements of A, i.e.,

q j = j k=1 R -1 (k, j)a k .
Remark 3.2.1. Notice that by construction the condition number of G is the square of the one of A. Consequently if the condition number of A associated with the set of input vectors A is larger than the inverse of the squared root of the working precision of the considered arithmetic, e.g., u 64 ≈ 10 -16 for 64-bit calculation, the associated Gram matrix G is numerically singular and its Cholesky decomposition is no longer defined. This constitutes the main practical drawback of this method.

This procedure produces an orthonormal basis starting from a set of linear independent vectors, which is naturally extended to TT-vectors. As described in Algorithm 15, given a set of TT-vectors A = {a 1 , . . . , a m } with a i ∈ R n 1 ו••×n d , we construct the Gram matrix

CHAPTER 3. ORTHOGONALIZATION SCHEMES IN TT-FORMAT

91

G ∈ R m×m through the tensor scalar product and we compute its Cholesky factorization getting the lower triangular matrix L ∈ R m×m . As in the matrix case, R ∈ R m×m the transpose of L expresses the same information as the TT-vectors of A, but with respect to a different basis. Following the matrix approach, we retrieve this basis, i.e., the orthonormal set Q whose element q i ∈ R n 1 ו••×n d is defined as

q i = i k=1 R -1 (k, i)a k .
Remark 3.2.2. In the matrix framework, the orthogonal vector q j are obtained from the elements of A by back-substitutions involving the R factor. However this approach cannot be easily translated into the tensor framework, where the inverse of R has to be explicitly computed.

As for the other orthogonalization techniques, in the tensor case, we prevent memory issues, monitoring the TT-ranks. Indeed, assuming that all the TT-vectors of A have TT-ranks bounded by r, then the i-th TT-vector constructed in line 12 has maximum TT-rank bounded by ir. Since this value grows linearly with m, in line 14 we introduce a rounding step at prescribed accuracy δ. As in Sections 3.2.1 and 3. for j = 2, . . . , i do construct the i-th new basis TT-vector as a linear combination of the (i -1) input TT-vectors 12: p = p + R -1 (i, j)a j 13: end for 14:

q i = TT-round(p, δ) round the TT-vector before adding it to the basis 15: end for 16: return: Q = {q 1 , . . . , q m }, R complexity of the TT-Gram algorithm is given by m TT-rounding operations. However, in this particular case, we can even estimate the cost of each single rounding step and consequently of the entire algorithm. Indeed, the maximum TT-rank of the rounded TTvector q i is bounded by i r, under the assumption that the maximum TT-rank and the maximum mode size of a i are bounded by r ∈ N and n ∈ N respectively. Consequently,

ORTHOGONALIZATION SCHEMES

the computational cost, that is the number of floating point operations, of each rounding operation, the most expensive step in the entire algorithm, is known and is equal O(dni 3 r 3 ); summing up over i ∈ {1, . . . , m}, we conclude that the TT-Gram algorithm cost is O(dnm 4 r) floating point operations.

Householder reflections

In the classical matrix framework, the Householder transformations are widely applied to generate orthogonal base, thanks to their remarkable stability properties, illustrated in the following sections. We briefly present the theoretical construction of a Householder transformation, underlining how to generate an orthogonal basis. The second half of this section describes in detail how we extend the Householder transformation to the tensor context, with a specific attention to the implementation of the Householder orthogonalization scheme in TT-format.

Given a vector x ∈ R n , the Householder reflector moves x along a chosen direction, which usually is an element of the canonical basis or a linear combination of them. We illustrate the construction of the Householder reflector in the general case. Let x ∈ R n be the vector we want to reflect along the normalized vector y ∈ R n , the Householder reflection or transformation is a linear operator H : R n → R n such that

H(x) = x y with y = 1.
The Householder reflector is represented with respect to the canonical basis of R n by the matrix H ∈ R n×n such that H = I n -2u ⊗ u where u ∈ R n is the Householder vector defined as

u = x -z x -z with z = x y. ( 3.3)
The matrix H representing the Householder reflection is unitary and entirely defined by the Householder vector u. Moreover the action of an Householder reflector is computed by one scalar product with the Householder vector u and one algebraic vector summation. Indeed given a vector z ∈ R n and an Householder reflector

H = I n -2u ⊗ u, the image of w through H is Hw = w -2 w, u u. (3.4)
In particular, if u is defined as in Equation (3.3), then we explicitly verify that Hx = x y. Firstly remark that

x -z 2 = x -x y, x -x y = x 2 -2 x x, y + x 2 y 2 = 2 x 2 -x x, y
since y = 1 by hypothesis. Thanks to this result and Equation (3.4), we get

Hx = x -2 x, u u = x - 2 2 x 2 -x x, y x, x -x y (x -x y) = x - 1 x 2 -x x, y x 2 -x x, y (x -x y) = x y.
The Householder transformations are generally used to compute the QR factorization of a matrix, but they find application also in situations where a set of vectors has to be converted into an orthogonal basis. We examine briefly the two possibilities. Given a matrix A ∈ R n×m , we construct m Householder reflections such that the k-th one moves the k-th column of A along a linear combination of the first k canonical basis vectors. Said differently, the k-th Householder transformation sets to zero the last (n-k) entries of the k-th column of A. Consequently, after m Householder reflections the matrix A ends up being upper triangular. We illustrate more in detail how the algorithm iteratively proceeds. Let a 1 ∈ R n be the first column of A, the first step consists of reflecting it along the first canonical basis vector e 1 , i.e., in constructing the Householder reflector H 1 such that H 1 a 1 = a 1 e 1 . Consequently the first Householder vector u 1 ∈ R n is u 1 = a 1 ± a 1 e 1 and normalized. For stability reasons, cf. [START_REF] Trefethen | Numerical Linear Algebra[END_REF], the sign of the norm of a 1 is determined by the sign of the first component of a 1 , that is a plus if a 1 (1) > 0, a minus otherwise. The first Householder reflector H 1 is applied to all the columns of A, that is ãj = H 1 a j for j ∈ {1, . . . , m}. Henceforth we denote by ãj the j-th column of A updated by all the previously defined (j -1) Householder transformations for j ∈ {2, . . . , m}. Notice that the first Householder transformation moves the first column of A along a multiple of the first canonical basis vector e 1 , i.e., it sets the last (n -1) entries of the first column of A to zero. Then we proceed reflecting the second column of A, updated by H 1 along a linear combination of the first two canonical basis vectors e 1 and e 2 . Let u 2 ∈ R n be the Householder vector defining the second Householder reflector H 2 , which updates a second time ãj the j-th column of A for j ∈ {2, . . . , m}. Thus, at this point ã2 only has its first two components nonzero. The k-th Householder reflection H k moves ãk ∈ R n the k-th column of A, updated by the first (k -1) Householder reflections, along a linear combination of the first k elements of the canonical basis of R n , i.e.,

H k ãk = k =1 α e with k =1 α 2 = ||ã k ||. Before normalization, the k-th Householder vector is u k ∈ R n such that u k = ãk - k =1 β e
(3.5)

ORTHOGONALIZATION SCHEMES

where for every ∈ {1, . . . , k -1} we have β = ãk ( ) and

β k = ± ||ã k || 2 - k-1 =1 β 2 .
Again for stability reasons, cf. [START_REF] Trefethen | Numerical Linear Algebra[END_REF], β k is positive if ãk (k) is positive, negative otherwise. Then u k is normalized.

H k ∈ R (n-k+1)×(n-k+1
) is defined from ûk and is applied only on the last (n -k + 1) components of ãj for j ∈ {k, . . . , m} and k ∈ {1, . . . , m}. This reduced approach is not reproducible in the tensor framework, where the object is expressed in compressed format. Therefore we present it even in the matrix case in the most general way, that is not the one adopted to make an implementation in matrix computation.

Once the k-th Householder transformation has been applied to ãj for j ∈ {k, . . . , m}, the vector ãk has its last (n -k) component equal to zero. The application of m Householder reflections to A leads to the upper triangular matrix R, i.e., the R factor of the QR decomposition. To compute the Q factor, we multiply the m Householder reflection matrices. For the majority of the applications, forming the Householder vectors and knowing the Householder transformations implicitly, as given in Equation (3.4), is sufficient. However, if we want to produce an orthogonal basis out of a generic set of m vectors A = {a 1 , . . . , a m } with a k ∈ R n , a further step has to be considered. As for computing the QR factorization, H k the k-th Householder transformation is defined by the k-th Householder vector u k , given in Equation (3.5), for every k ∈ {1, . . . , m}. To generate the set of orthonormal vectors Q = {q 1 , . . . , q m }, we apply H k the first k Householder transformations to the k-th canonical basis vector e k in reverse order, i.e.,

q k = H 1 • • • H k e k .
As in the previous sections, we extend with some modifications this approach to the tensor case. Let A be a set of m TT-vectors a i ∈ R n 1 ו••×n d for every i ∈ {1, . . . , m}. To construct the Householder transformations, we first define a canonical basis for a tensor subspace of dimension m of R n 1 ו••×n d . In order to do so, fixed N i = {1, . . . , n i } for every i ∈ {1, . . . , d} and n = d j=1 n j , let define the function ψ :

N 1 × • • • × N d → {1, . . . , n} such that ψ(i 1 , . . . , i d ) = i 1 + d α=2 (i α -1)m α with m α = α-1 β=1 n β .
Since ψ is invertible, we denote its inverse by φ : {1, . . . , n}

→ N 1 × • • • × N d such that φ(i) = (i 1 , . . . , i d ).
As consequence, ψ(φ(i)) = i and φ(ψ(i 1 , . . . , i d )) = (i 1 , . . . , i d ) for i ∈ {1, . . . , n} and i k ∈ {1, . . . , n k } with k ∈ {1, . . . , d}. The basis we fix for the subspace

of dimension m of R n 1 ו••×n d is E = {e 1 , . . . , e m } with e i = e i 1 ⊗ • • • ⊗ e i d with (i 1 , . . . , i d ) = φ(i), (3.6)
where e i k is the i k -th canonical basis vector of R n k for k ∈ {1, . . . , d}. Henceforth we use index i to denote the i-th element of the canonical basis, implying i = ψ(i 1 , . . . , i d ).

As stated in Remark 3.2.3, the first (k -1) components of u k are zeros, the last (n -k) components are equal to corresponding ones of ãk , while the k-th one is given by the difference of the k-th component of ãk and the quantity β k , defined in Equation (3.5). We want to transport this structure in the tensor case. However if the element of A are in TT-format, it is not possible to directly access to the tensor components and we need to recover them, either by multiplying the TT-cores with the correct index or by computing the scalar product with the element of E. Let ãk be the result of (k -1) Householder reflections applied to a k , the k-th Householder TT-vector is

u k ∈ R n 1 ו••×n d defined as u k = ãk - k j=1 R(j, k)e j
where R(j, k) = ãk , e j for every j ∈ {1, . . . , k} and R(k, k)

= ± ||ã(k) k || 2 -k-1
=1 R( , k) 2 as described in lines 8 and 10 of Algorithm 16 respectively. The k-th component of r k takes a positive sign if ãk , e k > 0, otherwise it takes a negative sign, extending in the tensor framework the stability preserving idea given in [START_REF] Trefethen | Numerical Linear Algebra[END_REF]. The j-th component of r k corresponds to the (j, k) component of the R factor. As previously pointed out, we have to ensure that the TT-rank of u k remains small for every k ∈ {1, . . . , m}. Assuming that the maximum TT-rank of ãk is bounded by r a , then after removing the first (k -1) components of ãk , i.e., after line 8, the TT-rank of u k is bounded by (r a + k -1). We introduce at this step the first TT-rounding call on the Householder TT-vector u k , whose TT-rank is decreased depending on the accuracy value δ. Then the k-th component of r k is subtracted, determining a further growth in the TT-rank of u k . Since u k plays a key role in the Householder transformation and consequently its TT-rank has a strong impact on the entire process, we decide to perform a further TT-rounding step over u k at accuracy δ.

After describing the construction of an Householder TT-vector, which is summarized in Algorithm 16, the focus is on the process that generates an approximately orthonormal TT-vector set from a generic TT-vector set A = {a 1 , . . . , a m }, as depicted in Algorithm 18. To reflect the k-th TT-vector of A along a linear combination of the first k elements of w = w -r(j)e j 8: end for 9: w = TT-round(w, δ) subtract from the norm of a the contribution of the components set to zero 10: r(i) = sign( a, e i ) ||a|| 2 -s 11: w = w -r(i)e i 12: w = TT-round(w, δ) 13: u = (1/||z||)w 14: return: u, r E, we generate with Algorithm 16 the k-th Householder TT-vector u k , which defines implicitly the Householder transformation H k . We store the first k components of r k in the k-th column of the upper triangular matrix R ∈ R m×m . Then H k is implicitly applied using u k to form ãj for every j ∈ {k, . . . , m}, as expressed in line 7 of Algorithm 18, following the same approach as in matrix case, i.e.,

H k (ã j ) = ãj -2 ãj , u k u k .
Algorithm 17 applies a given Householder reflection to a specific input vector. Remark Algorithm 17 b = apply-H-vec(a, u)

1: input: a, u ∈ R n 1 ו••×n d 2: b = a -2 a, u u
apply the Householder reflection defined by u to a 3: return: b that (k -1) transformations are performed on a k , computing ãk , before generating the k-th reflector, leading the maximum TT-rank of a k to be potentially much larger than its initial value. Therefore to keep the TT-rank of ãk reasonably small, we perform a TT-rounding before generating the associated Householder TT-vector, see line 10.

The final part of the TT-Householder transformation algorithm 18 generates the new set Q of orthonormal TT-vectors. Let q i be the i-th element of Q obtained applying the first i Householder reflections in reverse order to e i from the canonical basis E, see line 16

of Algorithm 18. To keep the maximum TT-rank of q i limited and to avoid running out of memory, each q i is rounded at accuracy δ, as stated in line 18 of Algorithm 18. u i , R(: i, i) = TTH-vec(w, F i , δ) with F i = {e 1 , . . . , e i } 6:

for j = i, . . . , m do 7:

a j = apply-H-vec(a j , u i ) update the j-th element of A 8:
end for 9:

if i < m then round the TT-vector that will define the successive Householder reflection 10: w = TT-round(a i+1 , δ)

q i = e i 15:
for j = i, . . . , 1 do 16:

q i = apply-H-vec(q i , u j ) for reflect the i-th element of E 17:
end for 18:

q i = TT-round(q i , δ) 19: end for 20: return: Q = {q 1 , . . . , q m }, R From the complexity viewpoint, the TT-Householder algorithm costs 4m TT-rounding operations: two for each Householder TT-vector, one for each TT-vector ãk after the (k -1)-th reflection and one for each q k orthogonal TT-vector. Consequently, the TT-Householder algorithm is more expensive than all the other orthogonalization methods, that is 4 times more expensive than CGS and MGS, twice more than CGS2 and MGS2.

Stability comparison

A central issue for the orthogonalization algorithms is the loss of orthogonality, i.e., how much the rounding errors propagate and affect the orthogonality of the computed basis. The loss of orthogonality of an orthogonalization scheme applied to the set A m = {a 1 , . . . , a m } is defined by the L2-norm of the difference among the identity matrix of size m and the Gram matrix defined from the m vectors generated by the orthogonalisation algorithm. We state more formally the definition. Let Q m = {q m , . . . , q m } be a set of m vectors, obtained from an orthogonalisation scheme applied to the m vectors of the input set A m . Let q i ∈ Q m be the i-th column of the matrix Q m ∈ R n×m for every i ∈ {1, . . . , m}, then the Gram matrix associated with the set

Q m is Q m Q m . Remark that the (i, j) element of Q m Q m is
the scalar product of q i and q j , that is Q m Q m (i, j) = q i , q j for every i, j ∈ {1, . . . , m}. Then, the loss of orthogonality of the considered algorithm

ORTHOGONALIZATION SCHEMES

for a basis of size m is equal to

||I m -Q m Q m || 2 .
(3.7)

In the classical matrix framework, an orthogonalization scheme is said numerically stable if the loss of orthogonality of the basis it computes is of the order of the unit roundoff u of the working arithmetic. The following theoretical results holding true for the six orthogonalization schemes in classical linear algebra constitute a base line for the comparison with the numerical results obtained in the tensor framework, discussed in Section 3.2.5.

In [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF]Theorem 1] the authors prove that given A m = {a 1 , . . . , a m } a set of m vectors, the loss of orthogonality for a basis obtained by CGS is bounded by a positive constant times the unit round-off u, Definition I.II.iii, times the squared condition number of the matrix A m ∈ R n×m whose j-th column is a j ∈ R n for j ∈ {1, . . . , m}, i.e.,

||I m -Q m Q m || 2 ∼ O uκ 2 (A m ) (3.8)
as long as κ 2 (A m )u 1. In [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF], an upper bound for the loss of orthogonality of MGS is provided . Indeed the loss of orthogonality for a basis of m vectors produced by MGS from {a 1 , . . . , a m } is upper bounded by a constant times the unit round-off u times the condition number of A m as previously defined from A m elements, i.e.,

||I m -Q m Q m || 2 ∼ O uκ(A m ) (3.9)
as long as κ(A m )u 1. The authors of [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF]Theorem 4.1] who proposed the Gram orthogonalization scheme, estimated also an upper bound for the loss of orthogonality of their orthogonalization technique. More precisely, the loss of orthogonality of a basis of m vectors produced by the Gram scheme from {a 1 , . . . , a m } satisfies the same upper bound as CGS, given in (3.8). Lastly the Householder orthogonalization algorithm is known for being the most stable one. Indeed the loss of orthogonality of a basis of m vectors produced by Householder transformations from {a 1 , . . . , a m } is bounded by a constant times the round-off unit, i.e.,

||I m -Q m Q m || 2 ∼ O u (3.10)
as proved in [START_REF] Wilkinson | The algebraic eigenvalue problem[END_REF]. When a further orthogonalization step is introduced in the classical and modified Gram-Schmidt, defining CGS2 and MGS2, their loss of orthogonality improves considerably, reaching Householder quality. Indeed, as proved in [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF][START_REF] Smoktunowicz | A note on the error analysis of classical Gram-Schmidt[END_REF], the loss of orthogonality of CGS2 and MGS2 satisfies the bound given in Equation (3.10), under the hypothesis κ 2 (A m )u 1 for CGS2, while it holds for MGS2 if κ(A m )u 1. A summary of all the loss of orthogonality bounds is presented in Table 3.1.

Numerical tensor experiments

In Sections 3.2.1 -3.2.3, we describe four orthogonalization methods which produce an orthonormal basis of TT-vectors, given a set of TT-vectors and a rounding accuracy δ. This section analyses two sets of results obtained from the orthogonalization schemes, highlighting similarities and differences with the known theoretical results in matrix computation. In all the experiments, the input set of TT-vectors A m = {a 1 , . . . , a m } is generated with a Krylov process. Let x 1 ∈ R n 1 ו••×n d be the TT-vector of ones, iteratively we compute x j+1 = -∆ d a j where ∆ d is the TT-matrix representing the discretization of the Laplacian operator of order d with Dirichlet boundary conditions, Equation (2.37), and a j is the normalized output of the TT-rounding algorithm applied to x j . Consequently, a j the j-th element of A m has TT-rank 1 for every j ∈ {1, . . . , m}. This TT-rank constraint facilitates the analysis of the memory requirement. By construction, the elements of A are generated as a sequence of m normalized (and rounded) Krylov TT-vectors, so that when we vectorize and arrange only the first k as columns of the matrix A k , its condition number κ(A k ) grows for k ∈ {1, . . . , m}. Henceforth A k denotes the subset of A m defined by its first k TT-vectors. The two experiments differ in the dimension of the problem, 3 for the first one and 6 for the second one, but they have the same mode size n = 15. We provide further details in the following sections.

Numerical loss of orthogonality

In this section, we examine the numerical results from two set of experiments from the viewpoint of the loss of orthogonality. The purpose is highlighting the similarities with the classical matrix orthogonalization methods. In particular, we investigate the loss of orthogonality

I k -Q k Q k 2
, where the (i, j) element of Q k Q k is computed by the scalar product of the i-th and j-th TT-vector of the orthogonal basis, i.e., Q k Q k (i, j) = q i , q j for q i , q j ∈ Q for i, j ∈ {1, . . . , k} for k ∈ {1, . . . , m}, where Q is the set of TT-vectors produced by the considered orthogonalization kernel.

For the first experiment, we set the order d = 3, the size mode n i = 15 for i ∈ {1, 2, 3} and the input number of TT-vectors m = 20. The results of this first group of experiments for the six different schemes and three different rounding accuracy values δ ∈ {10 -3 , 10 -5 , 10 -8 } are reported in Figure 3.1. The choice of a low dimensional problem enables us to convert in dense format and vectorize each TT-vector a j , storing it as the j-th column of A m ∈ R n 3 ×m . As consequence, we can estimate the condition number of A k ∈ R n 3 ×k , i.e., the submatrix of A m formed by its first k columns. In all the plots of Figure 3.1, together with the loss of orthogonality with continuous coloured curves, we display the constant rounding accuracy δ with a dashed black line, with coloured continuous lines the condition number κ(A k ) and its squared value scaled by u ≈ 10 -16 , as long as they are smaller than 1. We scale the condition number curves for the ease of comparison with the slope of the loss of orthogonality of TT-MGS and TT-CGS. All the curves are in function of the basis size k. Indeed, as previously stated, since the TT-vectors are generated as a sequence of normalized Krylov TT-vectors, for increasing values of k, the elements of A k get more linearly dependent, leading the associated condition number κ(A k ) to increase. For the interpretation ease, we present in the first line three plots, that are Figure 3.1A, 3.1B and 3.1C, with the loss of orthogonality of standard methods in tensor format, i.e., TT-Householder, TT-MGS, TT-CGS, TT-Gram; on the second line, Figure 3.1D, 3.1E and 3.1F display the loss of orthogonality of the methods with re-orthogonalization, that are TT-MGS2 and TT-CGS2, compared with the result of TT-Householder, TT-MGS and TT-CGS. In all the six plots of Figure 3.1, we observe similar behaviours. The TT-Householder loss of orthogonality in green stagnates around the rounding accuracy δ; this phenomenon is extremely clear in Figure 3.1A. This is consistent with the matrix theoretical expectation, stated in Equation (3.10), where the unit round-off u is replaced by the TT-rounding accuracy δ. The TT-MGS loss of orthogonality in red grows with the same slope of the condition number κ(A k ) in dashed green, matching the matrix upper bound stated in (3.9). Finally both the TT-CGS and and TT-Gram loss of orthogonality curves cross the rounding accuracy dashed line faster than TT-MGS, following the behaviour of the squared condition number κ 2 (A k ), as long as κ 2 (A k ) < 10 2 . Indeed TT-CGS and TT-Gram loss of orthogonality curves stagnates below 10 2 for k > 10 approximately, while κ 2 (A k ) continues to grow. Looking at the second line plots, Figure 3.1D, 3.1E and 3.1F show that introducing a second reorthogonalization loop improves considerably the loss of orthogonality. Indeed as long as k < 15 the loss of orthogonality of TT-CGS2 is close to the machine precision, around 10 -14 for δ ∈ {10 -3 , 10 -5 }, increasing after; for δ = 10 -8 it remains around 10 -14 . Consequently as long as the element of A k are not too much collinear, TT-CGS2 outcompetes TT-CGS, TT-MGS and TT-Householder, but not TT-MGS2. For the rounding accuracy δ ∈ {10 -5 , 10 -8 }, the loss of orthogonalization of TT-MGS2 stagnates around 10 -14 , while for δ = 10 -3 the loss of orthogonality jumps from 10 -14 to 10 -11 , where it seems to stagnates, when k > 16. Overall TT-MGS2 outperforms all the other algorithms. These results found for TT-CGS2 and TT-MGS2 are consistent with the matrix theory presented in Section 3.2.4. Moreover, we may hypothesise that the jumps arrive when the condition number κ(A k ), or its square, times the rounding accuracy is not any more sufficiently smaller than 1.

Loss of orthogonality with reorthogonalization comparison

To further validate our results and to study the applicability to large-scale problems, we consider a second experimental framework. Set the problem order to d = 6 with size mode n i = 15 for i ∈ {1, . . . , 6}, we generate m = 35 TT-vectors, defining the set A = {a 1 , . . . , a 35 }. For the rounding accuracy values δ ∈ {10 -3 , 10 -5 , 10 -8 }, we compute the loss of orthogonality for the four orthogonalization schemes, presented in Section 3.2.1 -3.2.3. The loss of orthogonality of these experiments is displayed in Figure 3.2. Because of the problem order d = 6 and size n = 15, in this case we do not add the curve of the condition number of the A k ∈ R n 6 ×k , whose k-th column would be the TT-vector a k ∈ A m uncompressed and vectorized. To compensate the absence of the condition number curve, we display the square of the TT-MGS loss of orthogonality values with a dashed line, that should exhibit the same slop as the CGS loss of orthogonality (and consequently of the squared condition number κ(A k )), if the matrix theory extends to the TT-framework. As in the previous case, the first line plots display standard orthogonalization algorithms in TT-format, while on the second line we display results from methods with re-orthogonalization. Figures 3.2A, 3.2B and 3.2C clearly show that the TT-Householder orthogonalization algorithm produces a basis whose loss of orthogonality stagnates around the rounding accuracy δ for every value in {10 -3 , 10 -5 , 10 -8 } after the basis size gets greater than approximately 10. This seems to support further our intuition that even in the tensor framework the bound expressed in Equation (3.10) is still holding true, with the unit round-off u replaced by the TT-rounding accuracy δ. In Figure 3.2A and 3.2B, when the basis size is smaller than about 15, the TT-MGS loss of orthogonality is smaller than the Householder one, but when the basis includes more then 15 TT-vectors, the relation reverses. For more accurate computation, that is for δ = 10 -8 , already when the basis size is around 5, the Householder loss of orthogonality outcompetes the TT-MGS one. Notice that for all the rounding accuracy values, the TT-MGS loss of orthogonality firstly grows linearly and stagnates after a while at different level, at 10 -2 for δ = 10 -3 , at 1 for δ ∈ {10 -5 , 10 -8 }. The stagnation of the TT-CGS and TT-Gram loss of orthogonality curves arrives almost immediately, when the basis size is greater than 10 for all the rounding accuracy values, i.e., in all the plots of Figure 3.2. This is probably due to the very bad condition number of the basis, so that the assumptions of the matrix theory are not longer valid, that is, condition number times rounding smaller than 1. Moreover mainly Figures 3.2B and 3.2C show that the TT-CGS and TT-Gram loss of orthogonality follow the square of the TT-MGS loss of orthogonality, supporting the idea that even in the TT-format, TT-MGS loss of orthogonality grows as the condition number, while TT-Gram and TT-CGS loss of orthogonality as the squared condition number. Figures 3.2D, 3.2E and 3.2F display the loss of orthogonality of TT-MGS2 and TT-CGS2, compared with the previously analysed results of TT-Householder, TT-MGS and TT-CGS. For all the considered rounding accuracies, TT-MGS2 outperforms all the other methods, with a loss of orthogonality which stagnates around 10 -5 for δ = 10 -3 , around 10 -10 for δ = 10 -5 and around 10 -13 for δ = 10 -8 . In all the three Figures 3.2D-3.2F, the TT-MGS2 curve presents a jump larger for greater values of δ, for 15 ≤ k ≤ 20 with δ ∈ {10 -3 , 10 -5 } and for k ∼ 10 with δ = 10 -8 . Also TT-CGS2 outcompetes the TT-Householder, TT-CGS and TT-MGS, following the behaviour of TT-MGS2, always for δ ∈ {10 -5 , 10 -8 } and as long as the TT-vectors are not too much collinear, i.e., for k < 20 when δ = 10 -3 . These results support the conclusion stated for the d = 3 experiments, that the bounds for the loss of orthogonality of TT-CGS2 and TT-MGS2 proved in the classical matrix framework still apply, probably under revised hypothesis.

Loss of orthogonality with reorthogonalization comparison

Memory usage estimation

This section aims to study the effect on the TT-rank and consequently on the memory requirement of the orthogonalization process from the numerical results. We investigate the growth of the TT-ranks, of the compression ratio, defined in Equation (1.7), and the compression gain curve, stated in Equation (1.8). We examine the TT-rank, the compression ratio and the compression gain of the new basis produced by the orthogonalization algorithms in the second set of experiments with d = 6, n i = 15, m = 35 and δ ∈ {10 -3 , 10 -5 , 10 -8 }, since this problem can already be considered a large scale one.

In the Householder algorithm 18 there are three sets of TT-vectors on which the TT-rounding is applied: the Householder TT-vector u k , the TT-vector a k to which we apply k Householder transformations and the orthogonal TT-vector q k obtained from the canonical basis TT-vectors with i successive Householder transformations. It is worthwhile studying the evolution of the maximum TT-rank, of the compression ratio and gain for each of these three TT-vector groups. In Figure 3.3 we display the maximum TT-rank, the compression ratio and the compression gain of u k , a k after the k-th reflection and q k for every k ∈ {1, . . . , 35} for all the values of the rounding accuracy δ. As expected, the maximum TT-rank and the compression ratio of u k , a k and q k grow for increasing basis sizes, because the number of terms in their computation grows with k. Remark that the maximum TT-rank of q k becomes larger than one of u k for a basis size greater than 10. This property is significant since in the majority of the practical vector computations only the Householder TT-vectors u k are stored and the orthogonal basis TT-vector q k is usually not explicitly formed. In Figures 3.3A, 3.3B and 3.3C we observe that the maximum TT-rank of a k after the k-th Householder reflection is extremely low, mainly if it is compared with those of q k and u k . Moreover, the rise of 1 of the maximum TT-rank of a k arrives every time the index k is equal to a multiple of the mode size n = 15, as we can see in Figure 3.3A and 3.3B. This behaviour is present also for δ = 10 -8 for Figure 3.3C, but it is less regular. We tried to investigate further this phenomenon from a theoretical viewpoint, but we did not arrive to a clear and convincing explanation. The compression ratio has closely the same shape as maximum TT-rank, but it enables us to observe the memory growth in percentage. Figures 3.3D, 3.3E and 3.3F present for all the values of the rounding accuracy δ a compression ratio smaller than 1, implying that for all these experiments the TT-format is still relevant to reduce the memory footprint. Moreover the compression ratio of a k stagnates around 10 -5 for all the three values of δ. Figures 3.3D and 3.3E show that the compression ratio of q k stagnates around 10 -1 ,

Compression ratio for the TT-vectors in Householder algorithm

Compression gain for the TT-vectors in Householder algorithm

ORTHOGONALIZATION SCHEMES

After describing the memory need of the TT-Householder algorithm, we compare the four orthogonalization schemes from the memory requirement viewpoint. Indeed, to fairly compare the orthogonalization schemes, we have to consider the memory consumptions together with the loss of orthogonality, studied in Section 3.2.5.1. In Figure 3.4 we display, for different accuracies δ, the maximum TT-rank, the compression ratio and the gain for the orthogonal TT-vectors q k generated by the orthogonalization schemes plus the Householder TT-vector u k ones, since, as already mentioned, in many applications q k are not computed. Remark that for every rounding accuracy δ, in the corresponding figure the curves get dashed and partially transparent when the corresponding loss of orthogonality become larger than 10 -1 . In all Figures 3.4A, 3.4B and 3.4C we see that the maximum TTrank of the orthogonal TT-vectors computed by TT-CGS and TT-Gram schemes stagnates around 10, but we do not have a clear theoretical justification for this phenomenon. The maximal TT-rank of q k from TT-Gram algorithm is theoretically bounded by k times the maximal TT-rank of a j for j ∈ {1, . . . , k}. When a j are generated, they are rounded by maximal TT-rank equal to 1, then the maximal TT-rank of q k is bounded by k in our experimental framework. On the other side, the maximal TT-rank of q k from TT-CGS is bounded by 1 + k(k -1)/2 knowing that the maximal TT-rank of a i is bounded by 1 for i ∈ {1, . . . , m} in our experiments. From the viewpoint of the maximal TT-rank, TT-Gram outcompetes TT-MGS and TT-Householder for basis size greater than 10, while TT-CGS establishes a lower bound in terms of maximal TT-rank. However the loss of orthogonality of TT-CGS and TT-Gram gets greater than 10 -1 around k = 10. On the other side, the TT-MGS maximal TT-rank curve gets dashed around k = 20, while the Householder one never, that is the loss of orthogonality arrives much after for TT-MGS or even does not arrive for TT-Householder. In Figure 3.4A, the maximal TT-rank of q k from TT-MGS overcomes the maximal TT-rank of the Householder TT-vector u k and of the Householder orthogonal TT-vector q k when the basis size k reaches 20 and 25 respectively. A similar relation among the maximal TT-rank for Householder and for MGS generated orthogonal TT-vectors appears in Figure 3.4B, but the turning point is set at a different basis size, that is 25 and 30 for the Householder TT-vector u k and q k respectively. For the last rounding accuracy value δ = 10 -8 , whose related results are displayed in Figure 3.4C, the maximal TT-rank of the MGS orthogonal TT-vector reaches the Householder q k when the basis size gets larger than 15, overcoming the maximal TT-rank of the Householder u k approximately at the same basis size. Compression ratio for the orthogonal basis

ORTHOGONALIZATION SCHEMES

As in analysis of TT-Householder algorithm memory requirement, we investigate also the compression ratio for the TT-vectors forming the orthogonal basis produced by the four orthogonalization methods. The compression ratio curves of Figures 3.4D, 3.4E and 3.4F have the same slopes of their corresponding maximal TT-rank curves, but they display clearly the memory needs. The orthogonal TT-vectors from TT-CGS and TT-Gram schemes demand approximately 1% of the total memory necessary to store the full format tensors, as illustrated in Figure 3.4D, 3.4E and 3.4F. The same figures highlight that when the basis size gets greater than 10 and the TT-vectors start to be more collinear, the curves gets dashed and these scheme resulting basis are very poor in terms of orthogonality. From both Figures 3.4D and 3.4E, we infer that storing the basis TT-vector generated by TT-Householder scheme request approximately 20% of the memory that we would need to store those tensors in full format. Similarly only 10% of the entire memory requested for full format storage is necessary to store the Householder TT-vector u k . Finally, for δ = 10 -8 storing cost for the Householder basis TT-vector q k reaches the same amount of memory demanded for storing them in full format, as depicted in Figure 3.4F. However from the same figure, we notice that storing the Householder TT-vectors even for δ = 10 -8 needs approximately 30% of the memory necessary to store the same tensor in full format. This property makes the TT-Householder algorithm extremely interesting, since usually it is sufficient to store the Householder TT-vectors. TT-Householder algorithm gets even more convenient when we compare its compression ratio curves with the TT-MGS, TT-MGS2 and TT-CGS2 ones. Indeed, for all the rounding accuracy values, the compression ratio curve of TT-MGS and TT-MGS2 always reaches 1, see Figures 3.4D, 3.4E and 3.4F, implying that the memory requested by the TT-vectors from these schemes reaches the same amount of memory we would use to store in full format the orthogonal basis tensors. This same consideration holds true for the compression ratio of the orthogonal basis produced by TT-CGS2, for δ ∈ {10 -5 , 10 -8 }, while for δ = 10 -3 , the TT-vectors from TT-CGS2 consume just 20% of the memory we would need to store the same tensors in dense format. The compression gain curves in Figures 3.4G, 3.4H and 3.4I present approximately the same behaviour for the different rounding accuracy values δ. The compression gain of TT-Gram has a pick at the very beginning, then it stagnates around 10 starting from k = 10, meaning at the compressing the j-th basis TT-vector reduce of 10 times the memory requirement for j ≥ k. The gain curves of TT-MGS, TT-MGS2 and TT-CGS2 have approximately the same shape, they increase up to k = 15, for the successive 5 iterations the gain curves drop down and around k = 22, they rise again. Also the TT-Householder basis gain curve has a similar patter, it first grows, reaching a pick, then it decreases and during the last iterations it rises again. The Householder TT-vector gain curve, as previously observed, has a tiny growth at the beginning, then it drops down and stagnates at a very low value from k > 10. Finally the TT-CGS gain curve increases for increasing dimension k of the basis, so when the last basis TT-vector is rounded, it leads to use 0.001 of the memory used to store the TT-vector before rounding it. However as highlighted by the dash style, the TT-CGS and TT-Gram basis have almost entirely lost the orthogonality already at k = 10.

Summary

From the viewpoint of the memory footprint, thanks to its stability property and the possibility of storing only the TT-vectors u i , the TT-Householder orthogonalization scheme appears to be the best in many cases, together with TT-CGS2 and TT-MGS2. Figures 3.4D, 3.4E and 3.4F display that when the input TT-vectors are not too much collinear, k < 15, TT-MGS2 and TT-CGS2 have a compression ratio approximately equal to TT-Householder. For a basis size between 15 and 25 (or 20 for δ = 10 -8 ) TT-Householder is more expensive than TT-MGS, TT-MGS2 and TT-CGS2 in memory terms. Finally when the input TT-vectors get more and more linearly dependent, the memory need of TT-MGS2 and TT-CGS2 basis is greater or equal than both the TT-Householder basis and Householder TT-vector. However from the viewpoint of the loss of orthogonality, TT-MGS2 performs better than TT-Householder for every rounding accuracy, while TT-CGS2 for δ ∈ {10 -5 , 10 -8 }. Finally from the computational cost side, TT-CGS2 and TT-MGS2 are cheaper than TT-Householder, that is 2m TT-rounding versus 4m.

Similarly to the matrix case, the choice of the orthogonalization scheme among TT-Householder, TT-CGS2 and TT-MGS2 depends strongly on the purpose and on the available computing resources. Indeed TT-Householder requires less memory, but it is computationally more expensive and its orthogonality stagnates around the rounding accuracy. On the other side, TT-MGS2 produces a basis of better orthogonality quality, as long as the input TT-vectors are not too collinear, and it is computationally cheaper than TT-Householder, but it is more expensive from the viewpoint of the memory consumption. The same considerations hold also for TT-CGS2, under the same hypothesis. The computational costs in tensor and matrix case are summarized in Table 3.1.

Matrix TT-vectors

Algorithm Computational cost in fp operations

I k -Q k Q k Computational cost in TT-rounding I k -Q k Q k Gram O(2nm 2 ) O(uκ 2 (A k )) m O(δκ 2 (A k )) CGS O(2nm 2 ) O(uκ 2 (A k )) m O(δκ 2 (A k )) MGS O(2nm 2 ) O(uκ(A k )) m O(δκ 2 (A k )) CGS2 O(4nm 2 ) O(u) 2m O(δ) MGS2 O(4nm 2 ) O(u) 2m O(δ) Householder O(2nm 2 -2m 3 /3) O(u) 4m O(δ)
Table 3.1 -Computational costs in floating point operations and in TT-rounding operations, and bounds for the loss of orthogonality, theoretical with respect to the unit round-off u and conjectured ones with respect to the rounding accuracy δ, for an input set of m vectors and TT-vectors respectively.

EIGENSOLVERS

Eigensolvers

The orthogonalization schemes presented in Section 3.2 are usually kernels of more elaborated algorithms. The purpose of this section is to illustrate an eigensolver as study case where they are applied and where their properties affect the final result. In Section 3.3.1 we describe the chosen eigensolver algorithm, that is the subspace iteration method, while Section 3.3.2 collects the numerical results.

Subspace iteration method

The subspace iteration method is an iterative eigensolver, that can be regarded as a block generalization of the power iteration method. This eigensolver is considered one of the simplest and most reliable, cf. [START_REF] Stewart | Matrix Algorithms[END_REF]Chapter 6] or [START_REF] Saad | Numerical Methods for Large Eigenvalue Problems[END_REF]Chapter 5], even though it is usually slow and should be accelerated with preconditioning techniques in practical computations. This section aims to describe the subspace iteration method extended to the tensor framework to solve tensor eigenproblems, i.e., finding eigenpairs (λ, v) of a tensor multilinear operator

A ∈ R (n 1 ×n 1 )ו••×(n d ×n d ) such that Av = λv with λ ∈ R and v ∈ R n 1 ו••×n d .
We briefly illustrate the subspace iteration method extended to tensor eigenproblems through the TT-formalism, henceforth referred to as TT-SUBSPace ITeration (TT-SUBSPIT), depicted in Algorithm 19. The TT-SUBSPIT is meant to compute nev TTeigenvectors associated with the largest nev eigenvalues of a given TT-matrix. Together with the TT-matrix whose TT-eigenpairs are searched, it takes as input parameters A a set of m ≥ nev TT-vectors, a rounding accuracy δ ∈ R + , a convergence threshold ε ∈ R + , a maximum number of iterations max_it and an integer p ∈ N, that defines the power of the TT-operator to use. As first step, TT-SUBSPIT orthogonalizes the m input TTvectors of A by an orthogonalization kernel, using the prescribed rounding accuracy δ and returning the orthogonal set Q, see line 5 of Algorithm 19. During the first iteration, we compute the j-th elements of U as the product of the TT-operator at power p and the j-th orthogonal TT-vector of Q, that is u j = A p q j . To avoid memory deficiencies, we round at accuracy δ and then normalize each element of U in line 14 and 15 of Algorithm 19. Then, the TT-vecotrs of U are orthogonalized with the previously used orthogonalization kernel at accuracy δ, see line 17, defining Q. Remark that at the very first iteration, the set V is empty and the set U as cardinality n = m. Then, we project the TT-operator A into the new orthogonal basis formed by the last n elements of Q, defining matrix G ∈ R n×n with n = m since no TT-eigenpair has converged yet. In line 23 of Algorithm 19, the eigenvalue decomposition of matrix G is computed, sorting the eigenvalues, and the correspondent eigenvector, in decreasing order. The n eigenvalues of G are by construction an approximation of the largest n eigenvalues of A. The eigenvectors of G are used to approximate of the first n TT-eigenvectors of A. Indeed, the TT-vector w j is obtained as a linear combination of the last n elements of Q, choosing as coefficients

EIGENSOLVERS

the entries of the eigenvectors of G, see line 27 of Algorithm 19. Before adding w j to the set W, we apply the TT-rounding algorithm at accuracy δ to prevent memory deficiencies, line 29 of Algorithm 19. Finally for every TT-vector of W we compute their residual, that is Aw j -λ j w j , scaled by |λ j |, where λ j is the j-th eigenvalue of G for every j ∈ {1, . . . , n}, see line 32. If the scaled residual is smaller than the convergence threshold ε, then the TT-vector w j and its eigenvalue λ j are locked, becoming the 1st converged TT-eigenvector and eigenvalue. Thus, by locking, we include w j in the set V and λ j in L, and we increase the counter of the number of converged eigenpairs c by one unit, lines 34 and 35 of Algorithm 19. Otherwise, the w j is used to update the (j + c)-th element of Q, see line 37 of Algorithm 19. During the following iterations, thanks to the locking strategy, we apply the TT-operator A at power p only to the last n = m -c elements of Q, so that the set U will have cardinality n. However, to ensure the orthogonality at line see line 17 of Algorithm 19 we apply the orthogonalizations scheme on the set of TT-vectors, whose first c elements are the TT-eigenvectors locked in V and the last n = m -c are those that have not converged yet in U, see line 17.

Numerical Experiments

In this section, we investigate the numerical results obtained with the TT-SUBSPIT with the six different orthogonalization schemes in TT-format, described in Section 3.2. Two sets of experiments are considered: in the first, we compute the TT-eigenpairs of the discrete Laplacian operator ∆ 3 of order d = 3 with mode size [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Coulaud | A note on GMRES in TT-format[END_REF], in the second, we examine the TT-eigenpairs for the discrete order d = 3 Laplacian with all mode sizes equal to 24. This operator is chosen since there exists an analytical expression for their eigenpairs. To compute an approximation of the first m = 7 TT-eigenpairs, we use the same value for the rounding accuracy δ and the convergence one ε in {10 -3 , 10 -5 } for both the experiment sets. As for the experiments of Section 3.2.5, the input set Z = {z 1 , . . . , z m } of TT-vectors are generated with a Krylov process, that is z 1 is a TTvector of ones and the (h + 1)-th one is obtained from the TT-rounding at TT-rank 1 of ∆ d z h for h ∈ {1, . . . , m -1}. As mentioned in [START_REF] Saad | Numerical Methods for Large Eigenvalue Problems[END_REF][START_REF] Stewart | Matrix Algorithms[END_REF], in the classical matrix case the higher the power value, the faster the convergence, but, at the same time, the more the vectors to orthogonalize get linearly dependent. Thus, to simplify the investigation of these preliminary results and highlight as much as possible the orthogonalization kernel effect, we set p = 1.

TT-Eigenpairs convergence

This section aims to study the convergence of the TT-SUBSPIT algorithm combined with the different orthogonalization schemes, described in Section 3.2. In particular we want to highlight the effect of the orthogonality stability on the quality of the computed TT-eigenpairs.

In Tables 3.2 and 3.3 we report the m = 7 computed eigenvalues λ sorted with decreasing value, the closest theoretical eigenvalue λ * , the relative distance between the two, the convergence order, i.e., in which order the eigenvalues converged and the spectrum order, that is which position λ * occupies in the spectrum. Notice that the 7 largest theoretical eigenvalues of the discrete Laplacian ∆ 3 with mode sizes [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Coulaud | A note on GMRES in TT-format[END_REF] that is there the largest one, followed by two clusters of three eigenvalues. Comparing the different orthogonalization kernel results in Table 3.2, we see that the Gram method leads to the worst approximations, missing the largest eigenvalue and getting close to just one out of 7 eigenvalues in the selected region. On the other hand, Householder algorithm provides the best output approximating six eigenvalues out of seven in the expected spectrum region; the other kernels find just two out of seven. Basically, TT-SUBSPIT with Householder approaches the largest one, three from the first cluster and two from the second one. With the other kernels, excluded Gram's one, TT-SUBSPIT leads to an estimate of the largest one and one from the first cluster, while the other estimated eigenvalues are closer to theoretical eigenvalues localized much far in the spectrum. From the convergence order view point, it is worthwhile remarking that MGS, MGS2, CGS and CGS2 computes the approximation in the exact same order. For all the seven approximated eigenvalues and the six orthgonalization kernels, the relative distance is lower than the rounding and convergence accuracy δ = ε = 10 -3 , with the largest error realised by Householder for the largest eigenvalue. Setting the rounding accuracy and the convergence one to 10 -5 changes slightly these observations. As previously, Householder outcompetes the other kernels providing an estimate of five eigenvalues in the chosen spectrum region, while all the other kernels return just four. However, TT-SUBSPIT Householder with accuracy 10 -5 approximates one eigenvalue in the right region less than with accuracy 10 -3 , suggesting that the accuracy and the inter-cluster distance may play a role in the computing capabilities of the TT-SUBSPIT algorithm. Notice that for δ = 10 -5 the approximation of the eigenvalues outside the right spectrum region are closer to the first theoretical seven than in the δ = 10 -3 case. Moreover, the relative distance between the approximated eigenvalues and the theoretical closest ones is significantly lower than the prescribed rounding and converging accuracy 10 -5 . Table 3.3m = 7 eigenvalues for the Laplacian of order d = 3 and mode size n = [19, [START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Coulaud | A note on GMRES in TT-format[END_REF] with δ = 10 -5 .

EIGENSOLVERS

EIGENSOLVERS

The information about the estimation of the discrete Laplacian of order 3 and mode sizes [START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF] eigenvalue are depicted in Table 3.4 and 3.5 for accuracy 10 -3 and 10 -5 respectively. The analytical value of the 7 largest eigenvalues of this operator are Thus, similarly to the previous experimental case, there is the largest eigenvalue, followed by two eigenvalues with multiplicity 3, that is two degenerate clusters of size 3. Table 3.4 highlights that, as in the previous experimental framework, TT-SUBSPIT with Householder manages to approximate six eigenvalues in the right spectrum region, while all the other kernels lead to eigenvalues located outside the selected region. The relative distance between the approximation and the theoretical eigenvalue is always lower than the rounding accuracy 10 -3 , with Householder realising the largest relative error. As in the previous case, MGS and CGS with and without re-orthogonalization return the eigenpair approximations in the same order. When the rounding and converging accuracy δ and ε are set equal to 10 -5 , the situation changes. Indeed, as reported in Table 3.5, Householder leads to four estimations in the right spectrum region, while all the other kernels to three. As in the other experiment set-up, when the accuracy value decreases, all the kernels performances improve, except for Householder, which remains in any case the most effective. Moreover, in this case all the kernels except Householder present the same convergence order. As previously, the relative distance between the numerical eigenvalue and the theoretical one is much lower than the rounding accuracy 10 -5 . Table 3.5m = 7 eigenvalues for the Laplacian of order d = 3 and mode size n = [24, 24, 24] with δ = 10 -5 .

To complete the investigation of the convergence quality depending on the chosen orthogonalization kernel, we study the minimum and the maximum value of the stopping criterion, that is the scaled residual; for simplicity we will refer to this by the term scaled residual envelope. More precisely, we study the scaled residual envelope in function of the number of orthogonalizations, counted by the i variable in Algorithm 19, and of the number of TT-rounding call performed, estimated by the iter variable in Algorithm 19. The number of orthogonalizations counts the number of loops, while the number of TT-rounding estimates the algorithm complexity. In Figures 3.5A and 3.5B, we display the scaled residual envelope in function of the number of orthogonalizations and of TT-rounding for the discrete order 3 Laplacian of mode sizes [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Coulaud | A note on GMRES in TT-format[END_REF]. From the orthogonalization counter point of view, Householder seems to be the optimal one, together with Gram for δ = 10 -3 , as seen in Figure 3.5A, while for δ = 10 -5 , CGS2 is the slowest one and Householder the second slowest one, see Figure 3.5B. From the number of TT-rounding instead, Householder is always the slowest, as shown in Figure 3.5C and 3.5D, i.e., it is the one with the highest complexity in terms of TT-rounding, the most expensive operation. However, it is important to remark that Householder for both the accuracy values leads to the best results in terms of robustness to compute the m largest eigenvalues. Remark that for δ = 10 -5 all the envelopes display many more peaks and valleys compared to the envelopes for δ = 10 -3 . Notice that also in the classical linear algebra scheme the convergence monotony of the subspace iteration method is not theoretically guaranteed.

The scaled residual envelope from the discrete Laplacian of mode sizes [START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF] experiments are displayed in Figure 3.6. In this case the Householder kernel is the one requiring more orthogonalizations and more TT-rounding steps to converge, but for both the accuracy values it leads to the best results in terms of robustness to compute the m largest eigenvalues. We highlight that as in the previous case, for δ = 10 -5 all the envelopes present peaks and valleys, while for δ = 10 -3 all the envelopes except the Householder one are monotonously decreasing. Residual envelope in function of the number of TT-rounding Residual envelope in function of the number of TT-rounding

EIGENSOLVERS

Memory requirement

This section is devoted to the analysis of the TT-eigenvector memory requirement through the study of the maximal TT-rank and the maximal compression ratio. In particular we display the maximum, among all the m = 7 computed TT-eigenvectors, the maximum TT-rank and compression ratio. Notice that, as in the previous section, we display both the maximal TT-rank and the compression ratio in function of the number of orthogonalization and of TT-rounding steps.

The memory requirement for the approximation of the TT-eigenvectors of the discrete order d = 3 Laplacian of mode sizes [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF][START_REF] Carme-N | Spanish National Health Survey[END_REF][START_REF] Coulaud | A note on GMRES in TT-format[END_REF] are shown in Figure 3.7. For δ = 10 -3 , all the orthogonalization kernels lead to some initial oscillations, higher for Householder, and then to a maximum TT-rank stagnating at 3. For δ = 10 -5 , the maximum TT-rank of the TT-eigenvectors approximated through TT-SUBSPIT with Householder or CGS2 stagnates at 3, while when the other orthogonalization kernels are used, the maximum TT-rank sets to 4. As comparison term, remark that the theoretical eigenvectors in TTformat have TT-rank equal to 1. From Figures 3.7E, 3.7F the compression ratio of TT-eigenvector approximations presents some peaks, which hit 16% of the memory needed to store the dense format tensor for Householder and 10% for the other schemes, and finally it stagnates around 2% -3%. Similarly, in Figures 3.7G and 3.7H, the compression ratio has some peaks reaching, for all kernels, 16% of the memory requested to store the same tensors in dense format and it sets between 2% and 3% at convergence, when the rounding accuracy is 10 -5 . In this case, the maximal TT-rank and the compression ratio curves in function of the number of TT-roundings or orthogonalizations do not present significantly different behaviours.

The memory requirement of TT-eigenvector approximations for the discrete order 3 Laplacian with all mode size equal to 24 are displayed in Figure 3.8. For the rounding and the converging accuracy equal to 10 -3 , the maximal TT-rank hits 10 when the Householder kernel is used and 8 when the other orthogonalization schemes are applied, stagnating in any case at 3 at convergence, as displayed in Figure 3.8A and 3.8B. For δ = 10 -5 , the maximal TT-rank associated with every orthogonalization schemes hits 10 during a first phase and sets to 2 for Householder and to 3 for the other kernels, see Figure 3.8C and 3.8D. From the compression ratio side, Figures 3.8E and 3.8F highlight that the compression ratio of the TT-eigenvectors approximated with the Householder kernel reaches higher peaks, as 0.2, that reads as 20% of the dense format memory requirement, than the other schemes, which do not overcome 0.13. However, at convergence the Householder compression ratio stagnates around 0.02 slightly below the other scheme curves. Similarly for δ = 10 -5 , all the compression ratio curves reach 0.2 during a first phase and they decrease to less than 0.02 for the Householder one and to 0.02 for the other schemes, as displayed in Figures 3.8G

Concluding remarks

This chapter aims to study the orthogonalization algorithms implemented with the TT-formalism with a particular focus on the TT-rounding effects. We firstly describe our extension to the tensor framework of six classical orthogonalization schemes through the TT-formalism. Then, we study the quality in terms of orthogonality of the computed basis and we relate our numerical results with the classical linear algebra theoretical ones. For completeness, we present an application of the considered kernels: an iterative eigensolver in TT-format. After describing the chosen eigensolver algorithm and highlighting its tensor specificities, we analyse its behaviour. Through a study case, we investigate how the chosen orthogonalization methods affect the quality of the computed eigenvalues and TTeigenvectors. In all the numerical experiments, particular attention is paid to the memory requirements, which are known to be a major constraint when tensors are involved.

In the first part, corresponding to Sections 3.2, six orthogonalization kernels are described. Firstly we present the Classical Gram-Schmidt (CGS) and its Modified version (MGS) extended to the tensor framework with the TT-formalism. To be as comprehensive as possible, the CGS and MGS algorithms with re-orthogonalization are addressed as well. After detailing the structure in TT-format of the Gram algorithm, the Householder transformation is introduced. This last scheme requires a more accurate generalization in TT-format, which is completely given in Section 3.2.3. To face the memory requirements, we include in all the orthogonalization schemes further rounding steps, that play a crucial role in maintaining the computations affordable and in determining the quality of the computed orthogonal basis. The theoretical classical linear algebra bounds for the loss of orthogonality of these six orthogonalization kernels are presented in Section 3.2.4 and compared with the numerical experiments reported in Section 3.2.5. We generate with a Krylov process an input set of TT-vectors, which become more and more collinear by construction, and we orthogonalize them through the considered methods. The two sets of experiments of different orders suggest that the theoretical classical bounds can be generalized to the tensor context with the TT-formalism replacing the IEEE unit round-off with the TT-rounding accuracy. We include an analysis of the memory requirement for the basis computation estimated through the maximal TT-rank and the compression ratio of TT-vectors forming the orthogonal basis. Moreover, the complexity of the algorithms is evaluated in terms of TT-rounding operations, which represent the major computational cost.

The second part, coinciding with Section 3.3, presents a possible application of these kernels, that is inside the SUBSPace ITeration eigensolver formulated in TT-format (TT-SUBSPIT). Section 3.3.1 includes the description of the TT-SUBSPIT algorithm, while the numerical results are found in Section 3.3.2. As for the orthogonalization schemes, also for TT -SUBSPIT some TT-rounding steps are required to prevent memory deficiencies. For the numerical experiments, we consider a classical study case in tensor format, i.e., the discrete order 3 Laplacian with different mode sizes. We compare the computed eigenvalues at different rounding accuracies with their analytical values, remarking that the Householder kernel seems to outcompete the other ones approximating more eigenvalues in the right spectrum region. For the sake of completeness, we analyse the convergence through the stopping criterion, that is the scaled residual, and the memory requirement through the maximal TT-rank and the compression ratio. Those preliminary results are a starting point for the investigation of the TT-SUBSPIT algorithm for solving tensor eigenproblems.

Data science, and every aspect related to this discipline, have been on the crest of a wave since 2010 [START_REF] Braschler | Applied Data Science: Lessons Learned for the Data-Driven Business[END_REF] when media and business popularized it, see [START_REF] Davenport | Data scientist: The sexiest job of the 21st century[END_REF][START_REF] Loukides | What is data science?[END_REF][START_REF] Manyika | Hal Varian on how the Web challenges managers[END_REF][START_REF] Patil | Building data science teams[END_REF]. Even if it has impacted economic, social and scientific levels [START_REF] Braschler | Applied Data Science: Lessons Learned for the Data-Driven Business[END_REF], data science as a field of study has not yet a clear and universally accepted definition. The problem of stating what are the topics of interest in data science was firstly addressed in 1962 by Tukey [START_REF] Tukey | The Future of Data Analysis[END_REF], who started the discussion inside the scientific community long before this discipline gained its notoriety. Notice that in 1962 Tuckey was using the term 'data analysis' and the expression 'data science' arrived later, in 1985 at the Chinese Academy of Sciences in Beijing during a talk by Jeff Wu [START_REF] Braschler | Applied Data Science: Lessons Learned for the Data-Driven Business[END_REF], who proposed it as a substitute for 'statistics'. A more wide and more comprehensive definition was proposed in 1998 by Hayashi [START_REF] Hayashi | What is Data Science ? Fundamental Concepts and a Heuristic Example[END_REF], who highlighted data science's multidisciplinary nature and the three main investigation topics, i.e., how to design, collect, and analyse data. Nowadays, a popular definition [START_REF] Braschler | Applied Data Science: Lessons Learned for the Data-Driven Business[END_REF][START_REF] Stadelmann | Applied data science in Europe : challenges for academia in keeping up with a highly demanded topic[END_REF] highlights the presence of methods and principles, coming from both theoretical areas, such as statistics or computer science, and applicative ones, such as economics, psychology, and biology.

A central topic in data science is analysing the data, extrapolating and visualizing the significant information. The increasing number of large-size datasets, for example, those coming from CERN experiments or Hubble Space Telescope images, from private technology companies such as Google, Amazon, and Facebook, or private sensors or devices, has acted as propulsion for developing new data analysis techniques able to handle huge data. Therefore, in recent years researchers from different fields started modelling and investigating their datasets, relying on tensor theory and algorithms. In the following chapters, we address two data analysis techniques, namely Correspondence Analysis (CA) and climate data analysis popular method referred to by Empirical Orthogonal Function (EOF).

II.I.I Correspondence Analysis background

In 1904 Pearson referred with the expression 'contingency tables' to datasets whose values are frequencies or counts of a combination of two categories belonging respectively to two different categorical variables, for example, the hair colour or the social class [START_REF] Pearson | On the Theory of Contingency and Its Relation to Association and Normal Correlation[END_REF]. The standard technique for analysing contingency tables and visualizing their information is Correspondence Analysis (CA), whose mathematical roots appeared during the 1930s in Richardson's and Kuder's [START_REF] Richardson | Making a rating scale that measures[END_REF], Horst's [START_REF] Horst | Measuring Complex Attitudes[END_REF] and Hartley's [START_REF] Hirschfeld | A Connection between Correlation and Contingency[END_REF] (whose former family name was Hirschfeld) works. During the 1940s, the notable statisticians, Fisher [START_REF] Fisher | The precision of discriminant functions[END_REF] and Guttman [START_REF] Guttman | The Quantification of a class of attributes: A theory and method of scale construction[END_REF] separately derived the same technique in different contexts, that is biometrics and psychometrics. The ideas of Guttman were further developed by Hayashi [66] between the 1940s and the 1950s, while in the same period Williams [START_REF] Williams | Use of scores for the analysis of association in contingency table[END_REF] worked on Fisher's proposal. All those contributions, sharing the same background procedure, are meant for returning a numerical result rather than a visual one [START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF]. Only with the advent of Benzécri's research during the 1960s, the first geometrical interpretations and outputs

II.I. INTRODUCTION

arrived together with the French expression 'analyse des correspondances' [START_REF] Benzécri | Statistical analysis as a tool to make patterns emerge from data[END_REF][START_REF] Benzécri | Sur l'analyse des tableaux binaires associés à une correspondance multiple[END_REF]. Hill's publication of 1974 [START_REF] Hill | Correspondence Analysis: A Neglected Multivariate Method[END_REF] diffused Benzécri's and his collaborators' results, previously limited to the French audience, suggesting the name correspondence analysis directly from the French translation. His later publication of 1982 [START_REF] Hill | Correspondence analysis[END_REF] defines CA with its geometric aspects.

In many domains, for example, social or biological sciences, frequently count data come from more than two categorical variables; these datasets are usually called multiway contingency tables. A common technique for studying multiway contingency tables is Multiple Correspondance Analysis (MCA). The principle of MCA is associating the multiway table with an indicator or Burt table [START_REF] Burt | The factorial analysis of qualitative data[END_REF], which is mathematically speaking, a matrix of only 0 and 1. The roots of these techniques can be found already in Guttman's paper of 1941 [START_REF] Guttman | The Quantification of a class of attributes: A theory and method of scale construction[END_REF]. The expression 'multiple correspondence analysis' was presented by Burt in 1950, but the complete formulation of MCA as analysing and visualizing methods was proposed during the 1970s by Benzécri and Lebart [START_REF] Benzécri | Sur l'analyse des tableaux binaires associés à une correspondance multiple[END_REF][START_REF] Lebart | Validité des résultats en analyse des données[END_REF][START_REF] Lebart | Techniques de la Description Statistique: méthodes et logiciels pour l'analyse des grands tableaux[END_REF]. Starting from the 1970s, several researchers developed MCA; the works of Kroonenberg, Kiers, Marcotorchino, Clausen, Beh, D'Ambra, and Lombardo contributed to the MCA investigation in the last thirty years, and an exhaustive description is presented in [START_REF] Beh | Correspondence Analysis[END_REF]. As pointed out in [START_REF] Beh | Correspondence Analysis[END_REF], MCA still relies on matrix representations of multiway tables, and consequently, it discards without the possibility of retrieving the information about the interactions among more than two variables. Thus, at the end of the 1990s [START_REF] Beh | Correspondence Analysis[END_REF][START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF], tensors were introduced to mathematically represent multiway tables, developing the MultiWay Correspondence Analysis (MWCA) technique, even if some previous attempts were already made between the 1970s and the 1980s [START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF]. Lombardo's [START_REF] Lombardo | Nonsymmetric correspondence analysis for three-way contingency tables[END_REF], Carlier's, and Kroonenberg's [START_REF] Carlier | Decompositions and biplots in three-way correspondence analysis[END_REF][START_REF] Carlier | Chapter 19 -The Case of the French Cantons: An Application of Three-Way Correspondence Analysis[END_REF][START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF][START_REF] Kroonenberg | History of multiway component analysis and three-way correspondence analysis[END_REF][START_REF] Kroonenberg | Three-mode principal component analysis: Theory and applications[END_REF] results are based on the CP [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF] and the Tucker [START_REF] Tucker | Some mathematical notes on three-modes factor analysis[END_REF] decomposition.

II.I.II Climate data analysis history

The weather and climate have always fascinated and intrigued humankind. Even if the collection of weather information started already during the XIX century, the scientific investigation of climate patterns through mathematical techniques made significant progress only from the 1920s [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF]. After the preliminary results of Walker and Ångström obtained between the 1920s and the 1930s, Obukhov and Bagrov introduced linear algebra tools in the study of climate data between 1940 and 1950 [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF]. The invention of Empirical Orthogonal Function (EOF), today's popular climate data analysis technique [START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF], is attributed to Lorenz [START_REF] Lorenz | Empirical Orthogonal Functions and Statistical Weather Prediction[END_REF], who first proposed the expression, even if other scientists as Obukhov [START_REF] Obukhov | Statistically homogeneous fields on a sphere[END_REF] and Fukuoka [START_REF] Fukuoka | The central meteorological observatory, a study on 10-day forecast (a synthetic report)[END_REF] already described a similar analysis procedure. Since the second half of the XX century, the EOF method has continuously developed with many theoretical and practical contributions; we refer the reader to [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF][START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF] and to the references therein for an exhaustive list of EOF-related works.

The EOF method is meant for the investigation and the independent visualization of data that depends on time and space, which are the usual variable of climate data. In particular, it works on dimensional reduction and pattern extraction [START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF]. Since climate datasets are usually of huge dimensions, it would be natural to generalize the EOF method part II. data analysis 131 through tensor compression techniques. We are not aware of any previous contribution that describes EOF in the tensor framework, even if in literature are present some studies of climate data through tensor techniques, as for example [START_REF] Zhang | A Parallel Nonnegative Tensor Factorization Algorithm for Mining Global Climate Data[END_REF].

II.II Statistics preliminaries

In this section, we introduce some basic statistical concepts, that we will use in the following chapters

The first key concept is the average or mean of an array, whether it represents a vector, a matrix, or a tensor. Definition II.II.i. Let x ∈ R n be a vector, whose i-th component represent the i-th observation of a random variable, the (sample) average or mean of x is µ x ∈ R defined as

µ x = 1 n n i=1
x(i).

The definition of average is generalized mode-wise to matrices and tensors. Given a tensor x ∈ R n 1 ו••×n d , its (sample) average or mean with respect to mode k is an order (d -1) tensor

x k ∈ R n 1 ו••×n k-1 ×n k+1 ו••×n d such that x k (i 1 , . . . , i k-1 , i k+1 , . . . , i d ) = 1 n k n k i k =1
x(i 1 , . . . , i k , . . . , i d ).

for every i h ∈ {1, . . . , n h } with h ∈ {1, . . . , k -1, k + 1, . . . , d} and k ∈ {1, . . . , d}. If d = 2, the considered data structure is a matrix with x 1 ∈ R n 2 and x 2 ∈ R n 1 its column and row average respectively. The variance is a mathematical object closely related to the average. As previously, we present the definition of variance for a vector and then we generalize it to tensors. Definition II.II.ii. Let x ∈ R n be a vector, whose i-th component represent the i-th observation of a random variable, the (sample) unbiased variance of x is s x ∈ R defined as

s x = 1 n -1 n i=1 (x(i) -µ x ) 2 = 1 n x -µ x 1 n , x -µ x 1 n = 1 n x -µ x 1 n 2 where 1 n ∈ R n is a vector of ones. The standard deviation is σ x = √ s x .
The (sample) unbiased variance with respect to mode k of an order d tensor

x ∈ R n 1 ו••×n d is an order (d -1) tensor s k ∈ R n 1 ו••×n k-1 ×n k+1 ו••×n d such that s k (i 1 , . . . , i k-1 , i k+1 , . . . , i d ) = 1 n k -1 n k i k =1 x(i 1 , . . . , i k , . . . , i d ) -x k (i 1 , . . . , i k-1 , i k+1 . . . , i d ) 2 = 1 n k -1 n k i k =1 x -1 n k ⊗ x k (i 1 , . . . , i k , . . . , i d ) 2
where 1 n k ∈ R n k is a vector of ones, for every i h ∈ {1, . . . , n h } with h ∈ {1, . . . , k -1, k + 1, . . . , d} and k ∈ {1, . . . , d}. The standard deviation with respect to mode k of x is the order (d -1) tensor defined as the square root of the mode k variance, that is

σ k = √ s k .
Another statistical definition, that we will use later on, is the covariance.

Definition II.II.iii. Given two vectors x, y ∈ R n whose components are the observations of two different random variables respectively, then c x,y ∈ R is the covariance of x and y computed as

c x,y = x -µ x 1 n , y -µ y 1 n .
More generally if X ∈ R n 1 ×n 2 is a matrix whose i-th column x i ∈ R n is the realization of a random variable for every i ∈ {1, . . . , p}, then its covariance matrix is

Q X ∈ R n 2 ×n 2 such that Q X (j, k) = n 1 i=1 X(i, j) -x 1 (j) X(i, k) -x 1 (k) = X -1 n 1 ⊗ x 1 X -1 n 1 ⊗ x 1
where 1 n 1 ∈ R n 1 is a vector of ones. The (j, k)-th element of Q X is actually equal to the covariance of the j-th and k-th column of X, if j = k, while Q X (j, j) is the (scaled) variance of the j-th column of X.

II.III Principal Component Analysis

A common technique to explore data is Principal Components Analysis (PCA) [START_REF] Atchley | Multivariate Statistical Methods: Among-Groups Covariation[END_REF][START_REF] Chambers | Computational Methods for Data Analysis[END_REF][START_REF] Gabriel | The biplot graphic display of matrices with application to principal component analysis[END_REF][START_REF] Hotelling | Analysis of a complex of statistical variables into principal components[END_REF][START_REF] Jolliffe | Principal Component Analysis. Second. Springer Series in Statistics[END_REF][START_REF] Jolliffe | Principal component analysis: a review and recent developments[END_REF][START_REF] Kendall | Multivariate analysis[END_REF][START_REF] Morrison | Multivariate Statistical Methods[END_REF][START_REF] Pearson | LIII. On lines and planes of closest fit to systems of points in space[END_REF]. The analysis procedures we investigate in the following chapters, i.e., CA in Chapter 4 and the EOF analysis in Chapter 5, rely on PCA. This method has three different natures, which are statistical, geometrical, and algebraical, that CA and EOF analysis inherit. We briefly present those three facets, under the same assumptions. Let X ∈ R n×p be a matrix whose i-th column x i ∈ R n is the realization of a random variable for every i ∈ {1, . . . , p}. In addition, we assume that every column of X has zero mean, i.e., x 1 (j) = 0 for every j ∈ {1, . . . , p}.

II.III.I Statistical viewpoint

The statistical purpose of PCA is to find a linear combination of the random variables that maximize their covariance. Mathematically, this problem seeks the first principal component y ∈ R n such that y = Xv * where v * ∈ R p is the said to be the loading and it part II. data analysis 133 is obtained as

v * = arg max ||v||=1 (s Xv ) = 1 p arg max ||v||=1 Xv, Xv = 1 p arg max ||v||=1 (v X Xv) = 1 p arg max ||v||=1 (v Q X v)
with the covariance matrix Q X = X X thanks to the assumption that the column average of X is zero. Since the matrix Q X is by construction symmetric, then it can be decomposed into the orthogonal basis formed by its eigenvectors, that is Q X = W ΛW with Λ = diag(λ 1 , . . . , λ p ) where w i and λ i are the i-th eigenvector and eigenvalue respectively. We assume that the eigenvalues, and consequently the eigenvectors, are sorted with decreasing absolute values. Thus, the PCA problem is

v * = 1 p arg max ||v||=1 (v W ΛW v) = 1 p arg max W v=z ||v||=1 (z Λz)
The solution to this maximization problem is z * = e 1 the first canonical basis vector and v * equal to w 1 the first eigenvector, associated with the largest eigenvalue. Consequently, the first principal component is y 1 = Xv * = Xw 1 . If we want to determine r principal components, we impose that the r loadings have norm 1 and are mutually orthogonal, getting that the i-th loading is the i-th eigenvector of the covariance matrix Q X and the i-th principal component is y i = Xw i for every i ∈ {1, . . . , r}. We refer the reader to [START_REF] Ma | Generalized Principal Component Analysis[END_REF]Theorem 2.1] for further details about the statistical viewpoint of PCA.

II.III.II Geometrical viewpoint

In this case, it is more convenient to assume that the observations are arranged columnwise, i.e., assuming that Y ∈ R p×n with n the number of observations and p the number of random variables considered. Remark that Y = X used in the previous section.

The geometrical facet of PCA regards the j-th column of Y as the coordinate array of the j-th observation in a space of dimension p. Thus, the aim of PCA from the geometrical viewpoint is finding a subspace of R p of dimension r which minimizes the projection error, i.e., the PCA is geometrically speaking a dimensionality reduction method. Mathematically the problem is determining S * ⊆ R p with dim(S * ) = r which minimizes the projection error of all the columns of Y , that is

S * = arg min S⊆R p n i=1 y i -π S (y i ) 2
where π S : R p → S is the projection map. Each subspace is determined by a basis, thus the projection map can be expressed using the selected basis. If the columns of U ∈ O(p × r) form an orthonormal basis of S, that is S = span(U ), then the projection by π S of z ∈ R p is expressed as π S z = U U z. Thanks to this result, the geometric PCA problem becomes determining the orthonormal basis U * ∈ O(p × r) which minimizes the projection error, i.e.,

U * = arg min U U =Ir n i=1 y i -U U y i 2 .
This is a classical least squares problem in linear algebra, whose solution is given by U ∈ O(p × r) the left orthogonal matrix of the SVD of Y = U ΣV truncated at the r-th column, which is also called loading matrix. The matrix U Y , which is equal to ΣV by construction, is referred as the principal components or coordinates of Y . The columns of U define the axis of the low dimensional space where we display the data, while the columns of ΣV represent the data coordinates in the new low dimensional subspace. Notice that the statistical and the geometrical results are coherent. Indeed, in the statistical loading matrix is W the eigenvector matrix of Q X , which is equal to the right orthogonal matrix of the SVD of X, that is X = Y = V ΣU by construction, i.e., the loading matrix found with the geometrical approach, cf. [START_REF] Ma | Generalized Principal Component Analysis[END_REF]Theorem 2.3].

II.III.III Algebraic viewpoint

In the algebraic context, given Y ∈ R n×p , PCA seeks a matrix Z * ∈ R n×p of rank r which minimizes the approximation error, that is

Z * = arg min rank(Z)=r Y -Z
with r < rank(Y ). Under this formulation, the best solution is guaranteed by the Eckart-Young theorem [START_REF] Eckart | The approximation of one matrix by another of lower rank[END_REF] through the SVD of Y . This algebraic approach is exactly equivalent to the previous two, leading indeed to the same solution, cf [START_REF] Ma | Generalized Principal Component Analysis[END_REF]Theorem 2.6]. This type of approach is often of interest in face recognition problems.

II.IV Tensor formalism

The generalization of PCA to tensor has been for many years a discussion topic, leading to different results, summarized in [START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF][START_REF] Kroonenberg | Three-mode principal component analysis: Theory and applications[END_REF], relying either on the Tucker model or on the Canonical Polyadic one. The study performed and reported in the following chapters are based on the Tucker model and its HOSDV realization, even if it is known that the HOSVD approximation at a prescribed accuracy or prescribed multilinear does not guarantee the optimality of the approximation as SVD does.

Chapter 4

A geometric framework for multiway correspondence analysis

Introduction

Correspondence Analysis (CA) [START_REF] Benzécri | 'analyse des correspondances[END_REF][START_REF] Gauch | Multivariate Analysis in Community Ecology[END_REF][START_REF] Gifi | Nonlinear Multivariate Analysis[END_REF][START_REF] Greenacre | Correspondence Analysis of HLA Gene Frequency Data from 124 Population Samples[END_REF][START_REF] Greenacre | Some objective methods of graphical display of a data matrix[END_REF][START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF][START_REF] Hill | Correspondence Analysis: A Neglected Multivariate Method[END_REF] is a well suited tool for the study of categorical data, which usually are stored in contingency tables 1 . In particular, this analysis technique makes possible the simultaneous visualization and interpretation of categories of two variables in a low dimension space. CA is the Principal Component Analysis (PCA), see Section II.III, of a contingency table with a specific metric and consequently it inherits the PCA three possible natures. Indeed, the PCA of a matrix has an algebraic, a geometric and a statistical guise simultaneously. A two variables table is algebraically represented by a matrix and from the algebraic viewpoint PCA looks for the best approximation of this matrix at a prescribed rank. As explained in [START_REF] Eckart | The approximation of one matrix by another of lower rank[END_REF], PCA relies on SVD for determining the best matrix approximation at a prescribed rank. Geometrically speaking, a two variable table is associated with two point clouds, i.e., the sets of row and the column vectors of the original matrix. These vectors are the coordinates of the different variables in the full dimensional space. Therefore, geometrically speaking, PCA looks for a subspace of low dimension which minimizes the projection error of the two original point clouds, as described in [START_REF] Pearson | LIII. On lines and planes of closest fit to systems of points in space[END_REF]. Then, in a statistical context, the table entries represent observations of two variables and the statistical purpose of PCA is maximizing the variance, identifying a small amount of linear independent variable combinations. As PCA, also CA looks for the best approximation at a given rank or equivalently for the optimal subspace of low dimension, but taking into account a specific weight metric equivalent to the classical one. Particularly, a barycentric relation links the two point clouds when projected in the low dimension subspace found by CA [START_REF] Lebart | Traitement des données statistiques: méthodes et programmes[END_REF]. This link justifies the simultaneous interpretation and visualization of the two point clouds.

When there are more than two variables in the contingency The remainder of this chapter is organized as follows. The first section presents the classical CA theory and then the MWCA with a focus on the geometric viewpoint, which is our theoretical contribution to the MWCA literature. In conclusion to this part, we compare the CA and the MWCA on two test examples, previously studied in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. The second section is centred on the analysis of the Malabar multiway contingency table [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF]. We apply the MWCA technique, which is known as the most suitable investigation tool for frequency data, using the theoretical results to interpret the dataset.

Correspondence Analysis

After recalling briefly the CA theory, we describe the construction of point clouds from multiway data, one per mode, and we link them together. More precisely, we show that each point cloud is associated with each mode of a tensor in MWCA in the same way that a point cloud is associated with either rows or columns of a matrix in CA. This theoretical work justifies the possibility of visualizing and interpreting simultaneously point clouds constructed in MWCA from multiway data. To prove the existence of a barycentric link between point clouds attached to each tensor mode and generalizing the correspondence between rows and columns for contingency tables, we pass through three steps. Firstly we express the link in a classical Euclidean space, after we generalize to any Euclidean space and finally we construct the MWCA metric, stating the barycentric relation.

Matrix case

This section describes briefly the main steps of CA and its main result, i.e., the barycentric relation linking row and columns point clouds, see [START_REF] Lebart | Traitement des données statistiques: méthodes et programmes[END_REF].

Let F ∈ R n 1 ×n 2 + be a contingency table, whose entry F (i, j) represents the relative 2 frequency of the i-th category for the first variable and the j-th category for the second one. As already stated, geometrically speaking, the matrix F is associated with two point clouds, i.e., two sets R = {r 1 , . . . , r n 1 } and C = {c 1 , . . . , c n 2 } where r i ∈ R n 2 is the i-th row of F and similarly c j ∈ R n 1 is the j-th column of F . The vectors r i and c j represent the components of the i-th category of the first variable in the space R n 2 and the j-th category of the second variable in the space R n 1 respectively. Henceforth we denote row and column marginals, norms and spaces by R and C respectively, and not by 1 and 2, to avoid any confusion with the classical meaning of norm-1 and norm-2, used in the previous chapters.

In the CA framework, we assume that the row space R n 2 and the column one R n 1 are endowed with two norms defined from the row and column marginals. Let f R ∈ R n 1 + be the row marginal vector, whose i-th entry is

f R (i) = n 2 j=1 r i (j) = n 2 j=1 F (i, j).
Similarly the column marginal is f C ∈ R n 2 + defined as

f C (j) = n 1 i=1 c j (i) = n 1 i=1 F (i, j).
ν R : (R n 2 , || • || R ) → (R n 2 , || • ||) and ν C : (R n 1 , || • || C ) → (R n 1 , || • ||) such that ν R (x) = D -1 R x and ν C (y) = D -1 C y are isometries, as they preserve the norm, that is ||ν R (x)|| = ||D -1 R x|| = ||x|| R and similarly for ν C . The pair of matrices (D -1 R , D -1 C ) enables us to endow a metric on the matrix space R n 1 ×n 2 . Let || • || M : R n 1 ×n 2 → R + be a metric such that ||X|| M = ||D -1 R XD -1 C ||.
We denote by S M the matrix space R n 1 ×n 2 endowed with the metric norm ||•|| M , while by S the same matrix space with the standard Euclidean norm. The bijection ν : S M → S defined as ν(X) = D -1 R XD -1 C is an isometry among S M and S, since it preserve the norm, that is ||ν(X)|| = ||X|| M . Remark that the action of ν can be seen as the combined action of ν C and ν C on the row and columns of an element of S M respectively.

The geometrical aim of CA is finding a low dimension space R s with s min{n 1 , n 2 } which minimizes the projection error of the two point clouds R and C, so that they can be 2 any contingency tables with absolute frequencies can be transformed into a relative frequency one 138

CORRESPONDENCE ANALYSIS

simultaneously visualized and interpreted easily. From the algebraic viewpoint this means finding the best approximation of a given contingency table at a prescribed rank in S M , thus it is not possible to directly apply the SVD to F ∈ S M . To retrieve the best low rank approximation, we transport F ∈ S M through the isometry ν into the Euclidean space S, where we can apply SVD. So given the contingency table F belonging to the space S M , let X ∈ S be its image through the isometry ν, i.e., X = ν(F ), then we consider the reduced SVD of

X X = U ΣV with U ∈ O(n 1 × r), V ∈ O(n 2 ×
Z R = D -2 R F Z C Σ -1 and Z C = D -2 C F Z R Σ -1
where the scaled principal components are Z i ∈ R n i ×r with Z i = D -2 i W i for i ∈ {R, C}. These relations state that the row scaled principal components in (R n 2 , ||•|| R ) are actually the scaled barycentre of the column scaled principal components in (R n 1 , || • || C ) and viceversa. Indeed component-wise, the previous relation writes

Z R (i, h) = 1 σ h n 2 j=1 F (i, j) f R (i) Z C (j, h) and Z C (j, h) = 1 σ h n 1 i=1 F (i, j) f C (j) Z R (i, h), (4.1)
which is a barycentric relation scaled by the h-th singular value, since the sum of weights of the principal components sum is equal to 1, i.e.,

n 2 j=1 F (i, j) f 1 (i) = n 1 i=1 F (i, j) f 2 (j) = 1
by the marginal definition. CA relies on this relation for justifying the simultaneous visualization and interpretation of the two contingency table variables. We refer to [START_REF] Lebart | Traitement des données statistiques: méthodes et programmes[END_REF] for further details on the barycentric relation linking the principal components in the CA framework.

Tensor case

Starting from the extension of principal component analysis to tensors with HOSVD, cf. Algorithm 1, we associate a point cloud with each mode, and show the existence of an algebraic link between them. The aim of this section is providing a geometrical interpretation of the point cloud relations algebraically proved in the tensor framework. For sake of simplicity, we first prove a link between the point clouds in the standard Euclidean 3-order tensor space, and then we generalize to d-order tensors. This structure choice is kept throughout this theoretical section. At the end of each section we present the geometric interpretation of the stated results. We naturally attach a point cloud to each matricization of 3-order tensor, see Definition 1.2.3. Then we search the optimal projection of each point cloud defined from the mode matricization in low dimension space. Finally, we show how their coordinates are linked and we extend this result to general Euclidean spaces of d-order tensors.

Let x ∈ R n 1 ו••×n d be a tensor of order d and let (U µ ) µ ∈ O(n µ × r µ ) be the Tucker decomposition basis obtained from the HOSVD algorithm at multi-linear rank r = (r 1 , . . . , r d ). The tensor x is expressed as

x = r 1 ,...,r d i 1 ,...,i d =1 c(i 1 , . . . , i d )u 1 i 1 ⊗ • • • ⊗ u d i d (4.2)
with c ∈ R n 1 ו••×n d being the HOSVD core tensor and u µ iµ ∈ R nµ the i µ -th column of U µ . The condition for Equation (4.2) to be a decomposition is r µ = rank(X (µ) ) for µ ∈ {1, . . . , d}, otherwise we have an approximation of the given tensor. To keep the notation plain, we denote by x both the approximation and the decomposition of the given tensor, when its nature is clear enough from the context. Let Σ µ be the diagonal singular value matrix of the matricization of x with respect to mode µ. For simplicity, the i µ -th diagonal element of Σ µ is denoted by σ (µ) iµ for every i µ ∈ {1, . . . , r µ } and for every µ ∈ {1, . . . , d}. The principal component of mode µ is Y µ ∈ R nµ×rµ defined as Y µ = U µ Σ µ for each µ ∈ {1, . . . , d}.

Principal components in the canonical Euclidean space

For sake of simplicity and clarity, we assume d equal to 3. The following proposition states a relation linking the three sets of principal components. Proposition 4.2.1. Let x be a tensor of R n 1 ×n 2 ×n 3 and let c be its HOSVD core at multi-linear rank r. Let Y µ be the principal components of mode µ. If r µ = rank(X (µ) ) for µ ∈ {1, 2, 3}, then

Y 1 = X (1) (Y 3 ⊗ K Y 2 )(B (1) ) , Y 2 = X (2) (Y 3 ⊗ K Y 1 )(B (2) ) , Y 3 = X (3) (Y 2 ⊗ K Y 1 )(B (3) ) , with b = (Σ -1 1 , Σ -1 2 , Σ -1 3 )c.

CORRESPONDENCE ANALYSIS

Proof. We start the proof for the first mode principal components. Let x be decomposed in the HOSVD basis as in Equation (4.2), i.e.,

x = r 1 ,r 2 ,r 3 i,j,k=1 c(i, j, k)u 1 i ⊗ u 2 j ⊗ u 3 k .
As previously stated in Corollary 1.2.4 the matricization of x with respect to mode 1 in the Tucker basis is expressed with the Kronecker product, defined in 1.2.4, as

X (1) = r 1 i=1 u 1 i ⊗ r 2 ,r 3 j,k=1 c(i, j, k)u 3 k ⊗ K u 2 j . (4.3)
The PCA of X (1) is

X (1) = Y 1 V 1 = r 1 i=1 σ (1) i u 1 i ⊗ v 1 i (4.4) with v 1 i the i-th column of V 1 ∈ O(n 2 n 3 × r 1 ) and σ (1) i u 1 i the i-th column of Y 1 ∈ R n 1 ×r 1 .
By comparing Equations (4.3) and (4.4) for a fixed index i, we get

σ (1) i u 1 i ⊗ v 1 i = u 1 i ⊗ r 2 ,r 3 j,k=1 c(i, j, k)u 3 k ⊗ K u 2 j .
Since Σ 1 is invertible by the hypothesis r 1 ≤ rank(A (1) ), we identify v 1 i with a linear combination of the Kronecker product of u 2 j and u 3 k scaled by σ

(1) i as (3) k , we express the i-th column of V 1 as a linear combination of the Kronecker product of the j-th and k-th column of Y 2 and Y 3 , i.e.,

v 1 i = 1 σ (1) i r 2 ,r 3 j,k=1 c(i, j, k)u 3 k ⊗ K u 2 j . ( 4
v 1 i = 1 σ (1) i r 2 ,r 3 j,k=1 c(i, j, k) σ (2) j σ (3) k σ (2) j σ (3) k u 3 k ⊗ K u 2 j = r 2 ,r 3 j,k=1 c(i, j, k) σ (1) i σ (2) j σ (3) k y 3 k ⊗ K y 2 j = r 2 ,r 3 j,k=1 b(i, j, k) y 3 k ⊗ K y 2 j (4.6) with b = (Σ -1 1 , Σ -1 2 , Σ - 1
3 )c, y 2 j and y 3 k the j-th and k-th column of Y 2 and Y 3 respectively. Remark that Y 3 ⊗ K Y 2 is a matrix of n 2 n 3 rows and r 2 r 3 columns whose -th column is y 3 k ⊗ K y 2 j with = jk for every j ∈ {1, . . . , r 2 }, k ∈ {1, . . . , r 3 } and ∈ {1, . . . , r 2 r 3 }, see Definition 1.2.3. The tensor b matricized with respect to mode 1 is a matrix of r 1 rows and r 2 r 3 columns, whose (i, jk)-th element is b(i, j, k) for all j ∈ {1, . . . , r 2 }, k ∈ {1, . . . , r 3 }. So the sum in the right-hand side of Equation (4.6) can be expressed as the matrix-product between Y 3 ⊗ K Y 2 and tensor b matricized with respect to mode 1 as 1) ) . ) ) .

V 1 = (Y 3 ⊗ K Y 2 )(B (
V 1 yields Y 1 = X (1) V 1 .
= X (1) V 1 = X (1) (Y 3 ⊗ K Y 2 )(B ( 1
The left and right hand side of this last equation proved right the proposition thesis. The other relations follow straightforwardly from this one, permuting the indices coherently.

From an algebraic view this first proposition shows that the principal components of each mode can be expressed as a linear combination of the principal components of the two other modes. However as pointed out in the preliminary section, principal components can be seen from different viewpoints.

From a geometric viewpoint a point cloud X µ is attached to the matricization with respect to mode µ of tensor x for µ ∈ {1, 2, 3}. Indeed the i µ -th row of X (µ) represents the coordinates of the i µ -th element of mode µ point cloud living in the space R n =µ where n =µ = n 1 n 2 n 3 nµ . Given a multi-linear rank r, we reformulate the problem of the Tucker approximation as a problem of dimension reduction. Indeed, we look for the subspace of R n =µ of dimension r µ which minimizes in norm the projection of point cloud X µ on it. This problem is solved with the HOSVD algorithm, which provides three orthogonal basis (U 1 , U 2 , U 3 ) of the corresponding subspaces. Therefore the i µ -th row of Y µ = U µ Σ µ represents the coordinates of the i µ -th element of X µ projected into the subspace of R n =µ of dimension r iµ for µ ∈ {1, 2, 3}. The Proposition 4.2.1 result is interpreted geometrically as each point cloud living in the linear subspace built from the Kronecker product of the other two. Proposition 4.2.1 is generalized to the d-case as follows.

Proposition 4.2.2. Let x be a tensor of R n 1 ו••×n d and let c be its HOSVD core at multilinear rank r. Let Y µ be the principal components of mode µ. If r µ ≤ rank(A (µ) ) for µ ∈ {1, . . . , d}, then

Y µ = X (µ) (Y d ⊗ K • • • ⊗ K Y µ+1 ⊗ K Y µ-1 ⊗ K • • • ⊗ K Y 1 )(B (µ) ) with b = (Σ -1 1 , . . . , Σ -1 d )c
for every µ ∈ {1, . . . , d}. Proof. The proof is very similar to that of Proposition 4.2.1, so we give only the main steps. Let start by the first mode principal components. The 1-mode matricization of x is expressed in the Tucker basis with the Kronecker product, see Corollary 1.2.4, as

X (1) = r 1 i 1 =1 u 1 i 1 ⊗ r 2 ,...,r d i 2 ,...,i d =1 c(i 1 , . . . , i d )u d i d ⊗ K • • • ⊗ K u 2 i 2 . (4.8)

CORRESPONDENCE ANALYSIS

The PCA of X (1) leads to

X (1) = Y 1 V 1 = r 1 i 1 =1 σ (1) i 1 u 1 i 1 ⊗ v 1 i 1 (4.9) with v 1 i 1 the i 1 -th unitary column of V 1 ∈ O(n =1 ×r 1 )
where n =1 = d µ=2 n µ and σ

(1)

i 1 u 1 i 1 the i 1 -th column of Y 1 ∈ R n 1 ×r 1 .
Comparing Equations (4.8) and (4.9) for a fixed index i 1 , we identify v 1 i 1 with a linear combination of the Kronecker product of u µ iµ for µ ∈ {2, . . . , d} scaled by σ

(1)

i 1 as v 1 i 1 = 1 σ (1) i 1 r 2 ,...,r d i 2 ,...,i d =1 c(i 1 , . . . , i d )u d i d ⊗ K • • • ⊗ K u 2 i 2 . (4.10)
Introducing in Equation (4.10) the singular values σ (µ) iµ , we express the i 1 -th column of V 1 as a linear combination of the Kronecker product of the i µ -th column of Y µ for every i µ ∈ {2, . . . , d} as

v 1 i 1 = r 2 ,...,r d i 2 ,...,i d =1 c(i 1 , . . . , i d ) σ (1) i 1 σ (2) i 2 . . . σ (d) i d σ (2) i 2 . . . σ (d) i d u d i d ⊗ K • • • ⊗ K u 2 i 2 = r 2 ,...,r d i 2 ,...,i d =1 b(i 1 , . . . , i d ) y d i d ⊗ K • • • ⊗ K y 2 i 2 (4.11)
with b = (Σ -1 1 , . . . , Σ -1 d )c and y µ iµ the i µ -th column of Y µ for µ ∈ {2, . . . , d}. Thanks to the correspondence between Kronecker product and matricization, the right-hand-side of Equation (4.11) is expressed as the matrix-product between Y d ⊗ K • • • ⊗ K Y 2 and tensor b matricized with respect to mode 1 and transposed as (1) ) .

V 1 = (Y d ⊗ K • • • ⊗ K Y 2 )(B
(4.12)

Multiplying Equation (4.9) on the right by V 1 yields Y 1 = X (1) V 1 and replacing V 1 by its expression of Equation (4.12), it finally follows 1) ) .

Y 1 = X (1) V 1 = X (1) (Y d ⊗ K • • • ⊗ K Y 2 )(B (
This last equation proves the thesis. The other relations follow straightforwardly from this one, permuting the indices coherently.

Extension to a generic Euclidean space for d-order tensors

As discussed in Section 1.3.1, the minimization problem faced with the HOSVD algorithm is expressed by the Frobenious norm, induced by an inner product. The standard inner product is defined by the identity matrix. However, whatever SPD matrix induces an inner product and the associated metric norm on a vector space, which is therefore isomorphic to the standard Euclidean space. We emphasize in this section the role of the metric on the relationships between point clouds, using this isomorphic relationship between Euclidean spaces with different inner products [START_REF] Franc | Etude Algébrique des multitableaux: apports de l'algèbre tensorielle[END_REF].

Let S be the Euclidean tensor space R n 1 ו••×n d endowed with the standard inner product. Given d SPD matrices M µ of size n µ , then the space S M is the tensor space R n 1 ו••×n d endowed with the metric norm defined by ||x|| M = ||(M 1 , . . . , M d )x||. As in the matrix case discussed in Section 4.2.1, we move the f belonging to S M into S, thanks to the isometry defined by ν : S M → S, such that x = ν(f ) = (M 1 , . . . , M d )f . Remark that ν acts independently on each mode by construction.

Let now x ∈ S be the HOSVD approximation of ν(f ) at multi-linear rank r and let (U µ ) µ ∈ O(n µ × r µ ) be the associated basis. Σ µ is the singular value matrix of the µmatricization X (µ) and define Y µ = U µ Σ µ the principal components of mode µ for tensor x for each µ ∈ {1, . . . , d}. The principal components of f in the tensor space S M are the given by W µ ∈ R nµ×rµ such that

W µ = M -1 µ Y µ . (4.13)
In the following proposition, we link the sets of principal components in the Euclidean space S M . As previously, for the sake of clarity the result is first proved for d = 3 and afterwards generalized to whatever order d.

Proposition 4.2.3. Let f be a tensor in S M and let x be its image through the isometry ν in the standard tensor space S. Let c be the HOSVD core tensor of x at multi-linear rank r such that r µ ≤ rank(X (µ) ) for µ ∈ {1, 2, 3}. Then for W µ the principal component of mode µ of f in the metric tensor space S M it holds

W 1 = F (1) (M 2 3 W 3 ⊗ K M 2 2 W 2 )(B (1) ) , W 2 = F (2) (M 2 3 W 3 ⊗ K M 2 1 W 1 )(B (2) ) , W 3 = F (3) (M 2 2 W 2 ⊗ K M 2 1 W 1 )(B (3) ) , with b = (Σ -1 1 , Σ -1 2 , Σ - 1
3 )c. Proof. This result comes straightforwardly from Proposition 4.2.1 proof by introducing the metrics matrices. For completeness, we illustrate the proof focusing on the link for the principal components of the first mode. Let (U µ ) µ=1,2,3 be the HOSVD basis of x at multi-linear rank r with U µ ∈ O(n µ × r µ ). Let Y µ = U µ Σ µ be the principal components of mode µ for tensor x, where Σ µ is the singular values matrix of X (µ) . Proposition 4.2.1 yields

Y 1 = X (1) (Y 3 ⊗ K Y 2 )(B (1) ) (4.14) with b = (Σ -1 1 , Σ -1 2 , Σ -1 3 )c. Notice that x = ν(f ) = (M 1 , M 2 , M 3 )f .
So thanks to the thesis of Corollary 1.2.4, we express X (1) in function of F (1) as

X (1) = M 1 F (1) (M 3 ⊗ K M 2 )
and replacing it into (4.14), it gets

Y 1 = M 1 F (1) (M 3 ⊗ K M 2 )(Y 3 ⊗ K Y 2 )(B (1) ) (4.15)
where the transposition symbol is discarded on M µ since they are SPD matrices. Remarking that Y µ = M µ W µ from the W µ definition in Equation (4.13), substituting it in the Equation (4.15) we obtain ) ) . (4.16)

M 1 W 1 = M 1 F (1) (M 3 ⊗ K M 2 )(M 3 W 3 ⊗ K M 2 W 2 )(B ( 1
Since M 1 is invertible, from the previous equation it follows the thesis. The other relations follow straightforwardly from this proof, permuting the indices coherently.

This result is easily generalized to d-order tensors as follows.

Proposition 4.2.4. Let f be a tensor in S M and let x be its image through the isometry ν in the standard tensor space S. Let c be the HOSVD core tensor of x at multi-linear rank r such that r µ ≤ rank(X (µ) ) for for µ ∈ {1, . . . , d}. Then for W µ the principal component of mode µ of f in the metric tensor space S M it holds

W µ = F (1) (M 2 d W d ⊗ K • • • ⊗ K M 2 µ+1 W µ+1 ⊗ K M 2 µ-1 W µ-1 ⊗ K • • • ⊗ K M 2 1 W 1 )(B (µ) ) with b = (Σ -1 1 , . . . , Σ -1 d )c for every µ ∈ {1, . . . , d}.
Proof. The proof is a direct a generalization of Proposition 4.2.3 proof. Let Y µ ∈ R nµ×rµ be the principal components of x, then from Proposition 4.2.2, it writes

Y µ = X (µ) (Y d ⊗ K • • • ⊗ K Y µ+1 ⊗ K Y µ-1 ⊗ K • • • ⊗ K Y 1 )(B (µ) ) (4.17)
with b = (Σ -1 1 , . . . , Σ -1 d )c. Since x is the image of f through the isometry ν, that is x = ν(f ) = (M 1 , . . . , M d )f , the mode µ matricization of X is expressed in function of the SPD matrices M µ as

X (µ) = M µ F (µ) (M d ⊗ K • • • ⊗ K M µ+1 ⊗ K M µ-1 • • • M 1 ). ( 4.18)
Replacing X (µ) in Equation (4.17) by this last equation, it follows that

Y µ = M µ F (µ) (M d Y d ⊗ K • • • ⊗ K M µ+1 Y µ+1 ⊗ K M µ-1 Y µ-1 ⊗ K • • • ⊗ K M 1 Y 1 )(B (µ) )
thanks to the distributive property of the Kronecker product. Remarking that M µ are SPD matrices, i.e., they coincide with their transpose, the previous equation gets

Y µ = M µ F (µ) (M d Y d ⊗ K • • • ⊗ K M µ+1 Y µ+1 ⊗ K M µ-1 Y µ-1 ⊗ K • • • ⊗ K M 1 Y 1 )(B (µ) )
and because of the relation between classical principal components and metric ones, that is Y µ = M µ W µ , we have

M µ W µ = M µ F (µ) (M 2 d W d ⊗ K • • • ⊗ K M 2 µ+1 W µ+1 ⊗ K M 2 µ-1 W µ-1 ⊗ K • • • ⊗ K M 2 1 W 1 )(B (µ) ) .
Since M µ is invertible, from this last equation the thesis follows.

Geometric view for the MultiWay Correspondence Analysis

In this conclusive section, we transport the previous results in the MultiWay Correspondence Analysis context. We firstly clarify the metric of the Euclidean space where we set our problem. Then we make explicit the point cloud relation in this particular framework. As final outcome, we are able to prove the correspondence between the point clouds attached to each mode.

In accordance with the correspondence analysis framework, we consider a d-way contingency table t ∈ N n 1 ו••×n d . The first step for performing CA is scaling t by the sum of all its components setting a new relative frequency tensor

f ∈ R n 1 ו••×n d + f = 1 n 1 ,...,n d i 1 ,...,i d =1 t(i 1 , . . . , i d ) t.
We first clarify the tensor space we will work with. Let f µ be the marginal of mode µ, i.e., the vector whose components are the sums of the slices of mode µ for all µ ∈ {1, . . . , d}.

For example the i 1 -th element of f 1 is

f 1 (i 1 ) = n 2 ,...,n d i 2 ,...,i d =1 f (i 1 , . . . , i d ) for all i 1 ∈ {1, . . . , n 1 }.
There is no loss of generality in assuming that f µ (i µ ) > 0 for every µ ∈ {1, . . . , d}. Indeed the i µ -th component of f µ is equal to zero if and only if all the entries of the i µ slice of f with respect to mode µ are zeros, i.e., f (i 1 , . . . , i µ , . . . , i d ) = 0 for every i k ∈ {1, . . . , n k } and k ∈ {1, . . . , µ -1, µ + 1, . . . , d}. If there exists a mode µ and an index i µ such that its marginal component is zero, that is f µ (i µ ) = 0, then the i µ category of the µ variable has zero frequency, or differently said, this variable case never arrives. Consequently it does not impact the global analysis and it is discarded.

Define D µ = diag( f µ ) ∈ R nµ×nµ for each µ ∈ {1, . . . , d} and assume that f belongs to R n 1 ו••×n d endowed by the metric induced by the matrices (D -1 1 , . . . , D -1 d ), since D -1 µ is SPD for every µ ∈ {1, . . . , d}. We denote by S M this Euclidean space and by S the tensor space R n 1 ו••×n d endowed with the standard inner product. Under this assumption, let ν be the isometry between the spaces S M and S and let x = ν(f ) = (D -1 1 , . . . , D -1 d )f . The general element of tensor x is written as

x(i 1 , . . . , i d ) = f (i 1 , . . . , i d ) f 1 (i 1 ) . . . f d (i d )
.

Performing the HOSVD over tensor x at multi-linear rank r leads to a new orthogonal basis (U µ ) µ=1,...,d , a core tensor c and principal components Y µ = U µ Σ µ for every µ ∈ {1, . . . , d} in the standard tensor space. Focusing on the principal components W µ = D µ Y µ of tensor f in S M , Proposition 4.2.4 entails

W µ = F (1) (M 2 d W d ⊗ K • • • ⊗ K M 2 µ+1 W µ+1 ⊗ K M 2 µ-1 W µ-1 ⊗ K • • • ⊗ K M 2 1 W 1 )(B (µ) )
by the definition of z 2 j and z 3 k for every i ∈ {1, . . . , n 1 }, j ∈ {1, . . . , n 2 }, k ∈ {1, . . . , n 3 } and ∈ {1, . . . , r 1 }. This final equation can be read as a mutual barycenter relation scaled by the inverse of the corresponding singular value. Indeed there is a list of weight terms which sum to 1, i.e.,

n 2 ,n 3 j,k=1 f (i, j, k) f 1 (i) = f 1 (i) f 1 (i) = 1
times a linear combination expressed through b 1 of z 2 j and z 3 k . Moving back to the geometric perspective, the Proposition 4.2.5 states that the scaled coordinates of a point cloud correspond to the barycenter of the other two point cloud scaled coordinates.

The two remaining barycentric relations follow from this proof, permuting coherently the indices.

Correspondence in CA refers to the correspondence of point cloud coordinates through the scaled barycentric relation. We proved that this well known relation in matrix framework is holding also in the tensor one, through HOSVD. Therefore we propose to refer to it as MultiWay Correspondence Analysis from HOSVD.

This final result is extended and verified straightforwardly for d-order tensors.

Proposition 4.2.6. Let f be a tensor in the tensor space S M endowed with the norm induced by the inner product matrices D µ = diag( f µ ) with f µ the µ mode marginal of f for every µ ∈ {1, . . . , d}. Let Z µ ∈ R nµ×rµ be the scaled principal components for tensor f of mode µ in S M . If r µ = rank(F (µ) ) for µ ∈ {1, . . . , d}, then

z µ iµ ( µ ) = 1 σ (µ) µ d η=1 η =µ nη iη=1 rη η =1 f (i 1 , . . . , i d ) f µ (i µ ) z d i d ( d ) • • • z µ+1 i µ+1 ( µ+1 )z µ-1 i µ-1 ( µ-1 ) • • • z 1 i 1 ( 1 )b µ ( 1 , . . . , d )
where b µ = Σ µ × µ b from Proposition 4.2.4.

Proof. From Proposition 4.2.4, under the CA metric choice, the principal components of the first mode of f in S M are expressed as 1) ) .

W 1 = F (1) (Z d ⊗ K • • • ⊗ K Z 2 )(B (
If the previous equation is multiplied by equation by D -2 1 and it has B (1) replaced by the equivalent Σ -1 1 B

(1)

1 , it gets

Z 1 = D -2 1 F (1) (Z d ⊗ K • • • ⊗ K Z 2 )(Σ -1 1 B (1) 1 ) . ( 4

.20)

Making explicit the 1 -th component of z 1 i 1 , the i 1 -th row of Z 1 , from Equation (4.20), it follows the thesis, i.e.,

z 1 i 1 ( 1 ) = 1 σ (1) 1 n 2 ,...,n d i 2 ,...,i d =1 r 2 ,...,r d 2 ,..., d =1 f (i 1 , . . . , i d ) f 1 (i 1 ) z d i d ( d ) • • • z 2 i 2 ( 2 )b 1 ( 1 , . . . , d ).
The proof for the other modes follows directly from this one permuting coherently the indices.

CORRESPONDENCE ANALYSIS

Examples

To validate our theoretical results and to highlight the benefit of a tensor approach, MWCA is compared with the classical CA, applied to the same data reorganized as a matrix. Since the SPD matrices defining the isometry of CA and MWCA are different, there exist a discrepancy between CA and MWCA. We rely on Figure 4.1 and the elements therein to compare the two approaches. Let f ∈ R n 1 ו••×n d + store the data relative frequencies as a tensor, then their matrix representation is given by F (k) the k-matricization mode of f , once mode k has been selected. The MWCA is performed on f , while CA is performed on F (k) . Let X (k) be the k-matricization of ν T (f ) where ν T is the MWCA isometry, as depicted in Figure 4.1B, similarly X k is the outcome of CA isometry ν M applied to F (k) , as in Figure 4.1A. Since the two approaches are different, the SPD matrices defining the isometries ν T and ν M transporting f and F (k) differ and so do X (k) and X k . To estimate this discrepancy between the two decomposed objects we define the relative error e(X (k)

F (k) X k Û Σ V U ΣV ν M PCA ν -1 M CA (A) Scheme for CA of k-th ma- tricization f x ( Û1 , . . . , Ûd )C (U 1 , . . . , U d )C ν T HOSVD ν -1 T MWCA (B) Scheme for MWCA
, X k ) as e(X (k) , X k ) = ||X (k) -X k || ||X (k) || . ( 4.21)
If the value of e(X (k) , X k ) is small, we expect the two approach to lead to similar interpretations. Conversely, if the error value is significantly large, the outcome may suggest a different interpretation for the variable category interactions. The SVD and HOSVD, over which CA and MWCA relay respectively, provide an orthogonal basis for the matrix or the decomposed tensor, which is unique up to an orthogonal rotation. As proposed in [START_REF] Bro | Resolving the sign ambiguity in the singular value decomposition[END_REF] and [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF], we orient these new basis selecting as leading direction the one where the majority of the data points out. To perform CA and MWCA we used python 3.6.9 and the library TensorLy 0.6.0, see [START_REF] Kossaifi | TensorLy: Tensor Learning in Python[END_REF].

The following example is based on [START_REF] Carme-N | Spanish National Health Survey[END_REF]. We select this dataset, which is already analysed with CA in [START_REF] Carme-N | Spanish National Health Survey[END_REF], to show that the multiway method results are coherent with CA ones. Table 4.1 -Data from the Spanish National Health Survey of 1997, see [START_REF] Carme-N | Spanish National Health Survey[END_REF].

Example 4.2.7. In [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF] data are reported from a survey over 6731 people of both genders, from 16 to over 75 years old, who were asked to evaluate their health status. Then the answers were organized in a 3-way table with dimensions n 1 = 2, n 2 = 7 and n 3 = 5, as in Table 4.1. On mode 1 we have the two genders: male, 'M', and female, 'F' and on the second mode there are 7 age groups. On the last mode we set 5 health grades, from 'Very Good' to 'Very bad'. The (1, 1, 1) entry of the multiway table is the number of men between 16 and 24 years old who judge 'Very good' their health status. Let f ∈ R n 1 ×n 2 ×n 3 + represent the data relative frequencies in tensor format for MWCA. CA is realized over A 3 the matricization of f with respect to mode 3, i.e., 'health grade', where on the row we set the health categories and on the columns all the possible combinations of ages and gender. In Figure 4.2A we recover the correspondence analysis of [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF] As in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF], both plots in Figure 4.2 display a gradient for the age categories: they distribute from the youngest in the bottom to the oldest near the top. The 5 health categories are at the extremes of this gradient, the better health levels at the bottom, worst

CORRESPONDENCE ANALYSIS

ones at the top. From this we infer that the health evaluation decreases for increasing ages. This phenomenon is highlighted by both the techniques, even if it is clearer in Figure 4.2B, thanks to the possibility of analysing the variables separately. Taking into account the gender in Figure 4.2B, let us remark that the male dot is close to the 'Good' one. Thanks to the barycentric relation, we know that the closer the points the more correlated they are. So men appear to provide optimistic evaluations of their health status. Women dot stands aside from the other, slightly closer to worst health status, suggesting that female health judging is balanced between the age classes, with a minor inclination toward pessimistic evaluation. In Figure 4.2A for increasing ages, male and female dots tend to increase their distances, suggesting that when men and women age, they tend respectively to be more optimistic and pessimistic in evaluating their health status. MWCA plot seems to point out that men are more optimistic than how much pessimistic women are. In contrast from Figure 4.2B we cannot conclude easily which age category presents the biggest difference in health evaluation for the two genders, information that can be inferred from 4.2A. Lastly comparing the health point cloud in Figures 4.2A and 4.2B, we observe that their principal coordinates are almost the same and we explain it in terms of relative error. Indeed for this data-set the relative error defined in Equation (4.21) is e(X (3)

, X 3 ) = ||X (3) -X 3 || ||X (3) || ≈ 0.035
is quite small.

To further investigate the tensor approach in comparison with the matrix one, we analyse the dataset available on [START_REF] Greenacre | Opinions about working women[END_REF]. In [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF], these data are fully studied and described in their interaction through a matrix approach.

Example 4.2.8. In [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF] a case study is given by the data coming from the Internation Social Survey Programme of 1994. The authors focus on the survey question if a woman with a schoolchild should work full-time, part-time or stay at home. The participants were also given the possibility of expressing a doubtful position. In total 33590 people of both genders took part in the survey, from 24 countries, among which East and West Germany, Great Britain and Northern Ireland were considered respectively separated. The results relative to the working conditions of a schoolchild mother were organized in a 3-way contingency table [START_REF] Greenacre | Opinions about working women[END_REF]. On mode 1 we set the n 1 = 24 countries ('AUS' for Australia, 'DW ' for West Germany, 'DE' for East Germany, 'GB' for Great Britain, 'NIRL' for Northern Ireland, 'USA' for United States of America, 'A' for Austria, 'H ' for Hungary, 'I ' for Italy, 'IRL' for Ireland, 'NL' for Netherlands, 'N ' for Norway, 'S' for Sweden, 'CZ ' for Czechoslovakia, 'SLO' for Slovenia, 'PL' for Poland, 'BG' for Bulgaria, 'Rus' for Russia, 'NZ ' for New Zealand, 'CA' for Canada, 'RP' for Philippines, 'IL' for Israel, 'J ' for Japan and 'E' for Spain). On the second mode, there are the four possible answers to the specific question about the working condition, i.e., full time working denoted by 'W', part-time working by 'w', stay at home by 'H' and unsure by question mark, that is n 2 = 4. The last mode has size n 3 = 2 for the two genders: male, 'M', and female, 'F'. As consequence, the (1, 1, 1)-th entry of the multiway table is the number of Australian men who judge that a woman with a schoolchild should work full-time. As in the previous example, we assume that f ∈ R n 1 ×n 2 ×n 3 + represents the data relative frequencies in tensor format for MWCA, while we perform CA over F (2) the matricization of f with respect to the working condition mode, i.e., the second one. Differently said, the columns of F (2) corresponds to men and women from different country, while the row to the four possible survey answers. The result of CA is displayed in Figure 4.3A, matching the outcome of the correspondence analysis performed in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. Figure 4.2B shows multiway correspondence analysis with the first two columns of Y µ = U µ Σ µ , defined in Section 4.2.2.3 for µ ∈ {1, 2, 3}.

Comparing Figure 4.3A and 4.3B, it is clearly evident that the four survey answers about the working conditions have the same layout. The staying at home 'H' direction is almost orthogonal to the working one, outlined by part-time 'w' and full-time 'W' work. On the intersection among these two directions there is the question mark for the unsure answer. This strong similarity may be explained by the relative error among mode 2 MCA and MWCA. Indeed for this data-set the relative error defined in Equation (4.21) is e(X (2)

, X 2 ) = ||X (2) -X 2 || ||X (2) || ≈ 0.04
which is quite small. This working condition pattern is similar, but not identical to the one found in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. Indeed in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF], staying at home and working, in both the full and part time possibilities, form orthogonal direction, but in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF] the unsure reply dot is in the middle between 'w' and 'W'. We may believe that this difference is due to the pretreatment (particular scaling and centring) considered in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF] and not in our analysis. From the

APPLICATION: THE MALABAR DATASET

viewpoint of the countries, in both the Figure 4.2 plots a bottom-up gradient emerges for the countries. On the bottom, with cold colours for Figure 4.3A, we find liberal nations as Sweden, West Germany, Netherlands, New Zealand or Great Britain, while on the top there are more conservative states as Philippines, Poland or West Germany, according to the social structure of 1994. This gradient appears clearly in the analysis of [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF]. As in Example 4.2.7, the separation among of country and gender in Figure 4.3B eases the perception and interpretation of the liberal to conservative bottom-up gradient. Moreover in Figure 4.3A, it is not as immediately clear as in Figure 4.3B the global influence of the gender in answering to the considered survey question. Indeed in the MWCA output, the women dot is clearly aligned along the working line, while men dot is set in the midway between the staying at home and working direction. As previously stated, the barycentric relation implies that the nearer the dots the more correlated they are likely to be. As consequence, since the women dot is closer to the part-time 'w' dot, we may infer that on a global scale women in 1994 believed on average that a schoolchild mother should also have a job, with a preference for a part-time one. On the other side, the midway position of the men dot may suggest that male opinions uniformly distribute among the two extrema possible answers, working or staying at home. This potential conclusion, stated in [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF], is not as conveniently achieved from MCA output in Figure 4.3A. On the other side, if the analysis is focused on the gender effect per country, i.e., how much different are on average men and women answers to the survey, the results are visible only in Figure 4.3A. For example, checking the pink square and star in Figure 4.3A, we may conclude on average Philippines people reply similarly to the survey neglecting the gender. On the contrary, Russian men, displayed by the light orange square, seem to favour on average the stay at home answer, if they are compared with the Russian women, i.e., the light orange star in Figure 4.3A. Exactly as in Example 4.2.7, the choice of the matrix or tensor approach for the multiway table analysis strongly depends on the interpretation aims. If the final purpose is highlighting specific differences among variable category combinations, then the matrix approach should be privileged. However if more global variable category interactions are searched, then the tensor approach is probably the most convenient, thanks to the complete separation of variable categories and the consequent displaying clarity.

Application: the Malabar dataset

This second main section is devoted to the original analysis of the Malabar dataset [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF] a contingency multiway table. We briefly describe the data trying to get hints of a possible behaviour.

Since the classical tensor data analysis literature [START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF] suggests to perform MCA or MWCA on categorical variables data, this dataset represents the ideal opportunity for applying the theory developed in Section 4.2. To highlight the advantages of the tensor approach, we interpret the dataset through MWCA and MCA, constructing and analysing the point clouds as in sections 4.3.3 and 4.2.1, as it was done for Example 4.2.7.

Data description

The data-set provided by the Malabar project (IFREMER, CNRS, INRAE, Labex COTE) [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF] is a multiway contingency table of order 4. In this metabarcoding project, 32 water samples have been collected in Arcachon Bay at four locations (Bouee13, Comprian, Jacquets, Teychan), during four seasons, and at two positions in water column (pelagic and benthic). DNA has been extracted, and Operational Taxonomic Units (OTUs) have been built. OTU are expected to correspond to species and, without entering too much into details here, the question which motivates Malabar project is to understand or quantify the role of locations, seasons and water column positions in the diversity of protists. To address this question, a four way contingency table t has been built, where t(i, j, k, ) is the number of sequences collected at location j, in season and position k which belong to OTU i. The size of the contingency tensor is n 1 = 3539, n 2 = 4 n 3 = 2 and n 4 = 4. Henceforth we assume to work with the relative frequency data, i.e., with f = (1/t)t where t = n 1 ,n 2 ,n 3 ,n 4 i,j,k, =1 t(i, j, k, ).

Average comparison and data preprocessing

Before proceeding with the analysis of the dataset with tensor techniques, we investigate and eventually preprocess them. Since the focus of the data exploration is on the OTUs, we compute the average number of OTU sequences per location and per season, summing over the first OTU mode and third water column one. The average matrix is

Ξ ∈ R n 2 ×n 4 such that Ξ(j, ) = 1 n 1 n 3 n 1 ,n 3 i,k=1 f (i, j, k, )
for every j ∈ {1, . . . , n 2 } and k ∈ {1, . . . , n 4 }. season for the four different locations. We observe that Jacquets and Teychan present the same average behaviour, with a significant increase in the average number of OTU between summer and autumn and a huge decrease from autumn to winter. In Comprian the average number of OTU grows between spring and summer and similarly to the previous two locations, it decreases significantly from autumn to winter. Finally, in Bouee13 the average number of OTU shows much lighter average variation, in a way similar to Jacquets and Teychan. Since the water column position variable has only two categories, 'pelagic', i.e., close to the ocean surface, and 'benthic', i.e., close to the seabed, we study the average number of OTU in function of location, water column position and season, defining, ξ ∈ R n 2 ×n 3 ×n 4 as

ξ(j, k, ) = 1 n 1 n 1 i=1 f (i, j, k, ).
The variation of the average number of OTU in function of the season is displayed in Figure 4.4B for each location and for both the water column positions. Remarkably, the averages for the 'pelagic' and 'benthic' position are clearly clustered and present slightly opposite tendencies. Indeed for all the locations, in the 'benthic' position the average number of OTUs seems to stay almost constant throughout the year, with a perceptible increase only for Bouee13 from spring to summer and a minor decrease for all the locations from summer to autumn. On the 'pelagic' side, the average number of OTU decreases significantly from spring to summer and increases from summer to winter for all the location except Bouee13. This last location presents even in the 'pelagic' case lower fluctuations in the average number of OTU during the season cycle. Before drawing a conclusion, it is worthwhile computing the the standard deviation of the OTU defined as λ ∈ R n 2 ×n 3 ×n 4 such that

λ(j, k, ) = 1 n 1 n 1 i=1 f (i, j, k, ) -ξ(j, k, ) 2
.

The mean Ξ P = ξ P (:, 1, :) and the standard deviation Λ P = λ P (:, 1, :) for the pelagic category are of order 10 -5 and 10 -4 respectively. On the other side, the orders of the benthic mean, Ξ B = ξ(:, 2, :), and of the benthic standard deviation, Λ B = λ B (:, 2, :), are respectively 10 -6 and 10 -5 . Since the standard deviation is of one order greater than the mean for both the pelagic and benthic subdataset, an independent study of the pelagic and benthic subtensor of order 3 is not motivated.

MultiWay Correspondence Analysis

The Malabar dataset is a multiway contingency table and thus according to the literature [START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF] the most suitable interpretation technique is MultiWay Correspondence Analysis or Multiple Correspondence Analysis, as previously mention in Section 4.3. On the basis of the result of Section 4.2, we perform MWCA on the Malabar dataset, constructing the point clouds associated with each mode of the multiway table and visualizing the first two columns of each mode principal components together in Figure 4.5 to obtain a more reliable interpretation. As for Examples 4.2.7 and 4.2.8, we perform also the canonical CA on the matricization with respect to mode 1 of the Malabar dataset, to emphasize which aspects each of the techniques best grasps.

In Figure 4.5A the result of CA performed over A 1 , the matricization of frequency tensor data f with respect to mode 1, where the rows represent the different OTUs and the columns the locations, water column positions and seasons combined. Next to it, Figure 4.5B displays the result of MWCA over the tensor data. In the CA case OTU point cloud spreads over two orthogonal directions, led by the two positions in the water column. Similarly in MWCA plot the water column point cloud seem to organize and to affect the position and clustering of the others. Because of mode combination in Figure 4.5A, it is not evident which position in the water column affects which location or season. However thanks to the barycentric relation and the multiway approach, we can infer some relations among seasons, locations and water column positions from Figure 4.5B, where they are independently displayed. Indeed pelagic point appears to affect the OTU present during spring and winter time. Similarly benthic position probably drives the distribution of OTU in autumn and summer. A similar argument can be repeated for location point cloud. Indeed the barycentric relation suggests that the benthic position leads the distribution of OTU in Comprian, Jacquets and Teychan, while the pelagic dot influences Bouee13 dot. The barycentric relation enables us to interpret the interaction among season and location point clouds. In Figure 4.5B winter and spring seem to be more correlated with Bouee13 and similarly summer and autumn are probably more correlated with the remaining three locations. Analysing the OTU and the season point clouds, we observe that winter and spring pull out one orthogonal direction over which part of the OTU points spread. Similarly summer pulls out the other OTU orthogonal direction. The autumn point appears to be neutral with respect to the OTU point cloud distribution. Similarly if we study the OTU and location point cloud interaction, one orthogonal direction is driven by Bouee13, one by Comprian and Jaquets, while Teychan seems to have a small effect on the OTU point distribution.

In Figures 4.5C and 4.5D, we filter the OTU by size, keeping just those with first principal components greater than 0.2 and the second greater than 0.12. Figure 4.5C confirms the idea of water-column position driving the OTU point cloud distribution. However in Figure 4.5B most of the OTU points are close to the origin. Indeed in this left plot we filtered 8 OTU which are probably more correlated with benthic or pelagic condition. Comparing the two filtered figures, we see that only 3 of the 8 selected OTU are common between them, i.e., OTU '22', '24' and '29'. These common OTU share the same correlations between the two methods. Indeed in both case, '22' is more related to benthic position, while '24' and '29' with pelagic one. Moreover most of the OTU in Figure 4.5D are concentrated around the origin. We may infer that the action of the water column point cloud is softened by the other modes, here independent, leading to a more balanced distribution of OTU. In conclusion notice that in MWCA the different point clouds are more spread than in the CA case, enabling a deeper analysis of the variables interactions. It could be linked to the significant relative error from Equation (4.21) e(X (1) , X 1 ) = ||X (1) -X 1 || ||X (1) || ≈ 0.14, which may also explain why CA and MWCA figures present a flipped y-axis, even though the SVD and HOSVD basis have been oriented according to the same criterion. Indeed the mode combinations including benthic have positive y-components in Figure 4.5A, while benthic point has negative y-component in Figure 4.5B. However since it affects in the same way all the point clouds, the two methods lead to a coherent interpretation.

Concluding remarks

We devoted this chapter to the study of multiway contingency tables through tensor techniques. The focus is set on the MultiWay Correspondance Analysis (MWCA), that associates multiway contingency tables with point clouds. After describing the point clouds construction process, we proved the algebraic relations linking them and we proposed a geometric interpretation. To highlight the benefits and the potential drawbacks of a tensor approach compared to a classical matrix one, we presented two examples. Then we applied this technique together with other classical ones for analysing and interpreting an original multiway contingency table.

The first part, i.e., Sections 4.2.1 and 4.2.2, collects the theoretical content of the entire chapter. We started describing the classical theory of Correspondence Analysis (CA) in the usual matrix context. We showed how points clouds are attached to a contingency table and how the geometrical spaces, where these point clouds live, are defined in CA. As conclusive point, we stated in Equation (4.1) the known barycentric relation linking point cloud associated with rows and columns of a contingency table. Next we moved to the tensor framework. We recalled the classical way to associate a point clouds with each mode of a multiway contingency table and to construct their principal components. Our contribution was investigating the link of point cloud principal components. First we proved the existence of this relation in the real order-d tensor space endowed with the classical Euclidean metric. Then we extended this relation to the same space endowed with a whatever metric, induced by a set of d SPD matrices. Once set these preliminary results, we derived the barycentric relation in the MWCA case, introducing the MWCA metric. We have taken care to supplement each algebraic result with its geometric interpretation. In conclusion to Section 4.2 we presented two examples, both from [START_REF] Greenacre | Correspondence Analysis in Practice. Data-sets are available at[END_REF], with the aim of confirming the consistency of MWCA with CA and of highlighting the main advantages of a tensor technique over a matrix one.

The second part focuses entirely on the original analysis of the Malabar dataset [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF], a 4-way contingency table. We introduced initially the dataset, remarking that each of its element represents the number of DNA sequences extracted from a sample of water collected in a specific location inside the Arcachon Bay during a certain season and at a certain level in the water column. We proceeded with a preliminary data analysis,

CONCLUDING REMARKS

investigating the variation over the season of the average number of microorganism DNA sequences per location, observing that each location seems to have a preferred seasonal pattern. Next we studied the seasonal average change in the number of microorganisms per location and per position in the water column, noticing that a clear distinction exists among seabed and surface microorganism numbers, for all the considered locations. This step already suggested that the position in the water column could play a key role in the analysis. The conclusive part of this second section is dedicated to the analysis of Malabar dataset with the MWCA. Especially we compare the MWCA interpretation with the mode one MCA of the tensor data. We remark here that MWCA simplifies significantly the data interpretation, justifying it as the preferred tool for this type of dataset.

Chapter 5

Tensor techniques and climate data

Introduction

Climate data are measurements of complex phenomena involving a huge number of interactions of many different variables [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF][START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF], among which two crucial ones are time and space. The Empirical Orthogonal Function (EOF) analysis is a widely used tool originally meant to separate and visualize time and space-related information from a climate dataset. In practice, the EOF analysis is a variation of the Principal Components Analysis, see II.III, suitably tuned for climate data. Thus, as PCA, the EOF analysis is also used as an exploratory technique to investigate the pattern captured in the data and as a dimensional reduction method. The EOF analysis is applied also to forecast, to compare models and observations [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF].

The classical EOF analysis aims to capture the maximum variance of the data, represented by a matrix, expressing it as the product of two matrices, an orthogonal one, whose columns are said Empirical Orthogonal Functions (EOF) and an un-correlated one, whose columns are referred to as Principal Components (PC). The EOF matrix collects the space information, while the PC matrix the time one. The basic version of the EOF analysis has been improved and modified over the years, for example relaxing the constraint of the orthogonality; we refer the reader to [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF][START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF] for variations of the EOF analysis and other climate data exploration methods.

Climate data can also be mathematically modelled by tensors, as presented in [START_REF] Zhang | A Parallel Nonnegative Tensor Factorization Algorithm for Mining Global Climate Data[END_REF] for example. Thus, also the two variants of the EOF analysis can be readily formulated in the tensor framework, but without a significant benefit from the computational viewpoint. However, if the climate data are high dimensional and made available in Tucker format 1.3.1, the tensor approach becomes more interesting. Under this strong assumption, we investigate how to retrieve the EOF analysis results from Tucker format data and when it could be beneficial. The results we present constitute a preliminary study of the EOF analysis.

The remainder of this chapter is organized as follows. Section 5.2 presents two variants of the classical EOF analysis: one based on the covariance matrix and one on the correlation matrix, both computed from matrix climate data. We underline the computational

EMPIRICAL ORTHOGONAL FUNCTION ANALYSIS

possible simplifications for both these two mathematical procedures. In Section 5.3, we represent the climate data through the Tucker model and we show how to perform the EOF analysis benefiting from this compressed format. This tensor approach for EOF analysis is applied and compared with the classical matrix one in Section 5.4. More in detail, we briefly study the first EOF and PC obtained from the classical matrix analysis of a real climate dataset, namely HadCRUT4_RecAI_tas_mon_188001-201512.nc. This dataset is approximated also at different accuracies in Tucker format. Following the approach described in Section 5.3, we compute the EOF and the PC matrices, studying the relative error in function of the different accuracies. For completeness, we display and investigate also the first column of the Tucker decomposition of the dataset, relating it to the classical EOF results.

Empirical Orthogonal Function Analysis

This section introduces the classical EOF analysis. As already mentioned, the EOF analysis aims to separate the information carried in the data due to the time variables from those linked to the space variables. After describing in Section 5.2.1 the data structure and the pretreatments applied, we present two mathematical processes considered to compute the PC and the EOF matrices. For completeness, the computational aspects of the EOF analysis are briefly described.

To highlight the nature of the different mathematical objects, we add a subscript c to denote the centred data, a subscript s for the scaled data, and a subscript sc for the scaled and centred data to the bold Latin letter indicating the tensor storing the data. As it is conventional, when an array stores a sample average, the letter denoting it is overlined.

Data description and pretreatments

In this section, we present the general structure of data we will analyse henceforth.

Let S ⊂ R 3 denote the Earth surface in this chapter. The function f : R + × S → R associates with each time value and each point of the Earth's surface the temperature or the pressure measured in that time-space point. In the climatology framework, the function f is usually referred to as space-time field. As the next step, the field is discretized into a three-dimensional array. To achieve this aim, we define a grid of size p 1 × p 2 (i.e., p = p 1 p 2 points), over the Earth surface denoting the discrete latitude variable by θ j for j ∈ {1, . . . , p 1 } and the discrete longitude by φ k for k ∈ {1, . . . , p 2 }. Let t i be the discretetime variable for i ∈ {1, . . . , n}, then {(t i , θ j , φ k )} (n,p 1 ,p 2 ) (i,j,k)=(1,1,1) is the time-space grid used to discretize the function f . Indeed, the discretization of f over the time-space chosen grid is the tensor f ∈ R n×p 1 ×p 2 of order 3 such that f (i, j, k) = f (t i , θ j , φ k ) (5.1) for all i ∈ {1, . . . , n}, j ∈ {1, . . . , p 1 } and k ∈ {1, . . . , p 2 }.

Henceforth, we assume data to be purged by the seasonal (or external) cyclic patterns. Let X ∈ R n×p with p = p 1 p 2 denote the tensor f matricized with respect to mode one, i.e., X = F (1) . This choice couples together the two space variables. Before proceeding with the classical EOFs analysis, the data are preprocessed, removing from X the average in time. The time average x ∈ R p is

x(h) = 1 n n i=1

X(i, h)

for every h ∈ {1, . . . , p}.

We compute X c ∈ R n×p centring each column of X by its average, i.e.,

X c = X -1 n ⊗ x (5.2)
where 1 n denotes the vector of ones in R n . In details each element of X c is of the form X c (i, h) = X(i, h) -x(h) for every i ∈ {1, . . . , n} and h ∈ {1, . . . , p}. In climatology matrix X c is called the anomaly field.

EOFs and PCs computation

This section describes how to compute the Empirical Orthogonal Functions (EOFs) and the Principal Components (PCs) to perform the EOFs analysis [START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF][START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF].

As previously mentioned, the purpose of the EOFs analysis is decomposing the anomaly field, dividing the time and the space contributions, i.e., expressing the factorization of X c as X c (t i , s k ) = k=1 v k (s j )c k (t i ) for every discrete time t i and coupled space s j point. The EOFs analysis states the problem as finding a linear combination of coupled grid points that maximizes the variance, see Definition II.II.ii. This linear combination will prove how to split time and space discrete variables. Mathematically, we seek for a vector a * ∈ R p such that X c a * has the maximal variance, i.e. U (i, h) and using it to define the EOF matrix, that is A c = 1 √ n (X c C c ). In all the following sections, we assume to compute the EOFs and PCs following this approach frequently referred to as correlation EOFs analysis, centring in time both the EOF and the PC matrix.

EOF Analysis with Tucker model

In this section assuming that the input data are given in the Tucker compressed format, we derive the EOFs and PCs relying on this formalism. For ease of reading, the core tensor is denoted by the same letter of the decomposed (or approximated) tensor with a prime as superscript.

Let the tensor f ∈ R n×p 1 ×p 2 defined in Equation (5.1) be expressed in the Tucker compressed format, see Definition 1. In a more compact format, the average in time writes

F s = r 2 ,r 3 β,γ=1 F s (β, γ)v β ⊗ z γ with F s (β, γ) = 1 n n,r 1 i,α=1
f (α, β, γ)u α (i).

The scaled data centred in time are expressed by f sc ∈ R n×p 1 ×p 2 , given by f sc (α, β, γ) z γ ⊗ K v β , (5.9) thanks to Lemma 1.2.3. Thanks to Lemma 1.2.4, this previous equation writes

f sc = f s -
X sc = U Υ(Z ⊗ K V ) (5.10)
where Υ ∈ R r 1 ×r 2 r 3 is the matricization with respect to mode 1 of the core tensor f sc , while U ∈ O(n × r 1 ), V ∈ R p 1 ×r 2 and Z ∈ O(p 2 × r 3 ) have the α-th, β-th and γ-th column equal to u α , v β and z γ respectively. In Section 5.2.3, the PCs are computed through the SVD of X sc , but it is not correct to identify directly Equation (5.6) and (5.10). Indeed, the right factor of Equation (5.10) is not orthogonal because of the linear transformation applied over the second mode, see Remark 5.3.1. Consequently the orthogonal matrix U in Equation (5.10) is not the left orthogonal factor from the SVD of X sc , i.e., it is different from the matrix U in Equation (5.6) by construction. However, we can still Let now compute the SVD of the (r 1 × r) matrix (ΥR ), as (ΥR ) = Û Σ V with Û ∈ O(r 1 × q) V ∈ O(q × r) and Σ a (q × q) diagonal matrix with q equal to the rank of (ΥR ). If the matrix product (ΥR ) is replaced by its SVD in Equation (5.11), it becomes

X sc = U (ΥR Q = (U Û ) Σ(Q V ) .
Thanks to this last equation, we can compute the PC matrix from (U Û ), which is equal up to a rotation, to the orthogonal matrix U of Equation (5.6). Thus, let the centred in time PC matrix be C c = ( √ n -1)U c where

U c = (U Û ) -1 n ⊗ u with u(h) = 1 n n i=1 (U Û )(i, h).
(5.12)

As described in Section 5.2.3, the EOF matrix is computed from the product of the PC matrix and X c , the matricization with respect to mode 1 of f centred in time. As explained in Remark 5.2.2, instead of computing the EOF matrix, we can construct the EOF tensor from the input data centred on time and the PC matrix. We start computing F ∈ R p 1 ×p 2 the average in time of the input data as f (α, β, γ)u α (i).

Then the tensor data centred in time f c ∈ R n×p 1 ×p 2 is such that

f c = f -
a c = 1 n (C c × 1 f c ) = √ n -1 n (U c × 1 f c ) = √ n -1 n r 1 ,r 2 ,r 3 α,β,γ=1
f c (α, β, γ)q α ⊗ w β ⊗ z γ , (5.13) where q α ∈ R r 1 is equal to U c u α for every α ∈ {1, . . . , r 1 }.

From the computational point of view, assuming the data to be already in Tucker format, this approach requests two main operation: the QR-factorization of a (p × r) matrix and the SVD of a (r 1 × r) matrix with r = r 2 r 3 and p = p 1 p 2 . On the other side, the classical matrix approach performs only one SVD over a (n 1 ×p) matrix. Consequently, if the multilinear rank components are considerably small, the tensor technique may be of interest.

Numerical results

The purpose of this section is to investigate through the Tucker model a temperature time series. The dataset HadCRUT4_RecAI_tas_mon_188001-201512.nc, provided by L. Terray, research scientist in Climate, Environment, Coupling, and Uncertainties (CECI), a joint research team between Cerfacs and CNRS, contains a month temperature value in Kelvin for each point of the Earth's surface grid and 136 years. We mathematically represent the data as a tensor f ∈ R n×p 1 ×p 2 where n = 1632 is the number of time points, that is 12 months for 136 years, p 1 = 36 the number of latitude discrete points and p 2 = 72 of longitude ones.

We first present the first EOF and PC, which are by construction the most significant ones, obtained from the matrix correlation EOF analysis, see Section 5.2.3. Then a tensor approach is considered. To set a baseline, we study the Tucker basis of f c the data after centring in time. In the conclusive section, we approximate f c through the Tucker model prescribing an accuracy and we study the errors in computing the PC and EOF.

EOF analysis

To perform the correlation EOF analysis, as described in 5.2.3, we rely on the python3 library eofs applied to F (1) the matricization with respect to the first mode of the tensor f storing the raw temperature time series. By construction, the first PC displayed in Figure 5.1A and the first EOF in Figure 5.1B are the most significant ones. The first PC in Figure 5.1A shows a time trend over the years for the temperature. Neglecting the many oscillations, we observe that the curve general behaviour is growing up to 1950, has a

NUMERICAL RESULTS

drop-down point between the 60s and the 80s, and a final steep growth from the 90s. This general pattern reflects the atmospheric concentration of carbon dioxide, cf. [START_REF] Stavins | The Problem of the Commons: Still Unsettled after 100 Years[END_REF]. The EOF plot 5.1B highlights the north hemisphere and the south pole as the ones with the higher covariance values. This result suggests that the temperature rise is more significant in some areas, the northern hemisphere, and the Antarctic pole, while other areas, such as the oceans, are lower affected. Those results are very classical in climatology, see for example [START_REF] Team | Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment[END_REF].

Baseline Tucker model

As a first step in the analysis of the temperature time series data through tensor methods, we decompose it by the HOSVD algorithm and study the properties of the Tucker basis so computed. Remark 5.4.1. During the computational phase, we observe that the relative error among the raw data f and its HOSVD decomposition is of the order 10 -7 , which is significantly larger than the machine precision of the working arithmetic. However, if we centre the data in time, defining f c , then the relative error among f c and its HOSVD decomposition drops down to 10 -15 , as expected. A clear explanation for this phenomenon is not present, but it may be linked to the high-rank nature of the time series. We choose to perform all the analysis on the Tucker decomposition by the HOSVD algorithm applied to f c . In Figure 5.2, we display the first column of the time, latitude and longitude basis. The profile of the first column of the time basis in Figure 5.2A presents a shape similar to the one of the 1st PC displayed in Figure 5.1A, even if on a different scale. The first column of the latitude basis in Figure 5.2B presents two peaks, a tiny one between the 90 • and 60 • south parallel and a major one between the 50 • and 90 • north parallel. The 90 • and 60 • south parallel isolate the South pole area, while the 50 • and 90 • the north hemisphere. On the other side, the longitude basis first column in Figure 5.2C has the main crest between the Greenwich meridian and the 150th one east, a minor one between the 90th meridian and the 150th one west. The area underlying the main crest goes from the east of Europe and Africa to the east of Asia and Australia, while the area included between the 90th and the 150th meridian west covers North and South America. Crossing the information coming from the latitude and longitude plots, the areas carrying the most significant information are the northern hemisphere and the Antarctic pole, as the first EOF shows in 5.1B.

Tucker model and EOF analysis

In this section, we investigate the error in constructing the PC and EOF from a Tucker approximation of f c the data centred in time given an approximation accuracy.

Given f c the data centred in time and the accuracy δ, we compute its Tucker approximation such that

f c -(U, W, Z)f c f c ≤ δ,
with f c ∈ R r 1 ×r 2 ×r 3 the core tensor, U ∈ O(n × r 1 ), W ∈ O(p 1 × r 2 ) and Z ∈ O(p 2 × r 3 ). Notice that the multilinear rank (r 1 , r 2 , r 3 ) depends on the prescribed accuracy. The

NUMERICAL RESULTS

accuracy values we consider belong to {0.5, 0.1, 10 -2 , 10 -3 , 10 -8 }. To obtain the PC and the EOF from the Tucker approximation, we follow the approach described in Section 5.3. The (n × r 1 ) PC matrix centred in time is defined in Equation (5.12) from the QR of (Z ⊗ K DV ) with D the scaling diagonal matrix and the SVD of (ΥR ) with Υ the matricization of the core tensor f c with respect to mode 1. To highlight that in this case an approximation of the data at accuracy δ is used instead of the decomposition, we denote the PC matrix centred in time by C δc ; a similar notation is used for the EOF tensor. The EOF tensor a δc ∈ R r 1 ×p 1 ×p 2 is computed from the PC matrix and the approximation at accuracy δ of f c . The PC and EOF so computed are compared with those obtained from the classical matrix analysis. In particular we estimate the relative error for each PC as e δ ∈ R r 1 defined as e δ (h) = C c (:, h) -C δc (:, h) C c (:, h) while for the EOF tensor the relative error is g δ ∈ R r 1 such that g δ (h) = a c (h, :, :)a δc (h, :, :) a c (h, :, :)

for every h ∈ {1, . . . , r 1 }. Before the analysis of the PC and EOF relative errors, we focus on the multilinear rank of the approximation for a given accuracy δ, reported in Table 5.1. Notice indeed that already for δ = 10 -2 , the second and third mode are full rank, since r 2 = p 1 and r 3 = p 2 . For δ = 10 -3 , even the first mode is almost full rank, with r 1 = n -1, while for δ = 10 -8 we obtain the data decomposition in Tucker format. From the point of view of the relative approximation error, when δ = 0.5, we are collecting approximately 60% of the total information, for δ = 0.1 we keep 90% of it. Remark that for δ ∈ {10 -2 , 10 -3 , 10 -8 } the relative error is at least one order less than δ.

Those peculiarities seem to be related to the high-rank nature of the time series data. For completeness, we include the compression ratio defined as the number of memory units needed to store the tensor in Tucker format over the memory units requested for storing the tensor in full format, i.e., where r h and n h are the h-th component of the multilinear rank and the mode size respectively. Checking the third row of Table 5.1, we conclude that a beneficial compression arrives only when δ is equal to 0.5 with a request of only 9% of the memory used to store the entire tensor. For all the other accuracy values, the memory request to store the tensor in the Tucker format is more than the amount necessary to store the data in full format.

technique are briefly introduced in Section 5.2. Section 5.3 focuses on the construction of the EOF and PC matrix from data mathematically expressed in Tucker format. Under this assumption, we show that the correlation EOF analysis can be performed keeping the data in compressed Tucker format, using the Tucker factors and core. Thus, if the multilinear rank of the considered data is low, the tensor approach appears beneficial from the computational viewpoint. Section 5.4 constitutes the second part of the chapter. It is devoted to the numerical experiments performed on a single dataset through python3 and the eof library. As the first step, we plot the first PC and EOF computed by the eof library; they are used as comparison terms for the other experiments. Then, we decompose the dataset by the HOSVD algorithm and analyse the components of the first column of each orthogonal basis. We illustrate the similarities among the first column of the time orthogonal basis and the first PC and the relations among the first latitude and longitude orthogonal basis column and the first EOF. In the last Section 5.4.3, the data are approximated by the HOSVD at a prescribed accuracy δ, then the core tensor and the orthogonal basis are used to compute an approximation of the PC and EOF matrix. As a final step, we compare the approximations of the EOF and PC matrices with those computed by the eof library. We display and investigate the relative errors for each PC and EOF depending on the approximation accuracy δ. Our study shows that the first and more important EOF and PC are better approximated than the last ones.

Conclusions and prospectives

The current fast technological development brings new mathematical and computational challenges while opening new research directions. A clear example of this is represented by linear algebra, which in the years has turned into multilinear algebra. Indeed, different scientific and social disciplines, that used to rely on linear algebra techniques, present nowadays problems that are naturally addressed in multilinear algebra, moving from matrix approaches to tensor ones.

The purpose of this work is to investigate the properties of some classical linear algebra methods, coming from numerical linear algebra and data analysis, once they are extended to the tensor framework through the classical tensor decomposition algorithms, recalled in Chapter 1. In the following sections, we summarize the main results and contributions, provided by our work, together with the main open questions and research directions that it has opened.

Numerical linear algebra

The topic of Part I is numerical linear algebra methods generalized to tensors though the Tensor-Train format [START_REF] Oseledets | Tensor-Train Decomposition[END_REF], presented in Section 1.3.2.

More in detail, Chapter 2 focuses on the Generalised Minimal RESidual (GMRES) iterative solver from the backward stability viewpoint. Firstly we describe the variable accuracy approach, which consists in decoupling the computational and storage accuracies. This choice describes mathematically those situations where memory constraints make working with the classical IEEE floating point format not affordable. Once the representation hypotheses are fixed, we study experimentally the norm-wise backward error of GMRES, introducing storage perturbations firstly controlled component-wise and secondly norm-wise. The TT realization of GMRES in TT-format (TT-GMRES) represents a study case for the variable accuracy approach. So, we report on the backward error of TT-GMRES for solving several classical tensor linear systems. In particular, we address theoretically and numerically the problem of solving many tensor linear systems at once increasing the order of the space.

In Chapter 3, we describe and investigate experimentally six orthogonalization schemes, namely Classical Gram-Schmidt (CGS), Modified Gram-Schmidt (MGS) [START_REF] Gram | Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate[END_REF][START_REF] Schmidt | Zur Theorie der linearen und nichtlinearen Integralgleichungen[END_REF], their version with re-orthogonalization (CGS2, MGS2) [START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF], the Gram procedure from [START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF] and the Householder transformation [START_REF] Householder | Unitary Triangularization of a Nonsymmetric Matrix[END_REF], implemented with the TT-format. Their TT-version data analysis use of Canonical Polyadic decomposition [START_REF] Hitchcock | The Expression of a Tensor or a Polyadic as a Sum of Products[END_REF] (CP) instead of the Tucker one. For the contingency table, for example, we could study if a similar barycentric relation may be inferred using the CP decomposition, or combining it with the Tucker model, to benefit also with the usual low dimension of the core tensor. On the other side, for the EOF analysis, the orthogonality of the factors does not represent a crucial constraint. However, climate data are known for being high-rank data, so tensor decomposition techniques beneficial when the low-rank property is present may not be a suitable tool. A much more general and open question is the search for new visualization techniques relying on tensor formats, but it may request knowledge and to be co-designed with other disciplines.

Figure 1 . 1 - 1 1. 1 .

 1111 Figure 1.1 -Tensors as multidimensional tensors of order d ∈ {1, 2, 3}.

4 :

 4 Ûd ), c = HOSVD(a) initialize the core tensor and the factors by the HOSVD 3: while error > δ do for i = 1, . . . , d do compute the core tensor without the i-th factor 5: c = ( Û 1 , . . . , Û i-1 , I n i , Û i+1 , . . . , Û d )a compute the truncated SVD of the core tensor without i-th factor 6:

16

 16

√ d-1 a scale the accuracy δ 3 : 6 :

 36 r 0 = 1 and p = (n 2 • • • n d ) 4: C 1 = matricize a, mode = 1 initialize the first TT-core 5: for i = 1, . . . , d -1 do Ûi , Σi , Vi = T-SVD(C i , δ ) compute the truncated SVD define the i-th TT-core 7:

p

  = p/n i+1 move the singular values and the right orthogonal basis on the right, to compute the next TT-core 10:

R 8 :

 8 end for 9: s 0 = 1 10: for i = 1, . . . , d -1 do 11:

  Example I.II.i. The IEEE standard arithmetic floating point system includes two formats. The single precision format has basis β = 2, precision t = 24 and lower exponential bound e min = -125 while the maximum is e max = 128. The double precision format has the same basis, but precision t = 53, e min = -1021 and e max = 1024.The Once a floating point system F is considered, we can introduce the rounding function. Definition I.II.ii. Given the basis β and precision t of the floating point system F, let G ⊂ R be the set of all real numbers x expressed as y = m × β e-t for every e ∈ Z and m ∈ N 0 . The rounding function f l : R → G associates each x ∈ R with the closest element in G, i.e., f l(x) = arg min y∈G |y -x|. Notice that F ⊂ G by definition. Thus, if |f l(x)| ≥ max{|y| with y ∈ F}, then the rounding overflows, while if 0 < |f l(x)| ≤ min{|y| with y ∈ F -{0}}, then the rounding underflows. Another key concept in finite precision arithmetic is the unit round-off. Definition I.II.iii. Let β ∈ N and t ∈ N be the floating point system F basis and precision respectively, then the unit round-

βe 1 -

 1 Hk y with Hk = H(: k + 1, : k) for j = k, . . . , 1 do 12:

. 4 )

 4 Based on the basic IEEE model f l(a op b) = (a op b)(1 + ε), with op ∈ {+, -, ×, ÷}, |ε| < u and u the unit round-off of the working precision, many theoretical results exist in the literature. In particular, it is known [145, p. 152-161] that the Householder QR factorization of a set of n-dimensional vectors S = {s 1 , . . . , s m } generates an orthonormal basis Q with orthonormality quality

Figure 2 . 1 ,

 21 that is O(δ), since δ u 64 . It can be seen that the loss of orthogonality of the MGS-Arnoldi basis (which is characterized by a condition number that deviates from one and becomes larger than 4/3 after the iteration marked by the vertical dashed blue line) is also affected by δ-representation of the basis vectors. More results on other test examples are depicted in [3, Appendix A].

Figure 2 . 1 -

 21 Figure 2.1 -Convergence history of η AM,b for gre-115 with ILU(10 -1 ) using δ-data component-wise representation. The horizontal dashed black line indicates δ. The vertical dash blue line represents the first iteration where κ(V k ) > 4/3 in MGS-GMRES.

Figure 2 . 2 -

 22 Figure 2.2 -Convergence history of η AM,b for gre-343 with ILU(2 • 10 -1 ) using δ-data component-wise representation with fp32 calculation (left) and fp64 calculation (right). The horizontal dashed black line indicates δ. The vertical dash blue line represents the first iteration where κ(V k ) > 4/3 in MGS-GMRES.

Figure 2 . 3 -

 23 Figure 2.3 -Convergence history of η AM,b for gre-115 with ILU(10 -1 ) using SZ-componentwise storage for the Arnoldi basis and Householder reflectors. The horizontal dashed black line indicates δ. The vertical dash blue line represents the first iteration where κ(V k ) > 4/3 in MGS-GMRES.

Figure 2 . 4 -

 24 Figure 2.4 -Convergence history of η AM,b for gre-115 with ILU(10 -1 ) using δ-data normwise representation. The horizontal dashed black line indicates δ. The vertical dash blue line represents the first iteration where κ(V k ) > 4/3 in MGS-GMRES.

Figure 2 . 5 -

 25 Figure 2.5 -Convergence history of η AM,b for e05r0000 with ILU(10 -2 ) using SZ-norm-wise representation of the Arnoldi basis and Householder reflectors. The horizontal dashed black line indicates δ. The vertical dash blue line represents the first iteration where κ(V k ) > 4/3 in MGS-GMRES.

2. 3 .

 3 TENSOR TRAIN GMRES Algorithm 9 x, hasConverged = TT-GMRES(A, b, m, ε, δ) 1: input: A, b, m, ε, δ. 2: r 0 = b, β = r 0 and v 1 = (1/β)r 0 3: for k = 1, . . . , maxit do 4: w = TT-round(Av k , δ) MGS variant 5:

19 :

 19 return: x = x k , hasConverged Algorithm 10 x, hasConverged = Right-GMRES(A, M, b, x 0 , m, ε, δ) 1: input: A, M, b, m, ε, δ. 2: hasConverged = False 3: x = x 0 4: while not(hasConverged) do Iterative refinement step with at most m GMRES iterations on AM 5:

  ) and (2.21), given x ∈ R p×n 1 ו••×n d and define the new vector r = Axb.

52

 52

54 2. 3 . 2 . 3 . 4 .

 543234 TENSOR TRAIN GMRESProposition Let the "all-in-one" operator A ∈ R (p×p)×(n 1 ×n 1 )×...(n d ×n d ) and righthand side b ∈ R p×n 1 ו••×n d be as in Equations (2.13) and (2.14) respectively, we consider the "all-in-one" system Ax = b.Let A ∈ R (n 1 ×n 1 )ו••×(n d ×n d )be the tensor operator as in Equation (2.15) and let b ∈ R n 1 ו••×n d be a tensor such that ||b || = 1, that defines the individual linear systems A y = b with A = A [ ] and b = b [ ] for every ∈ {1, . . . , p}. Let x k denote the k-th "all-in-one" iterate, we have

Proposition 2 . 3 . 5 .

 235 With the same hypothesis and notation as for Proposition 2.3.4 for η A,b (x k ) and η A ,b (x [ ] k ) associated linear systems Ax = b and A y = b respectively, for every ∈ {1, . . . , p}, we have

  0 for every ∈ {1, . . . , p}. The right-hand side b is defined similarly to the previous section, that is b [ ] = b . If the initial guess is x 0 ∈ R p×n 1 ו••×n d equal to the null tensor, then at the k-th iteration TT-GMRES minimizes with respect to x the norm of the residual Axb on the space K k (A, b) = span b, Ab, A 2 b, . . . , A k-1 b ,

  A h 0 b . Consequently the -th slices of the basis vectors of K k (A, b) span the Krylov space K k (A 0 , b ). It means that the individual solutions defined by the slices x [ ] k of the iterate from the "all-in-one" TT-GMRES scheme lie in the same space as the y ,k generated by TT-GMRES applied to the individual systems A 0 y = b with y ,0 = 0. While the two iterates belong to the same space, they are different since the former, x [ ] k , is built by minimizing the residual norm of Axb over K k (A, b) and the latter, y ,k , by minimizing the residual norm of A 0 y = b over K k (A 0 , b ). If we neglect the effect of the rounding, one can expect that b

Proposition 2 . 3 . 7 .

 237 Under the hypothesis of Proposition 2.3.5, if A = I p ⊗ A 0 , then for η A,v (x k ) and η A ,b (x [ ] k ) associated with the linear systems Ax = b and A 0 y = b respectively, for every ∈ {1, . .

Figure 2 . 6 -

 26 Figure 2.6 -Convergence history of TT-GMRES on a 3-d convection-diffusion problem, n = 64, for three different rounding accuracies δ.

  Figure 2.7 -TT-GMRES and relaxed TT-GMRES for the solution of 3-d convectiondiffusion problem with n = 63.

  Compression ratio for the entire Krylov basis

Figure 2 .

 2 Figure 2.8 -3-d Poisson problem using δ = ε = 10 -5 .

66

 66

  Convergence history for τ = 10 -2 and rounding δ = 10 -5 Convergence history for τ = 10 -8 and rounding δ = 10 -5

Figure 2 .

 2 Figure 2.9 -3-d Poisson problem, comparing preconditioners.

  Maximal TT-rank of the last Krylov vector for τ = 10 -2

Figure 2 .

 2 Figure 2.9 -3-d Poisson problem, comparing preconditioners.

Figure 2 .

 2 Figure 2.9 -3-d Poisson problem, comparing preconditioners.

  Maximal TT-rank of the last Krylov vector Compression ratio for the entire Krylov basis

Figure 2 .

 2 Figure 2.10 -3-d Convection-diffusion using δ = ε = 10 -5 .

  3.3.1 let define a grid of n points along each direction of Ω, then the final discrete operator of this PDE is A α = α∆ 3 + D with α ∈[START_REF] Abdelmalek | Round off error analysis for Gram-Schmidt method and solution of linear least squares problems[END_REF][START_REF] Beh | Correspondence Analysis[END_REF] and D defined in Equation(2.42). Similarly, the right-hand side c α ∈ R n×n×n depends on the parameter α ∈ [1, 10] because of the boundary conditions. To solve for multiple discrete values of α, getting an "all-in-one" problem and solution, we tensorize ∆ 3 and D by a diagonal matrices, adding a fourth dimension. The tensor operator for the simultaneous solution is A ∈ R (p×p)×(n×n)×(n×n)×(n×n) defined asA = A ⊗ ∆ d + I p ⊗ D ,where A = diag(α 1 , . . . α p ) with α i ∈ [1, 10] logarithmically distributed for i ∈ {1, . . . , p}. The right-hand side of the "all-in-one" problem is b ∈ R p×n×n×n such that b [ ] = 1 c α c α for ∈ {1, . . . , p} using the slice notation introduced in Section 2.3.2. By construction b = √ p, i.e., the

  θ ∈ [0, 10]. The function σ θ , rewritten as σ θ (x, y, z) = 1 + θ1 Ξ (x, y, z) where 1 Ξ is the indicator function of Ξ, provides a linear dependency on θ for the PDE. If Ξ x is the projection of set Ξ over the x-axis and similarly for Ξ y and Ξ z , then σ θ (x, y, z) = 1 + θ1 Ξx (x)1 Ξy (y)1 Ξz (z).

Figure 2 .

 2 Figure 2.11 -4-d Parametric convection-diffusion using δ = ε = 10 -5 .

  Convergence history in η b for n = 255

Figure 2 .Figure 2 .

 22 Figure 2.12 -4-d Parametric convection-diffusion η b bound using δ = ε = 10 -5 .

Figure 2 .

 2 Figure 2.14 -4-d Heterogeneous convection-diffusion using δ = ε = 10 -5 .

  Convergence history in η b for n = 63 Convergence history in η b for n = 127

Figure 2 .Figure 2 .

 22 Figure 2.15 -4-d Heterogeneous convection-diffusion η b bound using δ = ε = 10 -5 .

Figure 2 .

 2 Figure 2.17 -4-d multiple right-hand side Poisson problem using δ = ε = 10 -5 .

  Convergence history in η b for n = 127

Figure 2 . 127 0 1 (

 21271 Figure 2.18 -4-d Poisson problem η b bound using δ = ε = 10 -5 .

Figure 2 .

 2 Figure 2.19 -4-d multiple right-hand side Poisson problem η AM,b bound using δ = ε = 10 -5 .

Figure 2 .

 2 Figure 2.20 -4-d multiple right-hand sides convection-diffusion problem using δ = ε = 10 -5 .

  Convergence history in η b for n = 63 Convergence history in η b for n = 127

Figure 2 .

 2 Figure 2.21 -4-d convection-diffusion problem η b bound using δ = ε = 10 -5 .

Figure 2 .

 2 Figure 2.22 -4-d multiple right-hand side convection-diffusion problem η AM,b bound using δ = ε = 10 -5 .

  Maximal TT-rank of the lastKrylov vector

Figure 2 .

 2 Figure 2.23 -4-d multiple right-hand side problem with eigenvector.

  Residual norm for n = 255

Figure 2 .

 2 Figure 2.24 -4-d multiple right-hand side problem, residual comparison.

3. 2 . 2 :

 22 ORTHOGONALIZATION SCHEMES Algorithm 11 Q , R = TT-CGS(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + for i = 1, . . . , m do

Algorithm 13 Q 2 :

 132 , R = TT-CGS2(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + for i = 1, . . . , m do 3: p 0 = a i repeat twice the orthogonalization loop 4:

p 2 normalize p 2 14 : 2 :

 142 end for compute the R factor from the repeated orthogonalization loop 15: R = R 1 + R 2 16: return: Q = {q 1 , . . . , q m }, R Algorithm 14 Q , R = TT-MGS2(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + for i = 1, . . . , m do 3: p 0 = a i repeat twice the orthogonalization loop 4:

2 . 2 , 2 : 3 :

 2223 notice that the Algorithm 15 Q , R = TT-Gram(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + for i = 1, . . . , m do for j = 1, . . . , i do construct the Gram matrix through the inner product of the input TT-vectors 4: G(i, j) = G(j, i) = a i , a j 5: end for 6: end for 7: L = cholesky(G) compute the Cholesky factorization 8: R = L and R -1 = invert(R) define the R factor of the QR-factorization 9: for i = 1, . . . , m do 10: p = R -1 (i, 1)a 1 11:

Remark 3 . 2 . 3 . 2 with e 1 ∈

 32321 By construction the first (k -1) entries of u k are zeros, the last (n -k) ones are equal to the corresponding ones of a k . The k-th component of u k is given by the difference of the k-th component of a k and the quantityβ k , that is u k (k) = ãk (k) -β k .Said differently, only the last (n -k + 1) entries of ãk have a determinant role. Thanks to this property, in the matrix context, the Householder QR factorization has a reduced and simplified construction. The k-th Householder vector is ûk ∈ R n-k+1 defined from the norm of ãk with the first (k -1) entries being zeros, i.e., ûk = γ k e 1 where γ R n-k+1 for every k ∈ {1, . . . , m}. The k-th Householder transformation

96 3. 2 2 : s = 0 3 : w = a 4 :

 962234 . ORTHOGONALIZATION SCHEMES Algorithm 16 u, r = TTH-vec(a, E, δ) 1: input: a ∈ R n 1 ו••×n d , E = {e 1 , . . . , e i }, δ ∈ R + for j = 1, . . . , i -1 docompute the component of a along the j-th canonical basis TT-vector 5:r(j) = a, e j 6:s = s + r(j) 2set to zero the component of a along the j-th canonical basis TT-vector e j 7:

Algorithm 18 Q 3 : w = a 1 4 :

 1834 , R = TT-Householder(A, δ) 1: input: A = {a 1 , . . . , a m }, δ ∈ R + 2: let E = {e 1 , . . . , e m } be the canonical basis of a subspace of dimension m of R n 1 ו••×n d for i = 1, . . . , m do construct the i-th Householder TT-vector 5:

  δ = 10 -8

Figure 3 . 1 -

 31 Figure 3.1 -LOO and condition number for m = 20 TT-vectors of order d = 3 and mode size n = 15. The curves get dashed and partially transparent when they get greater than 1.

  δ = 10 -8

Figure 3 . 2 -

 32 Figure 3.2 -LOO for m = 35 TT-vectors of order d = 6 and mode size n = 15. The curves get dashed and partially transparent when they get greater than 1.

  vector uk Basis TT-vector qk Reflected TT-vector ak (C) δ = 10 -8 Maximal TT-rank for the TT-vectors in Householder algorithm vector uk Basis TT-vector qk Reflected TT-vector ak (D) δ = 10 -3 vector uk Basis TT-vector qk Reflected TT-vector ak (E) δ = 10 -5 vector uk Basis TT-vector qk Reflected TT-vector ak (F) δ = 10 -8

  vector uk Basis TT-vector qk Reflected TT-vector ak (I) δ = 10 -8

Figure 3 . 3 -

 33 Figure 3.3 -Householder memory requirement for m = 35 TT-vectors of order d = 6 and mode size n = 15.

  δ = 10 -8

Figure 3 . 4 -

 34 Figure 3.4 -Comparison of the orthogonal basis memory requirement for m = 35 TTvectors of order d = 6 and mode size n = 15. The curves get dashed and partially transparent when their corresponding loss of orthogonality gets greater then the prescribed rounding accuracy δ.

[

  7470.430, 7441.016, 7441.016, 7441.016, 7411.601, 7411.601, 7411.601 ].

Figure 3 . 5 -

 35 Figure 3.5 -Residual envelope for m = 7 TT-eigenpairs of order d = 3 and mode size n = [19, 24, 31].

Figure 3 . 6 -

 36 Figure 3.6 -Residual envelope for m = 7 TT-eigenpairs of order d = 3 and mode size n = [24, 24, 24].

Figure 3 . 7 -

 37 Figure 3.7 -Memory requirement for TT-eigenvector for m = 7 TT-eigenpairs of order d = 3 and mode size n = [19, 24, 31].

  Then the row norm || • || R : R n 2 → R + and the column norm || • || C : R n 2 → R + are defined as ||x|| R = ||D -1 R x|| and ||y|| C = ||D -1 C y|| where D R = diag( √ f R ) and D C = diag( √ f C ). The matrices D -1 R and D -1 C induce two isometries on the row space (R n 2 , || • || R ) and the column one (R n 1 , || • || C ), since they are both SPD matrices. Indeed, the functions

  r) and Σ = diag(σ 1 , . . . , σ r ) with r = rank(X). The principal component for the row point clouds are given by the columns of Y R = U Σ in a subspace of dimension r of R n 2 endowed with the canonical Euclidean norm, while the column principal coordinates are Y C = V Σ in a subspace of dimension r of R n 1 still with the canonical Euclidean norm. More precisely from a geometric viewpoint the (i, h) element of Y R represents the coordinate of the i-th category of the row variable along the h-th principal axis. Similarly Y R (j, h) is the coordinate of the j-th category of the column variable along the h-th principal axis for h ∈ {1, . . . , r}. As consequence the projections of the row and the column point clouds live in the subspaces of the same dimension r. They are usually simultaneously displayed and interpreted, because a barycentric relation links these principal components in the Euclidean space selected by CA. More specifically, let the W R = D R Y R be the scaled principal components of the row variable in the subspace of dimension r of (R n 2 , || • || R ) and similarly W C = D C Y C represents the columns principal components in the subspace of dimension r of (R n 1 , || • || C ), then it holds

140

 140

. 5 ) 2 ) j u 2 j and σ ( 3 ) k u 3 k

 5233 Notice that the j-th and k-th column of Y 2 = U 2 Σ 2 and Y 3 = U 3 Σ 3 are σ (respectively. So introducing in Equation (4.5) the singular values σ (2) j and σ

(4. 7 )

 7 Multiplying Equation (4.4) on the right by

Figure 4 . 1 -

 41 Figure 4.1 -Correspondence analysis: matrix versus tensor.

(

  B) Multiway correspondence analysis.

Figure 4 . 2 -

 42 Figure 4.2 -CA versus MWCA for data of [24].

CHAPTER

  Multiway correspondence analysis.

Figure 4 . 3 -

 43 Figure 4.3 -CA versus MWCA for data of [24].

  Seasonal variation of the OTU average per location and water column position.

Figure 4 . 4 -

 44 Figure 4.4 -Seasonal variation of the OTU average.

Figure 4 .

 4 Figure 4.4A shows the variation of the number of OTU sequences in function of the

  MWCA with filtered OTU.

Figure 4 . 5 -

 45 Figure 4.5 -CA vs MWCA biplot of Arcachon Bay data-set.

a * = argmax ||a||=1 Var(X c a) = argmax ||a||=1 1 n - 1 ||X c a|| 2 ( 5 . 3 )

 ||a||=11253 since X c columns have zero mean, see Equation (5.2). Let Y c ∈ R n×p be a matrix defined as Y of (5.3) is actually a solution of the Karush-Kuhn-Tucker equation (Y c Y c )a -λa = 0 with ||a|| = 1 which is solved by the eigenvalue decomposition of (Y c Y c ), i.e., (Y c Y c ) = AΛA 5.3. EOF ANALYSIS WITH TUCKER MODEL It is common to centre in time the PC matrix, i.e., defining C c = ( √ n -1)U c with U c = U -1 n ⊗ u with u

1 ,r 2 ,r 3 α

 13 3.1, that is f = (U, W, Z)fwhere U ∈ O(n × r 1 ), W ∈ O(p 1 × r 2 ) and Z ∈ O(p 2 × r 3 ), while f ∈ R r 1 ×r 2 ×r 3 is the core tensor. Equivalently this last equation can be written asf = r 1 ,r 2 ,r 3 α,β,γ=1 f (α, β, γ)u α ⊗ w β ⊗ z γwhere u α , w β and z γ are the α-th, β-th and γ-th column of U , W and Z respectively.We follow the approach described in Section 5.2.3 to compute the EOFs and PCs. Firstly we scale the input data by the diagonal matrix D = diag( √ cos θ 1 , . . . , cos θ p 1 ) along the latitude mode, definingf s ∈ R n×p 1 ×p 2 f s = D × 2 f = (U, DW, Z)f = r 1 ,r 2 ,r 3 α,β,γ=1 f (α, β, γ)u α ⊗ Dw β ⊗ z γ = r 1 ,r 2 ,r 3 α,β,γ=1 f (α, β, γ)u α ⊗ v β ⊗ z γwhere v β = Dw β by construction for every β ∈ {1, . . . , r 2 }. Remark 5.3.1. The matrix V ∈ R p 1 ×r 2 defined as V = DW is not orthogonal any more. The next step is centring the scaled data in time. Still keeping the data in Tucker format, we compute the average in time F s ∈ R p 1 ×p 2 as F s (j, k) ,β,γ=1 f (α, β, γ)u α (i) v β (j) z γ (k).

r 1 α=1 u α ⊗ F s = r 1 ,r 2 ,r 3 α

 113 ,β,γ=1 f (α, β, γ) -F s (β, γ) u α ⊗ v β ⊗ z γ = r 1 ,r 2 ,r 3 α,β,γ=1 f sc (α, β, γ)u α ⊗ v β ⊗ z γwhere f sc (α, β, γ) = f (α, β, γ) -F s (β, γ) for every α ∈ {1, . . . , r 1 }, β ∈ {1, . . . , r 2 } and γ ∈ {1, . . . , r 3 }.Denoted by X sc ∈ R n×p the matricization of f sc with respect to mode 1, then X sc gets X sc = F(1) sc =

5. 3 .

 3 EOF ANALYSIS WITH TUCKER MODELbenefit from the Tucker format, applying some further algebraic transformations. We proceed computing the QR-factorization of (Z ⊗ K V ), getting Q ∈ O(p × r) and the upper triangular matrix R ∈ R r×r with p = p 1 p 2 and r = r 2 r 3 . Introducing the QR-factors into Equation (5.10), it writes X sc = U Υ(QR) = U (ΥR )Q .(5.11)

F 1 ,r 2 ,r 3 αr 2 ,r 3 β, γ=1 F

 1323γ=1 ,β,γ=1 f (α, β, γ)u α (i) w β (j) z γ (k),which more compactly is expressed asF = (β, γ)w β ⊗ z γ with F (β, γ)

r 1 α=1 u α ⊗ F = r 1 ,r 2 ,r 3 α

 113 ,β,γ=1 f (α, β, γ) -F (β, γ) u α ⊗ w β ⊗ z γ = r 1 ,r 2 ,r 3 α,β,γ=1 f c (α, β, γ)u α ⊗ w β ⊗ z γwhere f c (α, β, γ) = f (α, β, γ) -F (β, γ) for every α ∈ {1, . . . , r 1 }, β ∈ {1, . . . , r 2 } and γ ∈ {1, . . . , r 3 }. Finally the tensor EOF a c ∈ R r 1 ×p 1 ×p 2 writes

Figure 5 . 1 -

 51 Figure 5.1 -PC and EOF obtained from the correlation EOF analysis.

  The data centred in time by f c writes in Tucker format 1.3.1 asf c = (U, W, Z)f c with U ∈ O(n × r 1 ) the time basis, W ∈ O(p 1 × r 2 )the latitude one and Z ∈ O(p 2 × r 3 ) the longitude one, while f c ∈ R r 1 ×r 2 ×r 3 is the core tensor.

  1st column of the time basis.

  1st column of the latitude basis.

  1st column of the longitude basis.

Figure 5 . 2 -

 52 Figure 5.2 -Tucker basis of f c .

d h=1 n h r h + h=1 r h d j=1 n j

  ••×m d 2 with just the h-th and k-th mode having the same size, then (a •

h k b) is their tensor contraction with respect to mode h and k respectively. If tensor

  ••×n d → R n 1

  ••×n d be a multilinear endomorphism and let the tensor

  1

  ••×n d , its best Tucker approximation with prescribed multilinear rank (r 1 , . . . , r d ) is the tensor a * such that aa * is minimal, i.e., a

* = argmin rank(B (k) )≤r k k=1,...,d ab where B (k) is the k-th mode matricization of b ∈ R n 1

  [START_REF] Affleck | Rigorous results on valencebond ground states in antiferromagnets[END_REF]

  • • • n d )is decomposed with the truncated SVD at accuracy δ . The left orthogonal matrix Û1 is reshaped into the first TT-core to have size (1, n 1 , r 1 ) where r 1 is the number of columns of Û1 , see line 7 of Algorithm 4. We define matrix C 2 ∈ R r 1 ×(n

  2

  1 and e ∈ Z is the exponent lower and upper bounded by e min ∈ Z and e max ∈ Z, i.e., e min ≤ e ≤ e max . The value β is called the base and t the precision of the floating point system F. The range of the non-zero floating point numbers in F is

Table 2 .

 2

	-1 )

1 -Percentage of memory saving using SZ-component-wise representation.

Table 2 .

 2 10 -12 10 -14

		gre_115 with ILU(10 -1 )
	MGS	5.7	8.8 32.9 31.3	23.6	14.3
	Householder 50.4 43.3 43.5 35.9	29.0	21.9
		gre_185 with ILU(10 -1 )
	MGS	7.2 25.6 37.9 27.7	18.6	7.5
	Householder 38.4 45.6 46.6 37.6	29.7	20.6
		gre_343 with ILU(10 -1 )
	MGS	22.3 42.0 51.8 46.9	40.3	30.5
	Householder 24.5 51.4 58.3 49.5	44.0	35.5
		arc130 with ILU(8 • 10 -4 )
	MGS	74.6 63.8 51.3 40.4	23.7	21.3
	Householder 75.3 70.3 60.9 34.1	14.9	11.5
		e05r0000 with ILU(10 -2 )
	MGS	5.6	8.4 40.7 32.3	21.2	11.3
	Householder 4.4 40.1 44.7 34.9	24.5	15.7
		e05r0400 with ILU(10 -2 )
	MGS	0.1 16.1 39.5 29.4	18.5	9.1
	Householder 19.7 42.2 45.5 35.5	25.1	16.5
			cavity03
	MGS	0.7 25.6 40.7 29.9	19.1	9.8
	Householder 12.5 45.9 47.0 36.9	26.6	18.0
		pde225 with ILU(3 • 10 -1 )
	MGS	10.8 32.1 37.7 29.4	19.2	8.0
	Householder 19.8 43.8 46.5 37.3	29.0	19.2

2 -Percentage of memory saving using SZ-norm-wise representation.

  The result follows from the thesis of Proposition 2.3.5, since A 2 = A 0 2

.32) CHAPTER 2. A ROBUST GMRES IN TT-FORMAT 57 Proof. Corollary 2.3.8. Given a sequence of iterative solutions {x k } k∈N and a value ν, if there exists a k * ∈ N such that | ||A x [ ]

  [START_REF] Daniel | Reorthogonalization and Stable Algorithms for Updating the Gram-Schmidt QR Factorization[END_REF] for every ∈ {1, . . . , p} and for every k ∈ N such that k ≥ k * * where k * * = max k * .

	Corollary 2.3.9. Under the hypothesis of Corollary 2.3.8, if there exists a k

Table 2 .

 2

	16	32	64	2	8	16	32	64

3 -Preconditioner properties for grid step n = 63.

  To plot the η AM,b bound from Proposition 2.3.5, we define a vector υ ∈ R w whose k-th component corresponds to the value of the coefficient ρ from Equation (2.26) evaluated for the solution at the k-th iteration, i.e., equal to the iteration number to reach the convergence. Let m and M the parameter index for which the norm of υ is minimal and maximal respectively, i.e.,

	υ (k) = ρ (t k )		for every	k ∈ {1, . . . , w}
	with w m = argmin	υ	and M = argmax	υ	(2.47)
	∈{1,...,p}			∈{1,...,p}

which in our specific case are equal to 1 and 14 respectively. In Figure

2

.

[START_REF] Benzécri | 'analyse des correspondances[END_REF]

we display in η AM,b (t k ) scaled by ρ (see Equation (2.26) from Proposition 2.3.5) and by ρ * (see Equation (2.33) from Corollary 2.3.8) versus η A M,b (t [ ] k ) for ∈ {5, 20} and for all the values of n.

  n×n and similarly for D y and D x . Remark that B 1 is a Laplacian-like operator, which is expressed in TT-format according to Equation (2.35) and(2.36). The final discrete TT-operator of Problem (2.48) is A is normalized, i.e., it is scaled by 1/n 3 . Since we want to solve for p values of θ in [0, 10] simultaneously, i.e., we want to solve p-times the discrete Problem(2.48)

θ = B 0 + θB 1 .

The right-hand side is c ∈ R n×n×n such that c(i 1 , i 2 , i 3 ) = 1 for i k ∈ {1, . . . , n} for k ∈ {1, 2, 3}. To study the quality of the bounds expressed in Proposition 2.3.4 and 2.3.5, the tensor c

Table 3 .

 3

			Householder	MGS	MGS2	CGS	CGS2	GRAM
		λ	8165.137	8166.140	8166.151	8166.156	8166.138	8136.768
		λ *	8166.432	8166.432	8166.432	8166.432	8166.432	8136.942
	1	|λ * -λ| |λ * |	1.586e -04 3.586e -05 3.447e -05 3.390e -05 3.607e -05 2.145e -
		convergence order	3	7	7	7	7	2
		spectrum order	1	1	1	1	1	4
		λ	8136.560	8136.758	8136.758	8136.758	8136.758	8058.973
		λ *	8136.942	8136.942	8136.942	8136.942	8136.942	8059.020
	2	|λ * -λ| |λ * |	4.697e -05 2.260e -05 2.260e -05 2.260e -05 2.260e -05 5.817e -
		convergence order	1	1	1	1	1	3
		spectrum order	4	4	4	4	4	15
		λ	8136.450	8058.985	8058.985	8058.985	8058.985	8058.601
		λ *	8136.942	8059.020	8059.020	8059.020	8059.020	8058.693
	3	|λ convergence order	5	3	3	3	3	1
		spectrum order	4	15	15	15	15	17
		λ	8135.066	8058.559	8058.559	8058.559	8058.559	8021.230
		λ *	8136.942	8058.693	8058.693	8058.693	8058.693	8021.672
	4	|λ convergence order	6	2	2	2	2	4
		spectrum order	4	17	17	17	17	22
		λ	8107.330	8021.229	8021.229	8021.229	8021.229	7980.728
		λ *	8107.528	8021.672	8021.672	8021.672	8021.672	7980.880
	5	|λ convergence order	7	4	4	4	4	5
		spectrum order	7	22	22	22	22	35
		λ	8106.774	7980.676	7980.676	7980.676	7980.676	7943.274
		λ *	8107.528	7980.880	7980.880	7980.880	7980.880	7943.348
	6	|λ * -λ| |λ * |	9.300e -05 2.546e -05 2.546e -05 2.546e -05 2.546e -05 9.334e -
		convergence order	2	5	5	5	5	6
		spectrum order	7	35	35	35	35	42
		λ	8058.094	7943.245	7943.245	7943.245	7943.245	7942.854
		λ *	8058.693	7943.348	7943.348	7943.348	7943.348	7943.054
	7	|λ convergence order	4	6	6	6	6	7
		spectrum order	17	42	42	42	42	43

* -λ| |λ * | 6.051e -05 4.330e -06 4.331e -06 4.330e -06 4.331e -06 1.145e -* -λ| |λ * | 2.305e -04 1.673e -05 1.673e -05 1.673e -05 1.673e -05 5.520e -* -λ| |λ * | 2.440e -05 5.527e -05 5.527e -05 5.527e -05 5.527e -05 1.894e -* -λ| |λ * | 7.444e -05 1.289e -05 1.289e -05 1.289e -05 1.289e -05 2.513e -2m = 7 eigenvalues for the Laplacian of order d = 3 and mode size n = [19, 24, 31] with δ = 10 -3 . * -λ| |λ * | 3.353e -09 3.018e -07 2.743e -08 4.553e -07 3.193e -09 1.366e -* -λ| |λ * | 3.534e -06 5.534e -09 5.534e -09 5.534e -09 5.534e -09 5.398e -* -λ| |λ * |

5.376e -09 6.350e -09 6.350e -09 6.350e -09 6.350e -09 6.381e -

Table 3 .

 3

			Householder	MGS	MGS2	CGS	CGS2	GRAM
		λ	7469.291	7382.018	7382.018	7382.018	7382.018	7382.010
		λ *	7470.430	7382.187	7382.187	7382.187	7382.187	7382.187
	1	|λ * -λ| |λ * |	1.525e -04 2.283e -05 2.283e -05 2.283e -05 2.283e -05 2.400e -
		convergence order	6	1	1	1	1	1
		spectrum order	1	11	11	11	11	11
		λ	7440.841	7266.610	7266.610	7266.610	7266.610	7266.611
		λ *	7441.016	7266.841	7266.841	7266.841	7266.841	7266.841
	2	|λ * -λ| |λ * |	2.349e -05 3.179e -05 3.179e -05 3.179e -05 3.179e -05 3.164e -
		convergence order	3	2	2	2	2	3
		spectrum order	3	36	36	36	36	36
		λ	7440.597	7266.598	7266.598	7266.598	7266.598	7266.603
		λ *	7441.016	7266.841	7266.841	7266.841	7266.841	7266.841
	3	|λ convergence order	5	3	3	3	3	2
		spectrum order	3	36	36	36	36	36
		λ	7439.689	7266.503	7266.503	7266.503	7266.503	7266.566
		λ *	7441.016	7266.841	7266.841	7266.841	7266.841	7266.841
	4	|λ convergence order	7	4	4	4	4	4
		spectrum order	3	36	36	36	36	36
		λ	7411.681	7151.317	7151.317	7151.317	7151.317	7151.279
		λ *	7411.601	7151.496	7151.496	7151.496	7151.496	7151.496
	5	|λ convergence order	4	6	6	6	6	7
		spectrum order	5	76	76	76	76	76
		λ	7410.233	7151.150	7151.149	7151.150	7151.150	7151.103
		λ *	7411.601	7151.496	7151.496	7151.496	7151.496	7151.496
	6	|λ * -λ| |λ * |	1.820e -04 4.837e -05 4.845e -05 4.832e -05 4.838e -05 5.485e -
		convergence order	2	7	7	7	7	5
		spectrum order	5	76	76	76	76	76
		λ	7379.785	7151.073	7151.073	7151.073	7151.073	7150.870
		λ *	7382.187	7151.496	7151.496	7151.496	7151.496	7151.496
	7	|λ convergence order	1	5	5	5	5	6
		spectrum order	11	76	76	76	76	76

* -λ| |λ * | 5.629e -05 3.342e -05 3.342e -05 3.342e -05 3.342e -05 3.275e -* -λ| |λ * | 1.783e -04 4.655e -05 4.655e -05 4.655e -05 4.655e -05 3.795e -* -λ| |λ * | 1.070e -05 2.496e -05 2.496e -05 2.496e -05 2.496e -05 3.025e -* -λ| |λ * | 3.234e -04 5.907e -05 5.907e -05 5.907e -05 5.907e -05 8.749e -4m = 7 eigenvalues for the Laplacian of order d = 3 and mode size n = [24, 24, 24] with δ = 10 -3 . * -λ| |λ * | 8.870e -09 8.889e -09 8.520e -09 8.886e -09 8.787e -09 8.175e -09 * -λ| |λ * | 1.896e -09 3.170e -09 3.170e -09 3.170e -09 3.170e -09 3.141e -09 * -λ| |λ * |

4.857e -09 6.020e -09 6.020e -09 6.018e -09 6.018e -09 6.011e -09

  table, i.e., the data are 4.2. CORRESPONDENCE ANALYSIS organized in a multiway table, it is convenient to move from matrix analysis techniques to tensor ones. As well underlined in[START_REF] Zare | Extension of PCA to Higher Order Data Structures: An Introduction to Tensors, Tensor Decompositions, and Tensor PCA[END_REF], it is still possible to investigate multiway data through matrix methods, matricizing the data, but the matricization introduces necessarly a variable coupling, making some variable interactions harder to identify. On the other side, analysis procedures based on tensor decomposition algorithms have the advantage of considering each variable independently and clearly show all the variable links. For example in the Tucker model, presented in Section 1.3.1, the strength of the variable correlation is expressed by the core tensor coefficients. Classically, PCA is extended to tensor data by the Tucker model and its realization through HOSVD, see Algorithm 1. The algebraic extension of CA to MultiWay Correspondence Analysis (MWCA) follows naturally and it has been studied in[START_REF]Multiway Data Analysis[END_REF][START_REF] Franc | Etude Algébrique des multitableaux: apports de l'algèbre tensorielle[END_REF][START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF][START_REF] Kroonenberg | Three-mode principal component analysis: Theory and applications[END_REF]. However the further investigation of the geometric nature of tensor MWCA was missing in the literature. The focus of this chapter is investigating the geometric viewpoint in CA in the tensor framework. Indeed we propose to fulfil this gap by proving the algebraic relation and the geometric interpretation of the point clouds associated with each mode of a multiway contingency table.

  Therefore multiplying Equation (4.7) by the matricization of x with respect to mode 1, the principal component Y 1 is expressed as linear combination of the Kronecker product of principal components Y 2 and Y 3 , i.e., Y 1

  , while in 4.2B we display multiway correspondence analysis with the first two columns of Y µ = U µ Σ µ , defined in Section 4.2.2.3 for µ ∈ {1, 2, 3}.

	Principal component 2	0.1 0.0 0.1 0.2	16-24M 25-34M 35-44M 45-54M 55-64M 65-74M 16-24F 25-34F 35-44F 45-54F 55-64F 65-74F +75F Very good Bad Very Bad +75M mode 1, gender per age mode 2, health	Regular	Good	Principal component 2	0.1 0.0 0.1 0.2	16-24 25-34 35-44 45-54 55-64 65-74 Very good Very Bad +75 Regular Bad mode 1, gender mode 2, age mode 3, health	M Good F
		0.2							0.2
			0.0	0.2	0.4 Principal component 1	0.6	0.8		0.0	0.2	0.4 Principal component 1	0.6	0.8
			(A) Mode 1 correspondence analysis.

It really happened!

La Generalized Minimal RESidual (GMRES) est une méthode itérative robuste permettant de résoudre des systèmes linéaires. Le GMRES repose sur la technique du sousespace de Krylov pour l'approximation de la solution. La solution approchée est calculée en minimisant la norme du résidu du système linéaire sur l'espace de Krylov associé, dont la dimension augmente de manière itérative. Le problème de minimisation est simplifié par la construction d'une base orthogonale de l'espace de Krylov à travers le MGS et le schéma de Householder, ce qui conduit à l'implémentation du MGS-GMRES et du Householder-GMRES présentés respectivement dans[START_REF] Saad | GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems[END_REF] et[START_REF] Walker | Implementation of the GMRES method using Householder transformations[END_REF]. Un critère d'arrêt significatif et fréquemment utilisé est basé sur l'erreur à rebours associée à la matrice et au côté droit du système linéaire. Ce choix est motivé par la stabilité à rebours des deux variantes de GMRES dans l'arithmétique classique en virgule flottante, prouvée respectivement dans[START_REF] Paige | Solution of Sparse Indefinite Systems of Linear Equations[END_REF] et[START_REF] Drkosova | Numerical stability of GMRES[END_REF]. Cette propriété de stabilité à rebours garantit que la solution calculée par GMRES peut être interprétée comme la solution du même système linéaire avec des perturbations de norme relative sur la matrice et le côté droit qui sont limitées par l'arrondi unitaire de la précision de travail.

In Statistics a contingency table is a table whose (i, j) entry is the frequency of the combination of the i-th category of the first variable with the j-th category of the second variable.

Acknowledgements

ORTHOGONALIZATION SCHEMES

while the one of u k around 10 -2 . In Figure 3.3F, it is not clear if the compression ratio of q k and u k stagnate around 1 and 10 -1 respectively. All the Figures 3.3G, 3.3I and 3.3H, the gain curves of u k , q k and a k present approximately the same behaviour. In particular the gain of compressing the k-th Householder TT-vector u k and the input TT-vector a k after the k-th reflection is almost constant through the iterations and minimal, but both the TT-vectors are compressed several times during the previous iterations. On the other side, the compression gain for storing the Householder basis TT-vector q k is significantly larger. For all the rounding accuracies, the compression gain curve of q k increases during the first 10 iterations, decreases slightly after and seems to grow again during the last 10 iterations. Remark that for the compression gain curve of the Householder basis vector the highest value is around 110 if δ = 10 -5 . This means that after the compression, it is actually requested slightly less than 1% of the memory used to store the same tensor in TT-format before the compression.

Algorithm [START_REF] Bro | Solving the sign indeterminacy for multiway models[END_REF] V , L = TT-SUBSPIT(A, Z, p, δ, ε max_it) u j = (1/ y )y with u j ∈ U 16:

end for orthogonalize the set formed by the locked TT-eigenvector and the TT-vector just updated 17:

end for 29:

end for 31:

for j = 1, . . . , n do Compression ratio in function of the number of orthogonalization or TT-rounding

Part II

Data analysis

CORRESPONDENCE ANALYSIS

where

µ W µ be the principal components scaled by the marginal inverse for µ ∈ {1, . . . , d} in tensor space S M . Henceforth, we denote by z µ iµ the i µ -th row of Z µ . Now we prove that each component of vector z µ iµ can be expressed as a scaling factor times the barycenter of the linear combinations of the other scaled principal component rows. We assume d equal to 3 to facilitate the comprehension of the following proof. Proposition 4.2.5. Let f be a tensor in the tensor space S M endowed with the norm induced by the inner product matrices

Proof. We describe the proof for the i-th row of Z 1 with i ∈ {1, . . . , n 1 }. From Proposition 4.2.3, under the CA metric choice, it follows that the principal components of f in the tensor space S M satisfy the relation ) ) .

Multiplying on the left this last equation by D -2 1 , we obtain ) )

and since

1 , it gets

Making explicit the -th component of z 1 i , the i-th row of Z 1 , from Equation (4.19), we have

EMPIRICAL ORTHOGONAL FUNCTION ANALYSIS

with A ∈ O(p × p) and Λ = diag(λ 1 , . . . , λ p ). The eigenvalues are positive since (Y c Y c ) is symmetric semipostive definite, see Equation (5.2), and for convenience we assume them to be sorted in decreasing order. The -th eigenvalue λ ∈ R p of (Y c Y c ) is a measure of the variance expressed by a the -th eigenvector, i.e., the -th column of A. In climatology the eigenvectors a are called Empirical Orthogonal Functions, or EOFs. Once the EOFs are computed, the anomaly field X c is projected in the subspace spanned by the eigenvectors, defining the so-called Principal Component (PC) matrix C ∈ R n×p as C = X c A. Thanks to this construction and the orthogonality of A, multiplying the equation defining the PC matrix on both sides by A T leads to a decomposition of the anomaly filed X c , that is

The components of the anomaly field X c are expressed as a linear combination of the EOF matrix column a and of the PC matrix column c , i.e.,

In climatology, EOFs and PCs are expressed as a function of the discrete space variable and of the time variable respectively. So, let s h be the discrete coupled space variables running over the space grid for h ∈ {1, . . . p} and let express the EOFs as a (h) = a (s h ). Similarly PCs can be written as a function of the discrete-time variable, that is the i-th component of c is denoted c (t i ) for every i ∈ {1, . . . , n}. If now we reintroduce the discrete-time and coupled space variables, we obtain an expression of X c where these two discrete variables are separated in EOFs and PCs, i.e.,

Remark 5.2.1. During the interpretation step, the -th EOF a ∈ R p is reshaped into a matrix of size (p 1 × p 2 ) with p = p 1 p 2 , that is the space grid size, and visualized over a Earth surface map.

To perform the EOF analysis, literature [START_REF] Hannachi | Empirical orthogonal functions and related techniques in atmospheric science: A review[END_REF] numerical advice is to compute the SVD of X c instead of the eigenvalue decomposition of (X c X c ). The reduced SVD of X c is

Then the eigenvalues of (X c X c ) are the squared singular values of X c , i.e., λ = σ 2 . By construction, the EOFs are equal to the eigenvectors of (X c X c ), which correspond to the columns of V , the right orthogonal matrix from the SVD of X c . Comparing Equation (5.5) and the SVD of X c , the PC matrix writes

with u the -th column of the left orthogonal matrix U . The scaling factor √ n -1 is due to the definition of covariance matrix from Y c , see Equation (5.4).

Covariance and correlation for EOFs

In climatology literature, see [START_REF] Baldwin | Spatial Weighting and Iterative Projection Methods for EOFs[END_REF][START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF], it is common to perform the EOFs analysis, not on the covariance matrix, defined in Equation (5.4), but on the correlation matrix, obtained by scaling the data by a symmetric matrix. Indeed, as suggested in [START_REF] Baldwin | Spatial Weighting and Iterative Projection Methods for EOFs[END_REF][START_REF] Hannachi | Patterns Identification and Data Mining in Weather and Climate[END_REF], this choice compensates for the non-uniform distribution of the grid point over the Earth's surface. So, we scale the input tensor data f by the diagonal matrix D = diag( √ cos θ 1 , . . . , cos θ p 1 ) along the latitude mode, that is the second one, defining the scaled tensor

The procedure to obtain the EOFs and the PCs requests now some tiny changes. Let X s ∈ R n×p denotes the matricization with respect to mode 1 of f s , then, as previously, the matrix data is centred in time defining X sc ∈ R n×p as

for every h ∈ {1, . . . , p}. Then instead of computing the eigenvalue decomposition of the correlation matrix (X sc X sc ), we directly compute the SVD of X sc as

where U ∈ O(n × p), V ∈ O(p × p) and Σ = diag(σ 1 , . . . , σ p ). In this particular context, the PC matrix is commonly chosen as C = ( √ n -1) U , i.e., with a different scaling. Since the EOFs are affected by the latitude scaling, as well explained in [START_REF] Baldwin | Spatial Weighting and Iterative Projection Methods for EOFs[END_REF], the EOF matrix A ∈ R p×p is then constructed projecting the anomaly field not scaled X c into the space spanned by the PCs, that is

(5.7)

Remark 5.2.2. As stated in Remark 5.2.1, for the ease of the interpretation a the -th column of the EOF matrix A is usually reshaped into a matrix of shape (p 1 ×p 2 ). However in the correlation EOF analysis, instead of computing the EOF matrix, we can compute a tensor whose -th slice with respect to the first mode is the -th EOF already in matrix format. To do it, we firstly define f c ∈ R n×p 1 ×p 2 , i.e., we centre in time the input tensor data

and by construction the matricization with respect to the first mode of a is equal to the EOF matrix A, that is A (1) = A. In Figure 5.3 we display the PC and EOF relative error e δ and g δ on the left and right column respectively for every δ ∈ {0.5, 0.1, 10 -2 , 10 -3 , 10 -8 }. In all the plots of Figure 5.3, the relative error e δ and g δ have increasing component values, which implies that the first and most important PC and EOF are better approximated than the last ones. As we could expect, the more precise the approximations, i.e., the lower the values of δ, the smaller the PC and EOF relative errors are in general. As long as δ < 10 -2 , the PC relative error components are smaller than the EOF ones, as shown in Figures 5.3A, 5.3B, 5.3C and 5.3D. Moreover the h-th component of both e δ and g δ reaches and stagnates around 1, the relative error maximum, for h > 100 if δ = 0.5 and for h > 400 if δ = 0.1. When δ = 10 -2 , the first 200 components of g δ appear, in Figures 5.3E and 5.3F, smaller than the correspondent ones of e δ , that is the first EOF are better approximated than the first PC. Moreover δ = 10 -2 is the only case where the components of e δ and g δ appear to be always increasing. The Figures 5.3G and 5.3I, 5.3H and 5.3J look similar, as we could expect since the two multilinear rank differ of one unit only on the first component. In these last two cases, we see that for h ≥ 400, the components of the PC and EOF relative error stagnate around 10 -6 .

Concluding remarks

The focus of this chapter is on the investigation of climate datasets through tensor methods. In particular, the Tucker format is introduced in the EOF analysis, studying the benefit and the flaws of this choice. We show theoretically that the EOF and PC matrix can be computed from the Tucker factors and core tensor taking advantage of the compressed format, under the strong assumption that climate data are made available in Tucker format. To complete our study, we approximate a temperature time series through the HOSVD algorithm prescribing the accuracy and we study the effects of compression in the computation of the PC and EOF matrix.

The first part, which includes Sections 5.2 and Section 5.3, collects the theoretical results. In Section 5.2, after describing the mathematical data modelling and the considered pretreatments, we present the EOF analysis. The purpose of this technique is to separate the space and time information of the data into the PC and the EOF matrix respectively. For completeness, Section 5.2 includes the EOF analysis based on the correlation and the covariance matrix associated with the data. The computational aspects of this numerical linear algebra includes additional compression steps to prevent memory issues. Firstly, we report on their behavior in terms of loss of orthogonality of the computed basis. Then we plug them into the TT-version of the SUBSPace ITeration eigensolver (TT-SUBSPIT), analysing their effect in terms of eigenvalue computation.

Concluding remarks

The variable accuracy approach allows us to show experimentally in Chapter 2 that the backward stability of GMRES theoretically proved in [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF], still holds when the computational and the storage accuracies are distinguished. This means that the attainable accuracy of the backward error of GMRES appears of the order of the maximum between the computational and the storage accuracy, even when the assumptions of the theoretical proofs of [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] are violated. This attainable accuracy property is observed numerically also for TT-GMRES. Our experiments lead us to conclude that our TT-GMRES algorithm is more robust than its previous relaxed version proposed in [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF]. Finally, the theoretical bounds, which we formulate when many tensor linear systems are solved at once increasing the space order, appear from our numerical experiments sharp enough to guarantee some practical applicability.

The first contribution of Chapter 3 is the realization of the Householder orthogonalization kernel in TT-format, which has never been presented previously to the best of our knowledge. Additionally, the numerical experiments of Chapter 3 suggest that the loss of orthogonality in each orthogonalization scheme satisfies the same bounds [START_REF] Björck | Solving linear least squares problems by Gram-Schmidt orthogonalization[END_REF][START_REF] Giraud | Rounding error analysis of the classical Gram-Schmidt orthogonalization process[END_REF][START_REF] Smoktunowicz | A note on the error analysis of classical Gram-Schmidt[END_REF][START_REF] Stathopoulos | A Block Orthogonalization Procedure with Constant Synchronization Requirements[END_REF][START_REF] Wilkinson | The algebraic eigenvalue problem[END_REF] theoretically proved in the matrix case, where the unit round-off of the current working arithmetic is replaced by the compression accuracy. Lastly, the experimental results from the TT-SUBSPIT algorithm appear to recommend the TT-Householder transformation as the most robust one for every computation.

Perspectives

A central key point to address in the future is the theoretical validation of the backward stability of GMRES in the variable accuracy approach. Indeed, when the storage perturbations are controlled only norm-wise the backward stability theorems of [START_REF] Drkosova | Numerical stability of GMRES[END_REF][START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] do not apply anymore, since the underlying floating point representation is not holding anymore. However, the analysis presented in [START_REF] Paige | Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES[END_REF] is based on the worst-case. So, other possible proving techniques could be considered, such as those presented in [START_REF] Carson | A New Analysis of Iterative Refinement and Its Application to Accurate Solution of Ill-Conditioned Sparse Linear Systems[END_REF][START_REF] Paige | Residual and Backward Error Bounds in Minimum Residual Krylov Subspace Methods[END_REF]. Similarly, the loss of orthogonality bounds for the six kernels should be studied theoretically. Moreover, studying the quality of the TT-eigenvectors as much as the relationship between the cluster distance and the rounding accuracy would complete the analysis of the TT-SUBSPIT algorithm. Finally, to link the two chapters, we could try to implement the flexibility described in [START_REF] Palitta | On the convergence of Krylov methods with lowrank truncations[END_REF] with the basis re-orthogonalization, to keep the memory advantages of the relaxed TT-GMRES from [START_REF] Dolgov | TT-GMRES: solution to a linear system in the structured tensor format[END_REF] without losing the attainable accuracy of our version.

Data Analysis

The data analysis techniques investigated in Part II are meant for contingency tables and climate data. Their extension to the tensor format relies on the Tucker model [START_REF] Tucker | Implications of factor analysis of three way matrices for measurements of change[END_REF], described in Section 1.3.1.

The central subject of Chapter 4 is Correspondence Analysis (CA) from the geometric point of view. This technique is a variation of Principal Component Analysis (PCA) meant for investigating and visualizing frequency tables, referred to as contingency tables. The purpose of CA is to find the subspace that minimizes the projection error of the cloud points, whose coordinates are the row and the column entries of the contingency table. The projection of the two point clouds can be visualized and interpreted together, thanks to the barycentric relation that links them. The results of Chapter 4 integrate the MultiWay Correspondence Analysis (MWCA) theory, which generalizes the CA through the Tucker format. To stress the benefits and the flaws of a tensor approach two examples of MWCA in comparison with the matrix CA over literature data [START_REF] Greenacre | Theory and Applications of Correspondence Analysis[END_REF][START_REF] Kroonenberg | Applied Multiway Data Analysis[END_REF] are presented. In the second half of Chapter 4, we apply the MWCA to investigate the Malabar dataset [START_REF] Auby | Malabar project: samples versus otus contingency tables[END_REF] relying on the barycentric relation. As for the other examples, the MWCA results are compared with the CA ones.

The Empirical Orthogonal Function (EOF) analysis is the subject studied in Chapter 5. This technique is a variation of the PCA tuned for climate data. The EOF analysis aims to separate time and space information, analysing them independently. After studying how to perform the EOF analysis when the data are available in Tucker format, we investigate numerically the effect of the Tucker compression on the EOF outcome, comparing them with the classical EOF analysis results.

Concluding remarks

In Chapter 4, we show through the Tucker model that the CA barycentric link holds theoretically also when tensors data are considered, i.e., in the MWCA. We stress mainly how to interpret geometrically this relation and how to use it in the visualization and interpretation of a multiway contingency table. So, we fulfil the gap in the MWCA, which has not been studied geometrically as far as we are aware. Moreover, we perform the analysis of an original dataset. In Chapter 5, we describe theoretically how to perform the EOF analysis starting from data expressed in Tucker format, without decompressing. In addition, our numerical studies suggest that the Tucker compression affects less the most important components of the final EOF outcome.

Perspectives

Both the CA and the EOF analysis could be regarded as visualization techniques relying on PCA. Nowadays, it is common to replace PCA with the Tucker model, which preserves some of the PCA properties. However, the optimality of the projection error is not among these. So a possible alternative research direction is represented by the