En Particulier Amina

Rabab

Arij

Joe Tarek

Mohamad Khayri

Ali Darwich

Keywords: Tensors, optimization algorithms, machine learning, clustering, Riemannian optimization, simultaneous matrix diagonalization, Gaussian-mixtures, complex optimization, manifolds

application to clustering in machine learning

Algorithmes d'optimisation pour le problème d'approximation des décompositions en rang tensoriel: application au clustering en apprentissage automatique Résumé

Les tenseurs sont une généralisation d'ordre supérieur des matrices. Ils apparaissent dans une myriade d'applications. La décomposition de rang de tenseur décompose le tenseur en une somme minimale de tenseurs simples de rang 1. En pratique, la présence de bruit dans les entrées du tenseur fait que le calcul d'une décomposition de petit rang approchée est plus pertinente que de son calcul exact. Ce problème est connu comme le problème d'approximation des décompositions en rang tensoriel. Dans cette thèse, nous étudions ce problème pour les tenseurs symétriques, c.à.d pour les tenseurs avec des entrées invariantes par les permutations d'indices. Nous considérons des tenseurs symétriques avec des valeurs complexes. Parsuite en utilisant le lien entre les tenseurs et les polynômes homogènes, ainsi que des techniques d'optimisation complexe, nous proposons une approche d'optimisation riemannienne et nous développons un algorithme de Newton riemannien et un algorithme de Gauss-Newton riemannien pour résoudre ce problème. Nous abordons également le problème de diagonalisation simultanée de matrices, qui est étroitement lié au problème de décomposition tensorielle. Nous considérons ce problème sous deux angles : la certification et l'approximation. Pour la première partie, nous développons une suite de type Newton à convergence quadratique locale, et nous proposons un teste de certification. Pour la deuxième partie, nous développons un algorithme de gradient conjugué riemannien qui calcule localement un faisceau de matrices simultanément diagonalisables approché. En combinant cet algorithme avec un problème linéaire de moindres carrés, nous introduisons un algorithme d'optimisation alterné qui calcule une approximation de la décomposition pour les tenseurs tridimensionnels, quand le rang d'approximation est supérieur à la dimension de deux premiers modes. Enfin, en se basant sur les deux approches : tenseurs symétriques et diagonalisation simultanée de matrices, nous abordons le problème de clustering en apprentissage automatique pour les modèles de mélanges de Gaussiènne sphériques. Nous utilisons ces méthodes pour implémenter la méthode des moments, afin de fournir un bon point initial pour l'algorithme de maximisation de vraissemblance. Mots-clés : Tenseurs, algorithmes d'optimisation, apprentissage automatique, clustering, optimisation riemannienne, diagonalisation simultanée de matrices, mélanges Gaussiènnes, optimisation complex, variétés différentielles.

Introduction

Tensors are multidimensional arrays. They constitute a powerful tool from multilinear algebra and have a central role in many important applications. The roots of tensor computation can be traced back to the end of the nineteenth century in differential calculus [START_REF] Curbastro | Résumé de quelques travaux sur les systèmes variables de fonctions associés a une forme différentielle quadratique[END_REF][START_REF] Reich | Die Entwicklung des Tensorkalküls : Vom absoluten Differentialkalkül zur Relativitätstheorie[END_REF]. Then they became essential in many fields of applications in conjunction with the widespread use of big datasets, where they showed high capability in extracting hidden structures in the data, outperforming in this regard matrix-based methods. Mathematically speaking, tensors are more than simply a data structure. Indeed, if f is a linear map on a K-vector space V onto another K-vector space V , i.e. f (α 1 x 1 + α 2 x 2 ) = α 1 f (x 1 ) + α 2 f (x 2 ), ∀x 1 , x 2 ∈ V, ∀α 1 , α 2 ∈ K, where the two vector spaces are of the same dimension then this map can be represented by a matrix of coordinates with respect to fixed basis on respectively V and V . Similarly, f is said to be multilinear map from V 1 × . . . × V d onto K, where V 1 , . . . , V d are K-vector spaces, if f is linear with respect to every variable x k in V k , ∀k ∈ {1, . . . , d}, and thus as in the linear case the multilinear map f is represented by an array of coordinates, once the basis of V k for k ∈ {1, . . . , d} have been fixed, such that the entries of this array depend on d indices. The tensor product of V 1 , . . . , V d denoted by V 1 ⊗ . . . ⊗ V d is defined by universal property such that any multilinear map f on V 1 × . . . × V d lift to a linear map on V 1 ⊗ . . . ⊗ V d . For simplicity, we will look at tensors as multidimensional arrays of data. The Canonical Polyadic Decomposition (CPD) of tensors is at the core of many applications such that Signal Processing and Machine Learning [START_REF] Cichocki | Tensor decompositions for signal processing applications : From two-way to multiway component analysis[END_REF], [START_REF] Nicholas D Sidiropoulos | Tensor decomposition for signal processing and machine learning[END_REF], Sensor array processing [START_REF] Sørensen | Coupled canonical polyadic decompositions and multiple shift invariance in array processing[END_REF], Chemometrics [START_REF] Bro | Parafac tutorial and applications[END_REF], Principal components analysis [START_REF] Jolliffe | Principal Component Analysis[END_REF], and recently in Deep Learning [START_REF] Oymak | Learning a deep convolutional neural network via tensor decomposition[END_REF][START_REF] Goel | A survey of methods for low-power deep learning and computer vision[END_REF][START_REF] Bacciu | Tensor decompositions in deep learning[END_REF]. It consists of expressing a given tensor as a sum of rank-1 indecomposables tensors. The tensor rank is by definition the smallest number of rank-1 tensors needed in the CPD to generate the tensor. The CPD is also known as the rank decomposition when the number of rank-1 components is equal to the rank of the tensor. In particular, symmetric tensors i.e. higher order generalization of symmetric matrices, can be decomposed as a linear combination of simple symmetric tensors of symmetric rank one. To illustrate the interest of tensor rank decomposition, let us present briefly two applications as motivational examples :

Example 1.0.1 -(Blind Source Identification). Blind source identification consists of recovering source signals from observed signals without knowing the recording environment. For instance, let us consider this problem for a sensor array consisting of n > 2 displaced but otherwise identical subarrays of l sensors i.e. ln := I sensors in total. The array output can be described by the following model

X = AS t + E,
where A ∈ C I×r is the global array response, S ∈ C m×r contains m snapshots of r sources, and E is an additive noise. Let J k ∈ C l×I be a row-selection matrix such that J k X ∈ C l×m is the k-th subarray for k ∈ {1, . . . , n}. Let X ∈ C l×m×n be the tensor of frontal slices (see 2 CHAPITRE 1 -Introduction Section 2.1) equal to the matrices J k X. This tensor has a unique rank-r decomposition (except of the elementary indeterminacies of scaling and permutation)

X r i=1 a 1 i ⊗ a 2 i ⊗ a 3 i ,
with the factor matrices L = [a 1 i ] 1≤i≤r ∈ C l×r , M = [a 2 i ] 1≤i≤r ∈ C m×r , N = [a 3 i ] 1≤i≤r ∈ C n×r . The generic uniqueness of rank-r decomposition of X stays valid even when the system is underdetermined i.e. the number of sources exceeds the number of sensors. Using the rank decomposition of X , J k A and the sources S can be identified as follows [START_REF] Sidiropoulos | Parallel factor analysis in sensor array processing[END_REF][START_REF] Cichocki | Tensor decompositions for signal processing applications : From two-way to multiway component analysis[END_REF] J k A = L diag(a 3 k,1 , . . . , a 3 k,r ), S = M.

Example 1.0.2 -(Gaussian mixtures). Suppose that we have a mixture of r Gaussian distributions with r spherical covariance matrices (i.e. each covariance matrix is equal to the identity matrix multiplied by a scalar) such that we aim to estimate the proportion w i , the mean µ i ∈ R n and the covariance matrix Σ i = σ 2 i I n of each Gaussian distribution within the mixture, for i ∈ {1, . . . , r}. Assume that we have enough number of samples N that allows us computing significant statistics. The following theorem of Hsu-Kakade [START_REF] Hsu | Learning mixtures of spherical gaussians : Moment methods and spectral decompositions[END_REF] can be used to implement an algorithm to find these latent variables.

Theorem 1.0.1. Assume r ≤ n. Let -σ2 be the smallest eigenvalue of E[(x -E[x]) ⊗ (x -E[x])] and v a corresponding unit eigenvector, -

M 1 = E[(v t (x -E[x])) 2 x], -M 2 = E[x ⊗ x] -σ2 I n , -M 3 = E[x ⊗ x ⊗ x] -n i=1 (M 1 ⊗ e i ⊗ e i + e i ⊗ M 1 ⊗ e i + e i ⊗ e i ⊗ M 1 )
, where (e i ) 1≤i≤n denotes the canonical basis of R n , and E is the expectation (can also called the mean). Then σ2 = r i=1 ω i σ 2 i and

M 1 = r i=1 ω i σ 2 i µ i , M 2 = r i=1 ω i µ i ⊗ µ i , M 3 = r i=1 ω i µ i ⊗ µ i ⊗ µ i .
This theorem tells us that the symmetric rank decomposition of the symmetric tensor M 3 contains information on the parameters w i and µ i that we want to find. We can notice in the first example the presence of noise in the model of the sensor array processing, and in the second example the symmetric tensor is built from empirical moments. In fact, this is often the case for many tensor decomposition applications, where the data from which the tensor is constructed is rarely free of noise. For this reason, the problem to solve in practice is to approximate the tensor by a tensor of low rank, rather than computing the exact rank decomposition. This consists of formulating the approximation problem as a cost function that measures the distance (in general by using the Frobenius norm) between the tensor to approximate and a tensor in the set of tensors with a bounded number of terms in their rank decompositions. In this thesis we first consider the low rank tensor approximation problem for symmetric tensors with complex coefficients, where we investigate the geometric structure of the constraint set and show how it is possible, when combining this with efficient computation tools, to develop concrete efficient approximation algorithms.

We then move to the simultaneous matrix diagonalization problem which is closely related to tensors. In fact, suppose that we have a pencil of matrices, a tensor of order three can be built by stacking the matrices of the pencil one after the other along the third dimension. Conversely, unfolding a three dimensional tensor along the third dimension yields a pencil of matrices (this can also generalized for higher order tensors). The rank decomposition of the tensor corresponds to a simultaneous diagonalization of the matrices in the pencil. The relation between these two representations is heavily exploited in the literature. We address the simultaneous matrix diagonalization problem as a certification and approximation problem and briefly address its connection to tensor rank approximation problem. Finally, we highlight the importance and the applicability of the proposed approaches throughout an interesting application in recovering hidden structure in spherical Gaussian mixtures.

CHAPITRE 1 -Introduction

In this thesis we use techniques from numerical optimization, tensor analysis, linear algebra, and differential geometry. Concretely, we utilize tools such as Riemannian optimization, manifolds, matrix decomposition, complex optimization, and several others.

We start with two new algorithms for the low rank tensor approximation problem for symmetric tensors with complex coefficients. The first algorithm is a Riemannian Newton algorithm with quadratic local convergence. The second algorithm is a Riemannian Gauss-Newton algorithm over Veronese manifolds. The local convergence of the aforementioned algorithm is not affected by large scale differences between rank-one components. The two algorithms show their competitiveness in comparison with other state-of-the-art methods in terms of accuracy and running time.

We then move to the simultaneous matrix diagonalization problem closely related to tensors, and discussing in this context the certification and the approximation problems. Indeed, for a pencil of simultaneously diagonalizable matrices, we introduce a Newton-type sequence that converges quadratically towards the solution when starting from an initial point verifying a sufficient condition. Moreover, we introduce a Riemannian conjugate gradient algorithm for the approximate simultaneous diagonalization of matrices problem, and use this algorithm to develop an algorithm for the low rank tensor approximation problem for three-dimensional tensors when the approximation rank is higher than the dimension of the two first modes.

Finally, we show how the studied approaches can be used in machine learning applications for instance in clustering problems. Mainly, when the dataset that we aim to cluster is obeying spherical Gaussian mixture distribution, we propose to apply the Expectation Maximization algorithm, with an initial point given by the method of moments and show the impact of this choice especially in term of accuracy in comparison with other state-of-the-art methods.

Context and literature review

In the sequel we state the main research axis that we will consider in this thesis equipped with an overview of some important related works.

Low rank symmetric tensor approximation problem A symmetric tensor T of order d and dimension n in

T d (C n ) = C n ⊗ • • • ⊗ C n
is a special case of tensors, where its entries do not change under any permutation of its d indices. We denote their set by S d (C n ). The symmetric tensor decomposition problem consists of decomposing a symmetric tensor T ∈ S d (C n ) into a linear combination of symmetric tensors of rank one i.e.

T = r i=1 w i v i ⊗ ... ⊗ v i d times , w i ∈ C, v i ∈ C n (1.1)
For a multilinear tensor, its decomposition as a minimal sum of tensor products of vectors is called the Canonical Polyadic Decomposition [START_REF] Frank | The expression of a tensor or a polyadic as a sum of products[END_REF]. We have a correspondence between S d (C n ) and the set of homogeneous polynomials of degree d in n variables denoted C[x 1 , . . . , x n ] d =: C[x] d . Using this correspondence, (1.1) is equivalent to express the homogeneous polynomial p associated to T as a sum of powers of linear forms, which is by definition the classical Waring decomposition i.e.

p = r i=1 w i (v i,1 x 1 + • • • + v i,n x n ) d , w i ∈ C, v i ∈ C n (1.2)
1.1 -Context and literature review 5

The smallest r such that this decomposition exists is by definition the symmetric rank of p denoted by rank s (p). Let d ≥ 3. The generic symmetric rank denoted by r g , is given by Alexander-Hirschowitz theorem [START_REF] Alexander | Polynomial interpolation in several variables[END_REF] as follows :

r g = 1 n n+d-1 d
for all n, d ∈ N, except for the following cases : (d, n) ∈ { [START_REF] Absil | An extrinsic look at the Riemannian Hessian[END_REF][START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF], (4,[START_REF] Absil | An extrinsic look at the Riemannian Hessian[END_REF], (4,4), (4,[START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF]}, where it should be increased by 1. We say that T is of subgeneric rank, if its rank rank s (T ) = r in (1.2) is strictly lower than r g . In this case, a strong property of uniqueness of the Waring decomposition holds [START_REF] Chiantini | On generic identifiability of symmetric tensors of subgeneric rank[END_REF], and the symmetric tensor T is called identifiable, unless in three exceptions which are cited in [START_REF] Chiantini | On generic identifiability of symmetric tensors of subgeneric rank[END_REF]Theorem 1.1], where there are exactly two Waring decompositions. This identifiability property forms an important key strength of Waring decomposition. It can explain why this decomposition problem appears in many applications for instance in the areas of mobile communications, in blind identification of under-determined mixtures, machine learning, factor analysis of k-way arrays, statistics, biomedical engineering, psychometrics, and chemometrics. See e.g. [START_REF] Comon | Tensor decompositions, state of the art and applications[END_REF][START_REF] Comon | Blind identification of under-determined mixtures based on the characteristic function[END_REF][START_REF] Lathauwer | A multilinear singular value decomposition[END_REF][START_REF] Smilde | Multi-way Analysis with Applications in the Chemical Sciences[END_REF] and references therein. The decomposition of the tensor is often used to recover structural information in the application problem.

The Symmetric Tensor Approximation problem (STA) consists of finding the closest symmetric tensor to a given symmetric tensor T ∈ S d (C n ), of low symmetric rank. Equivalently, for a given r ∈ N * , it consists of approximating a homogeneous polynomial p associated to a symmetric tensor T by an element in Σ r , where Σ r = {q ∈ C[x] d | rank s (q) ≤ r}, i.e.

(STA) min q∈Σr 1 2 ||p -q|| 2 d .

Since in many problems, the input tensors are often computed from measurements or statistics, they are known with some errors on their coefficients and computing an approximate decomposition of low rank often gives better structural information than the exact or accurate decomposition of the approximate tensor [START_REF] Allman | Identifiability of parameters in latent structure models with many observed variables[END_REF][START_REF] Anandkumar | Tensor decompositions for learning latent variable models[END_REF][START_REF] David | Algebraic geometry of Bayesian networks[END_REF].

For matrices, the best low rank approximation can be computed via Singular Value Decomposition (SVD). Higher Order Singular Value Decomposition (HOSVD) has been investigated to compute a multilinear rank approximation of a tensor [START_REF] Lathauwer | A multilinear singular value decomposition[END_REF][START_REF] Lathauwer | On the best rank-1 and rank-(R 1 , R 2 , . . ., R n ) approximation of higher-order tensors[END_REF][START_REF] Vannieuwenhoven | A new truncation strategy for the higher-order singular value decomposition[END_REF], this, in contrast to the matrix case, does not give the best multilinear rank approximation (see for instance inequality [START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF] in [START_REF] Kressner | Low-rank tensor completion by Riemannian optimization[END_REF]).

A classical approach for computing an approximate tensor decomposition of low rank is the socalled Alternating Least Squares (ALS) method. It consists of minimizing the distance between a given tensor and a low rank tensor by alternately updating the different factors of the tensor decomposition, solving a quadratic minimization problem at each step. See e.g. [START_REF] Carroll | Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart-Young decomposition[END_REF][START_REF] Chen | Maximum block improvement and polynomial optimization[END_REF][START_REF] Harshman | Foundations of the PARAFAC procedure : Models and conditions for an "explanatory" multi-modal factor analysis[END_REF][START_REF] Kolda | Tensor decompositions and applications[END_REF]. This approach is well-suited for tensor represented in T d (C n ) but it looses the symmetry property in the internal steps of the algorithm. The space in which the linear operations are performed is of large dimension n d compared to the dimension n+d-1 d of S d (C n ) when n and d grow. Moreover the convergence is slow [START_REF] Espig | On the convergence of alternating least squares optimisation in tensor format representations[END_REF][START_REF] Uschmajew | Local convergence of the alternating least squares algorithm for canonical tensor approximation[END_REF].

Other iterative methods such as quasi-Newton methods have been considered for low rank tensor approximation problems to improve the convergence speed. See for instance [START_REF] Hayashi | A new algorithm to solve Parafac-model[END_REF][START_REF] Paatero | The multilinear Engine-A Table-Driven, least squares program for solving multilinear problems, including the n-way parallel factor analysis model[END_REF][START_REF] Phan | Low complexity damped Gauss-Newton algorithms for CANDECOMP/PARAFAC[END_REF][START_REF] Berkant | Quasi-Newton methods on Grassmannians and multilinear approximations of tensors[END_REF][START_REF] Laurent | Optimization-based algorithms for tensor decompositions : Canonical polyadic decomposition, decomposition in rank-(l r , l r , 1) terms, and a new generalization[END_REF][START_REF] Tomasi | A comparison of algorithms for fitting the PARAFAC model[END_REF]. A Riemannian Gauss-Newton algorithm with trust region scheme was presented in [33], to approximate a given real multilinear tensor by one of low rank. The Riemannian optimization set is a Cartesian product of Segre manifolds (i.e. manifolds of real multilinear tensors of rank one). The retraction on the Segre manifold, called ST-HOSVD, is based on sequentially truncated HOSVD [START_REF] Kressner | Low-rank tensor completion by Riemannian optimization[END_REF][START_REF] Hackbusch | Tensor Spaces and Numerical Tensor Calculus[END_REF][START_REF] Vannieuwenhoven | A new truncation strategy for the higher-order singular value decomposition[END_REF]. Moreover, an algorithm, called hot restarts, was introduced in [33] to avoid ill-conditioned decompositions. Closely related to these iterative methods, the condition number of join decompositions such as tensor decompositions is studied in [32].

Optimization techniques based on quasi-Newton iterations for block term decompositions of multilinear tensors over the complex numbers have also been presented in [START_REF] Laurent | Unconstrained optimization of real functions in complex variables[END_REF][START_REF] Laurent | Optimization-based algorithms for tensor decompositions : Canonical polyadic decomposition, decomposition in rank-(l r , l r , 1) terms, and a new generalization[END_REF]. In [START_REF] Berkant | Quasi-Newton methods on Grassmannians and multilinear approximations of tensors[END_REF] quasi-Newton and limited memory quasi-Newton methods for distance optimization on products of Grassmannian manifolds are designed to deal with the Tucker decomposition of a tensor and applied for a low multilinear rank tensor approximation. In all these approaches, an approximation of the Hessian is used to compute the descent direction, and the local quadratic convergence cannot be guaranteed.

Specific investigations have been developed, in the case of best rank-1 approximation. The problem is equivalent to the optimization of a polynomial on the product of unitary spheres (see e.g. [START_REF] Lathauwer | On the best rank-1 and rank-(R 1 , R 2 , . . ., R n ) approximation of higher-order tensors[END_REF][START_REF] Xinzhen | The best rank-1 approximation of a symmetric tensor and related spherical optimization problems[END_REF]). Global polynomial optimization methods can be employed over the real or complex numbers, using for instance convex relaxations and semidefinite programming [START_REF] Nie | Semidefinite relaxations for best rank-1 tensor approximations[END_REF]. However, the approach is facing scalability issues in practice for large size tensors.

In relation with polynomial representation and multivariate Hankel matrix properties, another least square optimization problem is presented in [START_REF] Nie | Low rank symmetric tensor approximations[END_REF], for low rank symmetric tensor approximation. Good approximations of the low rank approximation are obtained for small enough perturbations of low rank tensors. More recently, a method for decomposing real even-order symmetric tensors, called Subspace Power Method (SPM), has been proposed in [START_REF] Kileel | Subspace power method for symmetric tensor decomposition and generalized PCA[END_REF]. It is based on a power method associated to the projection on subspaces of eigenvectors of the Hankel operators and has a linear convergence.

Simultaneous diagonalization of matrices Let us consider s diagonalizable matrices M 1 , • • • , M s in C n×n which pairwise commute. A classical result states that these matrices are simultaneously diagonalizable, i.e., there exists an invertible matrix E and diagonal matrices Σ i , 1 i s, such that EM i E -1 = Σ i , 1 i s, see e.g. [START_REF] Roger | Matrix analysis[END_REF]. Our objective is to compute numerically a solution (E, F, Σ) of the system of equations

f (E, F, Σ) := F E -I n F M E -Σ = 0 (1.3)
where Σ = (Σ 1 , . . . , Σ s ) and EM F -Σ := (EM 1 F -Σ 1 , . . . , EM s F -Σ s ). Notice that this system is multi-linear in the unknowns E, F, Σ. We verify that when s = 1 and M 1 is a generic matrix, this system has a solution set of dimension 2 n 2 -n 2 -(n 2 -n) = n. However, for s > 1 and generic matrices M i , there is no solution. To have a solution, the pencil M must be on the manifold of s-tuples of simultaneously diagonalizable matrices.

The system (1.3) can be generalized to the following system :

f (E, F, Σ ) := F M 0 E -Σ 0 F M E -Σ = 0 (1.4)
where Σ = (Σ 0 , Σ 1 , . . . , Σ s ), M 0 ∈ C n×n is replacing I n and Σ 0 is a diagonal matrix replacing I n in the first equation of (1.3). When the pencil M = (M 0 , M 1 , . . . , M s ) contains an invertible matrix, the solutions of the two systems are closely related. If M 0 is invertible, a solution (E, F, Σ ) of (1.4) for M = (M 0 , M 1 , . . . , M s ) gives the solution (F M 0 , EΣ -1 0 , ΣΣ -1 0 ) of (1.3) for M = (M -1 0 M 1 , . . . , M -1 0 M s ). A similar correspondence between the solution sets can be obtained if a linear combination M 0 = s i=1 λ i M i is invertible. As (1.4) can be seen as an homogeneization of (1.3) and appears in several contexts and applications, we will also study Newton-type methods for this homogenized system. To solve the system of equations (1.3), we propose to apply a Newton-like method and to analyze the Newton map associated to an iteration. These ideas have also been developed in the literature, for instance, in a technical report for the fast computation of the singular value decomposition [START_REF] Van Der Hoeven | Certified singular value decomposition[END_REF], in [START_REF] Mahony | The constrained newton method on a lie group and the symmetric eigenvalue problem[END_REF] where a Newton method is used for the symmetric eigenvalue problem.

We say that we have a quadratic sequence associated to a system of equations if the sequence converges quadratically towards a solution.

The classical Newton map defines (E + X, F + Y, Σ + S) from (E, F, Σ) in order to cancel the linear part in the Taylor expansion of f (E + X, F + Y, Σ + S). An easy computation shows that the perturbations X, Y and S are solutions of such a Sylvester-type linear system

F E -I n + F X + Y E F M E -Σ -S + XM F + EM Y = 0. (1.5)
The technical background to solve this linear system is the Kronecker product, see [START_REF] Horn | Topics in Matrix Analysis[END_REF]. In this way, the size of the linear system that one needs to invert is n 2 . The construction of the methods studied here is based on perturbations of such type (E(I n + X), (I n + Y )F, Σ + S) rather than (E + X, F + Y, Σ + S). More precisely the perturbations X, Y and S that we consider are perturbations which cancel the linear part of the Taylor expansion of f (E(I n + X), (I n + Y )F, Σ + S). In this case, we can produce explicit solutions for the linear system in X, Y and S given by :

Z + X + Y ∆ -S + ΣX + Y Σ = 0. (1.6)
where Z = F E -I n and ∆ = F M E -Σ. We will see that the linear system (1.6) admits an explicit solution (X, Y, S) with respect to Z and ∆ for s = 1, 2 in (1.3). This is because Σ is a diagonal matrix. From these considerations, we define and analyze a sequence that converges quadratically towards a solution of the system (1.3) without inverting a linear system at each step of this Newton-like method. Simultaneous matrix diagonalization is required by many algorithms as it was pointed out in [START_REF] Bunse-Gerstner | A chart of numerical methods for structured eigenvalue problems[END_REF]. There is quite a body of literature on exactly or approximately commuting matrices and exact or approximate joint diaonalization, see for instance [START_REF] Košir | On stability of invariant subspaces of commuting matrices[END_REF][START_REF] Van Der Veen | An analytical constant modulus algorithm[END_REF]. A numerical analysis for two normal commuting matrices is proposed in [START_REF] Bunse-Gerstner | Numerical methods for simultaneous diagonalization[END_REF] using Jacobi-like methods. Their method adjusts the classical Jacobi method in successively solving n(n-1) 2 two-real-variables optimization problems at each sweep of the algorithm. Their main result states a local quadratic convergence and can be summarized in the following way. Let off 2 (A, B) 2 = i =j |A i,j | 2 + |B i,j | 2 . Let {α 1 , . . . , α n } (resp. {β 1 , . . . , β n }) be the set of the eigenvalues of A (resp. B). Let A k and B k the matrices obtained at the step k of the Jacobi-like method and

ρ k = off 2 (A k , B k ). If ρ 0 < 1 2 δ := 1 4 min i =j (|α i -α j |, |β i -β j |), then ρ k+1 < 2n(9n -13) ρ 2 k δ .
We will see in Theorems 5.1.5 and 5.1.8 that the local conditions of the quadratic convergence do not depend on n. Many other papers studied the so-called Jacobi-like methods (see e.g. [START_REF] Luciani | Canonical polyadic decomposition based on joint eigenvalue decomposition[END_REF], [START_REF] Mesloub | Efficient and stable joint eigenvalue decomposition based on generalized givens rotations[END_REF] and references therein).

CHAPITRE 1 -Introduction

In [START_REF] Hoeven | Efficient certification of numeric solutions to eigenproblems[END_REF] a sequence with proof of its convergence towards a numerical solution of the system (1.3) when s = 1 i.e. for M 1 , with the assumption of M 1 being a diagonalizable matrix, is presented. It requires matrix inversion. Furthermore, under some extra assumptions, its quadratic convergence is established.

Simultaneous matrix diagonalization appears in many applications. For instance, in the solution of multivariate polynomial equations by algebraic methods, the isolated roots of the system are obtained from the computation of common eigenvectors of commuting operators of multiplication in the quotient ring and from their eigenvalues [START_REF] Cox | Using Algebraic Geometry[END_REF], [START_REF] Elkadi | Introduction à la résolution des systèmes polynomiaux[END_REF]. In the case of simple roots, this reduces to simultaneous diagonalization of a pencil of matrices. Further, simultaneous matrix diagonalization is used for blind source separation, direction of arrival estimation, multi-dimensional harmonic retrieval, Canonical Polyadic Decomposition (CPD), econometric (see e.g. [START_REF] Boudjellal | Separation of dependent autoregressive sources using joint matrix diagonalization[END_REF], [START_REF] Van Der Veen | Azimuth and elevation computation in high resolution doa estimation[END_REF], [START_REF] Haardt | Simultaneous schur decomposition of several nonsymmetric matrices to achieve automatic pairing in multidimensional harmonic retrieval problems[END_REF], [START_REF] De Lathauwer | Computation of the canonical decomposition by means of a simultaneous generalized schur decomposition[END_REF], [START_REF] Bonhomme | Nonparametric estimation of non-exchangeable latent-variable models[END_REF] and references therein).

The approximate simultaneous diagonalization problem aims to approximate locally a pencil of matrices to a pencil of simultaneously diagonalizable matrices. This problem is widely studied in the literature for a pencil of real symmetric matrices C = (C 1 , . . . , C s ), in particular several algorithms based on Riemannian optimization methods (see [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF]) have been developed in order to find an approximate joint diagonalizer for the pencil C (see e.g. [START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF][START_REF] Absil | Joint diagonalization on the oblique manifold for independent component analysis[END_REF][START_REF] Rahbar | Geometric optimization methods for blind source separation of signals[END_REF][START_REF] Joho | Joint diagonalization of correlation matrices by using Newton methods with application to blind signal separation[END_REF]). The idea is to find a local minimizer B ∈ R n×n of an objective function f which measures the degree of nondiagonality of the pencil (BC 1 B T , . . . , BC s B T ) over a Riemannian manifold (see [START_REF] Wang | Penalty function-based joint diagonalization approach for convolutive blind separation of nonstationary sources[END_REF][START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF][START_REF] Afsari | Sensitivity analysis for the problem of matrix joint diagonalization[END_REF] for some examples of objective functions). This Riemannian manifold is defined according to the geometric constraints considered on B. For instance, the diagonalizer is supposed to be orthogonal in some of these algorithms after a pre-whitening step (see e.g. [START_REF] Cardoso | Blind beamforming for non gaussian signals[END_REF][START_REF] Cardoso | Jacobi angles for simultaneous diagonalization[END_REF][START_REF] Flury | An algorithm for simultaneous orthogonal transformation of several positive definite symmetric matrices to nearly diagonal form[END_REF][START_REF] Rahbar | Geometric optimization methods for blind source separation of signals[END_REF][START_REF] Douglas | Self-stabilized gradient algorithms for blind source separation with orthogonality constraints[END_REF][START_REF] Joho | Joint diagonalization of correlation matrices by using Newton methods with application to blind signal separation[END_REF][START_REF] Nikpour | Algorithms on the Stiefel manifold for joint diagonalisation[END_REF][START_REF] Nishimori | Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold[END_REF]). Due to inaccuracies in the computation of the diagonalizer with orthogonality constraints (see. [START_REF] Yeredor | Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation[END_REF]), oblique constraints, i.e. all the rows of the diagonalizer have unit Euclidean norm, have also been considered instead of the former constraints in more recent works (see e.g. [START_REF] Absil | Joint diagonalization on the oblique manifold for independent component analysis[END_REF][START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF]).

The approach of approximate joint diagonalizer for a pencil of real symmetric matrices is used to solve Blind Source Separation (BSS) problem, with potential applications in wide domains of engineering (see e.g. [55]). In more general context, when the matrices of the pencil M to approximate are general square matrices, there exists algorithms to find an invertible matrix E such that (EM 1 E -1 , . . . , EM s E -1 ) is the most diagonal. The majority of these algorithms is based on Jacobi-like method (see e.g. [START_REF] Luciani | Joint eigenvalue decomposition of non-defective matrices based on the lu factorization with application to ica[END_REF][START_REF] Fu | Simultaneous diagonalization with similarity transformation for nondefective matrices[END_REF][START_REF] Iferroudjene | A new jacobi-like method for joint diagonalization of arbitrary non-defective matrices[END_REF]). Nevertheless, some other approaches have been addressed this problem. In [START_REF] Vollgraf | Quadratic optimization for simultaneous matrix diagonalization[END_REF] the authors split the optimization problem into a sequence of simpler second order subproblems, and present an algorithm that works with no restriction on the transformation matrix E. More recently, the authors in [START_REF] Riku | Approximate simultaneous diagonalization of matrices via structured low-rank approximation[END_REF] present an algorithm based on two main steps. The first step approximates the pencil to nearly simultaneously diagonalizable pencil of matrices and this by solving a structured low-rank approximation problem. The second step computes a transformation matrix that diagonalizes exactly the pencil of simultaneously diagonalizable matrices obtained from the first step. One advantage for this approach over the other optimization methods regarding this problem, is that it has a guaranty to find an exact common diagonalizer if the pencil to approximate is already simultaneously diagonalizable.

Simultaneous matrix diagonalization of pencils of matrices appears in the Canonical Polyadic Decomposition (CPD) of tensors [START_REF] De | A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization[END_REF]. Under certain conditions the rank decomposition (i.e. the CP decomposition with number of rank-1 simple tensors equal to the rank of the tensor), is unique 1.2 -Context and literature review 9 [START_REF] Nicholas | On the uniqueness of multilinear decomposition of n-way arrays[END_REF]. In this case simultaneous matrix diagonalization allows to compute the rank decomposition. Direct methods based on simultaneous diagonalisation of matrices built from slices of tensors have been investigated for 3rd order multilinear tensors, e.g. in [START_REF] Harshman | Foundations of the PARAFAC procedure : Models and conditions for an "explanatory" multi-modal factor analysis[END_REF][START_REF] Sanchez | Tensorial resolution : A direct trilinear decomposition[END_REF][START_REF] Leurgans | A Decomposition for Three-Way Arrays[END_REF]73,[START_REF] De Lathauwer | Independent component analysis and (simultaneous) third-order tensor diagonalization[END_REF] or for multilinear tensors of rank smaller than the lowest dimension in [START_REF] De | A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization[END_REF][START_REF] Luciani | Canonical polyadic decomposition based on joint eigenvalue decomposition[END_REF][START_REF] De Lathauwer | Computation of the canonical decomposition by means of a simultaneous generalized schur decomposition[END_REF]. For the low rank tensor approximation problem (i.e. approximation of the tensor to CP decomposition of lower rank) via simultaneous matrix diagonalization see for instance [START_REF] De Lathauwer | Computation of the canonical decomposition by means of a simultaneous generalized schur decomposition[END_REF][START_REF] Roemer | A semi-algebraic framework for approximate cp decompositions via joint matrix diagonalization and generalized unfoldings[END_REF]. In his proof on lower bounds of tensor ranks, Strassen showed in [START_REF] Strassen | Rank and optimal computation of generic tensors[END_REF]Theorem 4.1] that a 3rd order multilinear tensor is of rank r if it can be embedded into a tensor with slices of rank r matrices, which are simultaneously diagonalizable. The CPD of tensors plays a crucial role in numerous applications such that Psychometric [START_REF] Carroll | Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition[END_REF], signal processing and machine learning [START_REF] Cichocki | Tensor decompositions for signal processing applications : From two-way to multiway component analysis[END_REF], [START_REF] Nicholas D Sidiropoulos | Tensor decomposition for signal processing and machine learning[END_REF], sensor array processing [START_REF] Sørensen | Coupled canonical polyadic decompositions and multiple shift invariance in array processing[END_REF], arithmetic complexity [START_REF] Bürgisser | Algebraic complexity theory[END_REF], wireless communications [START_REF] Sørensen | Blind multichannel deconvolution and convolutive extensions of canonical polyadic and block term decompositions[END_REF], multidimensional harmonic retrieval [START_REF] Sørensen | Multidimensional harmonic retrieval via coupled canonical polyadic decomposition-part i : Model and identifiability[END_REF], [START_REF] Sørensen | Multidimensional harmonic retrieval via coupled canonical polyadic decomposition-part ii : Algorithm and multirate sampling[END_REF], Chemometrics [START_REF] Bro | Parafac tutorial and applications[END_REF], and Principal components analysis [START_REF] Jolliffe | Principal Component Analysis[END_REF].

Tensor decomposition for learning Gaussian mixtures from moments With the relatively recent evolutions of information systems over the last decades, many observations, measurements, data are nowadays available on a variety of subjects. However, too much information can kill the information and one of the main challenges remains to analyse and to model these data, in order to recover and exploit hidden structures.

To tackle this challenge, popular Machine Learning technologies have been developed and used successfully in several application domains (e.g. in image recognition [START_REF] He | Deep residual learning for image recognition[END_REF]). These techniques can be grouped in two main classes : Supervised machine learning techniques are approximating a model by optimising the parameters of an enough general model (e.g. a Convolution Neural Network) from training data. Unsupervised machine learning techniques are deducing the parameters characterising a model directly from the given data, using an apriori knowledge on the model. The supervised approach requires annotated data, with a training step that can introduce some bias in the learned model. The unsupervised approach can be applied directly on a given data set avoiding the costly step of annotating data, but the quality of the output strongly depends on the type of models to be recovered.

We consider the latter approach and show how methods from effective algebraic geometry help finding hidden structure in data that can be modelled by mixtures of Gaussian distributions. The algebraic-geometric tool that we consider is tensor decomposition. It consists in decomposing a tensor into a minimal sum of rank-1 tensors. This decomposition generalises the rank decomposition of a matrix, with specific and interesting features. Contrarily to matrix rank decomposition, the decomposition of a tensor is usually unique (up to permutations and scaling) when the rank of the tensor, that is the minimal number of rank-1 terms in a decomposition, is small compared to the dimension of the space(s) associated to the tensor (see for instance [START_REF] Chiantini | An algorithm for generic and low-rank specific identifiability of complex tensors[END_REF][START_REF] Chiantini | Effective criteria for specific identifiability of tensors and forms[END_REF][START_REF] Chiantini | On generic identifiability of symmetric tensors of subgeneric rank[END_REF]). Such a tensor is called identifiable. This property is of particular importance when the decomposition is used to recover the parameters of a model. It guaranties the validity of the recovering process and its convergence when the number of data increases.

In [START_REF] Hsu | Learning mixtures of spherical gaussians : Moment methods and spectral decompositions[END_REF], symmetric tensor decompositions for moment tensors are studied for spherical Gaussian mixtures. Moment methods have been further investigated for Latent Dirichlet Allocation models, topic or multiview models in [START_REF] Anandkumar | Tensor decompositions for learning latent variable models[END_REF][START_REF] Majid Janzamin | Spectral Learning on Matrices and Tensors[END_REF]. In [START_REF] Ruffini | Clustering patients with tensor decomposition[END_REF], a tensor decomposition technique based on Alternate Least Squares (ALS) is used to initialise the Expectation Maximisation (EM) algorithm [START_REF] Hastie | The Elements of Statistical Learning[END_REF][START_REF] Murphy | Machine learning : a probabilistic perspective[END_REF][START_REF] Xu | On Convergence Properties of the EM Algorithm for Gaussian Mixtures[END_REF], for a mixture of discrete distributions (which are not Gaussian distributions). An overview of tensor decomposition methods in Machine Learning can be found in [START_REF] Rabanser | Introduction to Tensor Decompositions and their Applications in Machine Learning[END_REF].

Contributions

In this section we summarize our main contributions in this thesis.

Low rank symmetric tensor approximation problem In Chapter 4, we present two Riemannian Newton-type algorithms for the low rank tensor approximation problem (STA) for symmetric tensors with complex coefficients.

• The first algorithm is a Riemannian Newton algorithm (Section 4.2). We use the parametrization of the set of tensors of rank at most r by weights and vectors on the unit sphere.

Exploiting the properties of the apolar product on homogeneous polynomials combined with efficient tools from complex optimization, we provide an explicit and tractable formulation of the Riemannian gradient and Hessian, leading to Newton iterations with local quadratic convergence. We prove that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the iteration (completed with a trust-region scheme) is converging to a local minimum (Proposition 4.2.13).

• The second algorithm is a Riemannian Gauss-Newton method on the Cartesian product of the manifolds of symmetric rank-1 tensors called Veronese manifolds (Section 4.2.2). We describe an explicit orthonormal basis of the tangent space of this Riemannian manifold. We use this basis to obtain the Riemannian gradient and the Gauss-Newton approximation of the Riemannian Hessian. We present an approximation method for a given homogeneous polynomial in C[x] d into linear form to the d th power, based on the rank-1 truncation of the SVD of Hankel matrix associated to the homogeneous polynomial. From this approximation method, we propose a new retraction operator on the Veronese manifold. The design of the algorithm depends on the geometry of the Veronese manifold and its tangent space. In this context, the Riemannian Gauss-Newton iteration that we describe is adapted to the symmetric setting by considering the reduced vector space C[x] d and by exploiting the apolar identities. In our approach we consider symmetric tensors with complex coefficients. The constraint set is parameterized via the complex Veronese manifolds, which leads us to a complex optimization problem with geometric constraints, and this, to the best of our knowledge, has not been addressed previously in tensor approximation.

• We analyze the numerical behavior of these methods, choosing for the initial point the approximate decomposition provided by the Simultaneous Matrix Diagonalisation (SMD) of a pencil of Hankel matrices [START_REF] Harmouch | Structured low rank decomposition of multivariate Hankel matrices[END_REF][START_REF] Mourrain | Polynomial-exponential decomposition from moments[END_REF] (Section 4.3.1). Numerical experiments (Section 4.3) show the good numerical behavior of the new methods for the best rank-1 approximation of real-valued symmetric tensors, for low rank approximation of sparse symmetric tensors, and against perturbations of symmetric tensors of low rank. Comparisons with existing state-of-the-art methods corroborate this analysis.

Simultaneous diagonalization of matrices

In Chapter 5, we investigate the simultaneous diagonalization of matrices from three points of view : Certification, approximation, and relation with tensor rank decomposition.

• In Section 5.1, our contributions are a new iteration for the simultaneous diagonalization of matrices, with a local quadratic convergence and its analysis. The iteration is different from a Newton iteration. It does not require to invert a large linear system, but performs simple matrix operations. We analyse the numerical behavior of the method and provide a certification test for the convergence. Sections 5.1.2, 5.1.3, 5.1.4, and 5.1.5 are devoted to respectively constructing a sequence to solve numerically :

-F E -I n = 0, -the system (1.3) when s = 1, -the system (1.4) when s = 1, -the system (1.3) for any s. Moreover, we provide for these cases, a certification that the sequence converges to a nearby solution, and a test to detect when this convergence is quadratic from an initial point. More precisely, in Section 5.1.3 we show that a triplet (E 0 , F 0 , Σ 0 ) must satisfy a property depending on the quantity

ε 0 := max(κ 2 0 K 2 0 Z 0 , κ 2 0 K 0 ∆ 0 ) to get a quadratic convergence where 1-Z 0 = E 0 F 0 -I n , 2-∆ 0 = E 0 M F 0 -Σ 0 , 3-κ 0 = max 1, max 1 j<k n 1 |σ 0,k -σ 0,j | , 4-K 0 = max k 1, |σ 0,k | ,
such that σ 0,1 , . . . , σ 0,n denote the diagonal entries of Σ 0 . The quantity κ is the condition number of the studied methods. Based on the same methodology of Section 5.1.3, in Sections 5.1.4 and 5.1.5 we exhibit a certification of the convergence of the sequence constructed to the studied case towards the solution with a sufficient condition on the initial point. In Section 5.1.6 we perform numerical experimentation to corroborate the theoretical analysis.

• In Section 5.2, we address the problem of approximate simultaneous diagonalization of matrices. We consider a pencil of square matrices M = [M 1 , . . . , M s ] without imposing any restrictions on the matrices to be diagonalized approximately, regarding their symmetry or definiteness. Unlike the common familiar case where one transformation or diagonalizer E is considered, we look in more general context, for two transformation matrices E and F that diagonalize approximately the pencil M in such a way that F M k E t is the most diagonal for k ∈ {1, . . . , s}. This structure allows for an implementation of an algorithm for the tensor rank approximation problem of three-dimensional tensors with approximation rank higher than the first two dimensional sizes (Section 5.3). The approximation problem is formulated as a minimization problem of a cost function that measures the norm of the off-diagonal terms of the pencil's matrices. Moreover, restrictions on E and F are imposed to be in the oblique manifold in order to avoid undesirable singular points.

Hence, the problem is formulated as a non-linear least squares problem over the Cartesian product of two oblique manifolds. We develop a Riemannian conjugate gradient algorithm to solve this problem. To develop our algorithm, we essentially use the framework presented recently in [START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF] regarding the Riemannian gradient-based optimization methods over the oblique manifold.

• In Section 5.3, we present a new algorithm for the low rank tensor approximation problem of trilinear tensors of size (n 1 , n 2 , n 3 ) with approximation rank r ≥ max(n 1 , n 2 ). We consider the pencil of matrices associated to the tensor to be approximated that contains the slices of this tensor obtained by the flattening according to the third mode. Then we extend the matrices of this pencil of size n 1 × n 2 to matrices of size r × r. The algorithm then works on the pencil of the extended matrices. At each iteration, the algorithm alternates between two steps. The first step finds E and F that diagonalize approximately the extended pencil, and this by using the Riemannian conjugate gradient algorithm for approximate simultaneous diagonalization of matrices (Section 5.2). The second step updates the entries of the extdended part in each matrix of the pencil by solving a linear least-squares problem. At the end an approximated rank-r decomposition is obtained for an extended tensor of size (r, r, n 3 ), from which we extract an approximated rank-r decomposition for the original tensor of size (n 1 , n 2 , n 3 ).

Tensor decomposition for learning Gaussian mixtures from moments Chapter 6 is the chapter where we show one application of symmetric tensors in machine learning, by addressing the clustering problem of a dataset following a spherical Multivariate Gaussian Mixture (MGM) distribution. Indeed, this done by using the method of moments. It has been shown in [START_REF] Chiantini | On generic identifiability of symmetric tensors of subgeneric rank[END_REF] that for symmetric tensors, if the rank of the tensor is strictly less than the rank r g of a generic tensor of the same size, then the tensor is generically identifiable, except in three cases. We show in Section 6.2.5 a more specific result : for a symmetric tensor T having a decomposition with r points, if the Hankel matrix associated to T in a degree strictly bigger than the degree of interpolation of the r points is of rank r, then the tensor is identifiable. We show in Section 6.2.3, that under some assumption on the spherical Gaussian mixtures, a tensor of moments of order 3 of the distribution is identifiable and its decomposition allows to recover the parameters of the Gaussian mixture. For symmetric tensor decomposition, a method based on flat extension of Hankel matrices or commutation of multiplication operators has been proposed in [START_REF] Brachat | Symmetric tensor decomposition[END_REF] and extended to multi-symmetric tensors in [START_REF] Bernardi | General tensor decomposition, moment matrices and applications[END_REF]. This approach is closely related to the simultaneous diagonalisation of tensor slices, but follows a more algebraic perspective. Eigenvectors of symmetric tensors have been used to compute their decompositions in [START_REF] Oeding | Eigenvectors of tensors and algorithms for waring decomposition[END_REF]. In [START_REF] Harmouch | Structured low rank decomposition of multivariate Hankel matrices[END_REF], Singular Value Decomposition and eigenvector computation are used to decompose a symmetric tensor, when its rank is smaller than the smallest size of its Hankel matrix in degree less than half the order of the tensor. In Section 6.2, we describe a new algorithm, involving Singular Value Decomposition and simultaneous diagonalisation, to compute the decomposition of an identifiable tensor, which interpolation degree is smaller that half the order of the tensor. In Section 6.3, we apply the method of moments [START_REF] Hsu | Learning mixtures of spherical gaussians : Moment methods and spectral decompositions[END_REF] for recovering Gaussian mixtures and show, throughout examples on synthetic and real datasets, its impact on providing good initialisation point in the Expectation Maximization algorithm, in comparison with other state-of-the-art approaches.

Main notation

In this section we introduce some of the notation we use throughout the thesis. The superscripts . t , . * and . -1 are used respectively for the transpose, Hermitian conjugate, and the inverse matrix. Let M ∈ C m×n , we denote by A the Frobenius norm of A given by M :=

n i=1 n j=1 |m i,j | 2 . Let A ∈ C n×n , We denote by √ A a matrix B ∈ C n×n such that A = B 2 .
The complex conjugate is denoted by an overbar, e.g., w. We use parentheses to denote vectors e.g. W = (w i ) 1≤i≤r , and the square brackets to denote matrices e.g. V = [v i ] 1≤i≤r where v i 1.4 -Organization of the thesis are column vectors. The concatenation of vectors v 1 , v 2 , . . . is denoted (v 1 ; v 2 ; . . .). The trace of a matrix A is denoted by tr(A). For a matrix M ∈ C n×n , let ddiag(M ) be the diagonal matrix with the same diagonal as M and let off(M ) be the matrix where the diagonal term of M are replaced by 0. We have M = ddiag(M ) + off(M ).We say that M is an off-matrix if M = off(M ). In addition, diag(M) returns a vector containing the diagonal entries of M , and for (λ 1 , . . . , λ n ) ∈ C n , diag(λ 1 , . . . , λ n ) is the diagonal matrix in C n×n of diagonal entries λ 1 , . . . , λ n , and for a family of matrices (A i ) 1≤i≤n , diag(A i ) 1≤i≤n is the matrix with diagonal blocks A i .

Organization of the thesis

The thesis is divided in three parts : Preliminaries, Contributions, and Conclusions and Perspectives.

Preliminaries

This part contains the background needed in the contributions part, with no original contributions.

• Chapter 2 : We introduce a review on tensors given the main notion and information in this regard. • Chapter 3 : We introduce the necessary concepts of Riemannian optimization.

Contributions

This part contains our contributions in this thesis.

• Chapter 4 : This chapter contains our contributions to low rank symmetric tensor approximation problem.

-Section 4.1 : We describe the set of non-defective rank-r symmetric tensors.

-Section 4.2 : This section contains the Riemannian Newton algorithm and the Riemannian Gauss-Newton algorithm for the STA problem -Section 4.3 : This section carries out numerical experiments. -Section 4.3 : This section contains our conclusions in this chapter.

• Chapter 5 : This chapter contains our contributions to simultaneous matrix diagonalization problem.

-Section 5.1 : This section contains the Newton-type methods for simultaneous matrix diagonalization problem. -Section 5.2 : This section contains the Riemannian conjugate gradient algorithm for the approximate simultaneous diagonalization of matrices problem. -Section 5.3 : This section contains the algorithm for the low rank tensor approximation problem for real three-dimensional tensors with approximation rank larger than the size of the first two modes. -Section 5.4 : This section contains our conclusions in this chapter.

• Chapter 6 : This chapter contains our contributions to tensor decomposition for learning Gaussian mixtures from moments.

-Section 6.1 : In this section we review Gaussian mixtures and moments methods.

-Section 6.2 : This section contains an algebraic symmetric tensor decomposition method for identifiable tensors.

• The contributions of Section 5.1 were a joint work with Jean-Claude Yakoubsohn and Bernard Mourrain, the content is in the paper : Rima Khouja, Bernard Mourrain, Jean-Claude Yakoubsohn. Newton-type methods for simultaneous matrix diagonalization. In preparation, 2022, submitted to Calcolo journal, and it is under revision. https://hal.archivesouvertes.fr/hal-03390265.

Preliminaries

CHAPTER 2

Tensors

Tensors can be seen as a generalization of matrices to multiple dimensions. Formally, a tensor of order d is an element of the tensor product of d vector spaces. It can be represented by a multidimensional array of numerical values from a field like R or C with respect to fixed basis on the vector spaces. Indeed, a scalar is a tensor of order zero, a vector is a tensor of order one and a matrix is a tensor of order two. Researchers pay high attention to tensors due to their applications in wide fields such as signal processing, numerical linear algebra, computer vision, numerical analysis, neuroscience, chemometrics, linguistics and more. In particular, the popularity of data science and machine learning has grown considerably during the last decade, and tensors have expanded to these domains to provide the required high computing capacities. For instance, tensors appear in the name of the popular Google's machine learning package "Tensorflow", since they are counted from the main tools used in this package, where they provide a convenient data format that allows applying very efficient operations in order to extract important information within the dataset and also to build strong predictive models. There is a number of comprehensive and important reviews concerning tensors and their applications. We mention for instance : Kolda et al. [START_REF] Kolda | Tensor decompositions and applications[END_REF], Bro [START_REF] Bro | Parafac tutorial and applications[END_REF], Sidiropoulos et al. [START_REF] Nicholas D Sidiropoulos | Tensor decomposition for signal processing and machine learning[END_REF], Cichoki et al. [START_REF] Cichocki | Tensor decompositions for signal processing applications : From two-way to multiway component analysis[END_REF], and Rabanser et al. [START_REF] Rabanser | Introduction to Tensor Decompositions and their Applications in Machine Learning[END_REF].

Introduction to tensors

Unless otherwise stated, in this chapter tensors are assumed real valued. Let R n 1 ⊗ . . . ⊗ R n d denotes the vector space of the tensor product of R n 1 , . . . , R n d . The vector space R n 1 ⊗ . . . ⊗ R n d is defined by the universal property [START_REF] Bourbaki | Algebra I : Chapters 1-3, Elements of Mathematics[END_REF] such that any multilinear map

f on R n 1 × . . . × R n d lift to a linear map on R n 1 ⊗ . . . ⊗ R n d .
An element A in the vector space of d-way arrays denoted by R n 1 ×...×n d can be written in the

u ⊗ v = [u i 1 u i 2 ] 1≤i 1 ≤n 1 ,1≤i 2 ≤n 2 .
This produces a matrix i.e. a 2-way array, thus the extension of this notion to the outer product of d vectors yields a d-way array, and more generally the outer product of a d-way array with a k-way array produces a d + k-way array. The so-called Segre map given by

ϕ : R n 1 × . . . × R n d → R n 1 ×...×n d (2.1) (a 1 , . . . , a d ) → a 1 ⊗ . . . ⊗ a d := [a 1 i 1 . . . a d i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d ,
is a multilinear map, thus by the universal property of the tensor product [START_REF] Bourbaki | Algebra I : Chapters 1-3, Elements of Mathematics[END_REF] there exists a unique linear map θ such that the following diagram commutes :

R n 1 ⊗ . . . ⊗ R n d R n 1 × . . . × R n d R n 1 ×...×n d . θ ϕ Since dim(R n 1 ×...×n d ) = dim(R n 1 ⊗ . . . ⊗ R n d ), θ is an isomorphism. Consider the canonical basis of R n 1 ⊗ . . . ⊗ R n d : e ( 1
)
i 1 ⊗ • • • ⊗ e (d) i d | 1 ≤ i 1 ≤ n 1 , . . . , 1 ≤ i d ≤ n d ,
where e ( ) 1 , . . . , e

( ) n

denotes the canonical basis in R n , = 1, . . . , d. Then θ may be described by

θ   n 1 ,...,n d i 1 ,...,i d =1 a i 1 ,...,i d e (1) i 1 ⊗ • • • ⊗ e (d) i d   = a i 1 •••i d n 1 ,...,n d i 1 ,...,i d =1 ∈ R n 1 ×...×n d .
Hence, a multilinear tensor of order or mode d and size

(n 1 , . . . , n d ) in R n 1 ⊗ . . . ⊗ R n d can be represented by a d-way array in R n 1 ×...×n d up to a choice of basis on R n 1 , . . . , R n d .
Thus, without loss of generality, we refer hereafter to a tensor in

R n 1 ⊗. . .⊗R n d by an element A = [a i 1 ,...,i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d in R n 1 ×...×n d ,
and we note that all the notion that we will work on in this chapter and throughout this thesis are independent from the choice of basis of the vector spaces.

We also mention that we can associate to a tensor A a multilinear polynomial of total degree d in the variables

x 1 = (x 1 1 , . . . , x n 1 1 ), . . . , x d = (x 1 d , . . . , x n d d ) of the form a i 1 ,...,i d x i 1 1 . . . x i d d , 1 ≤ i 1 ≤ n 1 , . . . , 1 ≤ i d ≤ n d .
This bijection between the set of tensors and the set of multilinear polynomials is important since it allows to study tensors from an algebraic geometric point of view (see for instance [START_REF] Landsberg | Tensors : Geometry and Applications[END_REF]). Some basics. We can create sub-dimensional sections of a tensor by fixing some of the tensor's indices. Fibers are the higher-order generalization of matrix rows and columns. They are obtained by fixing all the indices of the tensor but one. Slices of a tensor are obtained by fixing all the indices of the tensor except of two. In particular, if A is a three-dimensional tensor, then the fibers A[:, j, k], A[i, :, k] and A[i, j, :] are called respectively column, row and tube fibers. Further, the slices A[i, :, :], A[:, j, :] and A[:, :, k] are called respectively horizontal, lateral and frontal slices. The inner product of A = [a i 1 ,...,i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d and B = [b i 1 ,...,i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d such that A and B are in R n 1 ×...×n d is given by :

A, B = n 1 i 1 =1 . . . n d i d =1 a i 1 ,...,i d b i 1 ,...,i d .
The Frobenius norm of A is as follows :

A = A, A = n 1 i 1 =1
. . .

n d i d =1 |a i 1 ,...,i d | 2 .
It is analogous to the Frobenius matrix norm.

A tensor A ∈ R n 1 ×...×n d is called of rank one if it can be written as the outer product of d vectors as follows :

A = a 1 ⊗ . . . ⊗ a d ,
where

a k ∈ R n k , ∀1 ≤ k ≤ d.
We can notice that the set of rank-one tensors (also called simple tensors) is given by the image of the Segre map

ϕ : R n 1 × . . . × R n d → R n 1 ×...×n d (a 1 , . . . , a d ) → a 1 ⊗ . . . ⊗ a d = [a 1 i 1 . . . a d i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d .

Some types of tensors. A tensor

A ∈ R n 1 ×...n d is diagonal if a i 1 ,...,i d = 0 only if i 1 = . . . = i d .
A tensor is called cubical if each mode is of the same size i.e. n 1 = . . . = n d = n. We denote by

T d n (R) = R n ⊗ • • • ⊗ R n the outer product d times of R n . A cubical tensor A in T d n (R
) is said to be symmetric if each entries remains unchanged under any permutation of its d indices i.e.

a i 1 ...i d = a i σ(1) ...i σ(d) , ∀σ ∈ S d ,
where S d denotes the symmetric group of permutations on {1, . . . , d}. In particular, a tensor can be partially symmetric and it is called multi-symmetric. For example, a tensor A ∈ R n×n×s is symmetric in modes one and two if

a i 1 i 2 i 3 = a i 2 i 1 i 3 , ∀ 1 ≤ i 1 , i 2 ≤ n, ∀ 1 ≤ i 3 ≤ s,
in other words, all the frontal slices are symmetric matrices.

Since low rank symmetric tensor approximations constitutes a relevant part of our contributions in this thesis (Chapter 4), the next section is devoted to mention notation and definitions related to symmetric tensors, that will be used later in the contributions part.

Symmetric tensors

In this section, tensors are considered with complex coefficients. Let p be the associate homogeneous polynomial to A in C[x] d . This means that p can be written as the linear form given by the vector a to the d th power i.e.

p = (a t x) d . Definition 2.2.1. For p = |α|=d d α p α x α and q = |α|=d d α q α x α in C[x] d , their apolar pro- duct is p, q d := |α|=d d α pα q α .
Obviously, the apolar product for real symmetric tensors p and q in R[x] d is given by p,

q d = |α|=d d α p α q α . The apolar norm of p is ||p|| d = p, p d = |α|=d d α pα p α .
The apolar product is invariant by a linear change of variables of the unitary group U n : ∀u ∈ U n , p(u x), q(u x) d = p(x), q(x) d . The following properties of the apolar product can be verified by direct calculus :

Lemma 2.2.1. Let l = (v 1 x 1 + • • • + v n x n ) d := (v t x) d ∈ C[x] d where v = (v i ) 1≤i≤n is a vector in C n , q ∈ C[x] (d-1)
, we have the following two properties :

1. l, p d = p(v), p, l d = p(v), 2. p, x i q d = 1 d ∂ x i p, q d-1 , x i q, p d = 1 d q, ∂ x i p d-1 , ∀1 ≤ i ≤ n.
These properties are called the apolar identities.

Tensor reorderings : vectorization and matricization

Let A = [a i 1 ,...,i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d ∈ R n 1 ×...×n d .
• The vectorization of A is to stacking vertically all the entries of A in one vector.

vec(A) = [a 1,...,1 a 2,1,...,1 . . . a n 1 ,...,n d-1 ,n d -1 a n 1 ,...,n d ] t .

-Important matrix and tensor products

• Let k ∈ {1, . . . , d}. The kth mode of matricization or flattening of A is a rearrangement of the entries of A into a matrix A (k) such that the kth mode becomes the row index and all the other (d -1) modes become column indices, i.e. A (k) has n k rows and 1≤j =k≤d n j columns. More precisely, the matricization of mode k maps an element of the tensor A of index (i 1 , . . . , i d ) into an element of the matrix A (k) of index (i k , j) such that

j = 1 + d l=1 l =k (i l -1)I l with I l = l-1 m=1 m =k n m .
Example 2.3.1 -Let A be a tensor in R 3×2×4 such that its frontal slices are as follows :

A 1 =    1 4 2 5 3 6   , A 2 =    7
(A × k U ) i 1 ,...,i k-1 ,i,i k+1 ,...,i d = n k i k =1 a i 1 ,...,i k-1 ,i k ,i k+1 ,...,i d u i,i k .
This is nothing else but the multiplication of each mode-k fiber with the matrix U . Alternatively, the k-mode product can be expressed in terms of matricized tensors as follows :

Y = A × k U ⇔ Y (k) = U A (k) .
There is no matter concerning the order of multiplication i.e.

A × k U × l V = A × l V × k U (k = l), CHAPITRE 2 -Tensors
and when k = l, we have :

A × k U × k V = A × k (V U ).
The k-mode product also exists for tensors with vectors. Let v ∈ R n k , the kmode product of A with v denoted by A × k v is a tensor of order d -1 and size (n 1 , . . . , n k-1 , n k+1 , . . . , n d ). Element-wise, this can be expressed as follows :

(A × k v) i 1 ,...,i k-1 ,i k+1 ,...,i d = n k i k =1 a i 1 ,...,i k-1 ,i k ,i k+1 ,...,i d v i k .
• The Kronecker product of A ∈ R n 1 ×n 2 with B ∈ R m 1 ×m 2 is denoted by A ⊗ B and yields a matrix in R n 1 m 1 ×n 2 m 2 such that :

A ⊗ B =       a 1,1 B a 1,2 B . . . a 1,n 2 B a 2,1 B a 2,2 B . . . a 2,n 2 B . . . . . . . . . . . . a n 1 ,1 B a n 1 ,2 B . . . a n 1 ,n 2 B       = a 1 ⊗ b 1 a 1 ⊗ b 2 a 1 ⊗ b 3 . . . a n 2 ⊗ b m 2 -1 a n 2 ⊗ b m 2 ,
where a 1 , . . . , a n 2 (resp. b 1 , . . . , b m 2 ) denote the columns of the matrix A (resp. B). More precisely

(A ⊗ B) (i 1 -1)m 1 +j 1 ,(i 2 -1)m 2 +j 2 = a i 1 ,i 2 b j 1 ,j 2 .
We have the following property [START_REF] Kolda | Multilinear operators for higher-order decompositions[END_REF] which relates the Kronecker product of matrices to the k-mode product of tensors as follows :

Let A k ∈ R I k ×n k , Y = A× 1 A 1 . . .× d A d ⇔ Y (k) = A k A (k) (A d ⊗. . . A k-1 ⊗A k+1 ⊗. . .⊗A d ) t , ∀ 1 ≤ k ≤ d. (2.
2) • The Khatri-Rao product is the column-wise Kronecker product. For two matrices of the same number of columns A ∈ R n 1 ×m and B ∈ R n 2 ×m , their Khatri-Rao product denoted by A B is a matrix of size n 1 n 2 × m defined as follows :

A B = a 1 ⊗ b 1 a 2 ⊗ b 2 . . . a m ⊗ b m .
In particular, the Kronecker product of two vectors is identical to their Khatri-Rao product. • The Hadamard product is the element-wise matrix product. For two matrices A and B of the same size n 1 × n 2 , the Hadamard product is denoted by A * B, and it is given by :

A * B =       a 1,1 b 1,1 a 1,2 b 1,2 . . . a 1,n 2 b 1,n 2 a 2,1 b 2,1 a 2,2 b 2,2 . . . a 2,n 2 b 2,n 2 . . . . . . . . . . . . a n 1 ,1 b n 1 ,1 a n 1 ,2 b n 1 ,2 . . . a n 1 ,n 2 b n 1 ,n 2       .

Canonical Polyadic Decomposition

The Canonical Polyadic Decomposition (CPD) proposed by Hitchcock [START_REF] Frank | The expression of a tensor or a polyadic as a sum of products[END_REF], express the tensor as a sum of rank-one simple tensors in the form

A = r i=1 a 1 i ⊗ . . . ⊗ a d i , (2.3)
where r is a positive integer and

a k i ∈ R n k , ∀ 1 ≤ i ≤ r, ∀ 1 ≤ k ≤ d.
The factor matrices are given by :

A k = [a k 1 . . . a k r ] ∈ R n k ×r , ∀1 ≤ k ≤ d.
Thus, using the factor matrices the CPD decomposition can be expressed as follows

A = A 1 , . . . , A d = r i=1 a 1 i ⊗ . . . ⊗ a d i .
In particular, if A is a three-dimensional tensor, the frontal slices of A can be expressed in terms of factor matrices denoted by A, B and C as follows

A k = A diag(C[k, :])B t , ∀ 1 ≤ k ≤ n 3 .
This expression does not easily extend to tensors of order higher than three. In general, it is recommended from computation perspectives to normalize the columns of the factor matrices, by using a weight vector λ = (λ i ) 1≤i≤r ∈ R r so that

A = λ, A 1 , . . . , A d = r i=1 λ i a 1 i ⊗ . . . ⊗ a d i .
Equivalently, A can be expressed as a multilinear product with a diagonal tensor

A = D × 1 A 1 . . . × d A d ,
where

D is a diagonal tensor in T d r with D i,...,i = λ i , ∀ 1 ≤ i ≤ r. It follows from (2.
2) that the kth mode of matricization of A is given by

A (k) = A k Λ(A d . . . A k-1 A k+1 . . . A 1 ) t ,
where Λ = diag(λ). The Kronecker product in (2.2) is replaced here by the Hadamard product since all the factor matrices A k have the same number of columns equal to r.

Similarly, symmetric tensors can be expressed as a sum of symmetric rank-one tensors i.e. for A ∈ S

d n A = r i=1 w i a i ⊗ ... ⊗ a i d times , w i ∈ R, a i ∈ R n .
(2.4)

CHAPITRE 2 -Tensors

Equivalently, the homogeneous polynomial p associated to A is written as a sum of powers of linear forms i.e.

p = r i=1 w i (a i,1 x 1 + • • • + a i,n x n ) d , w i ∈ R, a i ∈ R n .
(2.5)

Tensor rank

The rank of a tensor A denoted by rank(A) is the minimal number of rank-one tensors needed in the CPD decomposition in (2.3) to generate the tensor A. The CPD decomposition with r equal to the rank of the tensor is called the rank decomposition. The tensor rank is not affected by mode permutation.

Similarly, the symmetric tensor rank of a symmetric tensor A denoted by rank s (A) is the minimal number of symmetric rank-one tensors in a decomposition of the form (2.5).

The decomposition (2.5) is called Waring decomposition when r is equal to the symmetric rank of the symmetric tensor. As a direct property, we have rank s (A) ≥ rank(A), since the constraint on the rank-one tensors to be symmetric may increase the rank. Nevertheless, Comon et al. show in [54] that equality holds when rank s (A) = 1, 2, and holds generically when rank s (A) ≤ n and when d is sufficiently large with respect to n. More recently, Shitov shows a counterexample in [START_REF] Shitov | A counterexample to comon's conjecture[END_REF] for a symmetric tensor in S 3 800 (C), that has rank strictly less than its symmetric rank. The rank may depend on the field, where it may be different over R and C. This idea is demonstrated throughout two examples : The first (can be found in [START_REF] Kolda | Tensor decompositions and applications[END_REF]) for a tensor A ∈ R 2×2×2 and the second for a symmetric tensor A ∈ S 3 2 (R) (from [54]). Example 2.5.1 -Let A ∈ R 2×2×2 such that the frontal slices are given by :

A 1 = 1 0 0 1 , A 2 = 0 1 -1 0 .
This tensor is of rank 3 over R and rank 2 over C. The factor matrices for the rank decomposition over R are as follows :

A = 1 0 1 0 1 -1 , B = 1 0 1 0 1 1 , C = 1 1 0 -1 1 1 ,
and over C are as follows :

A = 1 √ 2 1 1 -i i , B = 1 √ 2 1 1 i -i , C = 1 1 i -i .
Example 2.5.2 -Let A ∈ S 3 2 (R) such that the frontal slices are given by :

A 1 = -1 0 0 1 , A 2 = 0 1 1 0 .
This tensor is of symmetric rank 3 over R :

A = 1 2 1 1 ⊗3 + 1 2 1 -1 ⊗3 -2 1 0 ⊗3 ,
and symmetric rank 2 over C :

A = i 2 -i 1 ⊗3 - i 2 i 1 ⊗3 .
The maximum rank is the largest attainable rank. For instance, a week upper bound on the maximum rank that can be reached by a three-dimensional tensor in R n 1 ×n 2 ×n 3 is established in [START_REF]Rank, Decomposition, and Uniqueness for 3-Way and n-Way Arrays[END_REF] rank(A) ≤ min{n 1 n 2 , n 1 n 3 , n 2 n 3 }.

For example, it has been shown in the same reference [START_REF]Rank, Decomposition, and Uniqueness for 3-Way and n-Way Arrays[END_REF] that the maximum rank for 3 × 3 × 3 tensors over R is equal to 5.

Two other important notions in term of tensor rank is the typical rank and the generic rank.

The typical rank is any rank that occurs with probability greater than zero, whereas the generic rank is the rank which is true almost everywhere (i.e. with probability one), when their entries are chosen independently according to a continuous probability distribution.

For tensors over R there may be more than one typical rank, for example 2 × 2 × 2 tensors over R have two typical rank 2 and 3 [START_REF]Rank, Decomposition, and Uniqueness for 3-Way and n-Way Arrays[END_REF]. However, the smallest typical rank over R is the generic rank over C [START_REF] Blekherman | On maximum, typical, and generic ranks[END_REF], and thus the generic rank of C 2×2×2 is 2. When there is only one typical rank, then it may be called the generic rank. There is always a single typical rank over C, and is thus generic. The expected generic rank for real or complex tensors of size (n 1 , . . . , n d ) [START_REF] Blekherman | On maximum, typical, and generic ranks[END_REF] is given by

r g = n 1 n 2 . . . n d n 1 + n 2 + . . . + n d -d + 1 . (2.6)
The generic symmetric rank for real or complex symmetric tensors of order d and dimension n is given by Alexander-Hirschowitz theorem [START_REF] Alexander | Polynomial interpolation in several variables[END_REF] as follows

r g = 1 n n + d -1 d , ( 2.7)
except for when (d, n) ∈ {(3, 5), (4, 3), (4, 4), (4, 5)}, where it should be increased by 1.

For further discussion concerning typical and generic rank see for instance [START_REF] Blekherman | On maximum, typical, and generic ranks[END_REF]54,[START_REF] Chiantini | An algorithm for generic and low-rank specific identifiability of complex tensors[END_REF][START_REF] Chiantini | Effective criteria for specific identifiability of tensors and forms[END_REF].

The last type of tensor rank that we aim to present in this section is the border rank. A tensor A is of border rank r if it is the limit of tensors of rank r but not the limit of tensors of rank smaller than r. Rank and border rank of a tensor may actually be different, let us take the following example from [START_REF] Comon | Tensors : A brief introduction[END_REF] to illustrate this idea :

Example 2.5.3 -Let ε > 0 and u, v be two no-collinear vectors such that

A ε = 1 ε (u + εv) ⊗3 -u ⊗3 = A + O(ε), where A = u ⊗ u ⊗ v + u ⊗ v ⊗ u + v ⊗ u ⊗ u.
The tensor A is of rank 3 whereas its border rank is equal 2, since A is the limit of rank-2 tensors (A ε ) ε>0 .

Remark 2.5.1 -Unlike the matrix case, there is no straightforward algorithm to compute the rank of a given tensor as the problem is NP hard [START_REF] Håstad | Tensor rank is np-complete[END_REF].

Uniqueness

Let A ∈ R n 1 ×...×n d of rank r with rank decomposition given by

A = A 1 , . . . , A d = r i=1 a 1 i ⊗ . . . ⊗ a d i . (2.8)
Uniqueness means that the rank decomposition (2.8) is unique up to the elementary indeterminacies of scaling and permutation. In this case, the tensor A is called r-identifiable.

The scaling undeterminacy is due to the fact that we can scale vectors such that

A = r i=1 (λ i,1 a 1 i ) ⊗ . . . ⊗ (λ i,d a d i ), with λ i,1 . . . λ i,d = 1, ∀1 ≤ i ≤ r.
The permutation undeterminacy is due to the fact that rank-1 component tensors can be ordered arbitrarily such that

A = A 1 , . . . , A d = A 1 P, .
. . , A d P , for any r × r permutation matrix P.

One can notice that rank decomposition for matrices is not unique. Indeed, let A ∈ R n 1 ×n 2 be of rank r with rank decomposition

A = r i=1 a i ⊗ b i = AB t .
Let U ΣV t be the SVD decomposition of A. We can take A = U Σ and B = V , thus equivalently for some r × r orthogonal matrix W , it is possible to obtain a completely different rank-1 matrices (i.e. the rank-1 components in the rank decomposition of the matrix A) by taking A = U ΣW , B = V W . We remark that the SVD decomposition of a matrix is unique (for instance assuming that all the singular values are distinct) but this is only because of the strong condition on the matrices U and V (orthogonal matrices) with the diagonal matrix Σ of ordered singular values in the middle. Nevertheless, the uniqueness in the case of higher order tensors can be obtained with much weaker conditions. For instance, a classical sufficient condition for uniqueness for third order tensors is due to Kruskal [START_REF] Kruskal | Three-way arrays : rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics[END_REF]

k A + k B + k C ≥ 2r + 2,
where k M of a matrix M denotes its k-rank given by the maximum value k such that any k columns from M are linearly independent. This condition is extended to the general case of order-d tensors in [START_REF] Nicholas | On the uniqueness of multilinear decomposition of n-way arrays[END_REF] as follows

d i=1 k A i ≥ 2r + (d -1).
More generally, Liu and Sidiropoulos showed in [START_REF] Liu | Cramer-rao lower bounds for low-rank decomposition of multidimensional arrays[END_REF] the following necessary conditions for uniqueness On the other hand some conditions for generic uniqueness are established. For instance, Delathauwer has proved in [START_REF] De | A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization[END_REF] that the rank decomposition in (2.8) for third respectively fourth order tensors is generically (i.e. with probability one) unique if for third order tensors r ≤ n 3 and r(r

min k=1,...,d rank(A 1 . . . A k-1 A k+1 . . . A d ) = r,
-1) ≤ 1 2 n 1 (n 1 -1)n 2 (n 2 -1),
respectively for fourth order tensors r ≤ n 4 and r(r

-1) ≤ 1 4 n 1 n 2 n 3 (3n 1 n 2 n 3 -n 1 n 2 -n 1 n 3 -n 2 n 3 -n 1 -n 2 -n 3 + 3).
Further, Chiantini et al. investigate in [START_REF] Chiantini | An algorithm for generic and low-rank specific identifiability of complex tensors[END_REF][START_REF] Chiantini | Effective criteria for specific identifiability of tensors and forms[END_REF] the generic uniqueness or generic identifiability of tensors in C n 1 ×...×n d , where they show that, for instance, a generic tensor A ∈ C n 1 ×...×n d of subgeneric rank (i.e. of rank strictly lower than the expected generic rank in (2.6)) is r-identifiable if d k=1 n k ≤ 15000 unless some exceptions (see [START_REF] Chiantini | An algorithm for generic and low-rank specific identifiability of complex tensors[END_REF]Theorem 1]). They also studied this problem for symmetric tensors where they proved that the Waring decomposition of a generic symmetric tensor in S d n (C) of subgeneric rank (i.e. of rank strictly lower than the generic rank in (2.7)) is unique unless in three exceptions [START_REF] Chiantini | On generic identifiability of symmetric tensors of subgeneric rank[END_REF]Theorem 1.1]. For a thorough study of uniqueness and generic uniqueness see the aforementioned references in this section as well the references therein.

Exact computation

There exists in the literature various algorithms for exact computation of the rank decomposition. The approaches used to develop these algorithms are essentially based on linear algebra by solving sets of linear equations and computing generalized eigenvalue decomposition. We provide for instance some pointers among many others e.g. [START_REF] Sanchez | Tensorial resolution : A direct trilinear decomposition[END_REF]73,[START_REF] Jos | Kruskal's polynomial for 2×2×2 arrays and a generalization to 2×n×n arrays[END_REF][START_REF] Phan | Candecomp/parafac decomposition of high-order tensors through tensor reshaping[END_REF], and for the Waring decomposition of symmetric tensors e.g. [START_REF] Bernardi | Computing symmetric rank for symmetric tensors[END_REF][START_REF] Brachat | Symmetric tensor decomposition[END_REF][START_REF] Oeding | Eigenvectors of tensors and algorithms for waring decomposition[END_REF][START_REF] Harmouch | Structured low rank decomposition of multivariate Hankel matrices[END_REF]. For further algorithms for the rank decomposition based on simultaneous or joint diagonalization of matrices see e.g. [START_REF] De | A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization[END_REF][START_REF] De Lathauwer | Independent component analysis and (simultaneous) third-order tensor diagonalization[END_REF][START_REF] Sørensen | Canonical polyadic decomposition with a columnwise orthonormal factor matrix[END_REF][START_REF] Luciani | Canonical polyadic decomposition based on joint eigenvalue decomposition[END_REF][START_REF] Domanov | Canonical polyadic decomposition of thirdorder tensors : Relaxed uniqueness conditions and algebraic algorithm[END_REF].

Low rank tensor approximation problem

In practice measurements are rarely free of noise, consequently the rank decomposition of the tensor built from these measurements is rarely exact. For this reason, one often prefers to approximate the given tensor by a low rank tensor decomposition. This is the so-called low rank tensor approximation problem. It aims to best approximate a given tensor A ∈ R n 1 ×...×n d by a rank decomposition with number of components equal to the approximation rank r. More concretely, this can be done by minimizing the Frobenius norm between the given tensor A and the decomposition of rank r

min  A - , with  = r i=1 a 1 i ⊗ . . . ⊗ a d i = A 1 , . . . , A d . ( 2
A -λ 1 a 1 1 ⊗ . . . ⊗ a d 1 -• • • -λ r a 1 r ⊗ . . . ⊗ a d
r , such that the vectors a j i for 1 ≤ i ≤ r, and 1 ≤ j ≤ d are normalized or uniformly bounded, then at least some of the coefficients λ i become unbounded, i.e. the magnitude of some terms in the decomposition go to infinity. This phenomenon is referred to as degeneracy. More precisely, when there are k diverging terms such that their sum is bounded is called k-factor degeneracies [START_REF] Wim P Krijnen | On the non-existence of optimal solutions and the occurrence of "degeneracy" in the candecomp/parafac model[END_REF][START_REF] Jb Kruskal | How 3-mfa data can cause degenerate parafac solutions, among other relationships[END_REF][START_REF] Paatero | Construction and analysis of degenerate parafac models[END_REF][START_REF] Stegeman | Degeneracy in candecomp/parafac explained for p × p × 2 arrays of rank p + 1 or higher[END_REF][START_REF] Stegeman | Low-rank approximation of generic p×q×2 arrays and diverging components in the candecomp/parafac model[END_REF], this phenomenon also exists for symmetric tensors [54]. Any algorithm that minimizes the cost function in (2.9) will yield a degeneracy if this problem does not have an optimal solution (i.e. its infimum is not reached). Nevertheless, there exists in the literature a family of approaches to avoid degeneracies by imposing constraints in the CPD, like for instance imposing orthogonality between columns of factor matrices [55], bounding the coefficients λ i [START_REF] Paatero | Construction and analysis of degenerate parafac models[END_REF][START_REF] Lim | Nonnegative approximations of nonnegative tensors[END_REF], imposing a minimal angle between columns of factor matrices [START_REF] Lim | Blind multilinear identification[END_REF], imposing, when the tensor has positive entries, the non negativity on the components of the CP decomposition (i.e. that is impose on the entries of the rank-1 tensors to be positive) [START_REF] Lim | Nonnegative approximations of nonnegative tensors[END_REF] (for more references in this regard see [START_REF] Comon | Tensors : A brief introduction[END_REF] and references therein). However, in practice this problem remains open since there is no fully satisfactory solution.

Algorithms

We come now to methods for solving the low rank approximation problem. Several algorithms were proposed for this purpose in the literature. One of the common approaches is to parameterize the set of tensors of rank bounded by r using factor matrices and then to formulate the low rank approximation problem as an unconstrained optimization problem as follows min

(A 1 ,...,A d )∈D 1 2 A 1 , . . . , A d -A 2 , (2.10) with D := R n 1 ×r × . . . × R n d ×r , and A 1 , . . . , A d = r i=1 a 1 i ⊗ . . . ⊗ a d i .
General optimization methods such as, for instance, alternate least squares [START_REF] Carroll | Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart-Young decomposition[END_REF][START_REF] Richard A Harshman | FOUNDATIONS OF THE PARAFAC PROCEDURE : MODELS AND CONDITIONS FOR AN[END_REF][START_REF] Chen | Maximum block improvement and polynomial optimization[END_REF], conjugate gradient [START_REF] Paatero | The multilinear Engine-A Table-Driven, least squares program for solving multilinear problems, including the n-way parallel factor analysis model[END_REF] and quasi-Newton methods [START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF][START_REF] De | An adaptive algebraic multigrid algorithm for low-rank canonical tensor decomposition[END_REF][START_REF] Hayashi | A new algorithm to solve parafac-model[END_REF][START_REF] Paatero | A weighted non-negative least squares algorithm for three-way 'parafac' factor analysis[END_REF][START_REF] Phan | Low complexity damped Gauss-Newton algorithms for CANDECOMP/PARAFAC[END_REF][START_REF] Laurent | Optimization-based algorithms for tensor decompositions : Canonical polyadic decomposition, decomposition in rank-(l r , l r , 1) terms, and a new generalization[END_REF][START_REF] Tomasi | A comparison of algorithms for fitting the PARAFAC model[END_REF] are widely used in the literature to solve this problem. Alternate least squares (ALS) methods solve sequentially simple linear subproblems, where at each step the cost function is optimized for one of the factor matrices while the other factor matrices are considered fixed, which leads to solve overdetermined sets of linear equations. Despite its simplicity, the method has some drawbacks. For instance, it has a slow convergence especially in the presence of illconditioned cases [START_REF] Martin | Musings on multilinear fitting[END_REF][START_REF] Meisam Razaviyayn | A unified convergence analysis of block successive minimization methods for nonsmooth optimization[END_REF]. It has been shown that numerical optimization methods like conjugate gradient methods and quasi-Newton methods outperform the ALS-like strategies in this case [START_REF] Paatero | The multilinear Engine-A Table-Driven, least squares program for solving multilinear problems, including the n-way parallel factor analysis model[END_REF][START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF], though this comes at a higher computational cost per iteration. The aforementioned methods differ from each other for instance in terms of performance, computation complexity, and in how they address the ill-conditioning problem which appears in the iterative algorithms as the degeneracy phenomenon that we described previously and yields numerical difficulties. Let us mention one complication in the Gauss-Newton method which is a quasi-Newton method used in this context in many efficient algorithms e.g. [START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF][START_REF] Hayashi | A new algorithm to solve parafac-model[END_REF][START_REF] Paatero | A weighted non-negative least squares algorithm for three-way 'parafac' factor analysis[END_REF][START_REF] Phan | Low complexity damped Gauss-Newton algorithms for CANDECOMP/PARAFAC[END_REF][START_REF] Laurent | Optimization-based algorithms for tensor decompositions : Canonical polyadic decomposition, decomposition in rank-(l r , l r , 1) terms, and a new generalization[END_REF][START_REF] Tomasi | A comparison of algorithms for fitting the PARAFAC model[END_REF]]. An essential step in this method is to solve at each iteration the normal equation of the form J t Jx = -J t r such that r is the residual vector i.e. r = vec( A 1 , . . . , A d -A) and J is the Jacobian matrix of the objective function in (2.10). The matrix J t J, called the Gauss-Newton approximation of the Hessian, is never of full rank [214] and thus methods that use this approach apply strategies such that for example the use of a regularization term or the Moore-Penrose pseudoinverse. We mention that there exists efficient implementation in term of complexity and storage to compute the terms J t r [START_REF] Phan | Fast alternating ls algorithms for high order candecomp/parafac tensor factorizations[END_REF][START_REF] Vannieuwenhoven | A new truncation strategy for the higher-order singular value decomposition[END_REF] and J t J [START_REF] Paatero | A weighted non-negative least squares algorithm for three-way 'parafac' factor analysis[END_REF][START_REF] Phan | Low complexity damped Gauss-Newton algorithms for CANDECOMP/PARAFAC[END_REF][START_REF] Laurent | Optimization-based algorithms for tensor decompositions : Canonical polyadic decomposition, decomposition in rank-(l r , l r , 1) terms, and a new generalization[END_REF]. Recently, the structure of rank-1 components in the CP decomposition was exploited in [33], where the low rank approximation problem is formulated as a Riemannian optimization problem over the Cartesian product of rank-1 manifolds (namely Segre manifolds), and a Riemannian Gauss-Newton iteration is developed to solve the problem. One advantage in this approach, is that the local convergence of the Gauss-Newton method is not affected by large difference scaling between the rank-1 components. There exists also methods in the literature for the low rank symmetric tensor approximation problem, which aims to best approximates a given symmetric tensor to a Waring decomposition of low symmetric rank. We mention some of them. The algorithm CCPD-NLS from Tensorlab [START_REF] Vervliet | [END_REF] employs a Gauss-Newton iteration to solve, for a given A ∈ S d n (R), the non-linear least squares optimization problem

min A∈R n×r 1 2 A - r i=1 a ⊗d i 2 ,
where a i are the columns of A. In [START_REF] Nie | Low rank symmetric tensor approximations[END_REF] a method is developed for this purpose by expressing the linear relations among the entries of low rank symmetric tensors as polynomials called generating polynomials, and using approximate common zeros of these polynomials. Recently, Kileel and Pereira introduced the Subspace Power Method (SPM) [START_REF] Kileel | Subspace power method for symmetric tensor decomposition and generalized PCA[END_REF] to solve this approximation problem for real symmetric tensors of even orders, mainly the method uses the tensor power method called SS-HOPM [START_REF] Kolda | An adaptive shifted power method for computing generalized tensor eigenpairs[END_REF] on a modified tensor built from the flattening of the original tensor to a square matrix, and then applies deflation steps.

Applications of CPD

The CP decomposition is interesting in applications thank to its uniqueness property, which occurs for all higher order tensors. Because of this property, it is heavily used to recover CHAPITRE 2 -Tensors hidden structures in several applications including blind source separation, dimensionality reduction, pattern and image recognition, machine learning, data mining, data analysis, signal processing, biomedical engineering, chemometrics, and multidimensional harmonic retrieval ; we mention for instance [START_REF] Bro | Multi-way analysis in the food industry -models, algorithms, and applications. Technical report, MRI, EPG and EMA[END_REF][START_REF] Cichocki | Nonnegative Matrix and Tensor Factorizations : Applications to Exploratory Multi-way Data Analysis and Blind Source Separation[END_REF][START_REF] Cichocki | Era of big data processing : A new approach via tensor networks and tensor decompositions[END_REF][START_REF] Savas | Handwritten digit classification using higher order singular value decomposition[END_REF][START_REF] Smilde | Multi-way Analysis with Applications in the Chemical Sciences[END_REF][START_REF] Anandkumar | Tensor decompositions for learning latent variable models[END_REF][START_REF] Acar | Unsupervised multiway data analysis : A literature survey[END_REF][START_REF] Mørup | Applications of tensor (multiway array) factorizations and decompositions in data mining[END_REF][START_REF] Kolda | Tensor decompositions and applications[END_REF][START_REF] Sørensen | Multidimensional harmonic retrieval via coupled canonical polyadic decomposition-part i : Model and identifiability[END_REF][START_REF] Sørensen | Multidimensional harmonic retrieval via coupled canonical polyadic decomposition-part ii : Algorithm and multirate sampling[END_REF].

Tucker decomposition

In this thesis we are interested in particular in the rank decomposition. Nevertheless, in this section we present briefly another important type of tensor decomposition called the Tucker decomposition [START_REF] Ledyard | Some mathematical notes on three-mode factor analysis[END_REF].

• The Tucker decomposition can be viewed as a higher order Principal Component Analysis (PCA). It consists in decomposing the tensor into a so-called core tensor multiplied by a matrix along each mode : for A ∈ R n 1 ×...×n d we have

A = S × 1 A 1 × 2 . . . × d A d = r 1 i 1 =1
. . .

r d i d =1 s i 1 ...i d a 1 i 1 ⊗ . . . ⊗ a d i d := S; A 1 , . . . , A d ,
where

A i = [a i 1 , . . . , a i r i ] ∈ R n i ×r i , ∀ 1 ≤ i ≤ d, and S ∈ R r 1 ×.
..×r d is the core tensor. Tucker decomposition appears in many applications such as classification, feature extraction, and subspace-based harmonic retrieval [START_REF] Lu | A survey of multilinear subspace learning for tensor data[END_REF][START_REF] Vasilescu | Multilinear analysis of image ensembles : Tensorfaces[END_REF][START_REF] Haardt | Higher-order svd-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems[END_REF][START_REF] Huy | Tensor decompositions for feature extraction and classification of high dimensional datasets[END_REF]. Unlike the rank decomposition, the Tucker decomposition is generically not unique. • The multilinear rank of a tensor A ∈ R n 1 ×...×n d is the tuple (r 1 , . . . , r d ) such that r i is the column rank of the mode-i matricization of A, i.e. r i = rank(A (i) ). Any tensor of multilinear rank (r 1 , . . . , r d ) has an orthogonal Tucker decomposition such that the core tensor S ∈ R r 1 ×...×r d and the factor matrices are orthonormal i.e. A t i A i = I r i . • The Higher Order Singular Value Decomposition (HOSVD) [START_REF] Lathauwer | A multilinear singular value decomposition[END_REF][START_REF] Vannieuwenhoven | A new truncation strategy for the higher-order singular value decomposition[END_REF] is a multilinear singular value decomposition described by the successive application of the SVD to each mode of matricization of A ∈ R n 1 ×...×n d of multilinear rank (r 1 , . . . , r d ). Let U i ∈ R n i ×r i denotes the matrix of the left singular vectors from the SVD decomposition of A (i) , we have by orthonormality

A = S; U 1 , . . . , U d , 2.6 -Tucker decomposition such that S = A × 1 U t 1 × 2 . . . × d U t d .
• Truncated Higher Order Singular Value Decomposition (THOSVD) [START_REF] Lathauwer | A multilinear singular value decomposition[END_REF][START_REF] Vannieuwenhoven | A new truncation strategy for the higher-order singular value decomposition[END_REF]

A -Â ≤ √ d A -A t ,
where A t denotes the best multilinear rank (s 1 , . . . , s d ) approximation. • The Tucker decomposition has a link with the tensor rank decomposition. Indeed, let A ∈ R n 1 ×...×n d be a tensor of rank r with an orthogonal Tucker decomposition

A = S; U 1 , . . . , U d = r 1 i 1 =1
. . .

r d i d =1 s i 1 ...i d u 1 i 1 ⊗ . . . ⊗ u d i d ,
then one rank decomposition of A can be written as

A = r i=1 (U 1 s 1 i ) ⊗ . . . ⊗ (U d s d i ), where S = r i=1 s 1 i ⊗ . . . ⊗ s d
i is a rank decomposition of the core tensor S. This means once an orthogonal Tucker decomposition is computed to the tensor A it is sufficient to compute a rank decomposition of the core tensor S instead of the tensor A itself. In this regards, Tucker compression techniques are useful to reduce the computational complexity, especially when r i are significantly smaller than n i , there exists efficient algorithms for Tucker compression, for instance [START_REF] Caiafa | Generalizing the column-row matrix decomposition to multi-way arrays[END_REF]75,[START_REF] Oseledets | Tucker dimensionality reduction of three-dimensional arrays in linear time[END_REF].

CHAPTER 3

Riemannian Optimization

In this chapter we review Riemannian optimization techniques. Note that we address this topic in a concise way where we incorporate the essential information and notions to our purpose. Nevertheless, for a detailed, precised and pedagogic presentation of the notions introduced in this chapter, it is recommended to consult references like [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF][START_REF] Lee | Introduction to Smooth Manifolds[END_REF][START_REF] Godinho | An Introduction to Riemannian Geometry : With Applications to Mechanics and Relativity[END_REF]. In Section 3.1, we define Riemannian manifolds, then in Section 3.2 we describe the concept of Riemannian optimization on Riemannian manifolds. Next, Section 3.3 is devoted to present the main Riemannian optimization tools : the Riemannian gradient and Hessian, vector transport, and retraction operator, also we introduce these notions for Riemannian submanifolds in Section 3.3.5. First and second order Riemannian optimization algorithms that we will use are presented in Section 3.4 : Riemannian conjugate gradient algorithm, Riemannian Newton algorithm, Riemannian Gauss-Newton algorithm, and Riemannian trust-region scheme (with dogleg steps). Finally, in Section 3.5 we present the Riemannian manifolds that interest us in this thesis : Sphere, Segre and Veronese manifolds, the general linear group and the oblique manifold.

Riemannian manifold

A smooth manifold M is a set which is locally in diffeomorphism with a vector space, and which admits a globally defined differential structure. In other words, for each point x in M the tangent space T x M contains the tangent vectors on M at x which generalize the notion of a directional derivative. Let γ : R → M be a smooth curve on M that passes through the point x at 0 i.e. γ(0) = x, the derivative of γ at 0 is given by the classical formula :

γ(0) := lim t→0 γ(t) -γ(0) t .
The tangent space on M at x is {ξ x := γ(0) : γ is a smooth curve on M such that γ(0) = x}. It can be seen as a linear approximation of the manifold M in the neighborhood of x.

A Riemannian manifold is a couple (M, g) such that M is a smooth manifold and g is a smoothly varying inner product on M called a Riemannian metric on M. More precisely, at each point x the tangent space T x M is endowed with an inner product g x := ., . x (i.e. a bilinear, symmetric positive-definite form). This induces the following norm on

T x M ξ x x := ξ x , ξ x .

Riemannian optimization

A Riemannian optimization problem consists in minimizing a smooth function f : M → R on the domain M which is a Riemannian manifold i.e.

min f (x) s.t. x ∈ M.
Riemannian optimization methods aim to generate, starting from an initial guess x 0 ∈ M, a sequence x 1 , x 2 , . . . that remains in M and that converges to a local minimum of f constrained to M.

In a common basic form, Riemannian optimization methods are following the process given by :

x i+1 = R x i (tξ i ).
• The tangent vector ξ i ∈ T x i M is the search direction i.e. the direction according to which f will decrease locally, that is the directional derivative of f at x i according to the search direction also called descent direction ξ i denoted by Df (x i )[ξ i ] is strictly negative. It can be obtained using the first-order (gradient) or second-order (Hessian) information. • The scalar t > 0 is called the step size which is used to guarantee a sufficient decrease of f in x i+1 according to the descent direction ξ i . Two typical strategies to choose t are the line-search methods like Armijo backtracking, and trust region methods. • As we can see tξ i is in T x i M. To define the new point on the Riemannian manifold M we use a smooth operator R x : T x M → M called retraction which replaces the straight ray in T x i M with a curve that locally lies on M such that moving along this curve is considered as moving in the direction of ξ i . An advantage of Riemannian optimization is that there exists, in the general case, proofs of convergence of the Riemannian optimization algorithms. These proofs are closely related to the results from Euclidean unconstrained optimization (see [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF]). In our current context, we consider in particular the case where the real valued function f takes the form :

f : M → R, x → 1 2 F (x) 2 , with F : M → E, x → F (x), (3.1)
where E is a Euclidean space. Thus minimizing this f is a Riemannian least-squares minimization problem.

We review in Section 3.3 in more details the ingredients of Riemannian optimization methods (gradient, Hessian, retraction, . . . ). Then in Section 3.4 we recall the Riemannian optimization 3.3 -Riemannian optimization tools 35 algorithms which we will use in this thesis. In Section 3.5 we review the Riemannian manifolds involved in the Riemannian least-squares problems which we will consider in this thesis.

Figure 3.2 -Illustration of one step of a typical Riemannian optimization method. On the right at x a descent direction ξ which leads to a decrease in f (i.e. Df (x)[ξ] < 0). On the left, the new point is defined on the manifold M by using a retraction operator.

Riemannian optimization tools

Hereafter in this section we present the main tools for Riemannian optimization methods.

Riemannian gradient

Let f be a smooth real valued function on a Riemannian manifold M the gradient of f at x denoted by grad f (x) is defined as the unique element of T x M such that :

grad f (x), ξ x = Df (x)[ξ], ∀ξ ∈ T x M.
An interesting property of grad f (x) is that its direction is the steepest-ascent direction of f at

x. A tangent vector ξ in T x M is called a descent direction if grad f (x), ξ x < 0.
If we assume f is as in (3.1) then for all ξ ∈ T x M we have :

grad f (x), ξ x = Df (x)[ξ] = DF (x)[ξ], F (x) = ξ, (DF (x)) * [F (x)] ,
where ., . denotes the inner product on E, and (DF (x)) * denotes the adjoint of the operator DF (x) : T x M → E. Thus we have :

grad f (x) = (DF (x)) * [F (x)].
(3.2)

Riemannian Hessian

To define the Riemannian Hessian operator, we need first to introduce some important notions. A vector field ξ is a function such that for each x ∈ M it associates a tangent vector ξ x ∈ T x M.

An affine connection on a smooth manifold is also an important geometric object in differential geometry. It permits to connect nearby tangent spaces and generalizes directional derivatives of vector fields. Coming back to our Riemannian manifolds context, let M be a Riemannian manifold with metric ., . , the connection of Levi-Civita denoted by ∇ is the unique affine connection that verifies Koszul formula

2 ∇ ξx η x , ν x x = D η x , ν x x [ξ x ] + D ξ x , ν x x [η x ] -D ξ x , η x x [ν x ] + ν x , [ξ x , η x ] x + η x , [ν x , ξ x ] x -ξ x , [η x , ν x ] x ,
where ξ x , η x and ν x are the vector fields ξ, η and ν evaluated at x and [., .] is the Lie bracket * . The connection of Levi-Civita is very important on a Riemannian manifold M with a metric ., . , it permits for instance to define the Hessian operator, geodesics and distance on M. Note that the gradient operator of f is a vector field, since for each x ∈ M, grad f (x) is a tangent vector in T x M. Thus the Hessian operator is by definition the linear mapping from T x M to itself such that :

Hess f (x)[ξ x ] = ∇ ξx grad f (x). (3.3)
If we assume that f is as in (3.1) then for all ξ, η in T x M we have :

Hess f (x)[η], ξ x = DF (x)[ξ], DF (x)[η] + F (x), ∇ 2 F (x)[ξ, η] , (3.4)
where ., . denotes the inner product on E and ∇ 2 F denotes the second covariant derivative of F (see [2, Chapter 8 and Section 5.6] for the proof).

Retraction

The geodesics on a Riemannian manifold M with metric ., . generalize the notion of straight lines. They are given by curves γ : I ⊂ R → M on M with zero acceleration. In our context, this means by using the connection of Levi-Civita :

∇ γ(t) γ(t) = 0.
Further, they are characterised by the initial choice of the initial point γ(0) = x ∈ M and the initial direction γ(0) = ξ ∈ T x M. Geodesics allow to define distance on M with the metric ., . . Indeed, for x, y ∈ M and γ a geodesic on M such that γ(0) = x and γ(1) = y, the distance d(x, y) between x and y is defined by :

d 2 (x, y) = 1 0 γ(t), γ(t) γ(t) dt.
For every ξ ∈ T x M there exits an interval I = [0, 1] and a unique geodesic γ(t; x, ξ) :

I → M such that γ(0) = x, γ(0) = ξ. The mapping Exp x : T x M → M, ξ → γ(1; x, ξ),
is called the exponential map at x. It maps along geodesics in direction of the tangent vector. Intuitively, exponential map is a natural way from a differential geometry point of view used in Riemannian optimization methods, to turn an increment x + ξ in T x M into a new point on the Riemannian manifold M. Nevertheless, the exponential map may be very complicated or very expensive to compute. Herein, a retraction map R which corresponds to the first order approximation of the exponential map i.e.

R x (ξ) = x + ξ + o( ξ ),
is an adequate alternative choice. Formally, the definition of a retraction map is given by the following :

1. R x (0 x ) = x ; 2.
there exists an open neighborhood U x ⊂ T x M of 0 x such that the restriction on U x is well-defined and smooth ;

3. R x satisfies the local rigidity condition

DR x (0 x ) = id TxM ,
where id TxM denotes the identity map on T x M.

The rigidity condition insures that for every tangent vector ξ ∈ T x M, moving along the curve

γ ξ : t → R x (tξ) is moving in the direction of ξ, since γξ (0) = d dt R x (tξ) | t=0 = DR x (0 x )[ξ] = id TxM [ξ] = ξ.
It is an important property for Riemannian optimization purposes.

Finally, we recall the following easy-to-proof property of retraction on a Cartesian product of Riemannian manifolds :

Lemma 3.3.1. Let M 1 , . . . , M r be manifolds, x i ∈ M i and M = M 1 × • • • × M r and x = (x 1 , . . . , x r ) ∈ M. Let R i : T x i M i → M i be retractions. Then R x : T x M → M defined as follows : R x (ξ 1 , . . . , ξ r ) = (R x 1 (ξ 1 ), . . . , R xr (ξ r )) for ξ i ∈ T x i M i , 1 ≤ i ≤ r, is a retraction on M.

Vector transport

Another important notion in Riemannian optimization, is the vector transport [2, Section 8.1], which is used to transport a tangent vector ξ in T x M to a tangent vector in T Rx(η) M (see Figure 3.5). Similarly to the retraction which approximates the exponential mapping to the firstorder, vector transport is the first-order approximation of parallel translation along geodesics, and it is used rather than the parallel translation for the same reason which the retraction is used rather than the exponential map i.e. to reduce the computational cost while keeping the convergence properties of the algorithm. Concerning parallel transport, briefly, it corresponds to transport a tangent vector ν(0) in T γ(0) M along the curve γ(t) in such a way ν(t) in T γ(t) M corresponds to the tangent vector ν(0) in T γ(0) M (see Figure 3.4) in the sense that the transport along the curve γ is parallel according to the metric ., . on M. In other words, using the Levi-Civita connection, the parallel transport ν along the curve γ is given by the solution of : Coming back to vector transport, its definition uses the so-called Whitney sum given by :

∇ γ(t) ν(t) = 0.
T M ⊕ T M = x∈M {x} × T x M × T x M = {(x, ξ, η), x ∈ M, ξ ∈ T x M, η ∈ T x M}. Definition 3.3.2 (Vector transport). A vector transport on the manifold M is a smooth mapping : T M ⊕ T M → T M, (x, η, ξ) → T η (ξ) satisfying the following properties for all x ∈ M.
1. There exists a retraction R such that, for all (x, η, ξ)

∈ T M ⊕ T M, it holds that T η (ξ) ∈ T Rx(η) M. 2. T 0 (ξ) = ξ for all ξ ∈ T x M.
3. For all (x, η) ∈ T M, the mapping T η :

T x M → T Rx(η) M, ξ → T η (ξ) is linear.
Figure 3.5 -Vector transport on M.

Riemannian optimization tools for Riemannian submanifolds

Let M be an embedded submanifold of a Riemannian manifold M endowed with a metric ., . . For each x ∈ M, the tangent space T x M is a subspace of T x M. Hence, M turns into a Riemannian manifold, called a Riemannian submanifold of M, by simply inheriting M the metric ., . from M. Any element ξ ∈ T x M can be uniquely decomposed as follows :

ξ = P x (ξ) + P ⊥ x (ξ),
where P x denotes the orthogonal projection onto T x M and P ⊥ x denotes the orthogonal projection onto the orthogonal complement of T x M called the normal space to M at x and given by

(T x M) ⊥ := {η ∈ T x M : η, ξ x = 0, ∀ξ ∈ T x M}.
It follows that the connection of Levi-Civita on M denoted by ∇ can be deduced from the connection of Levi-Civita on M denoted by ∇ :

∇ ξx η x = P x (∇ ξx η x ), (3.5)
where x ∈ M, and ξ x , η x are vector fields evaluated at x. Moreover, if f is a smooth real valued function on M and f is the restriction of f on M then ∀x ∈ M we have :

grad f (x) = P x (grad f (x)). (3.6)
Further, the Hessian of f at x is obtained by substituting (3.5) and (3.6) in (3.3).

In particular if M is an embedded submanifold of a Euclidean space E, then (3.5) reads :

∇ ξx η x = P x (Dη x [ξ x ]),
with Dη x [ξ x ] is the classical directional derivative of η at x in the direction of ξ x . Also, in this case (3.6) reads :

grad f (x) = P x (∂ x f (x)),
where We note that in our context we will, in most cases, deal with the aforementioned type of Riemannian manifold i.e. embedded Riemannian submanifold in a Euclidean space.

∂ x f (x) is the classical Euclidean gradient of f at x. Finally, if x, y ∈ M such that y = R x (η) where η ∈ T x M, given a tangent vector ξ ∈ T x M, we have ξ ∈ T x M ⊂ T x E E, thus

Riemannian optimization algorithms

In this section we review some of the existing Riemannian optimization algorithms in the literature, which we will use in this thesis in the contributions part. Let f : M → R be a smooth real valued function defined on a Riemannian manifold M with a Riemannian metric ., . . We recall from Section 3.2 that a Riemannian optimization algorithm aims to solve locally the minimization problem :

min f (x) s.t. x ∈ M. (3.7)
We note that our review does not include an analysis for the convergence of these algorithms. For this purpose as well as for more details concerning these algorithms see for instance [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF].

Riemannian conjugate gradient method

The Riemannian conjugate gradient method generalizes the typical conjugate gradient method on Euclidean space. The search direction at step i in a classical conjugate gradient method is computed by combining the steepest descent direction i.e. ξ i := -grad f (x i ) with the search direction in the previous step η i-1 so that the new search direction is given by :

η i = ξ i + β i η i-1 ,
and then

x i+1 = x i + α i η i ,
where α i is the step-size that determines how far x i should moves along the direction η i .

To generalize the method to Riemannian manifolds, we have to take into account the following points :

-The search direction step involves the sum of an element η i-1 lies in

T x i-1 M with ξ i ∈ T x i M, thus η i-1 should be transported to T x i M by using vector transport T x i-1 →x i .
-We can choose one of the two following popular choice to compute β i :

β i = grad f (x i ) , grad f (x i ) -T x i-1 →x i (grad f (x i-1 )) grad f (x i-1 ) , grad f (x i-1 ) (Polak-Ribière), (3.8
)

β i = grad f (x i ) , grad f (x i ) grad f (x i-1 ) , grad f (x i-1 )
(Fletcher-Reeves).

(3.9)

-We need to use a retraction operator R x i : T x i M → M to map the tangent vectors to the manifold.

-Riemannian optimization algorithms

41

-Perform a line-search with the conjugate direction η i along the curve t → R x i (tη i ). A standard Armijo backtracking step size can be used to choose t : Let α > 0, β, σ ∈ (0, 1), find the smallest nonnegative integer m such that :

f (x i ) -f (R x i (β m αη i )) ≥ -σ grad f (x i ), β m αη i , take t = β m α.
We give the skeleton of the Riemannian conjugate gradient algorithm with Armijo backtracking line search step in Algorithm 3.1.

Require: Riemannian manifold M ; vector transport T on M with associated retraction R ; cost function f on M ; initial iterate x 1 ∈ M, tangent vector η 0 = 0.

1: for i = 1, 2, . . . do 2:
Compute the gradient

ξ i := grad f (x i ) ; 3: Compute a conjugate direction η i := -ξ i + β i T x i-1 →x i (η i-1 ) ; 4:
Perform Armijo backtracking to find the smallest integer m ≥ 0 such that for given α > 0, β, σ ∈ (0, 1)

f (x i ) -f (R x i (β m αη i )) ≥ -σ ξ i , β m αη i ; 5: Compute the next new point x i+1 := R x i (β m αη i ) ;

Riemannian Newton and Riemannian Gauss-Newton methods

A Riemannian Newton method is a geometric generalization of the classical Newton method. Recall that if F is a smooth function from R to R such that F (x * ) = 0, then Newton's method consists of starting with an initial guess x 0 ∈ R and generating a sequence x 1 , x 2 , . . . in R, such that :

x k+1 = x k - F (x k ) F (x k ) ,
where F is the derivative of F . By rewritten this expression as

F (x k ) + F (x k )(x k+1 -x k ) = 0,
this means graphically that x k+1 is the intersection of the tangent to the graph of F at x with the horizontal axis. This can be generalize to a function F : R n → R n by taking :

F (x k ) + DF (x k )[x k+1 -x k ] = 0, (3.10)
where DF (x k )[z] denotes the directional derivative of F at x along z. Now, if we consider the minimization problem in (3.7) but with constrained set equal to R n , since the aim is to find a local minimum of f , this means that we seek a critical point i.e. grad f (x * ) = 0, thus by replacing F by grad f , the Newton equation (3.10) reads :

grad f (x k ) + D(grad f )(x k )[x k+1 -x k ] = 0.
Finally, to generalize this approach to Riemannian manifolds, x k+1 -x k is replaced by a tangent vector

η k ∈ T x k M, grad f (x k ) is the Riemannian gradient of f at x k and D(grad f )(x k ) is replaced by the Riemannian Hessian Hess f (x k ). Using a retraction operator R on M, the new point x k+1 is obtained from η k by x k+1 = R x k (η k ).
Formally, a Riemannian Newton method for solving (3.7) [2, Chapter 6] consists of starting with an initial guess x 0 ∈ M and generating a sequence x 1 , x 2 , . . . in M, with respect to the following process :

x k+1 ← R x k (η k ) with Hess f (x k )[η k ] = -grad f (x k ); (3.11)
where grad f (x k ) and Hess f (x k ) are respectively the Riemannian gradient and Hessian of f at x k on M, and

R x k : T x k M → M is a retraction operator from the tangent space T x k M to M.
Riemannian Gauss-Newton method is a quasi-Newton method i.e. an approximation of the Riemannian Newton method for the case where f = 1 2 F 2 as in (3.1). Recall from (3.4), that with this f , we have :

Hess f (x)[η], ξ x = DF (x)[ξ], DF (x)[η] + F (x), ∇ 2 F (x)[ξ, η] , ∀ξ, η ∈ T x M.
The Riemannian Gauss-Newton method consists in approximating Hess f (x)[η], ξ x by the first term DF (x)[ξ], DF (x) [η] . By injecting this in the Riemannian Newton equation in (3.11), we obtain the so-called Riemannian Gauss-Newton equation

((DF (x k )) * • DF (x k ))[η k ] = -(DF (x k )) * [F (x k )], since in this case grad f (x k ) = (DF (x k )) * [F (x k )] (see (3.2)). The operator (DF (x k )) * • DF (x k
) is called the Riemannian Gauss-Newton approximation of the Riemannian Hessian Hess f (x k ). Thus, technically, the Riemannian Gauss-Newton method differs from the Riemannian Newton method by solving the Gauss-Newton equation instead of the exact Newton equation. Obviously, the Riemannian Gauss-Newton method is of lower computational complexity than the Riemannian Newton method. On the other hand, Riemannian Gauss-Newton method is in general not superlinearly convergent, in contrast with the Riemannian Newton method which has a local superlinear rate of convergence (see [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF]Theorem 6.3.2]). Nevertheless, in practice, Riemannian Newton and Gauss-Newton methods are combined with a line-search step or with a trust-region scheme in order to ensure a sufficient decrease in the cost function f at each iteration. The next section is devoted to describe the Riemannian trust-region scheme with a dogleg step.

Riemannian trust region scheme with dogleg steps

The Riemannian Newton (resp. Gauss-Newton) method is looking for a critical point of a real-valued function f , without distinguishing between local minimizer, saddle point and local maximizer. Furthermore, the convergence of this algorithm may not occur from the beginning. For these reasons, a trust region scheme is usually added to such algorithm in order to enhance the algorithm, with the desirable properties of convergence to a local minimum, with a local superlinear rate of convergence. In fact, trust region method ensures that f decreases at each iteration, which reinforces, when convergence occurs, the possibility of finding a local minimizer. Nevertheless, a global convergence to a local minimizer from any initial points is not guaranteed even after adding the trust region scheme (see [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF]Subsection 7.4.1] for the global convergence of Riemannian trust-region methods). The idea is to approximate the objective function f by its second order Taylor series expansion in a ball of center 0

x k ∈ T x k M and radius ∆ k denoted by B ∆ k := {η ∈ T x k M | ||η|| ≤ ∆ k },
and to solve the so-called trust-region subproblem

min η∈B ∆ k m x k (η), (3.12)
where

m x k (η) := f (x k ) + grad f (x k ), η x k + 1 2 H k [η], η x k , H k is some symmetric operator on T x k M (
in the case of Riemannian Newton method H k is the Hessian operator of f at x k and it is the Riemannian Gauss-Newton approximation (DF (x k )) * • DF (x k ) for the Riemannian Gauss-Newton method).

By solving (3.12), we obtain a solution η k ∈ T x k M. Accepting or rejecting the candidate new point x k+1 = R x k (η k ), as well as updating the trust region ∆ k is based on the quotient

ρ k = f (x k ) -f (x k+1 ) m x k (0) -m x k (η k ) .
One approach to solve the trust-region subproblem (3.12) is the so-called dogleg method : Let η N be the Newton direction given by H k [η N ] = -grad f (x k ), let η c denote the Cauchy point given by η c = -

grad f (x k ),grad f (x k ) x k H k [grad f (x k )],grad f (x k ) x k
grad f (x k ), and let η I be the intersection of the boundary of the sphere B ∆ and the vector pointing from η c to η N . Then the optimal solution η * of (3.12) by the dogleg method is given as follows :

η * =        η N if ||η N || ≤ ∆, - ∆ || grad f (x k )|| grad f (x k ) if ||η N || > ∆ and ||η c || ≥ ∆, η I otherwise.

Riemannian manifolds of interest

In this section we present five manifolds that we use thereafter in this thesis : the sphere, Segre and Veronese manifolds, the general linear group and the oblique manifold.

The unit sphere

The unit sphere in

R n S n-1 := {x ∈ R n | x T x = 1}; CHAPITRE 3 -Riemannian Optimization
is one of the simplest Riemannian manifolds. The inner product is inherited from the standard inner product on R n i.e. ξ, η x := ξ T η.

Its tangent space at x ∈ S n-1 is given by

T x S n-1 = {z ∈ R n | x T z = 0}.
The normal space is

(T x S n-1 ) ⊥ = {xα | α ∈ R}.
The projection on T x S n-1 respectively on (T x S n-1 ) ⊥ are given by

P x (ξ) = (I n -xx T )ξ, P ⊥ x (ξ) = xx T ξ. The map R x : T x S n-1 → S n-1 , ξ → x+ξ x+ξ is a retraction operator on S n-1 .
All the aforementioned information with their proofs can be found in [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF].

Segre manifold

The Segre manifold in R n 1 ×...n d consists of all tensors of rank 1 :

S n 1 ,...,n d := {a 1 ⊗ . . . ⊗ a d | a k ∈ R n k } -{0}.
It is an embedded submanifold in R n 1 ×...×n d , thus S n 1 ,...,n d is a Riemannian submanifold by tacking the metric induced by this ambient Euclidean space. The Segre manifold can be parameterized using the following local diffeomorphism :

Φ : R * + × S n 1 -1 × . . . × S n d -1 → S n 1 ,...,n d (α, a 1 , . . . , a d ) → αa 1 ⊗ . . . ⊗ a d .
Let x = (α, a 1 , . . . , a d ). This allows to identify the tangent space of S := S n 1 ,...,n d at x as follows :

T x (R * + × S n 1 -1 × . . . × S n d -1 ) ∼ = R × T a 1 S n 1 -1 × . . . × T a d S n d -1 , with D x Φ : R × T a 1 S n 1 -1 × . . . × T a d S n d -1 → T αa 1 ⊗...⊗a d S ( α, ẏ1 , . . . , ẏd ) → αa 1 ⊗ . . . ⊗ a d + α( ẏ1 ⊗ a 2 ⊗ . . . ⊗ a d + . . . + a 1 ⊗ . . . ⊗ a d-1 ⊗ ẏd ).
There exists a retraction operator on S called T-HOSVD retraction [START_REF] Kressner | Low-rank tensor completion by Riemannian optimization[END_REF]. It is defined as the rank-(1, . . . , 1) T-HOSVD approximation of A + p, such that A ∈ S, p ∈ T A S, as follows :

R A (p) = (q 1 q T 1 , . . . , q d q T d ).(A + p), where q k is the first left singular vector from the SVD decomposition of the matrix (A + p) (k) i.e. the kth mode of flattening of A + p.

Veronese manifold

Similarly to the Segre manifold, the Veronese manifold in T d (R n ), i.e. the space of real symmetric tensors of dimension n and order d, contains all rank-1 symmetric tensors in T d (R n ). By considering the correspondence between T d (R n ) and R[x] d , the Veronese manifold is given by the set of linear forms to the d th power :

V d n (R) := {(v t x) d | v ∈ R n } -{0}. Let p = (v t x) d ∈ V d n (R), the tangent space T p V d n (R)
is given by :

T p V d n (R) = {(u t x)(v t x) d-1 | u ∈ R n }.
Similarly to the Segre manifold, the Veronese manifold is an embedded submanifold in R[x] d . Herein, endowed with the metric inherited from R[x] d (for instance the apolar product defined on

R[x] d × R[x] d in Definition 2.2.1 which is an inner product on R[x] d ) V d n (R) is a Riemannian submanifold.
We will investigate this Riemannian manifold in Chapter 4 in more details especially in the complex field, in order to developp Riemannian based optimization algorithms for the symmetric low rank approximation problem for symmetric tensors with complex coefficients.

The general linear group

The information presented in this section can be found in [START_REF] Andruchow | The left invariant metric in the general linear group[END_REF][START_REF] Lee | Geometric direct search algorithms for image registration[END_REF][START_REF] Miller | The metric spaces, euler equations, and normal geodesic image motions of computational anatomy[END_REF][START_REF] Vandereycken | A riemannian geometry with complete geodesics for the set of positive semidefinite matrices of fixed rank[END_REF][START_REF] Zacur | Left-invariant riemannian geodesics on spatial transformation groups[END_REF][START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF]. The set of invertible matrices of size n × n denoted by GL n is open in R n×n , thus it is a manifold. In addition, it is a group under matrix multiplication. Consequently, GL n is by definition a Lie matrix group, called the general linear group. Since it is open in R n×n , its tangent space T B GL n at any point B ∈ GL n can be identified with R n×n . Since GL n is a Lie matrix group, thus an element ξ B ∈ T B GL n can be written as ξ B = BX or ξ B = XB such that X is in the Lie algebra which is by definition T In GL n R n×n , herein ξ B = BX or ξ B = XB with X ∈ R n×n . We will use the aforementioned information in Section 5.1 to apply perturbations on elements of GL n . In Section 5.1, GL n (C) is viewed as a complex Lie matrix group. However, the same information remains applicable in this case. The manifold GL n is equipped with either the left-invariant metric or the right-invariant which are for all B ∈ GL n , ξ B , η B ∈ T B GL n given by :

ξ B , η B B = tr(B -1 ξ B (B -1 η B ) t ), ξ B , η B r B = tr(ξ B B -1 (η B B -1 ) t )
Let γ : R → GL n , γ r : R → GL n be the geodesics of GL n with respect to respectively the leftand right-invariant metrics, for all B ∈ GL n , ξ B ∈ T B GL n they are given by :

γ (t) = B exp(t(B -1 ξ B ) T ) exp(t(B -1 ξ B -(B -1 ξ B ) T )) (3.13) γ r (t) = exp(t(ξ B B -1 -(ξ B B -1 ) T )) exp(t(ξ B B -1 ) T )B (3.14)
where exp(.) denote the matrix exponential. We denote the exponential map resulting from (3.13) and (3.13) by exp B :

T B GL n → GL n , exp r B : T B GL n → GL n .
Naturally, these exponential maps are retractions on GL n . The vector transports on GL n according to the left-and right-invariant metrics are given by :

T (B, ξ B , η B ) = exp B (ξ B )B -1 η B and T r (B, ξ B , η B ) = η B B -1 exp r B (ξ B ).
Finally, the Riemannian gradients of a cost function f : GL n → R at E ∈ GL n equipped with the left or right-invariant metric are given by :

grad f (B) = BB t grad Euc f (B), grad r f (B) = grad Euc f (B)B t B,
where grad Euc f (B) is the classical Euclidean gradient of f at B ∈ GL n .

Oblique manifold

The results that we are going to present in this section come from [

T B M o n = {ξ ∈ R n×n | ddiag(ξB t ) = 0}.
The manifold M o n is a Riemannian manifold inheriting the left or right-invariant metric of GL n . We summarize in Table 3.1, the necessary tools according to the left-or right-invariant metric, that we need for a Riemannian gradient based method (we refer the reader to [START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF] for the proofs of the formulas stated in the table).

P ob, B (Y ) = Y -BB t ∆B, ∆ is a diagonal matrix with diag(∆) = (BB t * BB t ) -1 diag(Y B t ) P ob,r B (Y ) = Y -∆BB t B, ∆ = ddiag(Y B t ) ddiag((BB t ) 2 ) -1 gradient grad ob f (B) = P ob, B (grad f (B)) grad r ob f (B) = P ob,r B (grad r f (B)) retraction R ob, B (ξ B ) = ∆(exp B (ξ B )) exp B (ξ B ) R ob,r B (ξ B ) = ∆(exp r B (ξ B )) exp r B (ξ B ) with ∆(Y ) = ddiag(Y Y t ) -1 2 vector transport T ob, B→R ob, B (ξ B ) (η B ) = P ob, R ob, B (ξ B ) (R ob, B (ξ B )B -1 η B ) T ob,r B→R ob,r B (ξ B ) (η B ) = P ob,r R ob,r B (ξ B ) (η B B -1 R ob,r B (ξ B ))

Contributions

CHAPTER 4

Riemannian Newton optimization algorithms for the symmetric tensor approximation problem

The Symmetric Tensor Approximation problem (STA) consists of approximating a symmetric tensor or a homogeneous polynomial by a linear combination of symmetric rank-1 tensors or powers of linear forms of low symmetric rank. In this chapter, we present two Riemannian Newton-type algorithms for low rank approximation of symmetric tensor with complex coefficients. The first algorithm uses the parametrization of the set of tensors of rank at most r by weights and unit vectors. Exploiting the properties of the apolar product on homogeneous polynomials combined with efficient tools from complex optimization, we provide an explicit and tractable formulation of the Riemannian gradient and Hessian, leading to Newton iterations with local quadratic convergence. We prove that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the Newton iteration (completed with a trust-region scheme) is converging to a local minimum. The second algorithm is a Riemannian Gauss-Newton method on the Cartesian product of Veronese manifolds. An explicit orthonormal basis of the tangent space of this Riemannian manifold is described. We deduce the Riemannian gradient and the Gauss-Newton approximation of the Riemannian Hessian. We present a new retraction operator on the Veronese manifold. We analyze the numerical behavior of these methods, with an initial point provided by Simultaneous Matrix Diagonalisation (SMD). Numerical experiments show the good numerical behavior of the two methods in different cases and in comparison with existing state-of-the-art methods. Keywords : symmetric tensor decomposition, homogeneous polynomials, Riemannian optimization, Newton method, retraction, complex optimization, trust region method, Veronese manifold.

In this chapter, we describe in section 4.1 the set of non-defective rank-r symmetric tensors. In subsection 4.2.1, we formulate the STA problem as a Riemannian least square optimization problem using the parametrization by weights and unit vectors. We compute explicitly the Riemannian gradient vector and the Hessian matrix in subsection 4.2.1.1 and describe the retraction in subsection 4.2.1.2. In subsection 4.2.2, we describe the Riemannian Gauss-Newton method on the product of Veronese manifolds. We present in subsection 4.2.2.1 a new retraction operator on the Veronese manifold with its analysis. In subsection 4.2.3, we recall the trust-region extension scheme, and prove under some regularity assumptions the convergence of the exact Riemannian Newton method with trust region steps to a local minimum of the distance function. Numerical experiments are featured in section 4.3. The final section 4.5 is for our conclusions.

The set of non-defective rank-r symmetric tensors

Let Σ r ⊂ S d n be the set of symmetric tensors of symmetric rank at most r. A symmetric tensor t ∈ Σ r is the sum of d th powers

t = r i=1 (v t i x) d , for v i ∈ C n . (4.1)
It is a point in the image of the following map :

ψ r : C n×r := C n × • • • × C n -→ C[x] d [v i ] 1≤i≤r -→ ψ r ((v i ) 1≤i≤r ) = r i=1 (v t i x) d .
The d th power (v t i x) d with v i = 0 are symmetric tensors of rank-1, which are on the so-called Veronese manifold.

Definition 4.1.1. Let ψ : C n → C[x] d , v → (v t x) d = |α|=d d α v α x α . The Veronese manifold in C[x] d denoted by V n,d is the set of linear forms in C[x] 1 -{0} to the d th power. It is the image of C n -{0} by ψ.
The Veronese variety studied in algebraic geometry is the algebraic variety of the projective space P s n,d -1 associated to V n,d , where [START_REF] Harris | Algebraic Geometry : A First Course[END_REF][START_REF] Zak | Tangents and Secants of Algebraic Varieties[END_REF][START_REF] Landsberg | Tensors : Geometry and Applications[END_REF]. The tangent space of V n,d at a point p = (v t x) d is the vector space spanned by

s n,d = dim C[x] d
x 1 (v t x) d-1 , . . ., x n (v t x) d-1 , that is the linear space T p (V n,d ) = {(u t x)(v t x) d-1 | u ∈ C n }.
The Zariski closure Σ r of Σ r is called the (r -1)th-secant variety of the Veronese variety. For r > 1, the algebraic variety Σ r is not smooth and contrarily to the case of matrices, singular points of Σ r can have a rank > r, as shown in the following example. For

d > 2, p = (v t 0 x)(v t 1 x) d-1 ∈ C[x] d with v 0 = v 1 ∈ C n is in the (Zariski) closure of Σ 2 since (v t 0 x)(v t 1 x) d-1 = lim δ→0 1 d δ (((v 1 + δv 0 ) t x) d -(v t 1 x) d ) but its symmetric rank is d > 2 [54, Proposition 5.6].
To avoid these singularities, we will restrict our theoretical analysis to points of Σ r where the map ψ r is a local embedding, since in the vicinity of singularities, the best low rank approximation problem is ill-posed (as shown by the previous example). The map ψ r is a local embedding at , where r g is the generic symmetric rank (except for (d, n) ∈ {(3, 5), (4, 3), (4, 4), (4, 5)} or d = 2) by Alexander-Hirschowitz theorem [START_REF] Alexander | Polynomial interpolation in several variables[END_REF]. Using "Terracini's lemma" (see e.g. [129, Lemma 5.3.1.1]), we have that Σ reg r is a dense open subset of Σ r iff the dimension of Σ r is the expected dimension n r. In this case, Σ r is also said to be non-defective. Alexander and Hirschowitz [START_REF] Alexander | Polynomial interpolation in several variables[END_REF] proved that Σ r is non-defective when r < r g (the exceptional defective cases for d > 2 being (d, n, r) ∈ {(3, 5, 7), (4, 3, 5), (4,4,[START_REF] Riku | Approximate simultaneous diagonalization of matrices via structured low-rank approximation[END_REF], (4, 5, 14)}).

y = [v i ] 1≤i≤r ∈ C n×r iff Jψ r (y) = d [x 1 (v t 1 x) d-1 , . . . , x n (v t 1 x) d-1 , . . . , x 1 (v t r x) d-1 , . . . , x n (v t r x) d-1 ]
It is also known that for r < r g , generic tensors of ψ r have a unique decomposition, i.e. a unique inverse image by ψ r up to permutations, except for (d, n, r) ∈ {(6, 2, 9), (4, 3, 8), (3, 5, 9)}, see [47, Theorem 1.1].

Riemannian optimization for the STA problem

In this section, we use the framework of Riemannian optimization [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF] to solve the STA problem. See also [33,[START_REF] Hackbusch | Tensor Spaces and Numerical Tensor Calculus[END_REF][START_REF] Kressner | Low-rank tensor completion by Riemannian optimization[END_REF] for real multilinear tensors. We develop a Riemannian Newton algorithm and a Riemannian Gauss-Newton algorithm exploiting the properties of symmetric tensors to obtain explicit and simplified formulation. We consider distance minimization problems for symmetric tensors with complex decompositions for both algorithms. Given p ∈ S d n ∼ C[x] d , we consider hereafter the following least square minimization problem

min y∈M f (y) (4.2)
where f :

M → R is half the square distance function to p i.e. f (y) = 1 2 F (y) 2 d with F (y) = Φ r (y)-p, such that Φ r : M → C[x] d is
a parametrization map of Σ r the set of symmetric tensors of symmetric rank bounded by r, and M is a Riemannian manifold. A Riemannian optimization method for solving (4.2) requires a Riemannian metric. Since we will assume that M is embedded in some space R M , we will take the metric induced by the Euclidean space R M .

We propose to parametrize Σ r , first via weights and unit vectors. We describe an exact Riemannian Newton method for this formulation in subsection 4.2.1. Secondly, we parametrize Σ r via sums of the d th power of linear forms that is as sums of tensors in V n,d . We develop a Riemannian Gauss-Newton method for this formulation in subsection 4.2.2. A dogleg trust-region scheme in subsection 4.2.3 is added to the two algorithms.

Riemannian Newton method for STA

We normalize the decomposition (4.1) by choosing unit vectors for v i and positive weights. Namely, we decompose a symmetric tensor p ∈ Σ r as p = r i=1 w i (v t i x) d with w i ∈ R * + and ||v i || = 1, for 1 ≤ i ≤ r ; by normalizing v i and multiplying by w i := ||v i || d if v i is not a unit vector. The vector (w i ) 1≤i≤r in this decomposition is called "the weight vector", and is denoted by W . Let V = [v i ] 1≤i≤r ∈ C n×r be the matrix of the normalized vectors.

The objective function expressed in terms of these weights and unit vectors is given by

f (W, V ) = 1 2 ||F (W, V )|| 2 d , with F (W, V ) = r i=1 w i (v t i x) d -p.

-Riemannian optimization for the STA problem 53

The function f is a real valued function of complex variables ; such function is non-analytic, because it cannot satisfy the Cauchy-Riemann conditions [START_REF] Remmert | Theory of Complex Functions[END_REF]. To apply the Riemannian Newton method, we need the second order differentials of f . As discussed in [START_REF] Laurent | Unconstrained optimization of real functions in complex variables[END_REF], we overcome the non-analytic problem by converting the optimization problem to the real domain, regarding f as a function of the real and imaginary parts of its complex variables.

Let N r = {(W, (V ), (V )) | W ∈ R * + r , V ∈ C n×r , ( (v i ), (v i )) ∈ S 2n-1 , ∀ 1 ≤ i ≤ r}, where S 2n-1 is the unit sphere in R 2n . Let ϕ r : (w, v 1 , . . . , v r , v 1 , . . . , v r ) ∈ N r → r i=1 w i ((v i + i v i ) t x) d .
Hereafter in this subsection, we use the following formulation to compute the different ingredients of a Riemannian Newton method :

(STA) Nr min y∈Nr f (y), where f (y) = 1 2 ||F (y)|| 2 d , with F (y) = ϕ r (y) -p.

Computation of the gradient vector and the Hessian matrix

In this section, we present the explicit expressions of the Riemannian gradient and Hessian on N r . We first describe an orthonormal basis of T y N r for y ∈ N r . Then we detail the computation of the gradient and Hessian in this basis, via the differentials of maps in complex and conjugate variables.

Lemma 4.2.1. Let y = (w, v 1 , . . . , v r , v 1 , . . . , v r ) ∈ N r . For all i = 1, . . . , r let vi = (v i ; v i ) ∈ S 2n-1 and let (I 2n -vi vt i ) = Q i R i P i be a rank-revealing QR-decomposition of the projector on v⊥ i in R 2n , where Q i Q t i = I 2n , R i is upper triangular, and P i is a permutation matrix.

Let Q i,re (resp. Q i,im ) be the matrix given by the first n rows (resp. the last n rows) and the first 2n -

1 columns of Q i . Let Q = Q re Q im ∈ R 2nr×(2n-1)r , where Q re = diag(Q i,re ) 1≤i≤r and Q im = diag(Q i,im ) 1≤i≤r . Then the columns of Q = diag(I r , Q) form an orthonormal basis of T y N r .
Proof. We have

T y N r T w (R * + ) r × T Z S r , where S r = {( (V ), (V )) | V ∈ C n×r , ||v i || 2 = 1, ∀1 ≤ i ≤ r} and Z = ( (V ), (V )) = (v 1 , . . . , v r , v 1 , . . . , v r ) ∈ R n×2r .
As T w (R * + ) r = R r , I r represents an orthonormal basis of T w (R * + ) r . We verify now that Q is an orthonormal basis of T Z S r . For i = 1, . . . , r, vi ∈ S 2n-1 ⊂ R 2n and the first (2n -1) columns of the factor Q i of a rank-revealing QR-decomposition of I 2n -vi vt i give an orthonormal basis of the image v⊥ i of (I 2n -vi vt i ), that is T vi S 2n-1 . The vector space T Z S r , of dimension r(2n -1), is the Cartesian product of the tangent spaces T vi S 2n-1 . Therefore, by construction, the columns of Q form an orthonormal basis of T Z S r .

We deduce that Q = diag(I r , Q) represents an orthonormal basis of

T y N r in the canonical basis of R 2nr . Let R r = {(W, (V ), (V )) ∈ R r × R n×r × R n×r | W ∈ R r , V ∈ C n×r }
and let f R be the function f seen as a function on R r . The gradient and the Hessian of f R at a point p R ∈ R r 54 CHAPITRE 4 -Riemannian Newton optimization algorithms for the symmetric tensor approximation problem are called the real gradient and the real Hessian. We denote them by G R and H R . We will describe their computation, after the next proposition, relating them to the Riemannian gradient and Hessian.

Proposition 4.2.2. Let p = (w, v 1 , . . . , v r , v 1 , . . . , v r ) ∈ N r , Q ∈ R (r+2nr)×(r+(2n-1)r) such that its columns form an orthonormal basis of T y N r . Let G R = (g 0 , g 1 , . . . , g r , g 1 ,. . . , g r ) ∈ R r+2nr (resp. H R ∈ R (r+2nr)×(r+2nr) ) be the gradient vector (resp. the Hessian matrix) of f R at p R in the canonical basis. The Riemannian gradient vector (resp. Hessian matrix) of f at p with respect to the basis Q is given by :

G = Q t G R , H = Q t (H R + S)Q,
where S = diag(0 r×r , S, S), with S = diag(s 1 I n , . . . , s r I n ),

s i = v i , g i + v i , g i .
Proof. Let y = (w, v 1 , . . . , v r , v 1 , . . . , v r ) ∈ N r . Let P y be the orthogonal projector on T y N r . Let Q ∈ R (r+2nr)×(r+(2n-1)r) such that its columns form an orthonormal basis of the image of P y or equivalently of T y N r . As the Riemannian gradient of f is the projection of Df R , the first order differentials of f R , on the tangent space

T y N r [2, Chapter 5], we have G = Q t G R
, where G R is the vector which represents the classical first order partial derivatives of f R at y R in the canonical basis.

Let η ∈ T y N r , z ∈ T y N ⊥ r . We have from [START_REF] Absil | An extrinsic look at the Riemannian Hessian[END_REF] that the Riemannian Hessian matrix of f at y is given by the formula :

Hη = P y H R η + U y (η, P ⊥ y G R )
, where H R is the matrix of the second order derivatives of f R at y R in the canonical basis, U y is the Weingarten map on N r at y given by U y (η, z) = P y D η Pz, where P is a matrix valued function on N r determined as follows : P : y ∈ N r → P y , and D η Pz represent the time derivative of y → P y z in terms of the time derivative of y i.e. ẏ ∈ T y N r applied at ẏ = η, and P ⊥ y = I -P y is the orthogonal projector on

T y N ⊥ r . As y ∈ N r we have w ∈ R * + r , and vi := (v i , v i ) ∈ S 2n-1 , ∀1 ≤ i ≤ r. Let u = (u 0 , u 1 , . . . , u r , u 1 , . . . , u r ) ∈ R r+2nr , such that ǔi = (u i , u i ), ∀1 ≤ i ≤ r. Let P w (resp. P vi ) denote the orthogonal projector on T w (R * + ) r = R r (resp. T vi S 2n-1
), we have that : P w (u 0 ) = u 0 , P vi ǔi = (I 2n -vi vt i )ǔ i , ∀1 ≤ i ≤ r, thus :

P y u =              u 0 ((I 2n -v1 vt 1 )ǔ 1 )[1 : n] . . . ((I 2n -vr vt r )ǔ r )[1 : n] ((I 2n -v1 vt 1 )ǔ 1 )[n + 1 : 2n] . . . ((I 2n -vr vt r )ǔ r )[n + 1 : 2n]              =              u 0 u 1 -v 1 vt 1 ǔ1 . . . u r -v r vt r ǔr u 1 -v 1 vt 1 ǔ1 . . . u r -v r vt r ǔr              , P ⊥ y u =              0 r v 1 vt 1 ǔ1 . . . v r vt r ǔr v 1 vt 1 ǔ1 . . . v r vt r ǔr             
.

Let U vi be the Weingarten map on S 2n-1 at vi . For η = (η 0 , η 1 , . . . , η r , η 1 , . . . , η r ) ∈ T y N r , and

z = (z 0 , z 1 , . . . , z r , z 1 , . . . , z r ) ∈ T y N ⊥ r with ηi = (η i , η i ) ∈ T vi S 2n-1 and ži = (z i , z i ) ∈ T vi S 2n-1 ⊥ , ∀1 ≤ i ≤ r, we have from [3] : U vi (η i , ži ) = -η i vt i ži .
Thus, with respect to the 4.2 -Riemannian optimization for the STA problem 55 parameterization that we consider we find that :

U y (η, z) = -              0 r η 1 vt 1 ž1 . . . η r vt r žr η 1 vt 1 ž1 . . . η r vt r žr             
.

Let G R = (g 0 ; g 1 ; . . . ; g r ; g 1 ; . . . ; g r ) ∈ R r+2nr and ǧi = (g i , g i ), l i = vi vt i ǧi for i = 1, . . . , r. We obtain U y (η, P ⊥ y G R ) by substituting ži by l i in U y (η, z). Since vt i vi = ||v i || 2 = 1, we find that

U y (η, P ⊥ y G R ) =              0 r η 1 vt 1 ǧ1 . . . η r vt r ǧr η 1 vt 1 ǧ1 . . . η r vt r ǧr             
= Sη, where S = diag(0 r×r , S, S), with S = diag(s 1 I n , . . . , s r I n ),

s i = vt i ǧi = v i , g i + v i , g i .
Since, U y (η, z) = P y D η Pz, and P y • P y = P y , we can write U y (η, z) = P y U y (η, z). Hence, U y (η, P ⊥ y G R ) = P y Sη = P y SP y η, since P y η = η for η ∈ T y N r . Thus we have : Hη = P y (H R + S)P y η, and then H = P y (H R + S)P y . Herein, H can be written with respect to the basis Q as follows : H = Q t (H R + S)Q, which ends the proof.

We describe now the real gradient and Hessian, by using complex variables and their conjugates. Recall from Brandwood [START_REF] Brandwood | A complex gradient operator and its application in adaptive array theory[END_REF] that transforming the pair ( (z), (z)) of real and imaginary parts of a given complex variable z into the pair (z, z) is a simple linear transformation, which allows us to achieve explicit and simple computation of the gradient and Hessian of f . Let

C r = (W, V, V ) ∈ R r × C n×r × C n×r | W ∈ R r , V ∈ C n×r } and K = I r 0 r×2nr 0 2nr×r J (4.3)
where J = I nr iI nr I nr -iI nr . The linear map K is an isomorphism between the R-vector spaces R r and C r . Its inverse is given by K -1 = I r 0 r×2nr 0 2nr×r 1 2 J * . Let f C be the function f seen as a function on C r . Considering f C for the computation of the gradient and the Hessian yields more elegant expressions than considering f R . For this reason, we compute first the gradient and the Hessian of f C , and then we use the isomorphism K in (4.3) to get the real gradient and the Hessian of f R . Lemma 4.2.3. The complex gradient G C can be transformed into the real gradient G R as follows :

G R = K t G C . ( 4

.4) approximation problem

Similarly H R and H C are related by the following formula :

H R = K t H C K. (4.5)
Proof. See [START_REF] Laurent | Unconstrained optimization of real functions in complex variables[END_REF] and the references therein.

Let us describe now explicitly the real gradient G R :

Proposition 4.2.4. The gradient G R of f R on R r is the vector G R =    G 1 (G 2 ) -(G 2 )    ∈ R r+2nr , where -G 1 = ( r i=1 w i ((v * j v i ) d ) -( p(v j ))) 1≤j≤r ∈ R r , -G 2 = (d r i=1 w i w j (v * i v j ) (d-1) vi -w j ∇ x p(v j )) 1≤j≤r ∈ C nr . Proof. We can write f C as f C = 1 2 (f 1 -f 2 -f 3 + f 4 ),
where

f 1 = r i=1 w i (v t i x) d 2 d = |α|=d d α r i=1 w i vα i r i=1 w i v α i
(by definition 2.2.1),

f 2 = r i=1 w i (v t i x) d , p d = r i=1
w i p(v i ) (by 1. in lemma 2.2.1),

f 3 = f2 = r i=1 w i p(v i ), and
f 4 = ||p|| 2 d . Let us decompose G C as G C =    G 1 G2 G3   , with G 1 = ( ∂f C ∂w j ) 1≤j≤r , G2 = ( ∂f C ∂v j ) 1≤j≤r and G3 = ( ∂f C ∂v j ) 1≤j≤r .
As f C is a real valued function, we have that ∂f C ∂v j = ∂f C ∂v j [START_REF] Nehari | Introduction to Complex Analysis[END_REF][START_REF] Remmert | Theory of Complex Functions[END_REF], thus G3 = G2 . Let us start by the computation of G 1 :

∂f 1 ∂w j = ∂ ∂w j |α|=d d α r i=1 w i vα i r i=1 w i v α i = |α|=d d α vα j r i=1 w i v α i + v α j r i=1 w i vα i = r i=1 w i (v * j v i ) d + r i=1 w i (v * i v j ) d = 2 r i=1 w i ((v * j v i ) d );
the third equality is deduced by using definition 2.2.1 and 1. of lemma 2.2.1. In addition, we have ∂f 2 ∂w j = ∂ ∂w j ( r i=1 w i p(v i )) = p(v j ), ∂f 3 ∂w j = p(v j ), and ∂f 4 ∂w j = 0. Thus,

∂f C ∂w j = r i=1 w i ((v * j v i ) d ) -( p(v j )).
w i vα i (w j α k v α-e k j ) = w j r i=1 w i ∂ x k (v t i x) d , (v t j x) d-1 d-1 = dw j r i=1 w i (v t i x) d , x k (v t j x) d-1 d = dw j r i=1 w i vi,k (v * i v j ) d-1 ,
the second (resp. third and fourth) equality are deduced by using lemma 2.2.1. Moreover, we have (4.4). By multiplication of these two matrices, we obtain :

∂f 2 ∂v j,k = 0, ∂f 3 ∂v j,k = w j |α|=d d α vα α k v α-e k j = w j ∂ x k p(v j ), and ∂f 4 ∂v j,k = 0. Thus, ∂f C ∂v j = 1 2 dw j r i=1 w i (v * i v j ) (d-1) vi -w j ∇ x p(v j ) . We have G R = K t G C from
G R =    G 1 G2 + G2 i( G2 -G2 )    =    G 1 2 ( G2 ) -2 ( G2 )   . Finally dividing by 2, we get G R =    G 1 (G 2 ) -(G 2 )   , where G 2 =
2 G2 , which ends the proof.

The matrix of the real Hessian can be computed as follows : Proposition 4.2.5. The real Hessian matrix H R is the following block matrix :

H R =    A (B) t -(B) t (B) (C + D) -(C + D) -(B) (D -C) (D -C)    ∈ R (r+2nr)×(r+2nr) , with -A = ([(v * i v j ) d ] 1≤i,j≤r ) ∈ R r×r , -B = [dw i (v * j v i ) d-1 vj + δ i,j (d r l=1 w l (v * l v i ) d-1 vl -∇ x p(v j ))] 1≤i,j≤r ∈ C nr×r , where δ i,j is the Kronecker delta, -C = diag[d(d -1)[ r i=1 w i w j v i,k v i,l (v * i v j ) d-2 ] 1≤k,l≤n -w j ∆ x p(v j )] 1≤j≤r ∈ C nr×nr , where ∆ x p(v j ) := [∂ x k ∂ x l p(v j )] 1≤k,l≤n , -D = [dw i w j (v * i v j ) d-2 ((v * i v j )I n + (d -1)v j v * i )] 1≤i,j≤r ∈ C nr×nr .
Proof. H C is given by the following block matrix :

H C =          ∂ 2 f C ∂w i ∂w j 1≤i,j≤r ∂ 2 f C ∂w i ∂v t j 1≤i,j≤r ∂ 2 f C ∂w i ∂v t j 1≤i,j≤r ∂ 2 f C ∂v i ∂w j 1≤i,j≤r ∂ 2 f C ∂v i ∂v t j 1≤i,j≤r ∂ 2 f C ∂v i ∂v t j 1≤i,j≤r ∂ 2 f C ∂v i ∂w j 1≤i,j≤r ∂ 2 f C ∂v i ∂v t j 1≤i,j≤r ∂ 2 f C ∂v i ∂v t j 1≤i,j≤r          . We have that ∂ 2 f ∂ z∂ zt = ∂ 2 f ∂z∂z t , and ∂ 2 f ∂z∂ zt = ∂ 2 f ∂ z∂z t ,
for a complex variable z and a real valued function with complex variables f . Using these two relations, we find that ∂ 2 f C ∂w i ∂w j 1≤i,j≤r , 58 CHAPITRE 4 -Riemannian Newton optimization algorithms for the symmetric tensor approximation problem

∂ 2 f C ∂v i ∂w j 1≤i,j≤r , ∂ 2 f C ∂v i ∂v t j 1≤i,j≤r
, and ∂f C ∂v i ∂v t j 1≤i,j≤r determine H C . We denote them respectively by A, B, C, and D. Herein, we can decompose H C as :

H C =    A Bt B * B C Dt B D C    .
The computation of these four matrices can be done by taking the formula of ∂f C ∂w j and ∂f C ∂v j obtained in the proof of proposition 4.2.4, and using the apolar identities in lemma 2.2.1. Using (4.5) we obtain :

H R =    A 2 ( B) t -2 ( B) t 2 ( B) 2 ( C + D) -2 ( C + D) -2 ( B) 2 ( D -C) 2 ( D -C)   .
Finally, for the simplification by 2, as in the previous proof, we redefine the formula of H R as it is given in proposition 4.2.5, where B, C, and D are respectively equal to two times B, C, and D.

Retraction on N r

To complete this Riemannian Newton method, we need to define a retraction operator on N r . Let us assume that the Riemannian Newton equation is solved at a point y = (w, v 1 , . . . , v r , v 1 , . . . v r ) ∈ N r , in local coordinates with respect to the basis Q as in lemma 4.2.1. It yields a solution vector η ∈ R r+r(2n-1) . The tangent vector η ∈ T y N r of size r + 2nr is given by η = Q η = (ν, η 1 , . . . , η r , η 1 , . . . , η r ). The new point R y (η) = ( w, ṽ1 , . . . , ṽr , ṽ 1 , . . . , ṽ r ) ∈ N r is defined using the product of the retractions on each component, that is the identity map on R r and the projection map on the sphere S 2n-1 [4] as follows :

-

w = R w (ν) = w + ν ; -(ṽ j , ṽ j ) = R (v j ;v j ) (η j , η j ) = (v j +η j ;v j +η j )
||(v j +η j ;v j +η j )|| . By lemma 3.3.1, this defines a retraction from T y N r to N r since R w (resp. R (v j ;v j ) ) is a retraction on R r (resp. S 2n-1 ).

Riemannian Gauss-Newton for STA

In this subsection, we consider the STA problem over the product of r Veronese manifolds V n,d . By separating the real and imaginary parts of the coefficients of a polynomial, the non-zero points The differential map DF = Dσ r at y = (y 1 , . . . , y r ) ∈ V r with

(v t x) d with v ∈ C n \ {0} form a smooth Riemannian variety in C[x] d . We equip the R-vector space C[x] d ∼ R 2s n,d with the real inner product : ∀p, q ∈ C[x] d , p, q R d = ( p, q d ). Let V r := V n,d × • • • × V n,d . The map σ r : y = (y 1 , . . . , y r ) ∈ V r → y 1 + • • • + y r ∈ C[x]
y i = (v t i x) d , v i ∈ C n is Dσ r (y) : T y 1 V n,d × • • • × T yr V n,d → T σr(y) C[x] d = C[x] d (η 1 , . . . , η r ) → η 1 + • • • + η r ,
where

T y i V n,d = (u t x)(v t i x) d-1 | u ∈ C n is of dimension 2n over R.
The Gauss-Newton equation is given by :

(DF (y)) * • (DF (y))[η] = -(DF (y)) * [F (y)], (4.6)
where DF (y) : T y V r → T y V r is the Gauss-Newton approximation of the Hessian of f at y. We are going to describe explicitly the matrix of this map in a convenient basis of T y V r . For a non-zero complex vector v ∈ C n , we define the inner product :

T y V r → C[x] d is
∀u, u ∈ C n , u, u v = u * u + (d -1)(u * v)(v * u ) v -2
It is a positive definite inner product on 2 . The symmetric matrix associated to this inner product in the canonical basis of R 2n is

C n ∼ R 2n since u, u v = u 2 +(d-1)|(v * u)| 2 v -2 ≥ 0 and it vanishes iff u = 0. Notice that v, v v = d v
M v := I 2n + (d -1) v -2 (v R v t R + v I v t I ) where v R = ( (v); (v)), v I = (-(v); (v))
are the vectors of R 2n obtained by concatenating the real and imaginary part (resp. opposite imaginary and real part

) of v ∈ C n . Let u 1 = v R v and u 2 = v I v .
We notice that u 1 and u 2 can be completed to an orthonormal basis of R 2n . Let U denotes the matrix of this basis i.e. U = [u 1 , . . . , u 2n ]. Then U U t = I 2n , so that an eigenvalue decomposition of the symmetric matrix M v of •, • v in the canonical basis of R 2n can be written as follows :

M v = U diag(1 + (d -1), 1 + (d -1), 1, . . . , 1)U t = U diag(d, d, 1, . . . , 1)U t . ( 4.7)
For shortness, we denote the strictly positive diagonal matrix diag(d, d, 1, . . . , 1) by S.

Lemma 4.2.6.

Let v = 0 ∈ C n ∼ R 2n and p = (v t x) d ∈ C[x] d . Let u 1 , . . . , u 2n ∈ C n be an orthonormal R-basis for the inner product •, • v with u 1 = v √ d v .
Then

q i = √ d v -d+1 (u t i x)(v t x) d-1 , i = 1, . . . , 2n is an orthonormal basis of T p V n,d for the inner product •, • R d .
Proof. Using the apolar identities in lemma 2.2.1, we have

q i , q j R d = √ d v -d+1 (u t i x)(v t x) d-1 , q j d = √ d -1 v -d+1 (u * i ∇ x q j )(v) = v -2d+2 (u * i u j )(v * v) d-1 + (d -1)(u * i v)(v * u j )(v * v) d-2 = (u * i u j ) + (d -1)(u * i v)(v * u j ) v -2 = u i , u j v .
We deduce that q i , q j R d = δ i,j and (q i ) i=1,...,2n is an orthonormal basis of T (v t x) d V n,d for the inner product •, • R d . We describe now how to compute an orthonormal basis for

•, • v . Lemma 4.2.7. Let M v = U SU t be the eigenvalue decomposition of M v as in (4.7). Let û1 = √ S -1 U t v R √ d v and let Q ∈ R 2n×2n
be the orthogonal factor of a rank-revealing QRdecomposition of I 2n -û1 ût 1 = QRP where R is upper triangular and P is a permutation matrix.

Let u R,1 = v R √ d v , u R,i = U √ S -1 Q [:,i-1] i = 2, . . . , 2n.
Then the orthonormal R-basis

u 1 , . . . , u 2n ∈ C n for •, • v is such that u i = (u R,i ) [1:n] + i (u R,i ) [n+1:2n] ∈ C n for i = 1, . . . , 2n.
Proof. As M v = U SU t with U U t = I 2n and S ∈ R 2n×2n a strictly positive diagonal matrix, we have

√ S -1 U t M v U √ S -1 = I 2n . Thus the column vectors of U √ S -1 form an orthonormal basis of R 2n for •, • v . The vector û1 = √ S -1 U t v R √ d v is representing the vector v R √ d v in this orthonormal basis. The first 2n-1 columns of the factor Q in a rank-revealing QR-decomposition of I 2n -û 1 ût 1 = QRP are orthonormal vectors û2 , . . . , û2n for •, • v , expressed in the basis U √ S -1 . An ortho- normal basis u R,1 , u R,2 , . . . , u R,2n ∈ R 2n for •, • v is thus given by u R,1 = v R √ d v , u R,i = U √ S -1 Q [:,i-1] , i = 2, . . . , 2n. The corresponding vectors ∈ C n are u i = (u R,i ) [1:n] + i (u R,i ) [n+1:2n] ∈ C n for i = 1, . . . , 2n.
Notice that when v is real and u, u are real such that v, u = v, u = 0, u, u v = u, u is the standard inner product of u, u . Consequently in the real case, an orthonormal basis (u i ) i=1,...,n ⊂ R n can be obtained directly from u 1 = v v and a rank-revealing QR-decomposition of I n -u 1 u t 1 . For y = (y 1 , . . . , y r ) ∈ V r with y i = (v t i x) d ∈ V n,d , ∀1 ≤ i ≤ r, let (q i,j ) j=1,...,2n be the orthonormal basis associated to v i defined in lemma 4.2.6 and let Q i = [q i,1 , . . . , q i,2n ] ∈ R 2s n,d ×2n be the coefficient matrix of the polynomials (q i,j ) j=1,...,2n in the canonical R-basis of C[x] d . The columns of the matrix

Q = diag(Q i ) 1≤i≤r ,
represent an orthonormal basis of T y V r for the inner product induced by •, • R d on each component. Therefore, the Jacobian matrix J of σ r at y, which is the matrix associated to Dσ r (y) = DF (y), with respect to the orthonormal basis Q on T y V r and the standard real basis on C[x] d is given by :

J = [Q 1 , . . . , Q r ] ∈ R 2s n,d ×2nr .
Proposition 4.2.8. The Gauss-Newton equation (4.6) in the orthonormal basis

Q of T y V r is of the form H η = -G,
where ηt = (η t 1 , . . . , ηt r ) ∈ R 2nr is the unknown coordinate vector of an element of the tangent space T y V r in the basis Q and

-G = [G k ] k=1,...,2nr with for 1 ≤ i ≤ r, 1 ≤ j ≤ 2n, G 2n(i-1)+j = √ d -1 v i -d+1 d r k=1 (u * i,j v k )(v * i v k ) d-1 ) -u * i,j ∇ x p(v i ) , -H = [H k,k ] 1≤k,k ≤2nr with for 1 ≤ i, i ≤ r, 1 ≤ j, j ≤ 2n, H 2n(i-1)+j,2n(i -1)+j = v i -d+1 v i -d+1 (u * i,j u i ,j )(v * i v i ) d-1 + (d - 1) u * i,j v i )(v * i u i ,j )(v * i v i ) d-2 .
Proof. As the matrix of Dσ r (y) = DF (y) in the orthonormal basis Q on T y V r and the standard real basis on C[x] d is J, we have that the Gauss-Newton equation (4.6) is

H η = -G with -G = J t vec(σ r (y) -p) = ( q i,j , σ r (y) -p R d ), -H = J t J = [Q 1 , . . . , Q r ] t [Q 1 , . . . , Q r ] = ( q i,j , q i ,j R d ).
By the apolar identities in lemma 2.2.1, we have

q i,j , σ r (y) -p R d = √ d -1 v i -d+1 r k=1 (u * i,j ∇ x (v t k x) d (v i ) -u * i,j ∇ x p(v i )) = √ d -1 v i -d+1 r k=1 d(u * i,j v k )(v * i v k ) d-1 -u * i,j ∇ x p(v i ) .
Similarly,

q i,j , q i ,j R d = v i -d+1 v i -d+1 u * i,j ∇ x ((u t i ,j x)(v t i x) d-1 )(v i ) = v i -d+1 v i -d+1 (u * i,j u i ,j )(v * i v i ) d-1 + (d -1)(u * i,j v i )(v * i u i ,j )(v * i v i ) d-2 ,
which ends the proof of the proposition.

The Gauss-Newton equation

H η = -G,
solved in local coordinate with respect to the basis Q, yields a vector η = (η 1 ; . . . ; ηr ) ∈ R 2nr . The components of the tangent vector η = (η 1 , . . . , η r ) ∈ T y V r ∈ C[x] d are then

η i = √ d||v i || -d+1 (v t i x) d-1 2n k=1
ηi,k (u t i,k x), i = 1, . . . , r.

Retraction on the Veronese manifold

We define the retraction of a tangent vector η ∈ T y V r to a new point ỹ on the manifold V r as follows :

ỹ = (ỹ 1 , . . . , ỹr ) = (R y 1 (η 1 ), . . . , R yr (η r )),
where R y i :

T y i V n,d → V n,
H k,d-k p = ( p, x α+β d ) |α|=k,|β|=d-k .
This matrix is also known as the Catalecticant matrix of the symmetric tensor p in degree (k, d -k) or the flattening of p in degree (k, d -k). In this definition, we implicitly assume that we have chosen a monomial ordering (for instance the lexicographic ordering on the monomials indexing the rows and columns of H k,d-k p ) to build the Hankel matrix. The properties of Hankel matrices that we will use are independent of this ordering. Such a matrix is called a Hankel matrix approximation problem since, as in the classical case, the entries of the matrix depend on the sum of the exponents of the monomials indexing the corresponding rows and columns.

When k = 1, using the apolar relations p,

x i x β d = 1 d ∂ x i p, x β d-1 , we see that H 1,d-1
p is nothing else than the transposed of the coefficient matrix of the gradient 1 d ∇ x p in the basis

x β d-1 β -1 |β|=d-1 . When p = (v t x) d ∈ V n,d , H 1,d-1 p
can thus be written as the rank-1 matrix

v ⊗ (v t x) d-1 .
Our construction of a retraction on V n,d is described in the following definition.

Definition 4.2.2. For v ∈ C n \ {0}, let π v : C[x] d → V n,d be the map such that ∀q ∈ C[x] d , π v (q) = ψ(v), q d ψ(v) 2 d ψ(v), ( 4.8)
where

ψ : v ∈ C n → (v t x) d ∈ V n,
R p : T p V n,d → V n,d q → π θ(p+q) (p + q).
The retraction that we are going to describe on the Veronese manifold is closely related to the one on the Segre manifold used in [33]. In fact, since the Segre manifold coincides with the manifold of tensors of multilinear rank (1, . . . , 1), the retraction in [33] is deduced from the truncated multilinear rank (1, . . . , 1) HOSVD of a real multilinear tensor, i.e. from the truncated rank one SVD of the matricization in the different modes [START_REF] Kressner | Low-rank tensor completion by Riemannian optimization[END_REF]. For a symmetric tensor, the matricization with respect to any mode gives the same Catalecticant matrix in degree ( 1, d-1). Hereafter, we show, by different techniques, that a single truncated SVD of the Catalecticant matrix in degree (1, d -1) gives a retraction on the Veronese manifold.

By the apolar identities, we check that R p (q) = (p(ū) + q(ū)) (u t x) d where u = θ(p + q). We also verify that π λ u = π u for any λ ∈ C \ {0} and any u ∈ C n \ {0}.

By the relation (4.8), for any v ∈ C n \ {0}, π v (q) is the vector on the line spanned by ψ(v), which is the closest to q for the apolar norm. In particular, we have π v (ψ(v)) = ψ(v).

We verify now that R p is a retraction on V n,d .

Lemma 4.2.9. Let p ∈ V n,d . Then, p is a fixed point by π u where u is the first left singular vector of

H 1,d-1 p . Proof. If p = (v t x) d = ψ(v) ∈ V n,d with v ∈ C n \ {0}, then the first left singular vector u of H 1,d-1 p is up to a scalar equal to v. Thus we have π u (p) = π v (ψ(v)) = ψ(v) = p.
Proposition 4.2.10. Let p ∈ V n,d . There exists a neighborhood U p ⊂ C[x] d of p such that the map ρ : q ∈ U p → π θ(q) (q) is well-defined and C ∞ smooth.

Proof. Let p ∈ V n,d and θ : q ∈ C[x] d → q ∈ C n where q is the first left singular vector of the SVD decomposition of

H 1,d-1 q . Let γ : C[x] d → V n,d = ψ • θ be the composition map by the parametrization map ψ of V n,d .
By construction, we have ρ : q → q, γ(q) d γ(q). Let O denotes the open set of homogeneous polynomials q ∈ C[x] d such that the Hankel matrix H 1,d-1 q has a nonzero gap between the As

ψ : v ∈ C n → (v t x) d ∈ V n,d
is a parametrization of the Veronese variety V n,d , the tangent space of V n,d at a point ψ(v) is spanned by the first order vectors Dψ(v) q of the Taylor expansion of ψ(v + t q) = ψ(v) + t Dψ(v) q + O(t 2 ) for q ∈ C n . We are going to use this observation to prove the rigidity property of R p . Proposition 4.2.11. For p ∈ V n,d , q ∈ T p (V n,d ),

p + t q -R p (t q) = O(t 2 ).
Proof. As p ∈ V n,d , q ∈ T p V n,d , there exist v, q ∈ C n such that p = ψ(v), q = Dψ(v) q. In particular, we have p + t q -ψ(v

+ t q) = O(t 2 ). This implies that H 1,d-1 p+t q -H 1,d-1 ψ(v+t q) = O(t 2 )
. By differentiability of simple non-zero singular values and their singular vectors [START_REF] Stewart | Matrix Algorithms : Volume II : Eigensystems[END_REF], we have u t -v t = O(t 2 ) where u t = θ(p + t q) and v t = θ(ψ(v + t q)) are respectively the first left singular vectors of H 1,d-1 p+t q and H 1,d-1 ψ(v+t q) . Since H 1,d-1 ψ(v+t q) is a matrix of rank 1 and its image is spanned by v + t q, v t is a non-zero scalar multiple of v + t q and we have π vt = π v+t q . By continuity of the projection on a line, we have

π ut (p + t q) = π vt (p + t q) + O(t 2 ) = π v+t q (p + t q) + O(t 2 ).
Since ψ(v + t q) = ψ(v) + t Dψ(v)q + O(t 2 ) = p + t q + O(t 2 ), we have

π v+t q (p + t q) = π v+t q (ψ(v + t q)) + O(t 2 ) = ψ(v + t q) + O(t 2 ).
We deduce that

p + t q -R p (t q) = p + t q -π ut (p + t q) = p + t q -ψ(v + t q) + (ψ(v + t q) -π vt (p + t q)) + (π vt (p + t q) -π ut (p + t q)) = ψ(v) + t Dψ(v)q -ψ(v + t q) + O(t 2 ) = O(t 2 ),
which proves the proposition. Proposition 4.2.12. Let p ∈ V n,d . The map R p :

T p V n,d → V n,d , q → R p (q) = π θ(p+q) (p + q)
is a retraction operator on the Veronese manifold V n,d .

Proof. We have to prove that R p verifies the three properties in definition 3.3.1.

1. R p (0 p ) = π θ(p) (p + 0 p ) = π θ(p) (p) =
p, by using lemma 4.2.9.

Let S

p : T p V n,d → C[x] d , q → p + q.
(p + tq) -R p (t q) = O(t 2 ),
which implies that d dt R p (t q) | t=0 = q, or equivalently DR p (0 p )q = q. Therefore we have DR p (0 p ) = id TpV n,d .

Adding a trust-region scheme

As discussed in 3.4.3, Riemannian Newton-type algorithm is usually combined with Riemannian trust region method to ensure a sufficient decrease in the cost function, and to enhance the algorithm, with the desirable properties of convergence to a local minimum, with a local superlinear rate of convergence. In this section, we add a Riemannian trust region scheme to the Newton (resp. Gauss-Newton) method described respectively in 4.2.1 and 4.2.2. Moreover, We prove in proposition 4.2.13 that under regularity assumptions, a local convergence for the Riemannian-Newton algorithm with trust region scheme can be obtained.

Let M denote the Riemannian manifold N r in subsection 4.2.1 (resp. V r in subsection 4.2.2), and let y k ∈ M. The subproblem to solve is

min η∈B ∆ k m y k (η), (4.9)
where

m y k (η) := f (y k ) + G t k η + 1 2 η t H k η, G k
is the gradient of f at y k and H k is respectively the Hessian of f at y k for the Riemannian Newton method and the Gauss-Newton approximation of f at y k for the Riemannian Gauss-Newton method, and

B ∆ k := {η ∈ T y k M | ||η|| ≤ ∆ k }.
By solving (4.9) using the dogleg method (3.4.3), we obtain a solution η k ∈ T y k M. Accepting or rejecting the candidate new point

y k+1 = R y k (η k ) is based on the quotient ρ k = f (y k )-f (y k+1 ) my k (0)-my k (η k ) .
If ρ k exceeds 0.2 then the current point y k is updated, otherwise the current point y k remains unchanged. The radius of the trust region ∆ k is also updated based on ρ k . We choose to update the trust region as in [33] with a few changes.

Let ∆ y 0 := 10 We take the initial radius as ∆ 0 = min{∆ y 0 , ∆ max }, if ρ k > 0.6 then the trust region is enlarged as follows :

∆ k+1 = min{2||η k ||, ∆ max }. Otherwise the trust region is shrinked by taking ∆ k+1 = min{( 1 3 + 2 3 (1 + e -14(ρ k -1 3 ) ) -1 )∆ k , ∆ max }.
The algorithm of the Riemannian Newton (resp. Gauss-Newton) method with trust region scheme for the STA problem is denoted by RNE-N-TR (resp. RGN-V-TR) and is given in pseudocode by Algorithm 4.1.

The Algorithm 4.1 is stopped when ∆ k ≤ ∆ min (by default ∆ min = 10 -3 ), or when the maximum number of iterations exceeds N max . Remark 4.2.1 -In order to handle ill-conditioned Hessian (resp. Gauss-Newton Hessian approximation) matrices in Algorithm 4.1, we use the Moore-Penrose pseudoinverse [START_REF] Ake | Numerical Methods for Least Squares Problems[END_REF][START_REF] Konstantinides | Statistical analysis of effective singular values in matrix rank determination[END_REF][START_REF] Stewart | Rank degeneracy[END_REF]. This can appear in cases where some vectors v i of the rank-r approximation span close lines, which Input : The homogeneous polynomial p ∈ C[x] d associated to the symmetric tensor to approximate, r < r g . Choose initial point y 0 ∈ N r (resp. y 0 ∈ V r ). while the method has not converged do 1. Compute the gradient vector and the Hessian matrix (resp. Gauss-Newton Hessian approximation) ;

2. Solve the subproblem (4.9) for the search direction η k ∈ B ∆ k by using the dogleg method ; Algorithm 4.1 -Riemannian Newton (resp. Gauss-Newton) algorithm with trust region sheme for the STA problem "RNE-N-TR"(resp. "RGN-V-TR") yields a singularity problem in the iteration. In particular, this is the case when the symmetric border rank of the symmetric tensor is not equal to its symmetric rank [33,54], [129, section 2.4]. For example, the tensor p

= (v t 0 x)(v t 1 x) d-1 + T , with v 0 , v 1 ∈ R n , T ∈ R[x] d and very small, is close to the tensor (v t 0 x)(v t 1 x) d-1 = lim δ→0 1 d δ (((v 1 + δv 0 ) t x) d -(v t 1 x) d ) of
W, V R , V I ) ∈ N r → r i=1 w i ((v R,i + i v I,i ) t x) d ⊂ Σ r ⊂ C[x] d is locally injective at a non-defective tensor, it defines a local diffeomorphism between N r,0 = ϕ -1 r (Σ 0 r ) and Σ 0 r . As B 0 is compact, N r,0 = ϕ -1 r (Σ 0 r
) is a compact Riemannian manifold. By construction, the distance between p and the iterates p i is decreasing in RNE-N-TR, so that their decomposition is in N r,0 = ϕ -1 r (Σ reg r ∩B 0 ). As N r,0 is a compact Riemannian manifold and f is smooth on N r,0 (as a polynomial function), [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF]Corollary 7.4.6] implies that the iterates of RNE-N-TR of Riemannian Newton method with a trust region sheme on N r,0 converge to a local minimum of the distance function f . The regularity assumption B 0 ∩ Σ r ⊂ Σ reg r implies that the ball centered at p and containing the initial point of the iteration does not contain a defective tensor. In this case, the iterates, which distance to p decreases, remain in the ball and the limit decomposition is a non-defective low rank tensor. This assumption, satisfied when p is far enough from the singular locus of Σ r , is a sufficient condition to ensure the regularity of the iteration points and their limit.

Numerical experiments

In this section, we present four numerical experiments using the RNE-N-TR and RGN-V-TR algorithms. These algorithms are implemented in the package TensorDec.jl * . We use a Julia implementation for the method SPM tested in subsection 4.3.4. The solvers from Tensorlab v3 [START_REF] Vervliet | [END_REF] are run in MATLAB 7.10. The experimentation was done on a Dell Windows desktop with 8 GB memory and Intel Core i5-5300U, 2.3 GHz CPU.

Choice of the initial point

The choice of the initial point is a crucial step in iterative methods. We use the direct algorithm of [START_REF] Harmouch | Structured low rank decomposition of multivariate Hankel matrices[END_REF], based on the computation of generalized eigenvectors and generalized eigenvalues of pencils of Hankel matrices (see also [START_REF] Mourrain | Polynomial-exponential decomposition from moments[END_REF]), to compute an initial rank-r approximation. This algorithm, denoted SMD, works only with r < r g such that ι ≤ d-1

2
where ι denotes the interpolation degree of the points in the rank-r decomposition [START_REF] Eisenbud | The Geometry of Syzygies : A Second Course in Commutative Algebra and Algebraic Geometry[END_REF]Chapter 4]. This implies that r < n+d -1

d where d = d-1
2 . It first computes a SVD decomposition of the Hankel matrix of the tensor t in degree d- 1 2 , d -d-1

2

, extracts the first r singular vectors, computes a simultaneous diagonalisation of the matrices of multiplication by the variables x i by taking a random combination of them, computing its eigenvectors and deducing the points and weights in the approximate decomposition of t. The rationale behind choosing the initial point with this method is when the symmetric tensor is already of symmetric rank r with r < r g and ι ≤ d- 1 2 , then this computation gives a good numerical approximation of the exact decomposition, so that the Riemannian Newton (resp. Gauss-Newton) algorithm needs few iterations to converge numerically. We will see in the following numerical experiments that this initial point is an efficient choice to get a good low rank approximation of a symmetric tensor.

Best rank-1 approximation and spectral norm

Let p ∈ S d n (R), a best real rank-1 approximation of p is a minimizer of the optimization problem dist 1 (p) := min

t∈S d n (R),ranks(t)=1 ||p -t|| 2 d = min (w,v)∈R×S n-1 ||p -w(v t x) d || d , (4.10)
where

S n-1 = {v ∈ R n | ||v|| = 1}
is the unit sphere. This problem is equivalent to

min t∈T d (R n ),rank(t)=1 ||p -t|| 2
F since at least one global minimizer is a symmetric rank-1 tensor [START_REF] Xinzhen | The best rank-1 approximation of a symmetric tensor and related spherical optimization problems[END_REF].

The real spectral norm of p ∈ S d n (R), denoted by ||p|| σ,R is by definition :

||p|| 2 σ,R := max v∈S n-1 |p(v)|. (4.11)
The two problems (4.10) and (4.11) are related by the following equality :

dist 1 (p) 2 = ||p|| 2 d -||p|| 2 σ,R ,
which we deduce by simple calculus and properties of the apolar norm (see also [START_REF] Lathauwer | On the best rank-1 and rank-(R 1 , R 2 , . . ., R n ) approximation of higher-order tensors[END_REF][START_REF] Xinzhen | The best rank-1 approximation of a symmetric tensor and related spherical optimization problems[END_REF]) :

dist 1 (p) 2 = min (w,v)∈R×S n-1 ||p -w(v t x) d || 2 d = min (w,v)∈R×S n-1 ||p|| 2 d -2 p, w(v t x) d d + ||w(v t x) d || 2 d = min (w,v)∈R×S n-1 ||p|| 2 d -2w p(v) + w 2 = min v∈S n-1 ||p|| 2 d -|p(v)| 2 = ||p|| 2 d -max v∈S n-1 |p(v)| 2 = ||p|| 2 d -||p|| 2 σ,R .
Therefore, if v is a global maximizer of (4.11) such that w = p(v), then w v ⊗d is a best rank-1 approximation of p. Herein, a rank-1 approximation w v ⊗d , such that w = p(v) and ||v|| = 1, is better when |w| is higher. Therefore, in the following experimentation, we report the weight w obtained by the different methods.

In [START_REF] Nie | Semidefinite relaxations for best rank-1 tensor approximations[END_REF] the authors present an algorithm called "SDP" based on semidefinite relaxations to find a best real rank-1 approximation of a real symmetric tensor by finding a global optimum of p on S n-1 . We choose two examples from [START_REF] Nie | Semidefinite relaxations for best rank-1 tensor approximations[END_REF], on which we apply the RNE-N-TR with initial point chosen according to the SMD algorithm adapted for 1 × 1 matrices. The reason behind using RNE-N-TR instead of RGN-V-TR is to take advantage of the local quadratic rate of convergence that distinguishes the exact Riemannian Newton iteration in RNE-N-TR [2, Theorem 6.3.2]. We compare these methods with the method CCPD-NLS which is a non-linear least-square solver for the symmetric decomposition from Tensorlab v3 [START_REF] Vervliet | [END_REF] in MATLAB 7.10. For CCPD-NLS we use two initialization strategies, in the first one we run 50 instances (i.e. 50 random initial points obeying Gaussian distributions), and we take the absolute value of the weight in average for this method. In the second one we use the same initialization as RNE-N-TR i.e. CCPD-NLS is initialized by SMD. We also compare with the algorithm CPD from Tensorlab v3 [START_REF] Vervliet | [END_REF] for multilinear tensors initialized by the determinist method from Tensolab called GEVD.

We denote respectively by |w sdp |, |w rne |, |w ccpd_smd |, and |w cpd_gevd | the weight in absolute value given respectively by SDP, RNE-N-TR, CCPD-NLS initialized by SMD, and CPD initialized by GEVD, and |w ccpd | denotes the absolute value of the weight in average given by CCPD-NLS. Note that |w sdp | is the spectral norm of p, since SDP gives a best rank-1 approximation. We report the time spent by SDP from [START_REF] Nie | Semidefinite relaxations for best rank-1 tensor approximations[END_REF] (resp. RNE-N-TR including the computation time of the initial point) in seconds (s) and we denote it by t sdp (resp. t rne ). We denote by N rne the number of iterations in RNE-N-TR. We denote by d 0 the norm between p and the initial point of RNE-N-TR, and by d * the norm between p and the solution obtained by RNE-N-TR. We denote by t ccpd_smd and N ccpd_smd the consumed time and number of iterations of CCPD-NLS initialized by SMD. We denote by t cpd_gevd the time spent by CPD initialized by GEVD. We denote by t ccpd (resp. N ccpd ) the time in seconds (s) (resp. number of iterations) in average for CCPD-NLS. The results in Table 4.1 show that the RNE-N-TR finds a global minimizer, starting from the initial point given by the SMD algorithm. The RNE-N-TR algorithm converges to this point in few iterations, and with very reduced time compared to the SDP algorithm especially when n grows, where the SDP method works in a greedy manner. On the other hand, |w ccpd | is smaller than |w sdp |, implying that CCPD computes, in several cases, a local minimum, which is not a global minimum i.e. a best rank-1 approximation. In comparison for these cases, RGN-V-TR took more iterations (∼20) than RNE-N-TR and consequently more time, while reaching the same optimimum.

(p) i 1 ,i 2 ,i 3 = (-1) i 1 i 1 + (-1) i 2 i 2 + (-1) i 3 i 3 , corresponding to the polynomial p = |α|=3 ( n i=1 α i (-1) i i ) 3 α x α .
The fact that RNE-N-TR finds the best rank-1 approximation in these examples comes from the good initial point provided by SMD algorithm. However, we have no guarantee that RNE-N-TR with this initial point will always converge to a best rank-1 approximation. This experimentation shows that RNE-N-TR combined with SMD algorithm for the initial point is an efficient method to get a good real rank-1 approximation of a real symmetric tensor.

Symmetric rank-r approximation

We consider two examples of a real and a complex valued sparse symmetric tensors, in order to compare the performance of RNE-N-TR and RGN-V-TR with state-of-the-art non-linear leastsquare solvers CCPD-NLS and SDF-NLS for symmetric decomposition from Tensolab v3 with random initial point following a standard normal distribution. We note that ccpd_smd means that CCPD-NLS is initialized by SMD as for the Riemannian Newton and Riemannian Gauss-Newton algorithms. These solvers employ factor matrices as parameterization and use a Gauss-Newton method with dogleg trust region steps called "NLS-GNDL". We also compare with the algorithm CPD from Tensorlab v3 [START_REF] Vervliet | [END_REF] for multilinear tensors initialized by the determinist method from Tensolab called GEVD, we mention that this algorithm does not impose symmetry. We fix 200 iterations as maximal number of iterations, and we run 50 instances for these methods and we report the minimal, median and maximal residual error denoted 'err', such that, err := ||p-p * || d , where p is the symmetric tensor to approximate and p * is the approximate symmetric tensor of rank-r. In the computation of the initial point by SMD algorithm in RNE-N-TR and RGN-V-TR, we compute eigenvectors of a random linear combination of multiplication operators. This computation is sensitive to the choice of the linear combination, when the operators are not commuting, which explains why we report also the minimal, median and maximal err for these two methods. The average of time t is in seconds, and the average number of iterations N is rounded to the closest integer.

(p) i 1 ,i 2 ,i 3 =        i 2 1 + 1 if i 1 = i 2 = i 3 , 1 if [i 1 , i 2 , i 3 ] ≡ [i, i, j] with i = j, 0 otherwise. ([i 1 , i 2 , i 3 ] ≡ [j 1 , j 2 , j 3 ]
iff there exists a permutation σ ∈ S 3 such that [i σ [START_REF] Absil | Joint diagonalization on the oblique manifold for independent component analysis[END_REF] , i σ(2) , i σ(3) ] = [j 1 , j 2 , j 3 ]). This sparse symmetric tensor corresponds to the polynomial p = 10 i=1 i 2 x 3 i + ( 10 i=1 x 2 i ) × ( 10 i=1 x i ).

Example 4.3.4 -Let p ∈ S 3 10 (C) such that :

(p) i 1 ,i 2 ,i 3 =        e √ i 1 +i 2 1 √ -1 + i 1 10 √ -1 if i 1 = i 2 = i 3 , i 10 √ -1 if [i 1 , i 2 , i 3 ] ≡ [i, i, j] with i = j, 0 otherwise.
This sparse symmetric tensor corresponds to the polynomial p = 10 i=1 e

√ i+i 2 √ -1 x 3 i + √ -1( 10 i=1 i 10 x 2 i ) × ( 10 i=1 x i ).
The numerical results in Table 4.2 show that the number of iterations of RNE-N-TR and RGN-V-TR method is low compared to the other methods that impose symmetry in this test i.e. CCPD-NLS initialized by SMD, CCPD-NLS and SDF-NLS with random initialization. The iterations in RNE-N-TR and RGN-V-TR are more expensive. The numerical quality of approximation is better for RNE-N-TR and RGN-V-TR than the other methods that impose symmetry. It is of the same order as the other methods for r = 3, 5 but much better for r = 10. On the other hand, we notice that the approximation obtained by CPD algorithm look better than the ones obtained by RNE-N-TR and RGN-V-TR, even though we checked that the decompositions obtained by CPD are not symmetric. CHAPITRE and CCPD-NLS (ccpd_smd) initialized by 50 points given by SMD, CPD (cpd_gevd) does not impose symmetry (i.e. the three factor matrices obtained by this method are not the same) and it is initialized by default by the Tensorlab's method GEVD, CCPD-NLS (ccpd) and SDF-NLS (sdf) initialized by 50 random initial points obeying Gaussian distributions. The methods RNE-N-TR and RGN-V-TR stop when the maximum number of iterations is reached (fixed to 200) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ). Tensorlab's methods stop when the stop criteria given by Display = 10 are verified.

Approximation of perturbations of low rank symmetric tensors

In this section, we consider perturbations of random low rank tensors. For a given rank r, we choose r random vectors v i of size n, obeying Gaussian distributions and compute the symmetric tensor t = r i=1 (v t i x) d of order d. We choose a random symmetric tensor t err of order d, with coefficients also obeying Gaussian distributions, normalize it so that its apolar norm is and add it to t : t = t + terr terr d . We apply the different approximation algorithms to t and compute the relative error factor ref := t * -t d between the approximation t * of rank r computed by the algorithm and the rank-r tensor t. We run this computation for 100 random instances and report the geometric average of the relative error. The average number of iterations N is rounded to the closest integer, and the average time t is in seconds.

As the initial tensor t is in a ball of radius centered at the tensor t of rank r, we expect t * to be at distance to t smaller than and the relative error factor to be less than 1.

We compare the RNE-N-TR and RGN-V-TR methods with the initial point computed by SMD algorithm, with the recent Subspace Power Method (SPM) of [START_REF] Kileel | Subspace power method for symmetric tensor decomposition and generalized PCA[END_REF] and the state-of-the-art implementation CPD-NLS of the package Tensorlab v3. Note that CPD-NLS is designed for the canonical polyadic decomposition [START_REF] Frank | The expression of a tensor or a polyadic as a sum of products[END_REF]. Nevertheless, in practice it is often observed that applying a general tensor rank approximation method (like CPD-NLS) from a symmetric starting point will usually result in a symmetric approximation. Since CPD-NLS is an efficient tensor decomposition routine of Tensorlab v3, we choose to compare our methods with this algorithm in this numerical experiment, using symmetric initial points and verifying that the obtained tensor approximations are symmetric. As SPM works for even order tensors with real coefficients, the comparison in Table 4.3 is run for tensors in S 4 10 (R). In Table 4.4, we compare CPD-NLS, RNE-N-TR, and RGN-V-TR for tensors in S d 10 (C) of order d = 4 and with complex coefficients. These tables also provide a numerical comparison with the low rank approximation methods tested in Example 5.4 of [START_REF] Nie | Low rank symmetric tensor approximations[END_REF], since the setting is the same. We also run this tensor perturbation test on some complex examples in which the approximation rank is higher than the mode size of the tensor (see Table 4.5). We test this with the three methods RNE-N-TR, RGN-V-TR, and CPD-NLS. We run 20 instances, for each example of tensor and .

The computational time for the methods RNE-N-TR and RGN-V-TR includes the computation of the initial point by the SMD algorithm. We fix 200 iterations as maximal number of iterations for RNE-N-TR, RGN-V-TR and CPD-NLS. For SPM, the iterations are stopped when the distance between two consecutive iterates is less than 10 -10 or when the maximal number of iterations (N = 400 in this experimentation) is reached.

In Tables 4.3, 4.4, the number of iterations of the RNE-N-TR and RGN-V-TR methods is significantly smaller than the number of iterations of the other methods. In SPM, the number of iterations to get an approximation of a single rank-1 term of the approximation is about 30, indicating a practical linear convergence as predicted by the theory [START_REF] Kileel | Subspace power method for symmetric tensor decomposition and generalized PCA[END_REF]Theorem 5.10]. As the method CPD-NLS is based on a quasi-Newton iteration, its local convergence is sub-quadratic, which also explains the relatively high number of iterations. The low number of iterations in RNE-N-TR and RGN-V-TR can be explained by the choice of the initial point by SMD algorithm. This provides a good initialization such that a solution by RNE-N-TR and RGN-V-TR can be obtained in a few number of iterations.

The cost of an iteration appears to be higher in RNE-N-TR and RGN-V-TR than in the other methods. Nevertheless, the total time is of the same order. Note that the cost of an iteration seems higher in RGN-V-TR than RNE-N-TR. Despite the fact that the first algorithm computes the Gauss-Newton approximation of the Hessian matrix, whereas the second algorithm computes the exact Hessian matrix. This can be explained by the use of a parametrization in the first algorithm (i.e. the Cartesian product of Veronese manifolds), which involves a more expensive retraction using SVD decomposition on larger matrices.

These experimentation also show a good numerical behavior for the Riemannian methods. In particular, the numerical quality of the low rank approximation is good for RNE-N-TR and RGN-V-TR, in comparison with SPM and CPD-NLS. The average of the relative error factor in RNE-N-TR and RGN-V-TR is less than 1. The numerical results in [START_REF] Nie | Low rank symmetric tensor approximations[END_REF]Example 5.4] for GP method and small perturbations ( ∈ {10 -2 , 10 -4 , 10 -6 }), shows that the numerical quality in GP-OPT method is worse than with these methods.

We also compare CPD-NLS, RNE-N-TR and RGN-V-TR for perturbation of random tensors of rank r > n and report the minimal and maximal relative error with the average number of iterations N (rounded to the closest integer) and the average time t (in seconds) in Table 4 normal distribution for each tensor instance) RNE-N-TR (rne), and RGN-V-TR (rgn) (initialized by the method SMD for each tensor instance) for rank-r approximations in S 4 10 (R). The method SPM stops when the distance between two consecutive iterates is less than 10 -10 or when the maximal number of iterations (fixed to 400) is reached. The methods RNE-N-TR and RGN-V-TR stop when the maximum number of iterations is reached (fixed to 200) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ). considered cases in Table 4.5 are for the degree d, the number of variables n and the rank r such that (d, n, r) is respectively [START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF]4,[START_REF] Alexander | Polynomial interpolation in several variables[END_REF], [START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF][START_REF] Azzalini | Clustering via nonparametric density estimation : The R package pdfcluster[END_REF][START_REF] Biernacki | Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models[END_REF], [START_REF] Acar | Unsupervised multiway data analysis : A literature survey[END_REF][START_REF] Acar | Scalable tensor factorizations for incomplete data[END_REF][START_REF] Amari | Nonholonomic orthogonal learning algorithms for blind source separation[END_REF], and [START_REF] Afsari | Sensitivity analysis for the problem of matrix joint diagonalization[END_REF][START_REF] Afsari | Some gradient based joint diagonalization methods for ica[END_REF][START_REF] Azzalini | Clustering via nonparametric density estimation : The R package pdfcluster[END_REF]. We see that the maximal relative error factor ref reached by RNE-N-TR and RGN-V-TR with initial point by SMD is less than 1. There is an exception in the first case when = 1, where a large number of iterations is needed for RNE-N-TR and RGN-V-TR. On the other hand, the minimal relative error of CPD-NLS is less than 1 in almost all Table 4.5, whereas its maximal relative error is higher than 1 in all Table 4.5.

This numerical experiment indicates that for these examples of random low rank tensors with random noise, SMD provides a good initial point, close enough to a good solution, so that RNE-N-TR and RGN-V-TR need a few number of iterations. In this context, the combination of an adaptive choice of initial point and a Newton-type method is successful.

Symmetric tensor with large differences in the scale of the weight vector

Consider the case of a real symmetric tensor t = r i=1 w i (v t i x) d , v i = 1, w i > 0, with large differences in the scale of the weights w i i.e. max i w i min i w i is large. More precisely, there are large differences in the norms of the rank-1 symmetric tensors w i (v t i x) d . We randomly sample real symmetric tensors of order d = 3 and dimension n = 7 with r ∈ {5, 10, 15, 20}, according to the normal distribution for each tensor instance), RNE-N-TR, and RGN-V-TR (initialized by the method SMD for each tensor instance) for rank-r approximations in S 4 10 (C). The method CPD-NLS stops when the stop criteria given by Display = 10 in Tensorlab are verified. The methods RNE-N-TR and RGN-V-TR stop when the maximum number of iterations is reached (fixed to 200) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ).

following model :

t = r i=1 10 is r (v t i x) d , v i = 1.
The components of the weight vector increase exponentially from 10 s r to 10 s . We aim to compare the performance of RNE-N-TR and RGN-V-TR methods (hereafter called respectively RNE and RGN for shortness) in this configuration. We run the following test :

-Take t as above, and create a perturbated tensor t p = t t + 10 -5 terr terr , where t err ∈ R[x] d is a random symmetric tensor with coefficients obeying Gaussian distributions ; -run 20 random initial points obeying Gaussian distributions ; -run RNE and RGN with a maximum of iterations N max = 500, and report in average respectively : the relative error (in geometric average) err rel := t t -t * d , where t * is a rank-r symmetric decomposition obtained by these methods ; the number of iterations N iter ; and the computation time t in seconds (s). We also report the number N opt of instances where err rel ≤ 1.1.10 The results in Table 4.6 show that RGN outperforms RNE. In fact, the average of the relative error in RGN is better, up to five order of magnitude, than in RNE. Moreover, starting from the same 20 random initial points in the two methods ; RGN succeeded to reach an optimum, at least in 9 instances with the different order of scale s, while RNE could not find any optimum. Notice that, as we mentionned before, the cost of one iteration in RGN is higher than in RNE. The good performance of RGN compared to RNE in this test was expected, since the orthonormal basis of the tangent space computed in RGN method is independent of the weight factor. This behavior was also observed in [33,Subsection 3.4] for real multilinear tensors, parametrized by Segre manifolds.

Practical session

We present two simple examples to show the implementation of the methods described in this chapter, available in the Julia package TensorDec.jl † .

[2]: using TensorDec using DynamicPolynomials using MomentTools using CSDP, JuMP # The function "Optimizer" is a global optimization solver based on positive␣ →semi-definite programming optimizer = CSDP.Optimizer using LinearAlgebra Example 1: The graph of P in polar coordinates on the sphere looks like this:

Let us use the function "optimizer" to minimize and maximize P on the unit sphere. The maximum evaluation of P in absolute value on the unit sphere (and that is why we have to use both maximize and minimize functions) gives the spectral norm of P and equivalently a best rank-1 approximation of the symmetric tensor associated to P. Dict{String,Any}("d*" => 2.8904022783502117,"d0" => 31.118041861573218,"nIter" => 27,"epsIter" => 0.001,"maxIter" => 500))

The reported error is the apolar norm between P and the approximated polynomial.

The initial error "d0" is ~31.11.

The algorithm rne_n_tr takes 10 iterations and decreases to the final error d* ~7.85, while the algorithm rgn_v_tr decreases to the final error ~2.89, after 27 iterations.

Conclusion

We presented two Riemannian Newton optimization methods for approximating a given complex-valued symmetric tensor by a low rank symmetric tensor. We used in subsection 4.2.1 the weighted normalized factor matrices parametrization for the constraint set. We developed an exact Riemannian Newton iteration with exact computation of the Hessian matrix (RNE-N-TR). We exploited in subsection 4.2.1.1 the properties of the apolar product and of partial complex derivatives, to deduce a simplified and explicit computation of the gradient and Hessian of the square distance function in terms of the points, weights of the decomposition and the tensor to approximate. We proved that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the iteration is converging to a local minimum. In subsection 4.2.2, we parametrized the constraint set via Cartesian product of Veronese manifolds. Taking into account the geometry of the Veronese manifold, we constructed a suitable basis for its tangent space at a given point on this manifold. Using this basis, we developed a Gauss-Newton iteration (RGN-V-TR). In subsection 4.2.2.1, we presented a retraction operator on the Veronese manifold. We showed that, combined with SMD method for choosing the initial point, the two methods have a good practical behavior in several experiments : in subsection 4.3.2 to compute a best real rank-1 approximation of a real symmetric tensor, in subsection 4.3.3 to compute a low rank approximation of sparse symmetric tensors, and in subsection 4.3.4 to compute low rank approximations of random perturbations of low rank symmetric tensors. In subsection 4.3.5, we showed that the numerical behavior of RNE-N-TR is affected by large differences in the scaling of the rank-1 symmetric tensor, where RGN-V-TR outperformed this algorithm in this case.

It was clear throughout the numerical experiments in this chapter, the good impact of the initial point chosen by SMD algorithm, which is based on simultaneous diagonalization of a pencil of matrices built from the symmetric tensor to approximate, on the numerical performance of the Riemannian Newton and the Riemannian Gauss-Newton algorithms presented in this chapter. Herein, in the next chapter, we will focus on the simultaneous diagonalization problem of a pencil of matrices, and its connection to the tensor rank approximation problem.

CHAPTER 5

On the simultaneous matrix diagonalization problem

In this chapter we study the simultaneous matrix diagonalization problem. Mainly, a pencil of matrices M = [M 1 , . . . , M s ] is called in this chapter simultaneously diagonalizable, if there exists two invertible matrices E and F such that Σ i := F M k E is a diagonal matrix, for k ∈ {1, . . . , s}. Our results are presented in three sections. In the first section, we assume that the pencil of matrices is simultaneously diagonalizable, and we construct a Newton-type sequence that converges quadratically towards the solution (E, F, (Σ i ) 1≤i≤s ). Moreover, we exhibit a certification test that the sequence converges towards the solution. In the second section, the considered pencil of matrices is not simultaneously diagonalizable, and thus the objective is to find two matrices E and F that diagonalize approximately the pencil, i.e. to approximate the pencil to a pencil of simultaneously diagonalizable matrices. To solve this problem, we present a Riemannian conjugate gradient algorithm. As an application of the aforementioned algorithm, the third section shows that it can be used to compute an approximated rank-r decomposition for a three dimensional tensor with r higher than the two first dimensional sizes. To this end in the third section, we connect tensor decomposition and simultaneous diagonalization of matrices. We develop an algorithm based on alternate optimization method that combines two steps. The first step uses the Riemannian conjugate gradient algorithm from the second section and the second step solves a linear least-squares problem. Keywords : Simultaneous diagonalization, Newton-type method, eigenproblem, eigenvalues, high precision computation, approximate simultaneous matrix diagonalization, Riemannian conjugate gradient algorithm, alternate optimization algorithm.

82

CHAPITRE 5 -On the simultaneous matrix diagonalization problem

-Newton-type methods for simultaneous matrix diagonalization 83

In this chapter, we present in Section 5.1 a Newton-type approach for the simultaneous matrix diagonalization problem. In Section 5.2, we present a Riemannian conjugate gradient algorithm for the approximate simultaneous matrix diagonalization problem. Finally, we introduce in Section 5.3, an algorithm based on simultaneous diagonalization of matrices for the low rank approximation problem for three dimensional real tensors with approximation rank higher than the first two mode dimensions.

Newton-type methods for simultaneous matrix diagonalization

In this Section we introduce a Newton-type method for the simultaneous matrix diagonalization problem. Sections 5.1.2, 5.1.3, 5.1.4, and 5.1.5 are respectively devoted to constructing a sequence that converges quadratically towards the numerical solution and to provide a certification test for its quadratic convergence for respectively the following systems :

-F E -I n = 0, -the system of one diagonalizable matrix, -the system of two simultaneously diagonalizable matrices, -the system of a pencil of simultaneously diagonalizable matrices. We perform numerical experimentation in Section 5.1.6.

Notation and preliminaries

Throughout this section, we will use the infinity vector norm and the corresponding matrix norm. For a given vector v ∈ C n and matrix M ∈ C n×n , they are respectively given by :

v max = max{|v 1 |, . . . , |v n |} M max = sup v max=1 M v max . Explicitly, M max = max{|m i,1 | + . . . + |m i,n | : 1 ≤ i ≤ n}.
For a second matrix N ∈ C n×n , we have

M + N max M + N max (sub-additivity) M N max M max N max (sub-multiplicativity).
Moreover, for a given matrix M ∈ C n×n , we denote by M L the following :

M L,Tri := max 1≤i≤n 1≤j≤i-1 |m i,j |,
i.e the max matrix norm of the lower triangular part of M, Furthermore, we consider in this section the regular case of diagonalizable matrices, that is, the matrices are diagonalizable with simple eigenvalues. Thus we will use the following notation

W n := {M ∈ C n×n | M with pairwise distinct eigenvalues}. It is well-known that W n is dense in C n×n .
We denote by D n ⊂ C n×n , the vector space of diagonal matrices of size n and D n denotes the subset of D n in which the diagonal matrices are of n distinct diagonal entries. Let E, F ∈ GL n and Σ ∈ D n . The perturbation of respectively E, F and Σ that we consider in this section are of the following form : E + Ė, F + Ḟ and Σ + Σ, where Ė and Ḟ are respectively in T E GL n and T F GL n and Σ is in T Σ D n . As GL n is a Lie group, Ė and Ḟ can be written as EX and Y F such that X, Y are in C n×n (see Section 3.5.4).

As D n is open in D n then T Σ D n = D n , herein Σ = S ∈ D n .
Finally, the perturbations of E, F and Σ that we consider are as follows : E + EX, F + Y F and Σ + S, such that X and Y are in C n×n and S is a diagonal matrix in C n×n .

We state the following lemma which will be used in some of the proofs in this section.

Lemma 5.1.1. Let ϕ(ε, u) = j 0 (1+uε 2 j )-1

εu

. Given ε 1 2 , u 1, and i 0, we have j 0

(1 + uε 2 j+i ) 1 + 2uε 2 i (5.1)
Proof. Modulo taking ε 2 i instead of ε, it suffices to consider the case when i = 0. Now ϕ(ε, u) is an increasing function in ε and u, since its power series expansion in ε and u admits only positive coefficients. Consequently, ϕ(ε, u) ϕ( 12 , 1) = 2.

5.1.2 Newton-type method for the system F E -I n = 0.

Let f : GL n × GL n → C n×n , (E, F ) → F E -I n .
We consider the following perturbations E + EX, F + Y F of respectively E and F where X, Y ∈ C n×n . To define the Newton sequence we have to solve the linear system obtained by canceling the linear part in the Taylor expansion of f (E + EX, F + Y F ). The same methodology will be adopted in the next sections for the other considered systems. Hereafter, we detail the computation of the Newton sequence associated to the system F E -I n = 0. Moreover, a sufficient condition on the initial point for the quadratic convergence of this Newton sequence will be established. Let Z = F E -I n . We observe that

f (E + EX, F + Y F ) = (F + Y F )(E + EX) -I n (5.2) = Z + (Z + I n )X + Y (Z + I n ) + Y (Z + I n )X.
(5.3)

We assume here that Z is of small norm i.e. we start from an initial point (E 0 , F 0 ) close from the solution of the system F E -I n = 0. Consequently, the linear system of first order terms to solve is

Z + X + Y = 0. (5.4)
Hence X = Y = -Z 2 is a solution of Constraint (5.4). Moreover we get, by substituting in Equation ( 5.3) X and Y by -Z 2 ,

(F + Y F )(E + EX) -I n = Z 2 - 3 4 I n + Z 4 . (5.5) Proposition 5.1.2. Let Z 0 = F 0 E 0 -I n . Define X 0 = -Z 0 2 , E 1 = E 0 (I n +X 0 ), F 1 = (I n +X 0 )F 0 and Z 1 = F 1 E 1 -I n . Assume that Z 0 max 1. Then Z 1 max Z 0 2 max
(5.6)

Proof. It follows easily from (5.5).

Theorem 5.1.3. Let E 0 and F 0 two complex square matrices of size n. Let Z 0 = F 0 E 0 -I n and assume that ε = Z 0 max < 1 2 . The sequences defined for i 0

Z i = F i E i -I n X i = - Z i 2 E i+1 = E i (I n + X i ) F i+1 = (I n + X i )F i
converge quadratically towards the solution of F E -I n = 0. Each E i , respectively F i are invertible and, if E ∞ and F ∞ are respectively the limits of sequences (E i ) i 0 and (F i ) i 0 we have for i 0,

E i -E ∞ max (1 + 2ε)2 -2 i+1 +1 ε E 0 max , F i -F ∞ max (1 + 2ε)2 -2 i+1 +1 ε F 0 max .
Proof. Let us prove by induction that Z k max 2 -2 k +1 ε. Since ε < 1 2 , we have

Z k+1 max Z k 2 max from (5.6) ε2 -2 k+1 +2 ε 2 -2 k+1 +1 ε. Consequently Z ∞ = 0. Since X k = -Z k 2 we deduce X k max 2 -2 k ε.
It follows X ∞ = 0. We have

E k = E k-1 (I n + X k-1 ) = E 0 (I n + X 0 ) • • • (I n + X k-1 )
.

Denoting W i = 0 k i (I n + X k ), W ∞ = k 0 (I n + X k ) we compute W ∞ -I n max k 0 (1 + 2 -2 k ε) -1 2ε by using Lemma 5.1.1.
Then W ∞ is invertible and

W -1 ∞ max 1 1 -2ε . Let E ∞ = E 0 W ∞ . Hence E 0 = E ∞ W -1 ∞ . In the same way F 0 = W -1 ∞ F ∞ .
Finally, the identity F ∞ E ∞ -I n = 0 permits to conclude that E 0 and F 0 are invertible. In the same way we prove easily that

W i -I n max 2ε. It follows that W i is invertible. Since E i = E 0 W i we deduce that E i is invertible. Moreover W i -W ∞ max W i max 1 - k i+1 (1 + X k max ) max (1 + W i -I n max ) k 0 (1 + 2 -2 k+i+1 ε) -1 max (1 + 2ε)2 -2 i+1 +1 ε by using Lemma 5.1.1.
We deduce that

E i -E ∞ max (1 + 2ε)2 -2 i+1 +1 ε E 0 max .
These properties also hold for the F i 's. The theorem is proved.

Newton-like method for diagonalizable matrices.

Let M ∈ W n , Σ ∈ D n , E, F ∈ GL n . We aim to construct Newton sequences which converge towards the numerical solution of f (E, F, Σ) = 0 where f :

GL n × GL n × D n → C n×n × C n×n , (E, F, Σ) → (F E -I n , F M E -Σ).
We consider in the same way as before the perturbations E + EX and F + Y F of respectively E and F and in addition the perturbation Σ + S of Σ such that S ∈ D n . We get with Z = F E -I n and ∆ = F M E -Σ :

(F + Y F )(E + EX) -I n = Z + (Z + I n )X + Y (Z + I n ) + Y (Z + I n )X (5.7) (F + Y F )M (E + EX) -Σ -S = F M E -Σ -S + F M EX + Y F M E + Y F M EX = ∆ -S + ΣX + Y Σ + ∆X + Y ∆ + Y (∆ + Σ)X (5.8)
As in the previous subsection we assume that (E, F, Σ) is sufficiently close to the solution of f (E, F, Σ) = 0, thus the linear system that we obtain from (5.7) and (5.8) is

Z + X + Y = 0 ∆ -S + ΣX + Y Σ = 0
The following lemma gives a solution of this linear system.

Lemma 5.1.4. Let Σ = diag(σ 1 , • • • , σ n ), Z = (z i,j
) 1≤i,j≤n and ∆ = (δ i,j ) 1≤i,j≤n be given matrices in C n×n . Assume that σ i = σ j for i = j. Let S, X and Y be matrices defined by S = ddiag(∆ -ZΣ)

(5.9) x i,i = 0

(5.10)

x i,j = -δ i,j + z i,j σ j σ i -σ j , i = j (5.11) y i,i = -z i,i (5.12
)

y i,j = δ i,j -z i,j σ i σ i -σ j , i = j.
(5.13)

5.1 -Newton-type methods for simultaneous matrix diagonalization 87

Then we have

Z + X + Y = 0 (5.14) ∆ -S + ΣX + Y Σ = 0.
(5.15)

Proof. It easy to verify that X + Y + Z = 0. In this way the equation (5.15) is equivalent to

∆ -S -ZΣ + ΣX -XΣ = 0.
Since ddiag(∆ -S -ZΣ) = ddiag(ΣX -XΣ) = 0 the formulas which define X follow easily.

In the next theorem we introduce the Newton sequences associated to the system f (E, F, Σ) = 0 with a sufficient condition on the initial point for its quadratic convergence.

Theorem 5.1.5. Let E 0 , F 0 ∈ GL n and Σ 0 ∈ D n be given such that they define the sequences for i 0,

Z i = F i E i -I n ∆ i = F i M E i -Σ i S i = diag(∆ i -Z i Σ i ) E i+1 = E i (I n + X i ) F i+1 = (I n + Y i )F i Σ i+1 = Σ i + S i ,
where S i , X i and Y i are defined by the formulas (5.9-5.13). Let us define

κ 0 = max 1, max i =j 1 |σ 0,i -σ 0,j | , K 0 = max(1, max i |σ 0,i |) and ε 0 = max(κ 2 0 K 2 0 Z 0 max , κ 2 0 K 0 ∆ 0 max ). Assume that ε 0 0.033. (5.16)
Then the sequences (Σ i, E i , F i ) i 0 converge quadratically to the solution of (F E -I n , F M E -Σ) = 0. More precisely E 0 and F 0 are invertible and

E i -E ∞ max 8.1 × 2 1-2 i+1 E 0 max ε 0 κK F i -F ∞ max 8.1 × 2 1-2 i+1 F 0 max ε 0 κK . Σ i -Σ ∞ max 1.85 × 2 1-2 i ε 0 κ 2 K .
Proof. Let us denote for each i 0,

ε = ε 0 ε i = max(κ 2 i K 2 i Z i max , κ 2 i K i ∆ i max ) κ = κ 0 κ i = max 1, max 1 j<k n 1 |σ i,k -σ i,j | K = K 0 K i = max 1≤k≤n 1, |σ i,k | ,
where σ i,1 , . . . , σ i,n denote the diagonal entries of Σ i . Let us show by induction on i that

ε i 2 1-2 i ε (5.17) Σ i -Σ 0 max (2 -2 2-2 i ) 2a κ ε (5.18) with a = 1 1 -8ε
. These inequalities clearly hold for i = 0. Assuming that the induction hypothesis holds for a given i and let us prove it for i + 1. We first prove that Σ i+1 -Σ 0 max

(2 -2 2-2 i+1 ) 2a κ ε under the assumption Σ i -Σ 0 max (2 -2 2-2 i ) 2a κ ε.
To do this, at the first step we show that this implies K -

4a κ ε K i ≤ K + 4a κ ε and 1 1 + 8aε κ κ i ≤ κ 1 -8aε . Let us prove K - 4a κ ε K i K + 4a κ ε.
We have

K i := Σ i max ≤ Σ 0 max + Σ i -Σ 0 max K + (2 -2 2-2 i ) 2a κ ε K + 4a κ ε K(1 + 4aε).
This implies simultaneously

K i K -|K -K i | K - 4a κ ε and K i K(1 -4aε). Let us show that κ i ≤ κ 1 -8aε
. In fact, if the σ i,j 's are the diagonal values of Σ i , the Weyl's bound [START_REF] Weyl | Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgleichungen (mit einer anwendung auf die theorie der hohlraumstrahlung)[END_REF] implies that

|σ i,j -σ 0,j | Σ i -Σ 0 max 4a κ ε for 1 j n.
So that for 1 j < k n, we obtain using 1 -8aε 0 :

|σ i,k -σ i,j | |σ 0,k -σ 0,j | -|σ i,k -σ 0,k | -|σ i,j -σ 0,j | |σ 0,k -σ 0,j |(1 -κ|σ i,k -σ 0,k | -κ|σ i,j -σ 0,j |) |σ 0,j -σ 0,k |(1 -8aε) 0.
Finally, we get :

κ i κ 1 -8aε .
On the other hand the inequality

|σ i,k -σ i,j | |σ 0,k -σ 0,j | + |σ i,k -σ 0,k | + |σ i,j -σ 0,j |
implies in the same way that above

κ i 1 1 + 8aε κ.
Next we prove (5.18) for i + 1. We know

S i = diag(∆ i -Z i Σ i ). Since ε i = max(κ 2 i K 2 i Z i max , κ 2 i K i ∆ i max ) and κ i , K i 1 then S i max ≤ 2 κ i ε i 2(1 + 8aε) κ 2 1-2 i ε.
It follows :

Σ i+1 -Σ 0 max S i max + Σ i -Σ 0 max 2(1 + 8aε) κ 2 1-2 i ε + (2 -2 2-2 i ) 2a κ ε 2 -2 1-2 i (2 -1) 2a κ ε since 1 + 8aε = a 2 -2 1-2 i 2a κ ε But it is eay to see that 2 1-2 i 2 2-2 i+1
. Finally we get

Σ i+1 -Σ 0 max 2 -2 2-2 i+1 2a κ ε.
Hence we can also write

K i - 2a κ i ε Σ i max -Σ i+1 -Σ i max K i+1 Σ i max + Σ i+1 -Σ i max K i + 2a κ i ε
Using more the Weyl's bound we can easily get that

κ i 1 + 4aε κ i+1 κ i 1 -4aε . Now we bound κ 2 i+1 K 2 i+1 Z i+1 max . We have Z i+1 = Z i X i + Y i Z i + Y i (Z i + I n )X i . Since X i max , Y i max ≤ κ i ( ∆ i max + K i Z i max ) 2 κ i K i ε i , we can write κ 2 i+1 K 2 i+1 Z i+1 max κ 2 i+1 K 2 i+1 κ 3 i K 3 i 4ε 2 i + κ 2 i+1 K 2 i+1 κ 4 i K 4 i 4ε 3 i + κ 2 i+1 K 2 i+1 κ 2 i K 2 i 4ε 2 i 4 (2 + ε i ) κ i+1 K i+1 κ i K i 2 ε 2 i 4 (2 + ε i ) 1 + 2aε 1 -4aε 2 ε 2 i
On the other hand

∆ i+1 = ∆ i X i + Y i ∆ i + Y i (∆ i + Σ i )X i .
Hence

κ 2 i+1 K i+1 ∆ i+1 max κ 2 i+1 K i+1 κ 3 i K 2 i 4ε 2 i + κ 2 i+1 K i+1 κ 4 i K 3 i 4ε 3 i + κ 2 i+1 K i+1 κ 2 i K i 4ε 2 i 4 (2 + ε i ) κ 2 i+1 K i+1 κ 2 i K i ε 2 i 4 (2 + ε i ) 1 + 2aε (1 -4aε) 2 ε 2
ε i+1 4 (2 + ε) 1 + 2aε 1 -4aε 2 ε 2 i 8 (2 + ε) 1 -6ε 1 -12ε 2 ε 2 1-2 i+1 ε 2 1-2 i+1 ε since 8 (2 + ε) 1 -6ε 1 -12ε 2 ε 1 for ε 0.033.
This completes the proof of the two induction hypothesis (5.17-5.18) at order i + 1.

Let W i = i k=0 (I n + X k ). Since X k max 2 κ k K k ε k 2(1 + 8aε) κK(1 -4aε) ε2 1-2 k 2 κK(1 -12ε) ε2 1-2 k
Consequently,

W ∞ -I n max i 0 (1 + 2 κK(1 -12ε) ε2 1-2 i ) -1 4 κK(1 -12ε) ε from Lemma 5.1.1 0.22 κK since ε 0.033.. Hence W ∞ is invertible and E 0 = E ∞ W -1 ∞ . This implies that E 0 is invertible. Moreover, W i -W ∞ max W i max 1 - k i+1 (1 + X k max ) max (1 + W i -I n max ) k 0 (1 + 2 κK(1 -12ε) ε × 2 1-2 k+i+1 ) -1 max (1 + 0.22) × 4 κK(1 -12ε) × 2 1-2 i+1 ε from Lemma 5.1.1 8.1 κK × 2 1-2 i+1 ε.
We deduce that

E i -E ∞ max 8.1 κK × 2 1-2 i+1 E 0 max ε.
In the same way we show that F 0 is invertible and

F i -F ∞ max 8.1 κK × 2 1-2 i+1 F 0 max ε.

-Newton-type methods for simultaneous matrix diagonalization 91

Finally

Σ i -Σ ∞ max k i Σ k+1 -Σ k max k i 2 κ 2 k K k ε k   k 0 2 -2 k   2 1-2 i 2 κ 2 K(1 -12ε)(1 -8ε) ε 0.82 × 2.25 × 2 1-2 i ε κK since k 0 2 -2 k 0.82 and ε 0.033. 1.85 × 2 1-2 i ε 0 .
The theorem is proved.

Proposition 5.1.6. The complexity of one Newton iteration in Theorem 5.1.5 is in O(n 3 ).

Proof. The computation of all the entries x i,j , y i,j of X i and Y i by the formulas (5.9-5.13) requires in total O(n 2 ) arithmetic operations. The computation of Z i , ∆ i , S i , E i+1 , F i+1 , which requires 6 backward stable matrix multiplications and diagonal matrix operations, has a complexity in O(n 3 ). Consequently, the complexity of each iteration is in O(n 3 ).

Remark 5.1.1 -It is possible to generalize this approach to the case where the diagonal matrices are replaced by Jordan matrices.

5.1.4 Newton-like method for two simultaneously diagonalizable matrices.

Let M 1 , M 2 be two commuting matrices in W n , thus M 1 and M 2 are simultaneously diagonalizable. We aim to find E, F ∈ GL n which diagonalize simultaneously M 1 , M 2 so that :

F M k E = Σ k | k ∈ {1, 2}, and Σ 1 , Σ 2 ∈ D n . This equivalent to find the numerical solution of f (E, F, Σ 1 , Σ 2 ) = 0 such that f : (E, F, Σ 1 , Σ 2 ) → (F M 1 E -Σ 1 , F M 1 E -Σ 1 )
We consider as before the perturbations

E + EX, F + Y F and Σ k + S k of respectively E, F and Σ k for k∈ {1, 2} . Letting Z k = FM k E -Σ k for k = 1, 2, we have : (F + YF)M k (E + EX) -(Σ k + S k ) = Z k -S k + Σ k X + Y Σ k + Z k X + YZ k +Y (Z k + Σ k )X (5.19)
By assuming Z 1 , Z 2 are of small norm, the linear system to solve from Equation (5.19) is the following

Z k -S k + Σ k X + Y Σ k = 0, k = 1, 2 (5.20)
A solution of (5.20) is given by the following lemma.

Lemma 5.1.7.

Let Σ k = diag(σ k 1 , • • • , σ k n ), Z k = (z k i,j ) 1≤i,j≤n be given matrices in C n×n for k ∈ {1, 2}. Assume that σ 1 j σ 2 j σ 1 i σ 2 i = 0 for i = j.
Let X, Y , and S k be the matrices defined by

x i,i = 0 (5.21)

x i,j = σ 1 j z 1 i,j σ 2 j z 2 i,j σ 1 i σ 1 j σ 2 i σ 2 j , i = j (5.22)
y i,i = 0 (5.23)

y i,j = - σ 1 i z 1 i,j σ 2 i z 2 i,j σ 1 i σ 1 j σ 2 i σ 2 j , i = j (5.24) S k = ddiag(Z k ), k = 1, 2. (5.25)
Then we have

Z k -S k + Σ k X + Y Σ k = 0, k = 1, 2 (5.26)
Moreover X max , Y max 2κεK (5.27)

where ε = max( Z 1 max , Z 2 max ), κ = max       1, max i =j 1 σ 1 i σ 1 j σ 2 i σ 2 j       , K = max(1, max i,k |σ k i |).
Proof. It is easy to verify that the equation (5.26) implies that for i = j,

σ k i x i,j + σ k j y i,j + z k i,j = 0
and that the solution of these equations is given by the formula (5.22), (5.24). Choosing

x i,i = y i,i =0, we take S k = ddiag(Z k + Σ k X + Y Σ k ) = ddiag(Z k ) since Σ k X + Y Σ k is an off-matrix,
to satisfy the equation (5.26). The bounds (5.27) follows easily from (5.22), (5.24).

Theorem 5.1.8.

Let E 0 , F 0 ∈ GL n and Σ 0,k = diag(σ k 0,1 , . . . , σ k 0,n ) ∈ D n , k = 1, 2
, be given and let define the sequences for i 0 and k = 1, 2 by :

Z i,k = F i M k E i -Σ i,k S i,k = diag(Z i,k ) E i+1 = E i (I n + X i ) F i+1 = (I n + Y i )F i Σ i+1,k = Σ i,k + S i,k ,
5.1 -Newton-type methods for simultaneous matrix diagonalization 93 where X i , Y i are defined by the formulas (5. 21-5.24). Let ε 0 = max( Z 0,1 max , Z 0,2 max ),

κ 0 = max       1, max i =j 1 σ 1 0,i σ 1 0,j σ 2 0,i σ 2 0,j       and K 0 = max(1, max j,k |σ k 0,j |). Assume that u := 4ε 0 κ 2 0 K 3 0 0.094.
(5.28)

Then the sequences (Σ i,k, E i , F i ) i 0 converge quadratically to the solution of F M k E -Σ k for k = 1, 2.
More precisely E 0 and F 0 are invertible and

E i -E ∞ max 1.46 × 2 1-2 i+1 E 0 max u F i -F ∞ max 1.46 × 2 1-2 i+1 F 0 max u.
Proof. Let us denote for each i 0,

ε = ε 0 ε i = max( Z i,1 max , Z i,2 max ) κ = κ 0 κ i = max       1, max 1 j<k n 1 σ 1 i,j σ 1 i,k σ 2 i,j σ 2 i,k       K = K 0 K i = max(1, max j,k (|σ k i,j |)),
where σ k i,1 , . . . , σ k i,n are the diagonal entries of Σ i,k . Let us show by induction on i that

ε i 2 1-2 i ε (5.29) Σ i,k -Σ 0,k max (2 -2 2-2 i )ε (5.30)
These inequalities clearly hold for i = 0. Assuming that the induction hypothesis holds for a given i and let us prove it for i + 1. We can notice that ε i ≤ 1. In fact by induction hypothesis, we have ε i ≤ 2 1-2 i ε 0 and from (5.28)

ε 0 = u 4κ 2 0 K 3 0 ≤ 1, since u ≤ 1 and κ 0 , K 0 ≥ 1. As 2 1-2 i ≤ 1, ∀i ≥ 0, we have ε i ≤ 1. We first prove that Σ i+1,k -Σ 0,k max (2 -2 2-2 i+1 )ε under the assumption Σ i,k -Σ 0,k max (2 -2 2-2 i )ε.
To do this, at the first step we show that this implies

K i ≤ K + 2ε and κ i ≤ κ 1 -8κε(K + ε)
. Let us prove K i K + 2ε. We have

K i := Σ i max ≤ Σ 0 max + Σ i -Σ 0 max K + (2 -2 2-2 i )ε K + 2ε. Let us show that κ i ≤ κ 1 -8κε(K + ε)
. In fact, if the σ i,j k 's are the diagonal values of Σ k i , we have |σ k i,j -σ k 0,j | Σ i,k -Σ 0,k max 2ε for 1 j n and k = 1, 2. It follows :

|σ 1 i,j σ 2 i,k -σ 1 0,j σ 2 0,k | = |σ 1 i,j σ 2 i,k -σ 1 0,j σ 2 i,k + σ 1 0,j σ 2 i,k -σ 1 0,j σ 2 0,k | = |σ 2 i,k (σ 1 i,j -σ 1 0,j ) + σ 1 0,j (σ 2 i,k -σ 2 0,k )| 2ε|σ 2 i,k | + 2ε|σ 1 0,j | 2ε(K + 2ε) + 2εK = 4ε(K + ε).

CHAPITRE 5 -On the simultaneous matrix diagonalization problem

Now,

|σ 1 i,j σ 2 i,k -σ 1 i,k σ 2 i+1,j | |σ 1 0,j σ 2 0,k -σ 1 0,k σ 2 0,j | -|σ 1 0,j σ 2 0,k -σ 1 i+1,j σ 2 i,k | -|σ 1 i,k σ 2 i,j -σ 1 0,k σ 2 0,j | |σ 1 0,j σ 2 0,k -σ 1 0,k σ 2 0,j |(1 -8kε(K + ε)).
Finally, we get :

κ i κ 1 -8κε(K + ε) .
To prove (5.30) it is sufficient to write

Σ i+1,k -Σ 0,k max S i,k max + Σ i+1,k -Σ 0,k max ε i + (2 -2 2-2 i )ε (2 1-2 i + 2 -2 2-2 i )ε (2 -2 2-2 i+1 )ε.
Let us prove (5.29). Since we have

Z i+1,k = Z i,k X i + Y i Z i,k + Y i (Z i,k + Σ i,k )X i .
we deduce

Z i+1,k max 2ε 2 i κ i K i + 2ε 2 i κ i K i + 4ε 2 i κ 2 i K 2 i (ε i + K i ) 4ε 2 i κ 2 i K i + 4ε 2 i κ 2 i K 2 i (1 + K i ) since ε i 1and κ i 1 3 × 4ε 2 i κ 2 i K 3 i = 12ε 2 i κ 2 i K 3 i since K i 1.
It follows

ε i+1 12κ 2 (K + 2ε) 3 (1 -8κε(K + ε)) 2 ε 2 i 12εκ 2 (K + 2ε) 3 (1 -8κε(K + ε)) 2 2 2-2 i+1 ε 3 1 + u 2 3 1 -2u 1 + u 4 2 u2 2-2 i+1 ε since ε K u 4 , κε u 4 2 1-2 i+1 ε since 3 1 + u 2 3 1 -2u 1 + u 4 2
2 -1 for u 0.094.

Let W i = i k=0 (I n + X k ). Since X l max 2κ l K l ε l 2 κ 1 -8κε(K + ε) (K + 2ε)ε2 1-2 l 1 + u 2 u 2 1 -2u 1 + u 4 2 1-2 l 0.65 × 2 1-2 l u since u 0.094.
W ∞ -I n max i 0 (1 + 0.65 × 2 1-2 i u) -1 1.3u from Lemma 5.1.1 1.3 × 0.094 = 0.1222 Hence W ∞ is invertible and E 0 = E ∞ W -1
∞ . This implies that E 0 is invertible. Moreover,

W i -W ∞ max W i max 1 - k i+1 (1 + X k max ) max (1 + W i -I n max ) k 0 (1 + 0.059 × 2 1-2 k+i+1 ) -1 max (1 + 0.1222) × 1.3 × 2 1-2 i+1 u 1.46 × 2 1-2 i+1 u.
We deduce that

E i -E ∞ max 1.46 × 2 1-2 i+1 E 0 max u.
In the same way we show that F 0 is invertible and

F i -F ∞ max 1.46 × 2 1-2 i+1 F 0 max u.
The theorem is proved.

Convergence of a pencil of simultaneously diagonalizable matrices.

In this subsection we present two strategies to solve the system (4.3) of a pencil of commuting matrices (M i ) 1≤i≤s in W n . The first strategy is trivial and consists of finding the common diagonalizers E and F of the pencil by numerically solving one of the systems

(F E -I n , F M 1 E -Σ 1 ) = 0 or (F M 1 E -Σ 1 , F M 2 E -Σ 1 )
= 0 using Theorem 5.1.5 or Theorem 5.1.8. Next we deduce the remaining diagonal matrices Σ i using the formulas

Σ i,k = E(:, k) * M i E(:, k) E(:, k) * E(:, k) 1 k n, 2 or 3 i s,
where E(:, k) is the k-th column in E.

In this strategy we use that a diagonalizer of one or two matrices of the pencil can diagonalize the other matrices of the pencil. We note that, in general, we don't have this property for simultaneously diagonalizable matrices, where, for instance, it is posssible to find a diagonalizer of M 1 which is not a common diagonalizer for the other matrices of the pencil. Nevertheless, this property holds here since we suppose that the matrices M i have simple eigenvalues. Another strategy is to find a "good" linear combination of the M i 's. This is based on Lemma 5.1.9 and Theorem 5.1.10.

CHAPITRE 5 -On the simultaneous matrix diagonalization problem

Lemma 5.1.9. Let us suppose that the M i commute pairwise and they are linearly independent i.e. s i=1

a i M i = 0 ⇒ a i = 0, i = 1 : s. Let E ∈ GL n and Σ i ∈ D n be such that E -1 M i E -Σ i = 0, i = 1 : s.
Let S ∈ C n×s and the column i of S is the diagonal of Σ i . Let σ = (σ 1, . . . , σ n ) and Σ = diag(σ).

Then the matrix S has a full rank and α = (S * S) -1 S * σ satisfies

s i=1 α i E -1 M i E -Σ = 0.
Proof. Since the matrices M i are simultaneously diagonalizable there exists E be such that

E -1 M i E -Σ i = 0. The condition s i=1 α i Σ i -Σ = 0
is written as Sα = σ where S ∈ C n×s . The assumption s i=1 a i M i = 0 ⇒ a i = 0, i = 1 : s implies that the matrix has a full rank. Consequently,

α = (S * S) -1 S * σ.
The lemma follows.

Theorem 5.1.10. Let M 1 , . . . , M p ∈ C n×n be p simultaneously diagonalizable matrices and verify the assumption of linearly independent. Let us consider matrices E 0 , F 0 and Σ 0,i = diag(F 0 M E 0 ), i = 1 : p. Let us define the matrix S ∈ C n×p in which the column i is the diagonal of Σ 0,i . Let σ = 1, e 2iπ n , . . . , e 2i(n-1)π n , Σ = diag(σ) and α = (S * S) -1 S * σ. We consider the system

EF -I n F M E -Σ = 0 (5.31)
where

M = p i=1 α i M i . If n 2 max( Z 0 max , ∆ 0 max ) 16 × 0.033
then (F 0 , E 0 , Σ) satisfies the condition (5.16) of Theorem 5.1.5.

Proof. In this case the quantity κ defined in the Theorem 5.1.5 is equal to

κ = 1 2 |sin π n | n 4 since |sin π n | 2 n for n 2.
Since K 0 = 1 we get

ε 0 = max(κ 2 0 K 2 0 Z 0 max , κ 2 0 K 0 ∆ 0 max ) ≤ n 2 16 max( Z 0 max , ∆ 0 max ).
The condition max( Z 0 max , ∆ 0 max ) ≤ 0.033 16 n 2 , gives the result.

Numerical illustration

We use a JULIA implementation of the Newton sequences in the numerical experiments. The experimentation has been done on a Dell Windows desktop with 8 GB memory and Intel 2.3 GHz CPU. We use the Julia package ArbNumerics for the computation in high precision.

Simulation

In this section we apply the Newton iterations presented in Theorem 5.1.5 (resp. Theorem 5.1.8) on examples of diagonalizable matrices (resp. of two simultaneously diagonalizable matrices). We validate experimentally the sufficiency of the condition established in Theorem 5.1.5 (resp. Theorem 5.1.8) to have a quadratic sequence (Tables 5.1, 5.2, 5.6, and 5.7). On the other hand, as this condition is sufficient but not necessary, we show through some other examples how this Newton sequence starting from an initial point which is not verifying this condition could converge quadratically (Tables 5.3, 5.4, 5.8, and 5.9). We note that the the computation in the aforementioned tables is done in high precision. Nevertheless, we test also the two Newton-type sequences using machine precision (Tables 5.5 and 5.10) and this to show that these sequences have the same numerical behavior of a classical Newton method, i.e., if the solution is in the neighborhood of the initial point the Newton-type iterations will converge towards this solution with a few number of iterations and the residual error obtained at the end is in double precision.

This allows us to have an heuristic estimation on the numerical dependency of the Newton sequences from this condition to converge. Furthermore, these examples reveal the possibility of achieving computation in such problem with high precision. For example, in the case of a diagonalizable matrix of simple eigenvalues, we can compute its eigenvalues using one of the solvers which works with a double precision. Then we take this point as an initial point for the Newton sequence of Theorem 5.1.5 in order to increase the precision. Hereafter, we give some details about the tests : Test-1 for Theorem 5.1.5 and Test-2 for Theorem 5.1.8, considered in this section.

Test-1. Let K = R or C, M = EΣE -1 + 10 -e A
, where e ∈ {3, 6}. The matrices E, Σ, and A ∈ K n×n are chosen randomly following standard normal distributions such that E is invertible, Σ is diagonal with n different diagonal entries and A is a random square matrix obeying normal distribution of size n and Frobenius norm equal to 1. Since M is a small perturbation of EΣE -1 , more precisely M -EΣE -1 = 10 -e , M is a diagonalizable matrix of simple eigenvalues. Herein, we apply the Newton iteration of Theorem 5.1.5 on M with initial point E 0 = E, F 0 = E -1 and Σ 0 = Σ. The residual error reported in this test at iteration k is given by :

err res = max( F k E k -I n max , F k M E k -Σ k max ). Test-2. Let K = R or C, M 1 = F -1 Σ 1 E -1 , M 2 = F -1 Σ 2 E -1
, where E, F , Σ 1 and Σ 2 ∈ K n×n are randomly sampled according to standard normal distributions, such that E and F are invertible, Σ 1 and Σ 2 are diagonal with n different diagonal entries. The Newton iteration in Theorem 5.1.8 is applied on M 1 and M 2 with initial point E 0 , F 0 , Σ 0,1 and Σ 0,2 , such that these matrices are obtained by applying a small perturbation on respectively E, F , Σ 1 and Σ 2 as follows :

E 0 = E + 10 -e A, F 0 = F + 10 -e B, Σ 0,1 = Σ 1 + 10 -e C, Σ 0,2 = Σ 2 + 10 -e D
, where e ∈ {3, 6}, A and B (resp. C and D) are random square matrices (resp. random diagonal matrices with different diagonal entries) of size n and Frobenius norm equal to 1, with entries in K following standard normal distributions. The residual error reported in this test at iteration k is given by :

err res = max( F k M 1 E k -Σ k,1 max , F k M 2 E k -Σ k,2 max ).
We notice that the condition established in Theorem 5.1.5 (resp. Theorem 5.1.8) is reached in Test-1 (resp. Test-2) for matrices of size 10 with order of perturbation equal to 10 -6 , and we can see in Tables 5.1, 5.2, 5.6, and 5.7 that the Newton sequences with initial point verifying the condition in the associated theorem converge quadratically. We can notice also that by increasing the perturbation up to 10 -3 (the initial point does not verify the condition in the associated theorem), the Newton sequences converge quadratically for different sizes of matrices n = 10, 50, 100 (see Tables 5.3, 5.4, 5.8, and 5.9). Moreover, we can notice in Table 5.5 the Newton-type iteration of Theorem 5.1.5 applied in double precision converges with a few number of iterations ∼ 5 and the final residual error measured with the Frobenius norm is of order machine precision ∼ 10 -14 and it is of the same order obtained by the standard Julia method eigen to compute the eigen decomposition. The same remarks are valid for Table 5.10 where the Newton-type sequence of Theorem 5.1.8 needs, in double precision, a few iterations to converges towards the solution given by using the Frobenius norm a residual error of order machine precision.

Table 5.1 -The computational results throughout 7 iterations of an example of implementation of Test-1 with K = R, n = 10 and e = 6 in precision 1024. Table 5.2 -The computational results throughout 7 iterations of an example of implementation of Test-1 with K = C, n = 10 and e = 6 in precision 1024. Table 5.5 -The residual error throughout 5 iterations given by the implementation of Test-1 with K = R, e = 3 and n = 10, 20, 30, in double precision.

Iteration ε := max(κ 2 0 K 2 0 Z 0 max , κ 2 0 K 0 ∆ 0 max ) ≤ 0.
Iteration ε := max(κ 2 0 K 2 0 Z 0 max , κ 2 0 K 0 ∆ 0 max ) ≤ 0.
Iteration n = 10 n = 20 n = 30 1 4.78e -3 1.01e -2 1.01e -2 2 4.71e -3 2.55e -3 1.14e -3 3 2.29e -5 1.97e -5 4.08e -7 4
1.43e -9 2.36e -10 2.26e -13 5 4.06e -15 1.23e -14 5.04e -14

M -E eigen Σ eigen E -1 eigen 9.49e -15 2.83e -14 7.45e -14 M -E newton Σ newton E -1 newton 2.96e -15 1.01e -14 3.42e -14

Cauchy matrix

In this section we present an example for a Cauchy matrix of size n = 13 of entries a i,j = 1 i+j , ∀1 ≤ i, j ≤ 13, that illustrates how the Newton-type iteration can be used to increase the accuracy of the eigenvalues. We take the eigen decomposition given by the standard JULIA method eigen from the package LinearAlgebra as an initial point of Newton sequences in Theorem 5.1.5 with 5 iterations. The computation is done with the precision 1024 using ArbNumerics package. The initial point given by eigen is in double precision. It is converted to the precision 1024 using ArbNumerics package, in order to apply Newtons iterations with this precision of 1024 bits. In Table 5.11 we report the eigenvalues given by eigen (σ eigen ) Table 5.6 -The computational results throughout 7 iterations of an example of implementation of Test-2 with K = R, n = 10 and e = 6 in precision 1024. and the eigenvalues rounded to the double precision given by Newton-type sequence (σ newton ) initialized with eigen. We also report the relative error |σnewton-σ eigen | σnewton in order to show the refinement amount realized by the Newton method. As we can see the matrix of this example is ill-conditioned (Cauchy matrices are in general ill-conditioned). There is a cluster of eigenvalues nearby zero. The accuracy enhancement obtained by applying Newton-type iterations can be clearly seen in Table 5.11, in particular for the first four smallest eigenvalues. For instance, the smallest eigenvalue returned by eigen is of order 10 -17 close to the second smallest eigenvalues of order 10 -16 . Newton-type method shows that the smallest eigenvalue of the order 10 -19 yields a large relative error ∼ 39.33. This also shows that all the eigenvalues are well-separated.

Iteration 4κ 2 K 3 ε ≤ 0.094

Sub-matrix iterations

It is possible to adapt the proposed method, taking into account the condition of the eigenvalue σ i given by the quantity

κ(σ i ) = max i =j 1, 1 |σ i -σ j |
Theoretical results imply that the computation of clusters of eigenvalues is ill-conditioned. However, one can apply Theorem 3 on sub-matrices to improve the well-conditioned eigenvalues. We denote δ = K ∆ 0 max 0.033 i ≤ p and i < j n. We adapt Newton iteration to the block associated with the well-conditioned eigenvalues by defining the matrices Table 5.11 -The relative error between σ eigen from the method eigen and σ newton from the Newton-type method for the Cauchy matrix X, Y and S as follows :

x i,i = 0

x i,j =    -δ i,j + z i,j σ j σ i -σj if |σ i -σ j | > δ 0 otherwise Y = -Z -X S = diag(-∆ + ZΣ).
Table 5.12 (resp. Table 5.13) shows the residual error err res as in Test-1 for the Cauchy matrix of size 200 (resp. the Rosser matrix of size 256 [START_REF] Rosser | Separation of close eigen-values of a real symmetric matrix[END_REF]) by applying the aforementioned sequences, the initial point is given by the Julia method eigen. The computation is done in precision 1024. Summary of this section. Taking a Newton approach towards systems of equations describing the simultaneous diagonalization problem of diagonalizable matrices, leads us to new algorithmic insights. We exhibit a Newton-type method without solving a linear system at each step as is the case of a classical Newton method. The numerical experiments corroborate the quadratic convergence predicted by the theoretical analysis.

Riemannian conjugate gradient algorithm for approximate simultaneous diagonalization of matrices

In the previous section, we studied the certification problem for the convergence of a pencil of simultaneously diagonalizable matrices. In this section, we consider the second part of the simultaneous matrix diagonalization problem, where the pencil of matrices is not necessarily simultaneously diagonalizable. Thus we aim to approximate it locally by a pencil of matrices which is simultaneously diagonalizable. We study this problem from an optimization point of view by taking into account the geometric constraints of the optimization problem and we present a Riemannian conjugate gradient algorithm. We consider the approximation problem over the real field R. Given a general pencil M = [M 1 , . . . , M s ] of real square matrices, such that M k ∈ R n×n , ∀k ∈ {1, . . . , s}, we aim to compute a simultaneously diagonalizable pencil that approximates M i.e. to find two invertible matrices E and F such that F M k E t is the most diagonal, ∀k ∈ {1, . . . , s}. For shortness we refer to the Approximate Simultaneous Diagonalization of matrices problem as ASD problem.

Notation. In this section we consider the real field. We denote by D * n the group of n × n nonsingular diagonal matrices, and by P n the group of n × n permutation matrices.

Cost function

The first cost function that comes in mind is the one which minimizes the Frobenius norm of the off-diagonal entries of each matrix in the pencil M :

f (E, F ) = 1 2 s k=1 F M k E t -ddiag(F M k E t ) 2 = 1 2 s k=1 off(F M k E t ) 2 , (E, F ) ∈ GL n ×GL n .
However, as discussed in [START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF], this function is not invariant by diagonal scaling i.e.

f (Σ 1 E, Σ 2 F ) = f (E, F ) with Σ 1 , Σ 2 ∈ D *
n , which might produce in practice undesirable CHAPITRE 5 -On the simultaneous matrix diagonalization problem degenerate solutions, since ASD is an optimization problem on GL n . For this reason, we choose to conduct the optimization method using the following cost function

f (E, F ) = 1 2 s k=1 M k -F -1 ddiag(F M k E t )E -t 2 , (E, F ) ∈ GL n × GL n , ( 5.32)
which generalizes the one proposed in [START_REF] Afsari | Sensitivity analysis for the problem of matrix joint diagonalization[END_REF] for a pencil of symmetric matrices, and it is invariant to the diagonal scaling. Herein, approximate simultaneous diagonalization of matrices problem (ASD) consists of minimizing the cost function (5.32) :

min (E,F )∈GLn×GLn f (E, F ) = min (E,F )∈GLn×GLn 1 2 s k=1 M k -F -1 ddiag(F M k E t )E -t 2 .
(5.33)

Oblique geometric constraints

It is easy to notice that if (E, F ) ∈ GL n × GL n is a solution of (5.33), then (EΣ

1 P 1 , P 2 Σ 2 F ), ∀P 1 , P 2 ∈ P n , ∀(Σ 1 , Σ 2 ) ∈ D * n × D *
n is also a solution. In practice, the permutation has no impact on the solution process. Contrarily, the diagonal scaling can conduct to degenerate solutions (we can construct sequences of non-singular diagonal matrices which converge towards singular matrices). Hence, to avoid degenerate solutions, some additional constraints must be added to the constraint set. In this regard, there exists in the literature, various approaches (see. [START_REF] Absil | Joint diagonalization on the oblique manifold for independent component analysis[END_REF][START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF][START_REF] Bouchard | Riemannian optimization and approximate joint diagonalization for blind source separation[END_REF][START_REF] Tichavsky | Fast approximate joint diagonalization incorporating weight matrices[END_REF][START_REF] Amari | Nonholonomic orthogonal learning algorithms for blind source separation[END_REF]). From the existing possibilities, we choose to fix the norm of the rows of E and F with the oblique constraint i.e. the rows of E and F are of unit Frobenius norm [START_REF] Absil | Joint diagonalization on the oblique manifold for independent component analysis[END_REF][START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF]. This means that we seek E and F in the oblique manifold M o n previously defined in Section 3.5.5. Finally, we formulate the ASD problem as a Riemannian least-squares problem on M o n × M o n as follows :

min (E,F )∈M o n ×M o n f (E, F ) = min (E,F )∈M o n ×M o n 1 2 s k=1 M k -F -1 ddiag(F M k E t )E -t 2 . (ASD)
We choose to equip the oblique manifold M o n with the right-invariant metric inherited from GL n (see Sections 3.5.4 and 3.5.5). In Definition 5.2.1 we define the metric that we equip with the Cartesian product of oblique manifolds

M o n × M o n . Definition 5.2.1. For (E, F ) ∈ M o n × M o n , ξ = (ξ E , ξ F ), η = (η E , η F ) ∈ T E M o n × T F M o n T (E,F ) M o n × M o n , then ξ, η r (E,F ) = (ξ E , ξ F ), (η E , η F ) r (E,F ) = ξ E , η E r E + ξ F , η F r F ,
where ., . r E (resp. ., . r F ) is as in Section 3.5.4.

The proposed Riemannian conjugate gradient method

We introduce a Riemannian conjugate gradient method with Armijo backtracking line-search step (RCG) (Section 3.4.1) to solve the (ASD). The algorithm (RCG) for ASD problem in pseudocode is given as follows :

5.2 -Riemannian conjugate gradient algorithm for approximate simultaneous diagonalization of matrices 105

Require: initial iterate X 1 := (E 1 , F 1 ) ∈ M o n × M o n , tangent vector η 0 := (η E 0 , η F 0 ) = (0, 0). 1: for i = 1, 2, . . . do

2:

Compute the gradient ξ i := grad ob f (X i ) ; 3:

Compute a conjugate direction η

i := -ξ i + β i T X i-1 →X i (η i-1 ) ; 4:
Perform Armijo backtracking to find the smallest integer m ≥ 0 such that

f (X i ) -f (R X i (0.5 m η i )) ≥ -0.1 × 0.5 m ξ i , η i r X i ; 5:
Compute the next new point X i+1 := R X i (0.5 m η i ) ;

6: end for Algorithm 5.1 -Geometric conjugate gradient for ASD.

Hereafter, the different steps of the algorithm will be explained in more details. In step 2 of Algorithm 5.1, the computation of the Riemannian gradient grad ob f (X) of f at X := (E, F ) ∈ M o n × M o n with respect to the right-invariant metric relies on some matrix trace properties that we recall in the following lemma * : Lemma 5. ., . is the canonical inner product in R n×n (i.e. A, B = tr(A t B)).

To compute this gradient, we first write f as a trace function by using Lemma 5.2.1 :

f (E, F ) = s k=1 f k (E, F ) such that, f k (E, F ) = 1 2 M k -F -1 ddiag(F M k E t )E -t 2 = 1 2 tr((M k -F -1 ddiag(F M k E t )E -t ) t (M k -F -1 ddiag(F M k E t )E -t )) = 1 2 tr(M k M t k -2E -1 ddiag(F M k E t )F -t F -1 ddiag(F M k E t )E -t ). Let f F := f (., F ) : M o n → R, E → f (E, F ); f E := f (E, .) : M o n → R, F → f (E, F ). We have, grad ob f (E, F ) = (grad ob f F (E), grad ob f E (F )) = s k=1 grad ob f F k (E), s k=1 grad ob f E k (F ) ,
where grad ob f F k (E) and grad ob f E k (F ) denote respectively the Riemannian gradient of

f F k = f k (., F ) : M o n → R, E → f k (E, F ) at E ∈ M o n and of f E k = f k (E, .) : M o n → R, F → 106
CHAPITRE 5 -On the simultaneous matrix diagonalization problem

f k (E, F ) at F ∈ M o n
, equipped with the right-invariant metric. We show how to compute grad ob f F k (E) then the same procedure can be applied to compute grad ob f E k (F ). To compute the Riemannian gradient of f F k at E in M o n endowed with the right-invariant metric, we recall from Table 3.1 that :

grad ob f F k (E) = P E ob,r (grad f F k (E)),
where P E ob,r is the orthogonal projection map into T E M o n according to the right-invariant metric already been given in the same table 3.1, and grad f F k (E) is the Riemannian gradient of f F k at E ∈ GL n equipped with the right-invariant metric given by (see Section 3.5.4 ) :

grad f F k (E) = grad Euc f F k (E)E t E, such that grad Euc f F k (E) is the Euclidean gradient of f F k at E ∈ GL n . Next, to compute grad Euc f F k (E) we compute the time derivative of f F k ( ḟ F k )
in terms of the time derivative of E ( Ė ∈ T E GL n ) by using the trace identities in Lemma 5.2.1 in order to obtain

ḟ F k (E) = tr(X Ė),
on the other hand we have that ḟ F k (E) = tr((grad Euc f F k (E)) t Ė), herein we deduce

grad Euc f F k (E) = X t .
Finally, the formulas of grad Euc f F k (E) and grad Euc f E k (F ) are given by :

grad Euc f F k (E) = [Q k (E, F ) ddiag(FM k E t ) -ddiag(Q k (E, F )) FM k E t ]E -t , grad Euc f E k (F ) = [Q k (E, F ) t ddiag(FM k E t ) -ddiag(Q k (E, F )) EM t k F t ]F -t , with Q k (E, F ) = (EE t ) -1 (EM t k F t -ddiag(FM k E t ))(F F t ) -1 . For shortness, ξ E i (resp. ξ F i ) denotes hereafter grad ob f F i (E i ) (resp. grad ob f E i (F i )). In step 3, η i := (η E i , η F i ) ∈ T E i M o n × T F i M o n such that : η i := -ξ i + β i T X i-1 →X i (η i-1 ) = -(ξ E i , ξ F i ) + β i (T ob,r E i-1 →E i (η E i-1 ), T ob,r F i-1 →F i (η F i-1 )),
where T ob,r E i-1 →E i (resp. T ob,r F i-1 →F i ) is the vector transport on M o n equipped with the right-invariant metric from Table 3.1. We use the Polak-Ribière formula in (3.8) for β i :

β i = ξ i , ξ i -T X i-1 →X i (ξ i-1 ) r X i ξ i , ξ i r X i
In step 5, we define the new point

X i+1 := (E i+1 , F i+1 ) ∈ M o n × M o n such that : X i+1 := R X i (0.5 m η i ) = R (E i ,F i ) (0.5 m η E i , 0.5 m η F i ) = (R ob,r E i (0.5 m η E i ), R ob,r F i (0.5 m η F i )),
where R ob,r E i (resp. R ob,r F i ) is the retraction on M o n for E i ∈ M o n (resp. for F i ∈ M o n ) according to the right-invariant metric from Table 3.1.

The algorithm stops when grad ob f (X i ) ≤ tolerence.

2. Extend the pencil M to a pencil M = [ M1 , . . . , Mn 3 ] with Mi of size r × r such that the block of the first n 1 rows with n 2 columns in Mi is equal to M i in M , and the remain entries of Mi are sampled randomly, for instance, according to a Gaussian distribution.

3. Approximate the pencil M locally by a pencil of simultaneously diagonalizable matrices M by using Algorithm 5.1 in Section 5.2, i.e. find (E, F ) ∈ M o r ×M o r that solves locally :

min (E,F )∈M o r ×M o r f 1 (E, F ) = min (E,F )∈M o r ×M o r 1 2 n 3 k=1 Mk -F -1 ddiag(F Mk E t )E -t 2 .
4. Recall from (5.35) that the tensor associated to the pencil of simultaneously diagonalizable matrices M can be written as a tensor rank-r decomposition with the factors E, F and Σ. This means, that the tensor associated to M is approximated locally by the tensor rank-r decomposition given by M .

5. Since our goal is to find a rank-r approximation of the tensor M, such that its slices M[:, :, k] are fixed blocks in the matrices Mk of the pencil M , the distance function

1 2 n 3
k=1 Mk -F -1 ddiag(F Mk E t )E -t 2 can be reduced, by fixing the obtained matrices E, F and optimizing all the entries of the matrices Mk except of those in the fixed blocks. This leads us to the following linear least-squares problem :

min M f 2 ( M ) = min M 1 2 n 3 k=1 Mk -F -1 ddiag(F Mk E t )E -t 2 , s.t. Mk [1 : n 1 , 1 : n 2 ] = M k , for k ∈ {1, . . . , n 3 }. (5.36)
In summary, we minimize the function (5.37) by alternating between two optimization minimization problems. The first updates E and F by minimizing f 1 in step 3 using the Riemannian conjugate gradient iteration from Section 5.2, and the second updates the entries which don't belong to the fixed blocks in Mk of the pencil M by solving the linear least-squares problem with f 2 in step 5.

f (E, F, M ) = 1 2 n 3 k=1 Mk -F -1 ddiag(F Mk E t )E -t 2 s.t. (E, F ) ∈ M o r × M o r , Mk [1 : n 1 , 1 : n 2 ] = M k , for k ∈ {1, . . . , n 3 },
6. Hence, we present an iterative algorithm such that at each iteration the alternate optimization method from step 5 is applied. This is an alternate optimization algorithm (AO). The algorithm stops when f (E, F, M ) ≤ tolerance, or when the number of iterations reaches a maximum number N max .

7. At the end, when the algorithm stops, an approximated rank-r decomposition is obtained for the tensor associated to a pencil M , from which we can extract an approximated rank-r decomposition for the targeted tensor M.

The Alternate Optimization algorithm (AO) is given in pseudo-code in Algorithm 5.2. We detail the linear least-squares problem in (5.36) Let H := Mk -F -1 ddiag(F Mk E t )E -t ∈ R r×r , let h i,j be the entry of H in the i-th row and the j-th column. We have h i,j = e i,j , H , where (e i,j ) 1≤i,j≤r is the canonical basis of R r×r . By developing the calculus using the trace properties in Lemma 5.2.1 in order to isolate our targeted variable Mk , we find :

h i,j = e i,j -F t ddiag(E -1 [j, :] ⊗ F -1 [i, :])E, Mk .
Hence, minimizing the Frobenius norm of h i,j is equivalent to minimizing the Frobenius norm of (vec(e i,j -F t ddiag(E -1 [j, :] ⊗ F -1 [i, :])E)) t vec( Mk ). This leads us to the following system :

GX = 0,
where G = [(vec(e i,j -

F t ddiag(E -1 [j, :] ⊗ F -1 [i, :])E)) t ] ∈ R r 2 ×r 2 , X = vec( Mk ) ∈ R r 2 .
Recall that we have n 1 n 2 fixed entries in Mk equal to the n 1 n 2 entries of M k , thus we write the system

GX = 0 as G = [G 1 , G 2 ], X = [X 1 , X 2 ], such that X 2 ∈ R n 1 n 2 represents the fixed known entries in Mk and G 2 ∈ R r 2 ×n 1 n 2 contains the corresponding columns to X 2 in G. Simi- larly, X 1 ∈ R r 2 -n 1 n 2 represents the vector of the unknown entries of Mk and G 1 ∈ R r 2 ×(r 2 -n 1 n 2 )
contains the corresponding columns to X 2 in G. Hence, the system to solve becomes :

G 1 X 1 = Y, ( 5.38)
where Y := -G 2 X 2 . Consequently, we can write

X 1 = G † 1 X 2 ,
where G † 1 denotes the pseudo-inverse of G 1 . Note that if G 1 is of full column rank then X 1 is the least-squares solution of the overdetermined system (5.38)

with G † 1 = (G t 1 G 1 ) -1 G t 1 .
For step 8 in Algorithm 5.2 the three factors A, B and C are computed by simply taking the block

( Mk -F -1 end ddiag(F end Mk E t end )E -t end )[1 : n 1 , 1 : n 2 ] =M k -(F -1 end ddiag(F end Mk E t end )E -t end )[1 : n 1 , 1 : n 2 ], since Mk [1 : n 1 , 1 : n 2 ] = M k , where (F -1 end ddiag(F end Mk E t end )E -t end )[1 : n 1 , 1 : n 2 ] =F -1 end [1 : n 1 , 1 : r] ddiag(F end Mk E t end )E -t end [1 : r, 1 : n 2 ].
Thus the pencil M is approximated by the pencil of simultaneously diagonalizable matrices

[F -1 end [1 : n 1 , 1 : r] ddiag(F end Mk E t end )E -t end [1 : r, 1 : n 2 ]]
A = F -1 end [1 : n 1 , 1 : r]; B = E -t end [1 : n 2 , 1 : r]; C = Σ t , where Σ is of size r × n 3 , such that Σ[:, k] = diag(F end Mend,k E t end ).

CHAPITRE 5 -On the simultaneous matrix diagonalization problem

Herein, the tensor associated to the pencil M is approximated by the rank-r decompostion given by the three factor matrices A, B and C. Remark 5.3.1 -If the tensor to approximate is of size (n, n, s) and the approximation rank r is equal to n, there is no need to have the extension step in the Algorithm 5.2, and the algorithm is reduced simply to Algorithm 5.1 for approximate simultaneous diagonalization of matrices.

Require: M ∈ T(n 1 , n 2 , n 3 ), approximation rank r ≥ max(n 1 , n 2 ).

1: Compute M the pencil associated to M ; 2: Extend M to the pencil M of n 3 matrices, such that Mk [1 : n 1 , 1 :

n 2 ] = M k for k ∈ {1, . . . , n 3 } ; 3: Set initial iterate (E 0 , F 0 , M0 := M ), with (E 0 , F 0 ) ∈ M o r × M o r . 4: for i = 1, 2, . . . do 5:
Obtain (E i , F i ) by applying Algorithm 5.1 initialized by (E i-1 , F i-1 ) to solve :

min (E,F )∈M o r ×M o r f 1 (E, F ) = min (E,F )∈M o r ×M o r 1 2 n 3 k=1 Mi-1,k -F -1 ddiag(F Mi-1,k E t )E -t 2 . 6:
Fix (E i , F i ) and update Mi-1 to Mi by solving : (5.39)

min M f 2 ( M ) = min M 1 2 n 3 k=1 Mk -F -1 i ddiag(F i Mk E t i )E -t
7: end for 8: Extract factor matrices A ∈ R n 1 ×r , B ∈ R n 2 ×r , C ∈ R n 3 ×r , such that :

A = F -1 end [1 : n 1 , 1 : r]; B = E -t end [1 : n 2 , 1 : r]; C = Σ t , where Σ is of size r × n 3 , such that Σ[:, k] = diag(F end Mend,k E t end ).
Algorithm 5.2 -Alternate optimization algorithm (AO) for tensor rank approximation of tensors in T(n 1 , n 2 , n 3 ) with r ≥ max(n 1 , n 2 ).

Example 5.3.1 -The operator of matrix multiplication of square matrices of size 2 × 2 can be represented by the homogeneous polynomial T (x, y, z) = x 1 y 1 z 1 + x 1 y 3 z 3 + x 2 y 1 z 2 + x 2 y 3 z 4 + x 3 y 2 z 1 + x 3 y 4 z 3 + x 4 y 2 z 2 + x 4 y 4 z 4 of degree 3 in three variables x = (x 1 , ..., x 4 ), y = (y 1 , ..., y 4 ) and z = (z 1 , ..., z 4 ). Let T ∈ T(4, 4, 4) be the associated tensor to the aforementioned polynomial. By taking all the monomials of degree three that we can construct with x i , y j and z k , for i, j, k ∈ {1, . . . , 4} and considering the lexicographic ordering on the monomials indexing the rows, columns and slices of T , the tensor T is as follows :

T [1, :, :] =     
1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0

     T [2, :, :] =      0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0      T [3, :, :] =      0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0      T [4, :, :] =      0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1     
The tensor T is of rank 7 (see. [START_REF] Winograd | On multiplication of 2 × 2 matrices[END_REF]), whereas the generic rank is 6. Since its rank is higher than the generic rank, the tensor T has many rank-7 decompositions. We apply the alternate optimization algorithm (Algorithm 5.2) to find a rank-7 decomposition of T . Starting with an initial point that gives an initial error

err 0 := 1 2 T - 7 i=1 A 0 [:, i] ⊗ B 0 [:, i] ⊗ C 0 [:, i] 2 = 412.63,
we found a rank decomposition after running the algorithm for 800 iterations that took around 90 seconds. The error at the final iteration is

err end := 1 2 T - 7 i=1 A end [:, i] ⊗ B end [:, i] ⊗ C end [:, i] 2 = 3.19e -12.
The three factor matrices A, B, C ∈ R 4×7 given by the algorithm are as follows : # The approximated decompostion of rank 6 given by pencil_ext, E and F is A0, B0, C0, T0=factors(pencil_ext, E, F,n1,n2,n3,r);

A =      -3.
# Error between tensor to approximate T and T0 the tensor of rank 6 with factor␣ →matrices A0, B0 and C0: print("Initial error: ", round(0.5*norm(T-T0)^2,digits=2)) Initial error: 10510.02

The error that we show for the algorithm alternate is:

1 2 n3 ∑ k=1 pencil_ext[k] -F -1 ddiag(Fpencil_ext[k]E t )E -t 2 .
The Julia function that computes this error is obj.

Initial error between T and T0: 10510.02 Final error between T and T_end: 0.0461

Conclusion

In this chapter, we studied the simultaneous diagonalization of matrices problem. The considered matrices were general square matrices. A pencil of matrices M = [M 1 , . . . , M s ] is simultaneously diagonalizable if there exists two matrices E and F such that F M k E is a diagonal matrix, for k ∈ {1, . . . , s}. In this regards, we addressed three topics. The first (Section 5.1) was about the certification problem of a pencil of simultaneously diagonalizable matrices, where we presented a Newton-type method associated to this problem and we proved, under a sufficient condition on the initial point, a certification of the convergence of the sequence quadratically towards the solution. In the second part (Section 5.2), we tackled the problem of approximate simultaneous diagonalization of matrices (ASD) and we developed a Riemannian conjugate gradient algorithm to locally approximate a pencil of matrices to a pencil of simultaneously diagonalizable matrices. In the last part (Section 5.3), we showed how to use this approach in tensor rank approximation problem, where there exists a link between simultaneous diagonalization of matrices and CP decomposition of tensors. We used approximate simultaneous diagonalization, more precisely the Riemannian conjugate gradient algorithm 5.1, to implement an algorithm (Algorithm 5.2) that computes an approximated rank-r decomposition for three-dimensional tensors when r is higher than the size of each of the first two dimensions. We note that the results of Section 5.2 and Section 5.3 constitute raw results which we aimed to integrate into this manuscript. Indeed, the investigations are still pursued in this direction, and we are driving more numerical experiments to have a better understanding of the numerical behavior of the two algorithms (5.1 and 5.2), which can help us, for instance, in the analysis of their convergence.

-Gaussian mixtures and high order moments 119

In this chapter we address the method of moments for recovering Gaussian mixtures. After reviewing Gaussian mixtures and moment methods in Section 6.1, we present in Section 6.2 an algebraic symmetric tensor decomposition method for identifiable tensors. In Section 6.3, we implement the method of moments to provide an initial point for the Expectation Maximization algorithm for recovering spherical Gaussian mixtures with some examples of synthetic and real datasets, in comparison with other state-of-the-art approaches.

Gaussian mixtures and high order moments

In this section, we review Gaussian mixture models and their applications to clustering.

Gaussian mixtures

Suppose that we wish to deal with some Euclidean data x ∈ R m , coming from a population composed of r homogeneous sub-populations (often called clusters). A reasonable assumption is then that each sub-population can be modelled using a simple probability distribution (e.g. Gaussian). This idea is at the heart of the notion of mixture distribution. The prime example of mixture is the Gaussian mixture, whose probability density over R m is defined as

p θ (x) = r j=1 ω j N (x|µ j , Σ j ), (6.1)
where N (•|µ, Σ) denotes the Gaussian density with mean µ ∈ R m and definite positive covariance matrices Σ ∈ S ++ m . The mixture is parametrized by a typically unknown θ = (ω 1 , ..., ω r , µ 1 , ..., µ r , Σ 1 , ..., Σ r ), composed of ω = (ω 1 , ..., ω r ), that belong to the r-simplex and correspond to the cluster proportions, -µ j and Σ j , that correspond respectively to the mean and covariance of each cluster j ∈ {1, ..., r}. Gaussian mixtures are ubiquitous objects in statistics and machine learning, and own their popularity to many reasons. Let us briefly mention a few of these.

Density estimation If r is allowed to be sufficiently large, it is possible to approximate any probability density using a Gaussian mixture (see e.g. [START_REF] Nguyen | Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces[END_REF]). This motivates the use of Gaussian mixtures as powerful density estimators that can be subsequently used for downstream tasks such as missing data imputation [START_REF] Di Zio | Imputation through finite Gaussian mixture models[END_REF], supervised classification [START_REF] Hastie | Discriminant analysis by Gaussian mixtures[END_REF], or image classification [START_REF] Sánchez | Image classification with the Fisher vector : Theory and practice[END_REF] and denoising [START_REF] Houdard | High-dimensional mixture models for unsupervised image denoising (HDMI)[END_REF].

Clustering Perhaps the most common use of Gaussian mixtures is clustering, also called unsupervised classification. The task of clustering consists in uncovering homogeneous groups among the data at hand. Within the context of Gaussian mixtures, each group generally corresponds to a single Gaussian distribution, as in Equation (6.1). If the parameters of a mixture are known, then each point may be clustered using the posterior probabilities obtained via Bayes's rule :

∀x ∈ R m , k ∈ {1, ..., r}, Pr(x belongs to cluster j) = ω j N (x|µ j , Σ j ) p θ (x) . ( 6.2)
Detailed reviews on mixture models and their applications, notably to clustering, can be found in [START_REF] Fraley | Model-based clustering, discriminant analysis, and density estimation[END_REF][START_REF] Bouveyron | Modelbased clustering and classification for data science : with applications in R[END_REF][START_REF] Mclachlan | Finite mixture models[END_REF].

Learning mixture models

The main statistical question pertaining mixture models is to estimate the parameters θ = (ω 1 , ..., ω r , µ 1 , ..., µ r , Σ 1 , ..., Σ r ) based on a data set x 1 , ..., x n . Typically, X 1 , ..., X n are assumed to be independent and identically distributed random variables with common density p data . The problem of statistical estimation is then to find some θ such that p θ ≈ p data . There are many approaches to this question, the most famous one being the maximum likelihood method. Maximum likelihood is based on the idea that maximising the log-likelihood function

(θ) = n i=1 log p θ (x i ), (6.3)
will lead to appropriate values of θ. One heuristic reason of the good behaviour of maximum likelihood is that (θ) can be seen as a measure of how likely the observed data is, according to the mixture model p θ . This means that the maximum likelihood estimate will be the value of θ that renders the observed data the likeliest. Another interesting interpretation of maximum likelihood in information-theoretic : when n -→ ∞, maximising the log-likelihood is equivalent to minimising the Kullback-Leibler divergence (an information-theoretic measure of distance between probability distributions) between p θ and p data , thus giving a precise sense to the statement p θ ≈ p data (see e.g. [21, Section 1.6.1]). For more details on the properties of maximum likelihood, see e.g. [210, Section 5.5].

In the specific case of a mixture model, performing maximum-likelihood is however complex for several reasons. Firstly, as shown for instance by [START_REF] Le | Maximum likelihood : an introduction[END_REF], finding a global maximum is actually often ill-posed in the sense that some problematic values of θ will lead to (θ) = ∞ while being very poor models of the data. While focusing on local rather global maxima will fix this first issue in a sense, iterative optimization algorithms are likely to pursue these unfortunate global maxima. Because of the peculiarities of mixture likelihoods, the most popular algorithm for maximising (θ) is the expectation maximization (EM, [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF]) algorithm, an iterative algorithm specialized for dealing with log-likelihoods of latent variable models. The EM algorithm is usually preferred to more generic gradient-based optimization algorithms [START_REF] Xu | On convergence properties of the em algorithm for gaussian mixtures[END_REF]. In a nutshell, at each iteration, the EM algorithm clusters the data using Equation (6.2), and then computes the mean and covariance of each cluster. This iterative scheme is related to another popular clustering algorithm known as k-means (the close relationship between the two algorithms is detailed in [START_REF] Christopher | Pattern recognition and machine learning[END_REF]Section 9]). A key issue when using the EM algorithm for a Gaussian mixture is the choice of initialization. Indeed, a poor choice may lead to degenerate solutions, extremely slow convergence, or poor local optima (see [START_REF] Baudry | Em for mixtures[END_REF] and references therein). We will see in this paper that good initial points can be obtained by using another estimation method called the method of moments (as was previously noted by [START_REF] Ruffini | Clustering patients with tensor decomposition[END_REF] in a context of mixtures of multivariate Bernoulli distributions).

The method of moments is a general alternative to maximum likelihood. The idea is to choose several functions g 1 : R m -→ R q 1 , ..., g d : R m -→ R q d called moments, and to find θ by attempting to solve the system of equations Of course, since p data is unknown, solving (6.4) is not feasible. However, one may replace the expected moments by empirical versions, and solve instead

     E x∼p data [g 1 (x)] = E x∼p θ [g 1 (x)] ... E x∼p data [g d (x)] = E x∼p θ [g d (x)].
     1 n n i=1 g 1 (x i ) = E x∼p θ [g 1 (x)] ... 1 n n i=1 g d (x i ) = E x∼p θ [g d (x)]. (6.5)
A very simple example of this, in the univariate m = 1 case, when g 1 (x) = x, and g 2 (x) = x 2 . Then, solving (6.4) will ensure that the distributions of the model p θ and the data p data have the same mean and variance. However, many very different distributions have identical mean and variance ! A natural refinement of the previous idea is to consider also higher-order moments g 3 (x) = x 3 , g 4 (x) = x 4 , .... This will considerably improve the estimates found using the method of moments. This approach was pioneered by [START_REF] Pearson | Contributions to the mathematical theory of evolution[END_REF] for learning univariate Gaussian mixtures. In the more general multivariate case m > 1, following [START_REF] Hsu | Learning mixtures of spherical gaussians : Moment methods and spectral decompositions[END_REF], the moments chosen can be tensor products, as we detail in the next section in case of a Gaussian mixture with spherical covariances.

Learning structure from tensor decomposition

In this section, we describe the moment tensors revealing the structure of spherical Gaussian mixtures and how it can be decomposed using standard linear algebra operations. We note that tensor methods have been introduced for GMM before, see for instance [START_REF] Nicholas D Sidiropoulos | Tensor decomposition for signal processing and machine learning[END_REF].

Let X = (X 1 , . . . , X m ) be a set of variables. The ring of polynomials in X with coefficients in C is denoted C[X]. The space of homogeneous polynomials of degree d ∈ N is denoted C[X] d . We recall that a symmetric tensor T of order d (with real coefficients) can be represented by an homogeneous polynomial of degree d in the variables X of the form

T (X) = |α|=d T α d α X α where α = (α 1 , . . . , α n ) ∈ N m , |α| = α 1 + • • • + α m = d, T α ∈ R, d α = d! α 1 !•••αm! , X α = X α 1 1 • • • X αm m .
A decomposition of T as a sum of d th power of linear forms is of the form

T (X) = r i=1 ω i (ξ i • X) d (6.6)
where

ξ i = (ξ i,1 , . . . , ξ i,m ) ∈ C m and (ξ i • X) = m j=1 ξ i,j X j .
When r is the minimal number of terms in such a decomposition, it is called the rank of T and the decomposition is called a rank decomposition (or a Waring decomposition) of T (X).

We say that the decomposition is unique if the lines spanned by ξ 1 , . . . , ξ r form a unique set of lines with no repetition. In this case, the decomposition of T is unique after normalization of the vectors ξ i up to permutation (and sign change when d is even). A tensor T with a unique decomposition is called an identifiable tensor. Then the Waring decompositions of T are of the form T

(X) = r i=1 ω i λ -d i (λ i ξ i • X) d for λ i = 0, i ∈ {1, . . . , r}. Given a random variable x ∈ R m , its moments are T α = E[x α 1 1 • • • x αm m ] for α = (α 1 , . . . , α m ) ∈ N m . The symmetric tensor of all moments of order d of x is E[(x • X) d ] = |α|=d E[x α 1 1 • • • x αm m ] d α X α .

The structure of the moment tensor

We aim at recovering the hidden structure a random variable, from the decomposition of its d th order moment tensor. This is possible in some circumstances, that we detail hereafter. Assumption 6.2.1. The random variable x ∈ R m is a mixture of spherical Gaussians of probability density (6.1) with parameters θ = (ω 1 , ..., ω r , µ 1 , ..., µ r , σ

[(x -E[x]) ⊗ (x -E[x])] and v a corresponding unit eigenvector, -M 1 (X) = E[(x • X)(v • (x -E[x])) 2 ], -M 2 (X) = E[(x • X) 2 ] -σ2 X 2 , -M 3 (X) = E[(x • X) 3 ] -3 X 2 M 1 (X). Then σ2 = r i=1 ω i σ 2 i and M 1 (X) = r i=1 ω i σ 2 i (µ i • X), M 2 (X) = r i=1 ω i (µ i • X) 2 , M 3 (X) = r i=1 ω i (µ i • X) 3 . (6.7)
To analyse the properties of the decomposition (6.7), we use the apolar product on symmetric tensors from Definition 2.2.1.

For an homogeneous polynomial T of degree d ∈ N (or equivalently a symmetric tensor of order d), we define the Hankel operator of T in degree k ≤ d as the map

H k,d-k T : p ∈ C[X] d-k → [ T, X α p d ] |α|=k ∈ C s k where s k = m+k-1 k = dim C[X] k is the number of monomials of degree k in X. The matrix of H k,d-k T in the basis (X β ) |β|=d-k is H k,d-k T = ( T, X α+β d ) |α|=k,|β|=d-k .
From the properties of the apolar product, we see that H 1,d-1

T : p → 1 d [ ∂ X i T, p d-1 ] 1≤i≤m . For ξ ∈ C m and k ∈ N, let ξ (k) = (ξ α ) |α|=k . We also check that if T = (ξ • X) d with ξ ∈ C m , then H k,d-k (ξ•X) d = ξ (k) ⊗ ξ (d-k
) is of rank 1 and its image is spanned by the vector ξ (k) .

Proposition 6.2.3. Assume that r ≤ m, w i > 0 for i ∈ {1, . . . , r} and µ 1 , . . . , µ r ∈ R m are linearly independent. The symmetric tensor M 3 (X) is identifiable, of rank r and has a unique Waring decomposition satisfying (6.7).

Proof. Assume that M 3 (X) has a decomposition of the form (6.7). Since the vector µ 1 , . . . , µ r are linearly independent, by a linear change of coordinates in Gl m , we can further assume that µ 1 = e 1 , . . . , µ r = e r are the first r vectors of the canonical basis of R m . In this coordinate system, M 3 (X) = r i=1 X 3 i and the matrix H 1,2 M 3 in a convenient basis has a r × r identity block and zero elsewhere. Thus H 1,2 M 3 is of rank r. Its kernel of dimension 1 2 m (m + 1) -r is spanned by the polynomials X i X j with (i, j) = (k, k) for k ∈ {1, . . . , r}. The kernel of H 1,2 M 3 is thus the space of homogeneous polynomials of degree 2, vanishing at e 1 , . . . , e r ∈ R n .

If M 3 (X) can be decomposed as M 3 (X) = r i=1 ω i (µ i • X) 3 with ω i ∈ C, µ i ∈ C m and r < r, then H 1,2 M 3 , as a sum of r < r matrices ω i H 1,2 (µ i •X) 3 of rank 1, would be of rank smaller 6.2 -Learning structure from tensor decomposition 123 than r < r, which is a contradiction. Thus a minimal decomposition of M 3 (X) is of length r and r is the rank of M 3 (X).

Let us show that the decomposition (6.7) of M 3 (X) is unique up to a scaling of the vector µ i , i.e. that M 3 (X) is identifiable. For any Waring decomposition M 3 (X) = r i=1 ω i (µ i • X) 3 , the vectors µ 1 , . . . , µ r are linear independant, since µ i spans imH 1,2 (µ i •X) 3 and H 1,2

M 3 = r i=1 ω i H 1,2 (µ i •X) 3 is of rank r.
As µ 1 , . . . , µ r can be transformed into e 1 , . . . , e r by a linear change of variables, ker H 1,2 M 3 is also the vector space of homogeneous polynomials of degree 2, vanishing at µ 1 , . . . , µ r ∈ C m . Therefore, the set of {µ 1 , . . . , µ r } coincides, up to a scaling, with the set of points {µ 1 , . . . , µ r } of another Waring decomposition of M 3 (X) = r i=1 ω i (µ i • X) 3 . This shows that M 3 (X) is identifiable.

Therefore, a Waring decomposition of M 3 (X) is of the form M 3 (X) = r i=1 ωi (μ i •X) 3 with ωi = λ -3 ω i , μi = λ i µ i and λ i = 0 for i ∈ {1, . . . , r}. As μ1 , . . . , μr are linearly independent, the homogeneous polynomials (μ 1 • X) 2 , . . . , (μ r • X) 2 are also linearly independant in C[X] 2 (by a linear change of variables, they are equivalent to X 2 1 , . . . , X 2 r ). Consequently, the relation

M 2 (X) = r i=1 ω i (µ i • X) 2 = r i=1 λ i ωi (μ i • X) 2
defines uniquely λ 1 , . . . , λ r , and M 3 (X) has a unique Waring decomposition, which satisfies the relations (6.7).

Under Assumption 6.2.1, the hidden structure of the random variable x can thus be recovered using Algorithm 6.1.

Input :

The moment tensors M 1 (X), M 2 (X), M 3 (X).

-Compute a Waring decomposition of

M 3 (X) to get ωi ∈ R, μi ∈ R m , i ∈ {1, . . . , r} such that M 3 (X) = r i=1 ωi (μ i • X) 3 . -Solve the system r i=1 ωi (μ i • X) 2 λ i = M 2 (X) to get λ i ∈ R and ω i = λ 3 i ωi ∈ R + , µ i = λ -1 i μi ∈ R m such that M 3 (X) = r i=1 ω i (µ i •X) 3 and M 2 (X) = r i=1 ω i (µ i •X) 2 . -Solve the system r i=1 ω i (µ i • X)σ 2 i = M 1 (X) to get σ 2 i ∈ R + . Output : ω i ∈ R + , µ i ∈ R n , σ 2 i ∈ R + for i ∈ {1, . . . , r}.
Algorithm 6.1 -Recovering the hidden structure of a Gaussian mixture This yields the parameters

ω i ∈ R + , µ i ∈ R m , σ i ∈ R + for i ∈ {1, . . . , r} of the Gaussian mixture x.
In the experimentation, the moments involved in the tensors M i will be approximated by empirical moments and we will compute an approximate decomposition of the empirical moment tensor M3 (X).

Decomposition of identifiable tensors

We describe now an important step of the approach, which is computing a Waring decomposition of a tensor. In this section, we consider a tensor T ∈ C[X] d of order d ∈ N with a Waring decomposition of the form T = r i=1 ω i (ξ i • X) d with ω i ∈ C, ξ i ∈ C m , that we recover by linear algebra techniques, under some hypotheses. Definition 6.2.1. The interpolation degree ι(Ξ) of Ξ = {ξ 1 , . . . , ξ r } ⊂ C m is the smallest degree k of a family of homogenous interpolation polynomials u 1 , . . . , u r ∈ C[X] k at the points Ξ (u i (ξ j ) = δ i,j for i, j ∈ {1, . . . , r}).

For any d ≥ ι(Ξ), there exists a family (ũ i ) i∈{1,...,r} of interpolation polynomials of degree d, obtained from an interpolation family (u i ) i∈{1,...,r} in degree ι(Ξ)

as ũi = (λ•X) d-ι(Ξ) (λ•ξ i ) d-ι(Ξ) u i for a generic λ ∈ C m such that λ • ξ i = 0 for i ∈ {1, . . . , r}.
Notice that if the points Ξ = {ξ 1 , . . . , ξ r } are linearly independent (and therefore r ≤ m), then ι(Ξ) = 1 since a family of linear forms interpolating Ξ can be constructed.

If k ≥ ι(Ξ), then the evaluation map e

(k) Ξ : p ∈ C[X] k → (p(ξ 1 ), . . . , p(ξ r )) ∈ C r is surjective.
Its kernel is the space of homogeneous polynomials of degree k vanishing at Ξ. Any supplementary space admits a basis u 1 , . . . , u r , which is an interpolating family for Ξ in degree k. A property of the interpolation degree is the following : Lemma 6.2.4. For k > ι(Ξ), the common roots of ker e

(k) Ξ is the union ∪ r i=1 C ξ i of lines spanned by ξ 1 , . . . , ξ r ∈ C m . Proof. As ι(Ξ) + 1 is the Castelnuovo-Mumford regularity of the vanishing ideal I(Ξ) = {p ∈ C[X] | p homogeneous, p(ξ) = 0 for ξ ∈ Ξ} [76][Ch.4], it is generated in degree k > ι(Ξ) and the common roots of ker e (k) Ξ = I(Ξ) k is ∪ r i=1 C ξ i .
Hereafter, we show that tensors T such that rankH k,d-k T = r for k > ι(Ξ) + 1 are identifiable and we describe a numerically robust algorithm to compute their Waring decomposition.

Let U = (U α,j ) |α|=k,j∈{1,...,r} ∈ C s k ×r be such that im U = im H k,d-k T and U i = (U e i +α,j ) |α|=k-1,j∈{1,...,r} be the submatrices of U with the rows indexed by the monomials divisible by X i for i ∈ {1, . . . , m}.

Theorem 6.2.5. Let T ∈ C[X] d with a decomposition T = r i=1 ω i (ξ i • X) d with ω i ∈ C and ξ i = (ξ i,1 , . . . , ξ i,n ) ∈ C m such that rankH k,d-k T = r for some k ∈ [ι(ξ 1 , . . . , ξ r ) + 1, d].
Then T is identifiable of rank r and there exist invertible matrices E ∈ C s k ×s k , F ∈ C r×r such that

E t U i F = ∆ i 0 (6.8)
with ∆ i = diag( ξ1,i , . . . , ξr,i ) for i ∈ {1, . . . , m}. For any pair (E, F ), which diagonalizes simultaneously [U 1 , . . . , U m ] as in (6.8), there exist unique ω 1 , . . . , ω r ∈ C such that

T = r i=1 ω i (ξ i • X) d with ξ i = ((∆ 1 ) i,i , . . . , (∆ m ) i,i ).
Proof. From the decomposition of T , we have for k ≤ d that

H k,d-k T = r i=1 ω i ξ (k) i ⊗ ξ (d-k) i is a linear combination of r Hankel matrices ξ (k) i ⊗ ξ (d-k) i of rank 1.
If T is of rank r < r, then using its decomposition of rank r , H k,d-k T would be of rank ≤ r < r, which is a contradiction. This shows that T is of rank r.

U F = [ ξ (k) 1 , . . . , ξ(k) r ]
For any polynomial p ∈ C[X] k , which coefficient vector in the monomial basis (X α ) |α|=k is denoted

[p], we have [p] t U F = [p( ξ1 ), . . . , p( ξr )] t . This shows that U ⊥ = {p ∈ C[X] | [p] t U = 0} is ker e (k)
Ξ . By Lemma 6.2.4 since k ≥ ι( Ξ), the common roots of the homogeneous polynomials in ker e (k) Ξ are the scalar multiples of Ξ. Consequently, the set of lines spanned by the vectors Ξ of a Waring decomposition of T is uniquely determined as the conjugate of the zero locus of U ⊥ ⊂ C[X] k and T is identifiable.

For any p ∈ C[X] k-1 represented by its coefficient vector [p] in the monomial basis (X α ) |α|=k-1 , we have

[p] t U i F = [x i p] t U F = [ ξ1,i p( ξ1 ), . . . , ξr,i p( ξr )] t .
(6.9)

Let E be the coefficient matrix of a basis u 1 , . . . , u r , v r+1 , . . . , v s k-1 of C[X] k-1 , such that u 1 , . . . , u r is an interpolating family for Ξ = { ξ1 , . . . , ξr } and v r+1 , . . . , v s k-1 is a basis of ker e (k-1) Ξ

. The matrix E is invertible by construction, and we deduce from (6.9) that

E t U i F = diag( ξ1,i , . . . , ξr,i ) 0 .
Let us show conversely that for any pair of matrices (E , F ), which diagonalizes simultaneously [U 1 , . . . , U m ] as in (6.8) with ∆ i = diag( ξ 1,i , . . . , ξ r,i ), there exist unique ω 1 , . . . , ω r ∈ C such that T = r i=1 ω i (ξ i • X) d . Let u 1 , . . . , u r , v r+1 , . . . , v s k-1 ∈ C[X] be the polynomials corresponding to the columns of E . Then for a generic λ = (λ 1 , . . . , λ r ) ∈ C m , we have

diag((λ • ξ 1 ), . . . , (λ • ξ r )) = m i=1 λ i [u 1 , . . . , u r ] t U i F = m i=1 λ i [u 1 , . . . , u r ] t U i F (F -1 F ) = [(λ • ξj ) u i ( ξj )] i,j∈{1,...,r} F -1 F = diag((λ • ξ1 ), . . . , (λ • ξr )) [u i ( ξj )] i,j∈{1,...,r} F -1 F .
As λ ∈ C m is generic and λ • ξi = 0 for i ∈ {1, . . . , r}, we deduce that ∆ = [u i ( ξj )] i,j∈{1,...,r} F -1 F is a diagonal and invertible matrix and that

ξ i = ∆i,i ξ i with ∆ i,i = 0. Then we have (ξ i • X) d = ∆d i,i (ξ i • X) d and T = r i=1 ω i (ξ i • X) d with ω i = ∆-d i,i ω i ,
which concludes the proof of the theorem. This leads to Algorithm 6.2 to compute a Waring decomposition of an identifiable tensor T .

Numerical experimentations

The model used in this section is the Gaussian Mixture Model (GMM) with differing spherical covariance matrices. Recall that if x = (x 1 , . . . , x n ) is a sample of n independent observations from r multivariate Gaussian mixture with differing spherical covariance matrices of dimension Input : T ∈ C[X] d , which admits a decomposition with r points Ξ = {ξ 1 , . . . , ξ r } and k > ι(Ξ).

-Compute the Singular Value Decomposition of H k,d-k T = U S V t ; -Deduce the rank r of H k,d-k T , take the first r columns of U and build the submatrices U i with rows indexed by the monomials (X i X α ) |α|=k-1 for i ∈ {1, . . . , n} ; -Compute a simultaneous diagonalization of the pencil [U 1 . . . , U m ] as E t U i F = diag( ξ1,i , . . . , ξr,i ) 0 and deduce the points ξ i = (ξ i,1 , . . . , ξ i,m ) ∈ C m for i ∈ {1, . . . , r} ; -Compute the weights ω 1 , . . . , ω r by solving the linear system T

= r i=1 ω i (ξ i • X) d ; Output : ω i ∈ C, ξ i ∈ C m s.t. T = r i=1 ω i (ξ i • X) d .
Algorithm 6.2 -Decomposition of an identifiable tensor m, and h = (h 1 , h 2 , . . . , h n ) is the latent variable that determine the component from which the observation originates, then :

x i | (h i = k) ∼ N m (µ k , σ 2 k I m ) where, Pr(h i = k) = ω k , for k ∈ {1, . . . , r}, such that r k=1 ω k = 1.
The aim of statistical inference is to find the unknown parametrs µ k , σ 2 k and w k , for k ∈ {1, . . . , r} from the data x. This can be done by finding the maximum likelihood estimation (MLE) i.e. finding the optimal maximum of the likelihood function associated to this model. The expectation maximization algorithm (EM) [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF], usually used for finding MLEs, is an iterative algorithm in which the initialization i.e. the initial estimation of the latent parameters is crucial, since various initializations can lead to different local maxima of the likelihood function, consequently, yielding different clustering partition. Thus, in this section we compare the clustering results obtained by different initialization of the EM algorithm against the initialization by the method of moments through examples of simulated (subsection 6.3.1) and real (subsection 6.3.2) datasets. We fix a maximum of 100 iterations of the EM algorithm. The different initialization considered in this section are the following :

-The k-means method [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF] according to the following strategy :

The best partition obtained out of 50 runs of the k-means algorithm. -The method of moments, where Algorithm 6.1 is applied to build the moments and Algorithm 6.2 is applied to the empirical moment tensor corresponding to M 3 (vX) (see Algorithm 6.2.2), with less than 5 Riemannian Newton iterations (Chapter 4) to reduce the distance between the empirical moment tensor and its decomposition. -The Model-based hierarchical agglomerative clustering algorithm (MBHC) [START_REF] Vaithyanathan | Model-based hierarchical clustering[END_REF][START_REF] Fraley | Algorithms for model-based gaussian hierarchical clustering[END_REF].

-The emEM strategy [START_REF] Biernacki | Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models[END_REF] as in [START_REF] Lebret | Rmixmod : The R package of the model-based unsupervised, supervised, and semi-supervised classification Mixmod library[END_REF] which makes 5 iterations for each of 50 short runs of EM, and follows the one which maximizes the log-likelihood function by a long run of EM. The k-means, MBHC and emEM are common strategies for initialising the EM algorithm for GMMs. The comparison among the different EM initialization strategies is based on three measures : The Bayesian Information Criterion (BIC) [START_REF] Schwarz | Estimating the Dimension of a Model[END_REF][START_REF] Fraley | How many clusters ? which clustering method ? answers via model-based cluster analysis[END_REF], the Adjusted Rand Index (ARI) [START_REF] Hubert | Comparing partitions[END_REF], and the error rate (errorRate). The BIC is a penalized-likelihood criterion given by the following formula BIC = -2 ( θ) + log(n)ν,

where is the log-likelihood function , θ is the MLE which maximizes the log-likelihood function and ν is the number of the estimated parameters. This criterion measures the quality of the model such that for comparing models the one with the largest BIC value among the other models is the most fitted to the studied dataset. The ARI criterion measures the similarity between the estimated clustering obtained by the applied model and the exact true clustering. Its value is bounded between 0 and 1. The more this measure is close to 1 the more the estimated clustering is accurate.

The error rate measure can be viewed as an alternative of the ARI. In fact this criterion measures the minimum error between the predicted clustering and the true clustering, and thus low error rate means high agreement between the estimated and the true clustering. The former criteria as well as the EM algorithm are used from the tools of the package mclust [START_REF] Scrucca | mclust 5 : clustering, classification and density estimation using Gaussian finite mixture models[END_REF] in R programming language.

Simulation

We performed 100 simulations from each of the two models described in examples 6.3.1 and 6.3.2. We counted the instances where each of the considered initialising strategies for the EM could find throughout the 100 simulated data and among the other initialization methods the largest BIC, the highest ARI, ARI≥ 0.99 (as in this case the clustering obtained is the most accurate) and the lowest errorRate. The values of the BIC, ARI, errorRate and consumed time of the different considered initialization strategies for one dataset sampled according to the model of Example 6.3.1 (resp. 6.3.2) are presented in Table 6.1 (resp. 6.3), and Figure 6.1 (resp. 6.2) shows a twodimensional visualization of the observations according to the first four features, the observations in the upper panels are labeled according to the actual clustering, while they are labeled in the lower panels according to the clustering obtained by the EM algorithm initialized by the method of moments. In order to have an estimation about the numerical stability of the obtained results, we repeat the same numerical experiment for each example 20 times and we compute the means (Table 6.2, 6.4) and the variances (values in parentheses in Table 6.2, 6.4) of the 20 percentages obtained of each of the BIC, ARI, ARI≥ 0.99 and errorRate values for the different initialising strategies. As we mentioned before the initialization strategies considered in this comparison against the method of moments are common and have, in general, good numerical behavior. Nevertheless, we cannot expect all the initialization strategies that exist for the EM algorithm to work well in all the cases [START_REF] Biernacki | Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models[END_REF][START_REF] Melnykov | Finite mixture models and model-based clustering[END_REF]. Hereafter, two examples are chosen in such a way to present some cases where the common initialization strategies k-means, MBHC and emEM have some difficulties to provide a good initialization to the EM algorithm for the GMMs with differing spherical covariance matrices, or in other words where the initialization by the method of moments outperforms the other considered initializations. For instance, we put in each of these two examples one cluster of small size (the blue cluster in Figure 6.1, the red cluster in Figure 6.2), we want to make the clusters overlap, since these initialization strategies could misscluster the dataset if the clusters are intersecting. We notice that this choice of the mean vectors and the different variances in each of the two examples yields a dataset with the expected clustering characteristic. Example 6.3.1 -In the first simulation example, a multivariate dataset (m=6) of n=1000 observations generated with r=4 clusters according to the following parameters : CHAPITRE 6 -Tensor decomposition for learning Gaussian mixtures from moments -The probability vector : ω = (0.2782, 0.0139, 0.3324, 0.3756) t . -The mean vectors : µ 1 = (-5.0, -9.0, 8.0, 8.0, 2.0, 5.0) t , µ 2 = (-7.0, 6.0, -1.0, 6.0, -8.0, -10.0) t , µ 3 = (-4.0, -10.0, -5.0, 1.0, 5.0, 4.0) t , µ 4 = (-6.0, 6.0, 5.0, 4.0, -1.0, -1.0) t . -The variances : σ 2 1 = 1.5, σ 2 2 = 2.5, σ 2 3 = 5.0, σ 2 4 = 15.0. -The mean vectors : µ 1 = (7.0, -4.0, -4.0, -6.0, -4.0) t , µ 2 = (2.0, -4.0, -6.0, -10.0, -3.0) t , µ 3 = (4.0, -4.0, -5.0, 6.0, 1.0) t . -The variances : σ 2 1 = 5.0, σ 2 2 = 10.0, σ 2 3 = 15.0. The Table 6.2, 6.4 show that in Example 6.3.1, 6.3.2 the best results among the considered initialising strategies are for the method of moments. In fact, in the former two tables we see that the method of moments found throughout the 100 simulated datasets, in average (by runing the numerical experiment 20 times), the largest BIC, highest ARI, ARI≥ 0.99 and lowest errorRate among the other initialization strategies in more instances than all the other considered initialization method, implying in this context marked outperformance for the moments initialization method. Note that the consumed time (see. Table 6.1, 6.3) tends to be higher in the method of moments than in the other initialization strategies. This is expected since stochastic approaches (to which the methods k-means, MBHC and emEM belong) outperform the deterministic approaches (as the method of moments) in this term.

Real data

In this subsection we present four examples of real datasets, for which we know already their number of clusters, and we report the different BIC, ARI and errorRate values as well as the consumed time attained by the EM algorithm initialized by the different considered initialization strategies and used with the GMM of different spherical covariance matrices. The explored real data are : The famous iris data [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF][START_REF] Dheeru | UCI machine learning repository[END_REF] widely used as an example of clustering to test the algorithms, Diabetes [START_REF] Reaven | An attempt to define the nature of chemical diabetes using a multidimensional analysis[END_REF], olive oil [START_REF] Azzalini | Clustering via nonparametric density estimation : The R package pdfcluster[END_REF], and MNIST [START_REF] Deng | The mnist database of handwritten digit images for machine learning research[END_REF]. On the other hand, the consumed time is clear higher in the moment method initialization.

Example 6.3.4 -The Diabete dataset [START_REF] Reaven | An attempt to define the nature of chemical diabetes using a multidimensional analysis[END_REF] contains three measurements : glucose, insulin and sspg ; made on 145 non-obese adult patients classified into three types of diabetes : Normal, Overt, and Chemical. Herein, in this example m = r = 3. We apply the different initialization strategies for the EM algorithm, the Table 6.6 shows the results. Despite the fact that k-means method is the fastest method in this example, the ARI and the BIC are noticeably lower than in the other methods. Concerning the method of moments, it succeeds to have quite similar scores to the other methods in this example, but with a bigger computation time.

Example 6.3.5 -The olive oil data set contains the chemical composition (8 chemical properties) of 572 olive oils. They are derived from three different macro-areas in Italy (South, Sardinia and Centre North). The dataset contains nine regions from which the olive oils were taken in Italy. Thus we can cluster this dataset according to the macro-areas (r = 3) or the region (r = 9). As the number of features in this dataset is m = 8, we choose r = 3, so that the condition r ≤ m for the method of moment is verified. The results show that the MBHC initialization strategy yields the largest BIC, the highest ARI and This shows that for these datasets which are not well fitted by the mixture of spherical Gaussians, the moment method can still give good initializations for the EM algorithm, in comparision with the common initialization strategies. Example 6.3.6 -The MNIST digit image database [START_REF] Deng | The mnist database of handwritten digit images for machine learning research[END_REF] is a large database that contains images of 28 × 28 pixels for handwritten digits (0 to 9). Each pixel contains an integer between 0 and 255 that represents the grayscale levels. The number of features is 28 × 28 = 784. We choose the MNIST digit image dataset which contains 60000 images. We take a subset of this dataset that contains the images of label 0 or 1. The size of the subset is 12665 images. Since the number of features is quite large (784), and we aim to test a spherical Gaussian mixture model, a good practice in this case is to apply one of the dimensionality reduction strategies. Roughly speaking, the dimensionality reduction strategies aim to reduce the number of features such that a high percentage of the information within the dataset is conserved. In other words, the performance in term of accuracy of the clustering methods will not be noticeably affected by this reduction, and on the other hand this will reduce considerably the time of computation. For this purpose, we choose to apply the Principal Component Analysis transformation (PCA) [START_REF] Pearson | on lines and planes of closest fit to systems of points in space[END_REF][START_REF] Jolliffe | Principal Component Analysis[END_REF]. We conserve the first five variables given by this transformation (see Figure 6.3). The dataset that we consider in this example contains 12665 observations, the number of clusters is r = 2, and the number of features is m = 5. We apply the different initialization strategies and we report the results in Table 6.8. As we can see, the results given by the method of moments in Table 6 comparison with the other initialization strategies with ARI= 0.9308. In particular, the method of moments clearly outperforms MBHC method in this regard, in term of accuracy and the time of computation. In fact, the MBHC takes 543.4 seconds without reaching a good ARI score. This example sheds some light on the performance of the method of moments. The large number of samples (in this example equal to 12665) does not have a high impact on the computation time, which is not the case, for the MBHC method, where this factor increases significantly its computation time. Moreover, it is true that a large number of features could have a negative impact on the computation time of the method of moments, but it is not a sever limitation since as we saw in this example, this can be efficiently remedied by applying one of the dimensionality reduction techniques. In this regard, some recent work [START_REF] João | Tensor moments of gaussian mixture models : Theory and applications[END_REF] studies how the computation complexity of the moment method can be reduced while conserving its desirable high accuracy property. Conducting more research in this direction, we believe that the method of moments will have more sophisticated and competitive (in term of computation time) developments in the future.

6.4 -Conclusion 135

Conclusion

We present an algorithm that use our low rank symmetric tensor approximation algorithms (4) to implement the method of moments. We propose to use this method to initialize the EM algorithm, and we show concretely throughout synthetic and real datasets examples the good impact of this choice in comparison with other state-of-the-art approaches. We prove that symmetric tensors with interpolation degree strictly less than half their order are identifiable, and we present an algorithm, based on simple linear algebra operations, to compute their decomposition.

Conclusions and Perspectives

CHAPTER 7

Conclusions and Perspectives

We summarize the result obtained in this thesis, and we describe some research perspectives.

Low rank symmetric tensor approximation problem We addressed the low rank tensor approximation problem for symmetric tensors with complex coefficients. Using the basic link between symmetric tensors and homogeneous polynomials we described two Riemannian Newton type optimization algorithms with trust region steps, which are Riemannian Newton algorithm and Riemannian Gauss-Newton algorithm. In the first algorithm, we used the parametrization of the constraint set (the set of symmetric tensors of symmetric rank bounded by the approximation rank r strictly less than the generic rank) by using the standard weighted normalized factor matrices parameterization. We computed explicitly the Hessian matrix by exploiting the apolar identities and partial complex derivatives tools. We proved that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the iteration completed with a trust region scheme is converging to a local minimum. In the second algorithm, we used for the first time the Veronese manifold to parameterize the constraint set. We presented a suitable basis for the tangent space of Veronese manifold which allowed us computing the different ingredients of a Riemannian Gauss-Newton iteration. We proposed to choose the initial point for strictly subgeneric ranks approximation with interpolation degree less than d-1 2 , with d is the order of the symmetric tensor, by the method that we called the SMD based on the computation of generalized eigenvectors and generalized eigenvalues of pencils of Hankel matrices, and we showed the good impact of this choice on the numerical performance of the algorithms. A series of numerical experiments were presented, through which we tested our algorithms versus other state-of-the-art algorithms. The algorithms that we presented are particularly well suited for the symmetric case. For instance, the Gauss-Newton algorithm is strictly connected with the geometry of varieties of rank-r symmetric tensors. Indeed, we showed throughout this work how taking into account the geometric structure of the constraint set where the target approximation lives can be relevant and can conduct to concrete efficient approximation algorithms. In this direction, we believe that a natural extension of this work, is to investigate new optimization algorithms for the low rank tensor approximation problem for other types of tensors by adopting the same methodology of this work, i.e. by interacting with the geometry of the problem while dealing with this approximation problem as a numerical optimization problem. For instance, for multisymmetric tensors this will lead to investigate the so-called Segre-Veronese manifold and to exploit the corresponding with multi-homogeneous polynomials. Also practical interesting questions interrogate the possibility of the use of some tensor's compression techniques such as the symmetric Tucker decomposition gradient algorithm to approximate the extended pencil by a pencil of simultaneously diagonalizable matrices, where this allows to compute a rank-r approximation for the tensor associated to the extended pencil. The second step solves a linear least-squares problem to update the entries that does not belong to the fixed blocks matrices that correspond to the matrices of the original pencil. Finally, the algorithm extract a rank-r approximation for the original tensor. This is a recent approach, that we continue to investigate and more analysis are needed to have a better understanding of the numerical performance. Indeed, the convergence of the method is not yet understood. Moreover, we observed that the method becomes slow for ill-conditioned cases. This point can be enhanced by developing a strategy to avoid ill-conditioned points, for instance by adding a regularization term to the ill-conditioned matrices. We note that because of this extension step, the applicability of the method could be limited to approximation ranks which exceed moderately the dimension of the two first modes. Finally, after dealing with these questions, it will be interesting to apply this algorithm to applications in data science for instance in image processing.

Tensor decomposition for learning Gaussian mixtures from moments In the context of unsupervised machine learning, the type of models to be recovered plays an important role. For Gaussian mixture models, where iterative methods such as Expectation Maximization algorithms are applied, the choice of the initialization is also crucial to recover an accurate model of a given dataset. We considered the method of moments to recover Gaussian mixtures, in particular spherical Gaussian mixtures, in order to achieve clustering tasks. We computed the first, second and third order moments in terms of the means and the covariances of the clusters and the vector of the cluster proportions. We used the estimation given by the method of moments as an initial point to the EM algorithm. We demonstrated in the experimentations that tensor decomposition techniques can provide a good initial point for the EM algorithm, and that the moment tensor method outperforms the other state-of-the-art initialization strategies in term of accuracy, when datasets are well represented by spherical Gaussian mixture models. For that purpose, we presented a new tensor decomposition algorithm adapted to the decomposition of identifiable tensors with low interpolation degree, which applies to a 3 rd order moment tensors associated to the data distribution as we have shown. One inconvenient side for the method of moments in comparison with the other approaches is that in general it consumes more computation time. This is not a limitation, since in general a successful method is the method that can make a trade-off between accuracy and complexity. Hence, since the method of moments works well in term of accuracy, a straightforward question is to try to reduce its complexity. For instance, we have seen that large number of features affects negatively the computation time of the method where we proposed to refine this using one of the dimensionality reduction strategies, in this regard, further approaches can be investigated. Another direction that can be explored, is the construction of moments method for non-spherical Gaussian mixtures. Finally, in term of tensor applications, we notice that low rank tensor approximation problems are recently widely explored for deep learning applications thus future works can be oriented in this direction.

Algorithmes d'optimisation pour le problème d'approximation des décompositions en rang tensoriel: application au clustering en apprentissage automatique Rima KHOUJA

Résumé

Les tenseurs sont une généralisation d'ordre supérieur des matrices. Ils apparaissent dans une myriade d'applications. La décomposition de rang de tenseur décompose le tenseur en une somme minimale de tenseurs simples de rang 1. En pratique, la présence de bruit dans les entrées du tenseur fait que le calcul d'une décomposition de petit rang approchée est plus pertinente que de son calcul exact. Ce problème est connu comme le problème d'approximation des décompositions en rang tensoriel. Dans cette thèse, nous étudions ce problème pour les tenseurs symétriques, c.à.d pour les tenseurs avec des entrées invariantes par les permutations d'indices. Nous considérons des tenseurs symétriques avec des valeurs complexes. Parsuite en utilisant le lien entre les tenseurs et les polynômes homogènes, ainsi que des techniques d'optimisation complexe, nous proposons une approche d'optimisation riemannienne et nous développons un algorithme de Newton riemannien et un algorithme de Gauss-Newton riemannien pour résoudre ce problème. Nous abordons également le problème de diagonalisation simultanée de matrices, qui est étroitement lié au problème de décomposition tensorielle. Nous considérons ce problème sous deux angles : la certification et l'approximation. Pour la première partie, nous développons une suite de type Newton à convergence quadratique locale, et nous proposons un teste de certification. Pour la deuxième partie, nous développons un algorithme de gradient conjugué riemannien qui calcule localement un faisceau de matrices simultanément diagonalisables approché. En combinant cet algorithme avec un problème linéaire de moindres carrés, nous introduisons un algorithme d'optimisation alterné qui calcule une approximation de la décomposition pour les tenseurs tridimensionnels, quand le rang d'approximation est supérieur à la dimension de deux premiers modes. Enfin, en se basant sur les deux approches : tenseurs symétriques et diagonalisation simultanée de matrices, nous abordons le problème de clustering en apprentissage automatique pour les modèles de mélanges de Gaussiènne sphériques. Nous utilisons ces méthodes pour implémenter la méthode des moments, afin de fournir un bon point initial pour l'algorithme de maximisation de vraissemblance.

3. 5 . 4

 54 The general linear group . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.5.5 Oblique manifold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Contributions Riemannian Newton optimization algorithms for the symmetric tensor approximation problem 49 4.1 The set of non-defective rank-r symmetric tensors . . . . . . . . . . . . . . . . . 51 CHAPTER 1

Figure 1 . 1 -

 11 Figure 1.1 -Illustration of a Gaussian mixture in the plane with 2 clusters. The first cluster has mean µ 1 = (-5.0, -3.0) t , variance σ 2 1 = 0.4 and proportion w 1 = 0.3. The second cluster has µ 2 = (-5.0, 4.0) t , σ 2 2 = 0.3 and w 2 = 0.7. Number of samples N = 1000. The yellow circles represent the means of the clusters and the red squares are the estimated means by the method of moments that we will detail in Chapter 6.

Figure 2 . 1 -

 21 Figure 2.1 -Illustration of tensors of order 1, 2, 3, 4 and 5.

Figure 2 . 2 -

 22 Figure 2.2 -CPD of a three-dimensional tensor.

  i ) ≥ r.

Figure 2 . 3 -

 23 Figure 2.3 -Tucker decomposition of a three-dimensional tensor.

Figure 3 . 1 -

 31 Figure 3.1 -Illustration of a tangent vector γ(0) of M at x.

Figure 3 . 3 -

 33 Figure 3.3 -Retraction.

Figure 3 . 4 -

 34 Figure 3.4 -Parallel transport ν on M along the curve γ.

40 CHAPITRE 3 -

 403 Riemannian Optimization ξ can be transported to T y M by simply use the orthogonal projector from E onto T y M (see [2, Section 8.1.3]) i.e.T η (ξ) := P Rx(η) (ξ).

6: end for Algorithm 3 . 1 -

 31 Riemannian conjugate gradient method with Armijo line-search.

Figure 3 . 6 -

 36 Figure 3.6 -Visualization of one iteration of Algorithm 3.1.

4. 2 -

 2 Riemannian optimization for the STA problem 57 Now, for the computation of G2 , let p = |α|=d d α vα x α , and 1 ≤ k ≤ n,

4 . 2 -

 42 d is a parameterization of the set Σ r of symmetric tensors of symmetric rank at most r. We formulate the STA problem as a Riemannian least square problem over V r as follows : (STA) Vr min y∈Vr f (y), where f (y) = 1 2 ||F (y)|| 2 d , with F (y) = σ r (y) -p for y ∈ V r . Riemannian optimization for the STA problem 59

  the differential map of F at y, and (DF (y)) * • (DF (y)) :

4. 2 - 1 p

 21 Riemannian optimization for the STA problem 63 first and the second singular values. It follows from [45] that the map θ is well-defined and smooth on O. As p is in V n,d and H 1,d-is of rank 1, p ∈ O. Let U p be a neighborhood of p in C[x] d such that ψ |Up is well-defined and smooth. As the apolar product •, • d and the multiplication are well-defined and smooth on C[x] d × C[x] d , ρ is well-defined and smooth on U p , which ends the proof.

-1 d r r i=1 ||w 0 i || 2

 2 in the Riemannian Newton iteration (resp. ∆ y 0 := 10 -1 d r r i=1 ||v 0 i || 2d in the Riemannian Gauss-Newton iteration), ∆ max := 1 2 ||p|| d .

  border rank 2 and symmetric rank d. It can be very well approximated by a tensor of rank 2, with two vectors of almost the same direction. Under some regularity assumption, it is possible to guarantee that RNE-N-TR algorithm converges to a local minimum of the distance function f . Proposition 4.2.13. Let p ∈ C[x] d , let p 0 ∈ Σ r be the initial point of RNE-N-TR and let B 0 = B(p, ||p -p 0 || d ) be the ball of center p and radius ||p -p 0 || d in C[x] d . Assume that B 0 ∩ Σ r ⊂ Σ reg r (i.e. all points of Σ r in B 0 are non-defective), then RNE-N-TR converges to a local minimum y ∈ N r of the distance function f to Σ r . Proof. Let Σ 0 r := B 0 ∩ Σ r = B 0 ∩ Σ reg r be the set of non-defective tensors of rank r in B 0 . As ϕ r : (

Example 4 . 3 . 1 -

 431 [START_REF] Nie | Semidefinite relaxations for best rank-1 tensor approximations[END_REF] Example 3.5]. Consider the tensor p ∈ S 3 n (R) with entries :

Example 4 . 3 . 3 -

 433 Let p ∈ S 3 10 (R) such that :

[ 3 ]

 3 : # Define the parameters X = @polyvar x1 x2 x3 [3]: (x1, x2, x3) [4]: # P is a homogeneous polynomial of degree 4 in 3 variables P = (x1+x2+0.75*x3)^4+1.5*(x1-x2)^4-2*(x1-x3)^4;

[ 5 ]

 5 : v1, M1 = minimize(P, [x1^2+x2^2+x3^2-1], [], X, 8, optimizer); v2, M2 = maximize(P, [x1^2+x2^2+x3^2-1], [], X, 8, optimizer); -0.5027488452521038 + 0.616685953317809im 0.9073543082281365 + 0.35258183865185777im -0.0088033550999379 -0.9139555194450166im],

107 5 . 4

 54 Practical session . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

i 90 CHAPITRE 5 -

 905 On the simultaneous matrix diagonalization problemIt follows

5. 1 -

 1 Newton-type methods for simultaneous matrix diagonalization 95 Consequently,

2 . 1 .

 21 Let A, B, C ∈ R n×n , then : tr(ABC) = tr(BCA) = tr(CAB) (invariant by cyclic permutation) tr(A t ) = tr(A) tr(aA) = a tr(A), a ∈ R tr(A ddiag(B)) = tr(ddiag(A)B) = tr(ddiag(A) ddiag(B))

  1≤k≤n 3 which correspond to a rank-r decomposition with A, B and C given by (see Proposition 5.3.1) :

i 2 ,

 2 s.t. Mk [1 : n 1 , 1 : n 2 ] = M k , for k ∈ {1, . . . , n 3 }.

( 6 . 4 ) 6 . 2 -

 6462 Learning structure from tensor decomposition 121

6. 3 -

 3 Numerical experimentations 125 As rankH k,d-k T = r, we deduce that the image of H k,d-k T is spanned by ξ (k) 1 , . . . , ξ(k) r and there exists an invertible matrix F ∈ C r×r such that

Figure 6 . 1 -

 61 Figure 6.1 -Scatterplot matrix for the sampled dataset of Example 6.3.1 projected onto the first four variables (features) : upper panels show scatterplots for pairs of variables in the original clustering ; lower panels show the clustering obtained by applying the EM algorithm initialized by the method of moments.

Figure 6 . 2 -

 62 Figure 6.2 -Scatterplot matrix for the sampled dataset of Example 6.3.2 projected onto the first four variables (features) : upper panels show scatterplots for pairs of variables in the original clustering ; lower panels show the clustering obtained by applying the EM algorithm initialized by the method of moments.

Example 6 . 3 . 3 -

 633 The iris dataset contains four physical measurements (length and width of sepals and petals) for 50 samples of three species of iris (setosa, virginica and versicolor). The number of features is m = 4 and the number of clusters is r = 3. The four initialization strategies yield the same BIC value. The ARI and the errorRate values are

Figure 6 . 3 -

 63 Figure 6.3 -Scatterplot for pairs of variables : upper panels show the first five features obtained by applying the PCA transformation on the dataset of Example 6.3.6. The graphs points marked according to the true two classes 0 and 1.

TABLE DES MATIÈRES

 DES

18 CHAPITRE 2 -

 182 Tensors form of an arrayA = [a i 1 ,...,i d ] 1≤i 1 ≤n 1 ,...,1≤i d ≤n d , such that a i 1 ,...,i d ∈ R is the (i 1 , . . . , i d )-entry of the array.Let u, v be two vectors in respectively R n 1 and R n 2 . Their outer product is defined by :

  , where x := (x 1 , . . . , x n ) is the vector of the variables x 1 , . . . , x n , α = (α 1 , . . . , α n ) is a vector of the multi-indices inN n , |α| = α 1 + • • • + α n , p α ∈ C, x α := x α 1

	C[x 1 , . . . , x n ] d := C[x] d . This allows to reduce the dimension of the ambient space from n d (di-mension of T d n (C)) to s n,d := n+d-1 d (dimension of S d n (C) ∼ C[x] d ). Bold letters such as p, q denote homogeneous polynomials in C[x] d or equivalently elements in S d n (C). A homogeneous polynomial p in C[x] d can be written as : p = |α|=d d α p α x α 1 . . . x αn n and d α := d! α 1 !...αn! . A symmetric tensor A in S d n (C) is of symmetric rank one if it can be written as the outer product d times of the same vector a in C n i.e.
	A = a ⊗ ... ⊗ a	.
	d times
	The set of complex valued symmetric tensors in T d n (C) is denoted S d n (C). We have a correspon-dence between S d n (C) and the set of the homogeneous polynomials of degree d in n variables

  R n 1 ×...×n d with a matrix U ∈ R I×n k is a tensor of order d and size (n 1 , . . . , n k-1 , I, n k+1 , . . . , n d ) denoted by A × k U such that

					10		 13 16		 19 22	
				8 11 9 12	 , A 3 =	  14 17 15 18	 , A 4 =	  20 23 21 24	 .
	We have :
	vec(A) = [1 2 3 4 5 6 7 8 9 . . . 22 23 24] t ,
			 1 4 7 10 13 16 19 22 
	A (1) =	  2 5 8 11 14 17 20 23 3 6 9 12 15 18 21 24   ,
	A (2) =	1 2 3 7 8 9 . . . 19 20 21 4 5 6 10 11 12 . . . 22 23 24	,
					1 2 3 4 5 6	
		A (3) =	    13 14 15 16 17 18 7 8 9 10 11 12	   	.
				19 20 21 22 23 24
	2.4 Important matrix and tensor products
	• The k-mode (matrix) product of a tensor A ∈

  allows to approximate a tensor A of multilinear rank (r 1 , . . . , r d ) by a tensor  of multilinear rank (s 1 , . . . , s d ) such that s i ≤ r i , ∀ 1 ≤ i ≤ d, by applying successive truncated rank-s i SVD to each mode-i flattening A (i) . It is well-known that truncated SVD for matrices yields best low rank matrix approximation, but it is not the case for truncated HOSVD where in general it does not yield best multilinear rank (s 1 , . . . , s d ) approximation. Nevertheless, we have the following quasi best approximation property[START_REF] Lathauwer | A multilinear singular value decomposition[END_REF]

  {B ∈ GL n : ddiag(BB t ) = I n } denote the set of all invertible matrices in GL n with normalized rows. The set M o n is a submanifold of GL n . For B ∈ M o

	25, 1].
	Let
	M o n :=

n , its tangent space T B M o n is as follows :

Table 3 .

 3

1 -Main tools for Riemannian optimization on the oblique manifold M o n .

  approximation problem is of rank n r. The tensors ψ r (y) with y ∈ C n×r such that rank Jψ r (y) = n r are called nondefective. The set of non-defective tensors of rank r, locally embedded in C[x] d , is the image by a local diffeomorphism of a Riemannian manifold and it is denoted Σ reg r . The map ψ r is a local diffeomorphism between an open subset of C n×r and Σ reg r ⊂ Σ r . Hereafter, we consider the cases where d > 2 and the rank r is strictly subgeneric, i.e. r < r g = 1

		n+d-1
	n	d

  d is a retraction operator on the Veronese manifold for i ∈ {1, . . . , r} that we describe hereafter (see lemma 3.3.1).We will use the following matrix construction to define the retraction on V n,d .

Definition 4.2.1. The Hankel matrix of degree (k, d -k) associated to a polynomial p in C[x] d is given by :

  The map S p is well-defined and smooth on T

p V n,d . By proposition 4.2.10, π is well-defined and smooth in a neighborhood U p of p ∈ V n,d . Thus R p = ρ • S p is well-defined and smooth in a neighborhood U p ⊂ T V n,d of 0 p . approximation problem 3. By proposition 4.2.11,

  3. Compute the candidate next new point y k+1 = R y k (η k ) ; 4. Compute the quotient ρ k ; 5. Accept or reject y k+1 based on the quotient ρ k ; 6. Update the trust region radius ∆ k . Output : y * ∈ N r (resp. y

* ∈ V r ).

Table 4 .

 4

	CHAPITRE 4 -Riemannian Newton optimization algorithms for the symmetric tensor
	68								approximation problem
	Example 4.3.2 -[155, Example 3.7]. Consider the tensor p ∈ S 5 n (R) given as :
			(p) i 1 ,...,i 5 = (-1) i 1 log(i 1 ) + • • • + (-1) i 5 log(i 5 ),
	corresponding to the polynomial p = |α|=5 ( n i=1 α i (-1) i log(i)) 5 α x α .
				Example 4.3.1				Example 4.3.2
	n	10	20	30	40	50	5	10	15	20	25
	|wrne|	17.8	34.2	50.1	65.9	81.6	1.100e+2	8.833e+2	2.697e+3	6.237e+3	11.504e+3
	d 0	32.4	28.4	44	64.6	78.3	526.1	6.559e+3	26.318e+3	64.268e+3	132.213e+3
	d *	13.2	28.3	43.8	59.5	75.3	477.5	6.096e+3	24.643e+3	60.435e+3	121.892e+3
	trne	0.038	0.304	1.5	3.3	12.1	0.058	0.282	3.8	18.3	34.8
	Nrne	5	4	4	4	6	5	4	6	6	6
	|w ccpd |	14.0	29.3	43.3	60.0	75.6	78.9	8.68e+2	2.354e+3	6.148e+3	10.587e+3
	t ccpd	0.173	0.109	0.105	0.122	0.143	0.093	0.187	1.2	5.5	16.7
	N ccpd	27	25	22	23	22	19	29	16	23	17
	|w ccpd_smd |	17.8	34.2	50.1	65.9	81.6	1.100e+2	8.833e+2	2.697e+3	6.237e+3	11.504e+3
	t ccpd_smd	0.194	0.158	0.118	0.123	0.192	0.228	0.261	1.2	4.8	14.1
	N ccpd_smd	33	32	24	23	27	40	31	19	13	21
	|w cpd_gevd |	17.8	34.2	50.1	65.9	81.6	1.100e+2	8.833e+2	2.697e+3	6.237e+3	11.504e+3
	t cpd_gevd	0.148	0.152	0.207	0.289	0.177	0.521	0.361	0.947	3.4	9.7
	|w sdp |	17.8	34.2	50.1	65.9	81.6	1.100e+2	8.833e+2	2.697e+3	6.237e+3
	t sdp	2.0	6.0	30.0	245.0	1965.0	1.0	22.0	78.0	1350.0

1 -Symmetric rank-1 approximation for Example 4.3.1 and Example 4.3.2 : RNE-N-TR (rne),

CCPD-NLS (ccpd_smd) initialized by SMD (Section 4.3.1), CCPD-NLS (ccpd) initialized by 50 random initial points obeying Gaussian distributions and the reported results for this method are in average, CPD (cpd_gevd) is initialized by default by the Tensorlab method called GEVD. The method RNE-N-TR stops when the maximum number of iterations is reached (by default 500) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ). Tensorlab's methods stop when the stop criteria given by Display = 10 are verified. The method SDP (sdp) is a global optimization to which we compare the aforementioned local optimization methods if they can find best symmetric rank-1 approximation.

Table 4 .

 4 4 -Riemannian Newton optimization algorithms for the symmetric tensor approximation problem

			Example 4.3.3				Example 4.3.4
	r	min	errrne med	max	trne avg	Nrne avg	r	min	errrne med	max	trne avg	Nrne avg
	3	70.6	96	134.3	0.03	2	3	22.4	28.8	30.9	0.04	2
	5	33.3	54.2	91.8	0.08	3	5	14.1	17.4	24.6	0.07	3
	10	0.884	0.884	94.1	0.465	6	10	0.164	0.168	0.369	0.113	2
	r	min	errrgn med	max	trgn avg	Nrgn avg	r	min	errrgn med	max	trgn avg	Nrgn avg
	3	70.6	96	136.8	0.064	3	3	22.4	27.6	36.1	0.065	3
	5	33.3	48.8	105.3	0.149	4	5	14.1	17.1	24.6	0.101	3
	10	0.886	0.886	10.1	0.836	7	10	0.162	0.164	0.169	0.219	2
	r	min	err ccpd med	max	t ccpd avg	N ccpd avg	r	min	err ccpd med	max	t ccpd avg	N ccpd avg
	3	71	102	137.1	0.067	14	3	22.9	26.8	35.2	0.084	14
	5	34.2	54.7	121	0.116	26	5	14.9	17	26.6	0.104	18
	10	7.8	7.8	9.7	0.5	90	10	4.8	4.8	11.2	0.506	60
	r	min	err ccpd_smd med	max	t ccpd_smd avg	N ccpd_smd avg	r	min	err ccpd_smd med	max	t ccpd_smd avg	N ccpd_smd avg
	3	71	96.3	142.5	0.044	5	3	22.9	27	30.1	0.068	10
	5	34.2	47.4	105.7	0.06	8	5	14.9	17.1	24.6	0.084	11
	10	7.8	7.8	47.1	0.726	44	10	4.8	4.8	4.8	0.09	12
	r	min	err cpd_gevd med	max	t cpd_gevd avg	N cpd_gevd avg	r	min	err cpd_gevd med	max	t cpd_gevd avg	N cpd_gevd avg
	3	70.9	70.9	70.9	0.069	1	3	22.6	22.6	22.6	0.065	1
	5	33.8	33.8	33.8	0.071	1	5	14.2	14.2	14.2	0.071	1
	10	2.3e-14	2.3e-14	2.3e-14	0.049	1	10	5.3e-15	5.3e-15	5.3e-15	0.049	1
	r	min	err sdf med	max	t sdf avg	N sdf avg	r	min	err sdf med	max	t sdf avg	N sdf avg
	3	71	96.3	136	0.155	14	3	22.9	27.4	35.2	0.254	15
	5	34.2	49.4	105.3	0.212	16	5	14.9	17.8	26.5	0.35	19
	10	7.8	8.2	38.3	2.3	158	10	4.8	6.2	12.6	2.5	144

2 -Computational results for Examples 4.3.3 and 4.3.4 : RNE-N-TR (rne), RGN-V-TR (rgn),

Table 4 .

 4

	.5. The

3 -Computational results of SPM (spm) (initialized by a random vector of size n = 10 obeying

Table 4 .

 4

	r		ref cpd	t cpd	N cpd	refrne	trne	Nrne	refrgn	trgn	Nrgn
		1	0.117	0.05	10	0.11	0.06	2	0.115	0.069	3
		10 -1	0.118	0.046	10	0.108	0.054	2	0.112	0.084	3
	1	10 -2	0.116	0.043	10	0.107	0.044	2	0.11	0.06	2
		10 -4	0.114	0.042	10	0.107	0.037	2	0.227	0.038	2
		10 -6	0.113	0.037	11	0.112	0.036	2	0.237	0.037	2
		1	0.167	0.072	14	0.162	0.078	2	0.166	0.118	3
		10 -1	0.169	0.077	14	0.164	0.063	2	0.167	0.111	3
	2	10 -2	0.162	0.071	14	0.163	0.061	2	0.163	0.09	2
		10 -4	0.171	0.071	14	0.163	0.062	2	0.204	0.063	2
		10 -6	0.175	0.069	13	0.162	0.062	2	0.23	0.064	2
		1	0.201	0.115	16	0.204	0.135	2	0.204	0.163	3
		10 -1	0.223	0.109	17	0.206	0.091	2	0.203	0.157	3
	3	10 -2	0.228	0.117	17	0.209	0.086	2	0.203	0.152	2
		10 -4	0.202	0.103	15	0.205	0.091	2	0.243	0.093	2
		10 -6	0.284	0.124	19	0.211	0.088	2	0.234	0.091	2
		1	0.235	0.149	18	0.234	0.192	3	0.234	0.23	3
		10 -1	0.232	0.165	19	0.244	0.132	2	0.238	0.215	3
	4	10 -2	0.237	0.142	17	0.25	0.113	2	0.232	0.219	3
		10 -4	0.238	0.158	19	0.25	0.112	2	0.255	0.117	2
		10 -6	0.232	0.161	19	0.254	0.111	2	0.274	0.116	2
		1	0.275	0.21	22	0.261	0.269	3	0.261	0.345	3
		10 -1	0.264	0.186	19	0.269	0.211	2	0.261	0.288	3
	5	10 -2	0.266	0.211	22	0.305	0.148	2	0.264	0.292	3
		10 -4	0.265	0.169	18	0.293	0.158	2	0.299	0.163	2
		10 -6	0.266	0.206	21	0.298	0.158	2	0.301	0.161	2

4 -Computational results of CPD-NLS (initialized by a random symmetric initial point obeying

Table 4 . 5

 45 -5 . approximation problem

	d		ref cpd	t cpd	N cpd	refrne	trne	Nrne	refrgn	trgn	Nrgn
	n
	r		min	max	avg	avg	min	max	avg	avg	min	max	avg	avg
	5 4 10	1 10 -2 10 -4 10 -6	0.803 0.849 1.5 776.4	7.8 881.2 9.8e+4 1.9e+7	2.1 2.3 2 2	162 172 157 184	0.834 0.718 0.711 0.789	25.7 0.933 0.933 0.912	3.4 0.197 0.0379 0.042	273 16 3 3	0.815 0.718 0.711 0.789	1.1 0.933 0.933 0.912	1.2 0.078 0.0535 0.071	49 4 3 4
	5 15 20	1 10 -2 10 -4 10 -6	0.15 0.149 0.152 0.155	1.9e+3 1.2e+5 9.2e+6 1.4e+9	9.8 12.7 13.8 11.9	45 62 67 59	0.153 0.151 0.152 0.156	0.172 0.183 0.181 0.173	22.6 13.6 13.6 13.8	3 2 2 2	0.153 0.148 0.152 0.156	0.172 0.169 0.181 0.174	27.1 26.9 14.7 14.8	3 3 2 2
	6 5 12	1 10 -2 10 -4 10 -6	0.515 0.519 0.518 1.1	109.7 2.4e+4 9.6e+5 9.7e+7	1.4 3.8 2.9 2.3	61 143 137 112	0.467 0.472 0.493 0.647	0.706 0.62 0.622 0.6	0.342 0.155 0.222 0.098	4 3 4 2	0.467 0.472 0.493 0.492	0.622 0.62 0.622 0.591	0.353 0.205 0.35 0.211	4 3 5 3
	7 8 15	1 10 -2 10 -4 10 -6	0.183 0.174 0.168 0.179	2.1e+3 5.3e+4 3.5e+6 1.1e+9	54.9 52.3 63.1 75.6	46 47 54 65	0.171 0.137 0.138 0.142	0.21 0.171 0.169 0.177	8.3 4.7 4.5 4.4	3 2 2 2	0.171 0.169 0.138 0.142	0.21 0.201 0.169 0.177	8.5 8.1 4.7 4.6	3 3 2 2
				r = 5							r = 10
	s	1		2		3		s		1		2	3
	Alg	RNE	RGN	RNE	RGN	RNE	RGN	Alg	RNE	RGN	RNE	RGN	RNE	RGN
	err rel	0.456	5.8e-6	0.411	5.5e-6	0.246	1.4e-5	err rel	0.372	9.4e-6	0.195	1.6e-6	0.224	6.8e-5
	N iter	120	39	165	61	175	77	N iter	423	87	270	186	392	206
	t	2.0	1.1	2.4	1.4	2.5	1.8	t	16.9	6.3	10.8	13.7	15.5	15.0
	Nopt	0	20	0	20	0	17	Nopt	0	18	0	20	0	9
				r = 15							r = 20
	s	1		2		3		s		1		2	3
	Alg	RNE	RGN	RNE	RGN	RNE	RGN	Alg	RNE	RGN	RNE	RGN	RNE	RGN
	err rel	0.198	9.9e-6	0.242	9.9e-6	0.184	1.1e-5	err rel	0.098	9.9e-6	0.165	1.0e-5	0.221	9.9e-6
	N iter	500	17	476	21	426	25	N iter	500	12	469	12	483	12
	t	37.3	2.1	34.6	2.6	30.9	3.2	t	54.4	2.4	59.3	2.9	59.1	2.6
	Nopt	0	20	0	20	0	20	Nopt	0	20	0	20	0	20

-Computational results of CPD-NLS (initialized by a random symmetric initial point obeying normal distribution for each tensor instance), RNE-N-TR, and RGN-V-TR (initialized by the method SMD for each tensor instance). The method CPD-NLS stops when the stop criteria given by Display = 10 in Tensorlab are verified. The methods RNE-N-TR and RGN-V-TR stop when the maximum number of iterations is reached (fixed to 200) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ).

Table 4 .

 4 6 -Computational results of RNE-N-TR and RGN-V-TR for scaled weights. The two methods are initialized for each s by the same 20 random initial points obeying Gaussian distributions. They stop when the maximum number of iterations is reached (fixed to 500) or when the radius of the trust region ∆ k is less than ∆ min (by default 10 -3 ).

  Notation and preliminaries . . . . . . . . . . . . . . . . . . 83 5.1.2 Newton-type method for the system F E -I n = 0. . . . 84 5.1.3 Newton-like method for diagonalizable matrices. . . . . . . 86 5.1.4 Newton-like method for two simultaneously diagonalizable matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.1.5 Convergence of a pencil of simultaneously diagonalizable matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.1.6 Numerical illustration . . . . . . . . . . . . . . . . . . . . 97 5.1.7 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.1.8 Cauchy matrix . . . . . . . . . . . . . . . . . . . . . . . . 99 5.1.9 Sub-matrix iterations . . . . . . . . . . . . . . . . . . . . . 100 5.2 Riemannian conjugate gradient algorithm for approximate simultaneous diagonalization of matrices . . . . . . . . . . . . . . 103 5.2.1 Cost function . . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.2 Oblique geometric constraints . . . . . . . . . . . . . . . . 104 5.2.3 The proposed Riemannian conjugate gradient method . . . . 104 5.3 Simultaneous matrix diagonalization and tensor decomposition .

	5.1 Newton-type methods for simultaneous matrix diagonalization . 83
	5.1.1

Table 5 .

 5 3 -The residual error throughout 7 iterations given by the implementation of Test-1 with K = R, e = 3 and n = 10, 50, 100 in precision 1024.

	033	err res

Table 5 .

 5 4 -The residual error throughout 7 iterations given by the implementation of Test-1 with K = C, e = 3 and n = 10, 50, 100 in precision 1024.

	Iteration	n = 10	n = 50	n = 100
	1	8.84e -3	9.75e -2	1.61e -2
	2	8.59e -6	6.39e -5	1.03e -4
	3	3.91e -11	3.99e -9	4.68e -9
	4	9.87e -22	1.87e -17	3.13e -17
	5	7.60e -43	4.42e -34	8.84e -34
	6	5.14e -85	2.50e -67	9.45e -67
	7	2.64e -169 8.28e -134 1.05e -132

Table 5 .

 5 7 -The computational results throughout 7 iterations of an example of implementation of Test-2 with K = C, n = 10 and e = 6 in precision 1024.

	err res

Table 5 .

 5 8 -The residual error throughout 7 iterations given by the implementation of Test-2 with K = R, e = 3 and n = 10, 50, 100 in precision 1024.

	Iteration	n = 10	n = 50	n = 100
	1	2.91e -2	4.57e -3	1.01e -2
	2	7.97e -5	1.03e -6	1.31e -6
	3	4.21e -9	1.69e -11	3.71e -11
	4	1.07e -16	2.42e -23	1.23e -22
	5	3.92e -33	1.18e -44	1.46e -43
	6	2.63e -64	1.02e -89	1.67e -86
	7	1.71e -128 3.20e -177 9.01e -172

Table 5 .

 5 9 -The residual error throughout 7 iterations given by the implementation of Test-2 with K = C, e = 3 and n = 10, 50, 100 in precision 1024.

	Iteration	n = 10	n = 50	n = 100
	1	7.33e -3	3.14e -3	5.52e -3
	2	3.49e -6	7.48e -7	1.35e -6
	3	2.91e -12	1.11e -13	1.19e -13
	4	2.04e -24	2.54e -27	1.68e -27
	5	8.23e -49	3.04e -54	2.19e -54
	6	1.88e -97 3.41e -108 1.50e -108
	7	1.31e -194 1.91e -215 4.53e -216

Table 5 .

 5 10 -The residual error throughout 5 iterations given by the implementation of Test-2 with K = R, e = 3 and n = 10, 20, 30, in double precision. M 1 -EΣ 1 E -1 , M 2 -EΣ 2 E -1 ) 3.74e -15 4.13e -14 8.21e -14 p+1 , . . . , σ n ) and |σ i -σ j | > δ for all 1

	Iteration			n = 10	n = 20	n = 30
	1		2.71e -3	1.21e -2	4.64e -3
	2		1.36e -6	4.91e -6	2.24e -6
	3		1.39e -12 2.57e -11 4.74e -11
	4		6.16e -15 8.97e -14 1.55e -13
	5		7.04e -15 8.09e -14 1.53e -13
	max( and p the index such that Σ =	Σ p	Σ n-p	, Σ p = diag(σ 1 , . . . , σ p ), Σ n-p =
	diag(σ

Table 5 .

 5 12 -The residual error throughout 6 iterations with the Cauchy matrix of size 200.

	Iteration p = 12, δ = 4.51e -7 p = 5, δ = 4.51e -7
	1	2.45e -15	2.35e -15
	2	9.63e -26	3.75e -29
	3	1.56e -36	1.21e -53
	4	1.54e -45	1.81e -83
	5	1.15e -54	3.49e -110
	6	5.08e -64	8.67e -137

Table 5 .

 5 [START_REF] Anandkumar | Tensor decompositions for learning latent variable models[END_REF] -The residual error throughout 6 iterations with the Rosser matrix of size 256.

	Iteration p = 11, δ = 1.11e -3 p = 5, δ = 1.11e -3
	1	7.15e -12	1.65e -12
	2	7.18e -20	7.18e -20
	3	1.42e -40	1.81e -41
	4	1.73e -53	1.56e -85
	5	7.17e -66	1.75e -119
	6	8.79e -79	8.11e -153

  -1 ddiag(F Mk E t )E -t 2 , s.t. Mk [1 : n 1 , 1 : n 2 ] = M k .

	5.3 -Simultaneous matrix diagonalization and tensor decomposition	109
	as follows :
	For each k in {1, . . . , n 3 }, we consider the linear least-squares problem
	min Mk ∈R r×r	1 2	Mk -F
			and step 6 of Algorithm 5.2. It can be solved

  2 1 I m , , ..., σ 2 r I m ) such that r ≤ m.

	Theorem 6.2.2 ([107]). Under the previous assumption, let
	-σ2 be the smallest eigenvalue of E

Table 6 .

 6 1 -Numerical results of one data set of Example 6.3.1

	Method	BIC	ARI	errorRate time(s)
	em_km	-29590.48 0.8281	0.168	0.045
	em_mom -29492.11	1.0	0.0	0.547
	em_mbhc -29594.97 0.8574	0.099	0.287
	em_emEM -29593.18 0.8366	0.132	0.171

Table 6 .

 6 2 -Estimation of the stability of Example 6.3.1 resultsExample 6.3.2 -In the second simulation example, a multivariate dataset (m=5) of n=1000 observations generated with r=3 clusters according to the following parameters :-The probability vector : ω = (0.0930, 0.2151, 0.6918) t .

	Method	BIC	ARI	ARI ≥ 0.99	errorRate
	em_km	38.35% (37.82) 47.6% (21.41) 48.85% (21.61)	47.6% (21.2)
	em_mom	74.8% (41.01) 88.75% (15.36) 83.4% (18.36) 88.60% (14.46)
	em_mbhc 10.75% (12.41) 15.9% (17.57) 15.55% (22.99) 15.9% (19.46)
	em_emEM	7.3% (8.43)	14.5% (8.05)	12.6% (17.83)	14.95% (7.52)

Table 6 .

 6 3 -Numerical results of one data set of Example 6.3.2

	Method	BIC	ARI	errorRate time(s)
	em_km	-28360.30 0.4352	0.309	0.051
	em_mom -28246.02 0.9498	0.03	0.504
	em_mbhc -28358.67 0.3197	0.384	0.292
	em_emEM -28360.42 0.4408	0.296	0.141

Table 6 .

 6 4 -Estimation of the stability of Example 6.3.2 results

	Method	BIC	ARI	ARI ≥ 0.99	errorRate
	em_km	0.45% (0.576)	0.05% (0.05)	0.0% (0.0)	0.1%(0.095)
	em_mom	50.0% (18.63) 92.35% (9.82) 0.0% (0.0) 92.1% (7.46)
	em_mbhc 49.35% (19.82) 2.45% (3.63)	0.0% (0.0)	2.45% (2.58)
	em_emEM	0.3% (0.326)	5.2% (4.48)	0.0% (0.0)	5.9% (5.36)

Table 6 .

 6 5 -Numerical results of Example 6.3.3

	Method	BIC	ARI	errorRate time(s)
	em_km	-1227.6656 0.6199	0.167	0.007
	em_mom -1227.6676 0.6410	0.153	0.203
	em_mbhc -1227.6696 0.6199	0.167	0.007
	em_emEM -1227.6495 0.6302	0.160	0.045

slightly better with the moment initialization among the other considered initialization strategies.

Table 6 .

 6 6 -Numerical results of Example 6.3.4

	Method	BIC	ARI	errorRate time(s)
	em_km	-5363.06 0.3371	0.289	0.007
	em_mom -5222.11 0.6355	0.144	0.380
	em_mbhc -5221.32 0.6355	0.144	0.008
	em_emEM -5221.33 0.6207	0.151	0.049

Table 6 .

 6 7 -Numerical results of Example 6.3.5 values among the other initialization strategies. Nevertheless, the initialization by the moment method comes in second position after the MBHC strategy in terms of the BIC, ARI and errorRate values, while the K-means and the emEM initialization strategies attain almost the same values of the previously mentionned criteria.

	Method	BIC	ARI	errorRate time(s)
	em_km	-10948.64 0.4018	0.262	0.021
	em_mom -10946.46 0.4532	0.210	0.508
	em_mbhc -10625.59 0.5003	0.185	0.080
	em_emEM -10948.72 0.4040	0.260	0.087
	the lowest errorRate

Table 6 .

 6 .8 are very satisfactory in 8 -Numerical results of Example 6.3.6

	Method	BIC	ARI	errorRate time(s)
	em_km	-384977.3 0.9304	0.017	0.537
	em_mom -384978.2 0.9308	0.017	1.87
	em_mbhc -382746.2 0.2445	0.252	543.4
	em_emEM -384977.6 0.9301 0.0177655	1.80

Mots-clés : Tenseurs, algorithmes d'optimisation, apprentissage automatique, clustering, optimisation riemannienne, diagonalisation simultanée de matrices, mélanges Gaussiènnes, optimisation complex, variétés différentielles.

Remerciements

approximation problem CSDP 6.2.0 Iter: 0 Ap: 0.00e+000 Pobj: 0.0000000e+000 Ad: 0.00e+000 Dobj: 0.0000000e+000 Iter: 1 Ap: 6.21e-001 Pobj: -4.7686654e+001 Ad: 3.23e-001 Dobj: 7.7359277e+000 Iter: 2 Ap: 1.00e+000 Pobj: -6.7310911e+002 Ad: 3.23e-001 Dobj: 9.7555745e+000 Iter: 3 Ap: 1.00e+000 Pobj: -6.2816076e+002 Ad: 8.92e-001 Dobj: 1.2402191e+000 . . . Iter: 20 Ap: 9.60e-001 Pobj: -7.7015795e+000 Ad: 9.55e-001 Dobj: -7.7015795e+000 Success: SDP solved Primal objective value: -7.7015795e+000 Dual objective value: -7.7015795e+000 Relative primal infeasibility: 1.69e-014 Relative dual infeasibility: 1.46e-010 Real Relative Gap: -2.15e-011 XZ Relative Gap: 9.09e-010 DIMACS error measures: 3.03e-014 0.00e+000 4.40e-009 0.00e+000 -2.15e-011 9.09e-010 CSDP 6.2.0 Iter: 0 Ap: 0.00e+000 Pobj: 0.0000000e+000 Ad: 0.00e+000 Dobj: 0.0000000e+000 Iter: 1 Ap: 6.19e-001 Pobj: -4.7582096e+001 Ad: 3.23e-001 Dobj: -8.7051506e+000 Iter: 2 Ap: 1.00e+000 Pobj: -6.7805576e+002 Ad: 3.21e-001 Dobj: -1.1060713e+001 Iter: 3 Ap: 1.00e+000 Pobj: -6.3256373e+002 Ad: 8.92e-001 Dobj: -1.4167349e+000 . . . Iter: 21 Ap: 9.60e-001 Pobj: -6.5652500e+000 Ad: 9.53e-001 Dobj: -6.5652500e+000 Success: SDP solved Primal objective value: -6.5652500e+000 Dual objective value: -6.5652500e+000 Relative primal infeasibility: 2.80e-015 Relative dual infeasibility: 4.47e-010 Real Relative Gap: -7.76e-011 XZ Relative Gap: 2.82e-009 DIMACS error measures: 5.01e-015 0.00e+000 1.35e-008 0.00e+000 -7.76e-011 2.82e-009

The minimum evaluation of P on the unit sphere is: -7.701579459519532.

The maximum evaluation of P on the unit sphere is: 6.565249952183416. Thus, the maximum weight in absolute value which is the spectral norm of P is: 7.701579459519532.

The unit vectors that give this value are: [0.6805571886747267, 0.048429787707634814, -0.7310926539221407] and [-0.6805571886747267, -0.048429787707634814, 0.7310926539221407].

Let us compute a rank-1 approximation of P. We will compute an initial point by the method SMD, the Julia function that corresponds to this method in the package TensorDec is called "decompose", then we will use the Riemannian Newton algorithm with trust region scheme for the real case, the corresponding Julia function in the package TensorDec is called "rne_n_tr_r".

We can notice that the weight given by decompose is close to the one given by Optimizer. Let us refine this point by using a few number of iterations of rne_n_tr_r, for example 5 iterations.

[26]: w_end, V_end = rne_n_tr_r(P, w1, V1, Dict{String,Any}("maxIter" => 5,"epsIter"␣ →=> 1.e-3)) [START_REF] Bouchard | Riemannian optimization and approximate joint diagonalization for blind source separation[END_REF]: ([-7.701576525649196], [0.68061769889553; 0.04831896554028091; -0.7310436550024019], Dict{String,Any}("d*" => 8.819736152809341,"d0" => 8.841950556430849,"nIter" => 5,"epsIter" => 0.001,"maxIter" => 5))

The weight in absolute value given by rne_n_tr_r initialized by decompose for rank-1 symmetric tensor approximation is: 7.701576525649196.

The unit vector given by rne_n_tr_r initialized by decompose for rank-1 symmetric tensor approximation is: [0.68061769889553; 0.04831896554028091; -0.7310436550024019].

Verifying with the global optimization method "optimizer" from the package CSDP, the symmetric rank-1 approximation w end (v t end X) 4 of P given by "rne_n_tr_r" initialized by "decompose" is a best rank-1 approximation.

Example 2:

Let us take a random symmetric tensor normally distributed with complex coefficients of order 4 and dimension 3 (the generic symmetric rank is 5), and let us compute by the Riemannian Newton algorithm "rne_n_tr" and the Riemannian Gauss-Newton algorithm "rgn_v_tr" initialized by a random initial point obeying normal distribution an approximated rank-3 symmetric tensor. [59]: # Adjust the initial point to use with rgn_v_tr since this function takes only␣ →the matrix V as parameter without the weight vector for i in 1:r

# Apply rgn_v_tr

V_end, Info = rgn_v_tr(P, V, Dict{String,Any}("maxIter" => 500,"epsIter" => 1.

→e-3))

[59]: (Complex{Float64}[-1.197237252086016 + 0.4498363042506125im 0.6784383624116359 + 0.5660791048163021im -0.7849864638452732 -0.01483593513452497im; 0.3788396401411271 + 0.2511463936422992im -0.974262084689121 -0.2574674292040156im -0.6433234546249749 -0.670531077035846im;

Simultaneous matrix diagonalization and tensor decomposition

In this section, we show the connection between simultaneous matrix diagonalization and tensor decomposition. For real tensor rank approximation of real multilinear tensors of dimension 3 and size (n 1 , n 2 , n 3 ) with approximation rank r ≥ max(n 1 , n 2 ), we develop an alternate optimization algorithm, based on approximate simultaneous diagonalization of matrices.

We denote by T(n 1 , n 2 , n 3 ) the set of real multilinear tensors of dimension 3 and size

where the slice M [:,:,i] is the matrix M i . Let M = [M 1 , . . . , M s ] be a pencil of simultaneously diagonalizable matrices M i ∈ R n×n , i.e. there exists matrices E, F ∈ GL n with

Proof. To a family M = [M 1 , . . . , M s ] of n×n matrices, corresponds the tensor M ∈ T(n, n, s) such that M [:,:,i] = M i . If M is simultaneously diagonalizable as in (5.34), then the corresponding tensor M can be written as

The matrices E, F, Σ involved in (5.34) (resp. (5.35)) are called the decomposition factors of M (resp. M). Now, suppose that we have a tensor M ∈ T(n 1 , n 2 , n 3 ) that we aim to approximate to a tensor of rank r ≥ max(n 1 , n 2 ), i.e. find three factor matrices A, B, C respectively in R n 1 ×r , R n 2 ×r and R n 3 ×r , that minimize the following non-linear least-squares function :

Our approach, based on alternate optimization method, to find such factor matrices A, B and C can be summarized as follows :

1. Let M be the pencil of n 3 matrices of size n 1 × n 2 associated to M.

CHAPTER 6

Tensor decomposition for learning Gaussian mixtures from moments

In data processing and machine learning, an important challenge is to recover and exploit models that can represent accurately the data. We consider the problem of recovering Gaussian mixture models from datasets. We investigate symmetric tensor decomposition methods for tackling this problem, where the tensor is built from empirical moments of the data distribution. We consider identifiable tensors, which have a unique decomposition, showing that moment tensors built from spherical Gaussian mixtures have this property. We prove that symmetric tensors with interpolation degree strictly less than half their order are identifiable and we present an algorithm, based on simple linear algebra operations, to compute their decomposition. Illustrative experimentations show the impact of the tensor decomposition method for recovering Gaussian mixtures, in comparison with other state-of-the-art approaches. Keywords : Method of moments, Gaussian mixtures, clustering, symmetric tensors.

for T is a symmetric tensor in S d (R n ), S is a symmetric tensor in S d (R l ) and A ∈ R n×l is an orthogonal matrix. For symmetric rank-r approximation, we can try to compute this symmetric Tucker decomposition for l bounded by r and then to apply our symmetric rank-r approximation algorithms on the core tensor in such a way that if

we can take T ≈ r i=1 (As i ) ⊗ . . . ⊗ (As i ), as a low rank approximation for T . This approach could be interesting especially to reduce the computation complexity. This also allows us to combine the two formats of the approximation problem which are tensors and polynomials, where the symmetric tensor T is used to compute a Tucker decomposition and the homogeneous polynomial associated to the core tensor is used to compute a low rank approximation, and thus to take advantage from the reduction of the computation cost than can be realized using each of the two formats. Another interesting question is to construct a method based on simple linear algebra operations that can provide a good initialization for our algorithms for all strictly subgeneric symmetric rank without the limitation regarding the bound on the interpolation degree on which depends the method SMD that we use.

Simultaneous matrix diagonalization

We addressed the simultaneous matrix diagonalization problem. We presented a Newton-type sequence that converges quadratically towards the solution of the system of equations associated to a pencil of simultaneously diagonalizable matrices. We considered the case of one diagonalizable matrix and two simultaneously diagonalizable matrices, then based on the resolution of the two aforementioned cases we concluded on the case of a family of simultaneously diagonalizable matrices. Moreover, we gave a certification on the quadratic convergence towards the solution when the initial point verify a sufficient condition that we established. This approach allows the computation in high precision, where this type of computation could be important in certain circumstances. Also, it is an iterative method that allows to obtain the numerical solution of a system of equations with the desirable property of local quadratic convergence without depending on computing the inverse of matrices such in a classical Newton iteration. We focused on the regular case. Some improvements and extensions can be considered, such as the treatment of clusters of eigenvalues. Further, we described a Riemannian conjugate gradient algorithm that approximates a pencil of matrices no necessarily simultaneously diagonalizable by a pencil of simultaneously diagonalizable matrices based on solving a Riemannian optimization problem over the Cartesian product of two oblique manifolds. This approach can be considered as a generalization of the recent work in [START_REF] Bouchard | Approximate joint diagonalization with Riemannian optimization on the general linear group[END_REF] where the matrices of the pencil are considered symmetric and the problem depends on one diagonalizer matrix E with its inverse. We employed this algorithm to construct an alternate optimization algorithm which for a tridimensinal tensor of size (n 1 , n 2 , n 3 ) and approximation rank r such that r ≥ max(n 1 , n 2 ) takes the pencil of matrices that correspond to the frontal slices of the tensor, then extends the pencil of n 1 × n 2 matrices to r × r matrices. The algorithm then works on the extended pencil and alternate between two steps. The first step uses the conjugate 7.0 -