Thesis Summary

This PhD thesis concerns the development of flexible Krylov subspace iterative solvers for the solution of large sparse linear systems of equations with multiple right-hand sides. Our target application is the solution of the acoustic full waveform inversion problem in geophysics associated with the phenomena of wave propagation through an heterogeneous model simulating the subsurface of Earth. When multiple wave sources are being used, this problem gives raise to large sparse complex non-Hermitian and nonsymmetric linear systems with thousands of right-hand sides. Specially in the three-dimensional case and at high frequencies, this problem is known to be difficult. The purpose of this thesis is to develop a flexible block Krylov iterative method which extends and improves techniques already available in the current literature to the multiple right-hand sides scenario. We exploit the relations between each right-hand side to accelerate the convergence of the overall iterative method. We study both block deflation and single right-hand side subspace recycling techniques obtaining substantial gains in terms of computational time when compared to other strategies published in the literature, on realistic applications performed in a parallel environment.

iii

Résumé de la Thèse

Les travaux de ce doctorat concernent le développement de méthodes itératives pour la résolution de systèmes linéaires creux de grande taille comportant de nombreux seconds membres. L'application visée est la résolution d'un problème inverse en géophysique visant à reconstruire la vitesse de propagation des ondes dans le sous-sol terrestre. Lorsque de nombreuses sources émettrices sont utilisées, ce problème inverse nécessite la résolution de systèmes linéaires complexes non symétriques non hermitiens comportant des milliers de seconds membres. Dans le cas tridimensionnel ces systèmes linéaires sont reconnus comme difficiles à résoudre plus particulièrement lorsque des fréquences élevées sont considérées. Le principal objectif de cette thèse est donc d'étendre les développements existants concernant les méthodes de Krylov par bloc. Nous étudions plus particulièrement les techniques de déflation dans le cas multiples seconds membres et recyclage de sous-espace dans le cas simple second membre. Des gains substantiels sont obtenus en terme de temps de calcul par rapport aux méthodes existantes sur des applications réalistes dans un environnement parallèle distribué. v List of Tables

Introduction

In this thesis we are concerned with the development of iterative solvers for large sparse linear systems with multiple right-hand sides with application to an industrial problem in geophysics and geology, the acoustic full waveform inversion [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF][START_REF] Claerbout | Imaging the earth's interior[END_REF]. It consists of an optimization problem which targets to generate an approximate model of the velocity of propagation of acoustic waves in the subsurface of Earth. Experimental data is gathered by triggering an acoustic wave source at a certain position in Earth's surface, and as these waves propagate through the subsurface and encounter discontinuities, they are scattered and propagated back. Special tools called geophones (a special type of microphone) record information concerning the waves that were propagated back. This process is repeated for several positions and the data is again recorded several times. The acoustic full waveform inversion then builds and improves a velocity model (supposing that a proper initial guess model is known) of the subsurface iteratively until a reliable approximation is obtained. To determine whether an approximation is reliable or not it is thus necessary to simulate how the waves would have propagated through the velocity model. In this thesis, we are concerned with simulating the wave propagation phenomena given a velocity model using the Helmholtz equation

-∆u(x) - 2πf v(x) 2 u(x) = s(x), x ∈ R 3
where u(x) denotes the wave pressure, v(x) is the velocity of the propagation of the wave and s(x) is the source term, and using perfectly matched layers [START_REF] Berenger | A perfectly matched layer for the absorption of electromagnetic waves[END_REF] (or PML) as boundary condition in order to simulate an infinite domain. The discretization of the Helmholtz equation with PML using finite difference techniques yields a sparse linear system of the form

AX = B,
where A ∈ C n×n is a nonsingular, non-Hermitian and non-symmetric matrix, B ∈ C n×p is supposed to be full rank, X ∈ C n×p and p is the number of acoustic wave sources triggered (usually of the order of O(10 4 )). The difficulties for solving this problem come when the acoustic wave sources are triggered at a high frequency. In this case, n can be of the order of O (10 9 ) meaning that the memory needed for solving this problem with direct solvers might be prohibitively large. In this situation iterative solvers are preferred since they allow the control of the memory used. Also, at high frequencies, the matrix A may present properties that complicate the preconditioning of iterative solvers, culminating in a slow convergence [START_REF] Ernst | Why it is difficult to solve Helmholtz problems with classical iterative methods[END_REF]. Recent publications show the interest of using geometric multigrid based techniques with an approximate coarse solution as preconditioner [START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF], characterizing thus a variable preconditioner and imposing the use of flexible block Krylov subspace methods. Several approaches have been used in the literature to solve the resulting linear system. Among the direct and hybrid solver solutions we mention [START_REF] Bollhöfer | Algebraic multilevel preconditioner for the solution of the Helmholtz equation in heterogeneous media[END_REF][START_REF] Haidar | On the parallel scalability of hybrid linear solvers for large 3D problems[END_REF][START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF][START_REF] Sourbier | FWT2D : a massively parallel program for frequency-domain full-waveform tomography of wide-aperture seismic datapart 2: numerical examples and scalability analysis[END_REF], but these often require a prohibitively large memory storage which grows proportionally with frequency being used, being thus suitable only for low and medium frequencies range. Recent publications [3,[START_REF] Wang | Acoustic inverse scattering via Helmholtz operator factorization and optimization[END_REF][START_REF] Wang | On 3d modeling of seismic wave propagation via a structured parallel multifrontal direct Helmholtz solver[END_REF][START_REF] Wang | Massively parallel structured multifrontal solver for time-harmonic elastic waves in 3-d anisotropic media[END_REF] use the low-rank approximation techniques to reduce the memory cost of direct solvers for Helmholtz, being thus to successfully solve three-dimensional problems for high-frequencies. Other techniques as the sweeping preconditioner [START_REF] Engquist | Sweeping preconditioner for the helmholtz equation: Moving perfectly matched layers[END_REF][START_REF] Poulson | A parallel sweeping preconditioner for high frequency heterogeneous 3d helmholtz equations[END_REF] report low storage cost for solving problems at mid frequencies.

Nevertheless, due to their optimal memory control and scalability, we opt for focusing on Krylov iterative methods. The solution of systems with multiple right-hand sides using Krylov subspace methods has been addressed in the literature. The so called block Krylov subspace methods are increasingly popular in many application area in computational science and engineering (e.g. electromagnetic scattering (monostatic radar cross section analysis) [START_REF] Boyse | A block QMR method for computing multiple simultaneous solutions to complex symmetric systems[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Soudais | Iterative solution methods of a 3-D scattering problem from arbitrary shaped multidielectric and multiconducting bodies[END_REF], lattice quantum chromodynamics [START_REF] Sakurai | Application of block Krylov subspace algorithms to the Wilson-Dirac equation with multiple right-hand sides in lattice QCD[END_REF], model reduction in circuit simulation [START_REF] Freund | Krylov-subspace methods for reduced-order modeling in circuit simulation[END_REF], stochastic finite element with uncertainty restricted to the right-hand side [START_REF] Elman | Efficient iterative algorithms for the stochastic finite element method with application to acoustic scattering[END_REF], and sensitivity analysis of mechanical systems [START_REF] Barbella | Block Krylov subspace methods for the computation of structural response to turbulent wind[END_REF] to name a few). Denoting by X 0 ∈ C n×p the initial guess for the system and by R 0 = B -AX 0 the initial block residual associated with such initial guess, a block Krylov space method for solving the p systems is an iterative method that generates approximations

X m ∈ C n×p with m ∈ N such that range (X m -X 0 ) ⊂ K m (A, R 0 )
where the block Krylov space K m (A, R 0 ) (in the unpreconditioned case) is a generalization of the well known Krylov subspace, defined as

K m (A, R 0 ) = range R 0 AR 0 . . . A m-1 R 0 ⊂ C n .
We refer the reader to [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] for a recent detailed overview on block Krylov subspace methods and note that most of the standard Krylov subspace methods have a block counterpart (see, e.g., block GMRES [START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF], block BiCGStab [START_REF] Guennouni | A block version of BICGSTAB for linear systems with multiple right-hand sides[END_REF], block IDR(s) [START_REF] Du | A block IDR(s) method for nonsymmetric linear systems with multiple right-hand sides[END_REF] and block QMR [START_REF] Freund | A block QMR algorithm for non-Hermitian linear systems with multiple right-hand sides[END_REF]). To be effective in terms of computational operations it is recognized that block iterative methods must incorporate a strategy for detecting when a linear combination of the systems has approximately converged [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF]. A simple strategy to remove useless information from a block Krylov subspace -called initial deflation -consists in detecting possible linear dependency in the block right-hand side B or in the initial block residual R 0 ( [64, §12] and [78, §3.7.2]). When a restarted block Krylov subspace method is used, this block size reduction can be also performed at each initial computation of the block residual, i.e., at the beginning of each cycle [64, Section 14]. In addition Arnoldi deflation [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] may be also considered; it aims at detecting a near rank deficiency occurring in the block Arnoldi procedure to later reduce the current block size. These strategies based on rank-revealing QR-factorizations [START_REF] Businger | Linear least squares solutions by Householder transformations[END_REF] or singular value decompositions [START_REF] Golub | Matrix Computations[END_REF] have been notably proposed both in the Hermitian [START_REF] Nikishin | Variable block CG algorithms for solving large sparse symmetric positive definite linear systems on parallel computers, i: General iterative scheme[END_REF][START_REF] Ruhe | Implementation aspects of band Lanczos algorithms for computation of eigenvalues of large sparse symmetric matrices[END_REF] and non-Hermitian cases [2,[START_REF] Bai | ABLE: an adaptive block Lanczos for non hermitian eigenvalue problems[END_REF][START_REF] Cullum | Two-sided Arnoldi and non-symmetric Lanczos algorithms[END_REF][START_REF] Freund | A block QMR algorithm for non-Hermitian linear systems with multiple right-hand sides[END_REF][START_REF] Loher | Reliable Nonsymmetric Block Lanczos Algorithms[END_REF][START_REF] O'leary | The block conjugate gradient algorithm and related methods[END_REF] for block Lanczos methods. They have been shown to be effective with respect to standard block Krylov subspace methods.

While initial deflation or deflation at the beginning of a cycle are nowadays popular, BGMRES based methods incorporating deflation at each iteration have been rarely studied. In [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] Robbé and Sadkane have introduced the notion of inexact breakdown to study block size reduction techniques in block GMRES. Two criteria have been proposed either based on the numerical rank of the generated block Krylov basis (W-criterion) or on the numerical rank of the block residual (R-criterion). Numerical experiments on academic problems of small dimension with a reduced number of right-hand sides illustrated the advantages and drawbacks of each variant versus standard block GMRES. Further numerical experiments can be found in [START_REF] Khabou | Solveur itératif haute performance pour les systèmes linéaires avec seconds membres multiples[END_REF]. Another method relying on such a strategy is the Dynamic BGMRES (DBGMRES) [START_REF] Cunha | Dynamic block GMRES: an iterative method for block linear systems[END_REF], which is an extension of block Loose GMRES [START_REF] Baker | An efficient block variant of GMRES[END_REF]. However, the combination of block Krylov subspace methods performing deflation at each iteration and variable preconditioning has been rarely addressed in the literature.

As an alternative to block methods, it was proposed in the literature the subspace recycling techniques [1,[START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF], for the case in which the right-hand sides are not all known a priori, but instead, the solution of one system is used to compute the right-hand side of the next linear system. Traditionally augmented methods or methods with deflated restart (e.g. [START_REF] Baker | A technique for accelerating the convergence of restarted GMRES[END_REF][START_REF] De Sturler | Truncation strategies for optimal Krylov subspace methods[END_REF][START_REF] Eiermann | Analysis of acceleration strategies for restarted minimal residual methods[END_REF][START_REF] Morgan | A restarted GMRES method augmented with eigenvectors[END_REF][START_REF] Saad | Analysis of augmented Krylov subspace methods[END_REF]) could retain their augmented subspace or the harmonic Ritz pairs (which approximate the smallest eigenvalue and its respective eigenvector of a matrix; cf. [START_REF] Morgan | Computing interior eigenvalues of large matrices[END_REF]) from the solution of one linear system to accelerate the convergence of the next linear system, if the proper modifications in the algorithm are performed. In [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] the GCRO-DR, a variant of GMRES-DR [START_REF] Morgan | GMRES with deflated restarting[END_REF] is proposed with the ability to recycle harmonic Ritz information. This is also particularly interesting for the case in which multiple left-hand sides situation is being addressed

A (i) X (i) = B,
for 1 ≤ i ≤ l and some relations between each A (i) hold. In the full waveform inversion scenario, the multiple left-hand sides situation is not all uncommon, as some techniques may generate l different models per iteration, thus requiring the solution of l block linear systems with p right-hand sides.

The main purpose of this thesis is to derive a class of flexible minimal block residual methods that incorporate block size reduction at each iteration. We carefully extend the theory available for single righthand side case to a unified view in the block case, and we draw important conclusions for the variable preconditioner scenario which to the best of our knowledge is new for both single and multiple right-hand sides scenario (cf. Theorem 2.4.3). Using the theory we developed, we will introduce a method belonging to this class, the deflated minimal block residual or DMBR. It consists of a variant of BGMRES using a generalization of the deflation technique depicted in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] which is able to discard subspaces at the beginning of each iteration. Our proposed variant uses the singular values of the scaled block residual to decide which part of the block Krylov subspace should be built for the current iteration, saving matrixvector operations and preconditioner applications. Since we are considering flexible preconditioner which are potentially the most expensive part of the iterative procedure, avoiding preconditioner applications can bring a substantial computational gain to the overall method. We compare DMBR with recently proposed flexible block space methods using deflation at the beginning of the cycle only [START_REF] Calandra | Two-level preconditioned Krylov subspace methods for the solution of three-dimensional heterogeneous Helmholtz problems in seismics[END_REF][START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF] as well as with the method proposed in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF], which is able to deflate at the end of each iteration. We observe the advantage of deflating at the beginning of each iteration in practice in several of our real life application numerical experiments.

Along with that, in the single right-hand side scenario, we extended the GCRO-DR method for the variable preconditioner case, as subspace recycling techniques might be of interest in our geophysical application. We then compared FGMRES-DR [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF] method with our new proposed method FGCRO-DR [START_REF] Carvalho | A flexible generalized conjugate residual method with inner orthogonalization and deflated restarting[END_REF], and we demonstrate that for the variable preconditioner case, FGMRES-DR and FGCRO-DR are not algebraically equivalent unless a specific collinearity condition hold for every iteration. We thus show the interest of using subspace recycling techniques with variable preconditioner with numerical experiments.

The outline of this thesis is thus as follows:

• In Chapter 2 we study restarted flexible block methods that satisfy a minimum norm property. We discuss key aspects of minimum norm methods and we extend and generalize concepts as well as properties which are well known for the single right-hand side case. Among the key generalizations, we mention partial convergence (in Definition 2.8.1), partial breakdown (in Definition 2.6.1) and partial stagnation (in Definition 2.9.1), concepts which are not trivially obtained from the single right-hand side case. The definitions and properties discussed in this chapter are going to be extensively used in the Chapter 3 when deflation is discussed. Also in Chapter 2 we demonstrate that the flexible block Arnoldi algorithm (cf. Algorithm 2.5.1) always spans a block Krylov subspace whenever the variable preconditioner holds a rank-preserving condition. This result is new to the best of our knowledge and helps the understanding of flexible Krylov methods both for single and multiple right-hand sides.

• In Chapter 3 we use the development of Chapter 2 to characterize which information we would like to deflate at the beginning of each iteration of the BGMRES algorithm. We show the relation between the block residual and the block Krylov subspace being built in absence of partial stagnation phenomena, and that using the singular values of the scaled block residual we are able to focus on the minimization of the residual associated only with the right-hand sides which did not converge yet, thus avoiding to perform expensive computations to further improve already converged approximate solutions. The final deflation strategy we propose in this chapter is based on [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] but it is not algebraically equivalent whenever restarting occurs in the algorithm. We compare our proposed algorithm, which we call deflated minimal block residual or DMBR, with other methods which are known to be efficient for solving the Helmholtz problem, and we obtain substantial gains in terms of computational time when performing experiments on real life problems.

• In Chapter 4 we introduce in more detail the acoustic full waveform inversion, clarifying some of the recent developments done in several areas to improve its performance. We discuss several issues involving the preconditioning and discretization of the Helmholtz equation and we finally discuss a software implementation using FORTRAN03 and object orientation strategies in order to obtain a modular code which can be easily adapted and modified to conform the newest technologies and techniques for the solution of the full waveform inversion. We use this software to perform numerical experiments of considerable dimension (from O(10 8 ) to O(10 9 )) at a relevant frequency range (from 5HZ to 12Hz) using no more than 128 cores. We successfully compute, using block Krylov methods, the wavefield for multiple sources given at once using an average of 11.8 Gb of memory per shot for realistic problems at 5Hz. This numerical experiments reinforces the interest in using deflation techniques for multisource scenario.

• In Chapter 5 we discuss the subspace recycling techniques, more specifically the development and analysis of FGCRO-DR method and how it is related to FGMRES-DR method in the single righthand side case. We conclude that in spite of the similarities, the methods are not algebraically equivalent and produce different iterates in our numerical experiments. We conclude that an algebraic equivalence could be achieved if a certain collinearity condition holds for every iteration, but that such situation is rare in practice. Numerical experiments allow us to witness the advantages of using subspace recycling techniques when solving sequences of linear systems.

• In Chapter 6 we present the final remarks of this thesis and the plans for future research.

Notation

Through this entire thesis we denote by e i the i-th vector of the canonical basis of the appropriate dimension. We write ||.|| for any unitarily invariant norm. Some important norms used include the Frobenius norm denoted by ||.|| F , the spectral norm ||.|| 2 when used for a matrix or the well known Euclidean norm when referring to a vector and the psi-norm ||.|| ψ defined as

||M || ψ = max i ||m i || 2
where m i is the i-th column of M (this norm can be found, for instance, in [START_REF] Simoncini | Convergence properties of block GMRES and matrix polynomials[END_REF][START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF] and it is unitarily invariant). We denote the condition number in the . 2 of a nonsingular matrix A by Whenever we mention that a given matrix V ∈ C n×t with n ≥ t is orthonormal, it means that it has orthonormal columns meaning that V H V = I t×t . When t = n we say that the matrix is unitary instead.

κ(A) = A 2 A -1
Regarding the algorithmic part, we adopt notation similar to those of MATLAB in the presentation. For instance, U (i, j) denotes the U ij entry of matrix U , U (1 : m, 1 : j) refers to the submatrix made of the first m rows and first j columns of U and U (:, j) corresponds to its jth column.

Whenever we refer to a subspace V, we implicitly suppose that V is a subset of H, where H is the Hilbert subspace. The notation V W reads "V is a proper subset of W" (i.e. V is contained in W but V = W) and the notation V ⊂ W reads "V is a subset of W" (i.e. V is contained in W and V = W may be true). Also, for any matrix V ∈ C n×p range (V ) , rank (V ) , null (V ) denote respectively the range of V , the rank of V and the nullity of V . We also use dim(V) to denote the dimension of the subspace V.

Let P ∈ C a×c and Q ∈ C c×b . Following Matlab standards, we abuse the notation allowing c = 0 resulting in operations as M = N P = P N = N for any N ∈ C a×d for a given positive integer d. We reinforce that in such a situation P is undefined as a mathematical entity, and we are simply defining P N = N . Similarly, we allow

S = Q T = T Q = T
with T ∈ C d×b for a given positive integer d. In our pseudo-code examples, we write P = [] and Q = [] without specifying dimensions.

Chapter 2

Introduction to Block Iterative Solvers

Introduction

In this chapter we address the basis for the development of methods for solving the problem

AX = B (2.1.1)
where A ∈ C n×n is any nonsingular complex non-Hermitian nonsymmetric matrix, and X, B ∈ C n×p , always considering p n, and that rank (B) = p, and for the sake of simplicity, we consider the zero initial guess (that is, X 0 = 0) for the moment.

Although some of the content of this chapter can be found in other publications (as [START_REF] Eiermann | Geometric aspects in the theory of Krylov subspace methods[END_REF][START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF][START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF][START_REF] Gutknecht | The block grade of a block Krylov space[END_REF][START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF]), here we present, reorder and generalize these key concepts aiming at establishing grounds for the more advanced techniques we are going to show in further chapters. We explicitly mention whenever new concepts or proofs are presented. Notably we bring to the attention of the reader the results formalized later in Theorem 2.4.3 and Proposition 2.5.3.

To solve (2.1.1) the most straightforward approach is to separate such system in p linear systems as AXe i = Be i , i = 1, ..., p where e i represents the i-th vector of the canonical basis in C p , and then apply the known preconditioned GMRES [START_REF] Saad | GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems[END_REF] method (or any other convenient iterative solver) to each system independently. A technique known as "subspace recycling" [START_REF] De Sturler | Recycling subspace information for diffuse optical tomography[END_REF][START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF][START_REF] De Sturler | Large scale topology optimization using preconditioned Krylov subspace recycling and continuous approximation of material distribution[END_REF] could be used here to gather information from A (as information concerning the approximation of the invariant subspace, approximate spectral information such as harmonic Ritz pairs [START_REF] Morgan | Computing interior eigenvalues of large matrices[END_REF], etc) when solving AXe k = Be k and use this knowledge to accelerate the convergence of the chosen method for solving AXe k+1 = Be k+1 . This strategy is particularly common for the cases in which Be k+1 depends on Xe k , meaning that each linear system has to be solved in sequence.

In this chapter however, we do not address the situation, and we suppose that all the columns of the right-hand side B are known beforehand.

In such a scenario, we can thus consider a generalization of iterative methods like GMRES for multiple right-hand side (see [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] for a recent detailed overview on block Krylov subspace methods). This generalization is often called in the literature "block iterative method". Among the reasons for considering such a generalization is the possibility of exploiting BLAS-3 operations rather than BLAS-2. From a more theoretical point of view, block iterative solvers span a larger search subspace than its single right-hand side counterpart and thus potentially finds an approximation for (2.1.1) with less computational effort. This behaviour has been shown experimentally in a number of publications [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF][START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF][START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF] and we discuss it with more details through this chapter.

We focus on restarted block methods that satisfy a minimum norm property as introduced in [107, §6.12]. We first discuss key aspects of minimum norm methods and we extend properties which are known in the single right-hand side case to the block case. We propose a generalization of concepts common to the single right-hand side scenario as convergence (cf. Definition 2.8.1), breakdown (Definition 2.6.1) and stagnation (Definition 2.9.1) to the multiple right-hand side scenario. We present also in this chapter the block Arnoldi (Algorithm 2.5.1) in Section 2.5 and the block GMRES (BGMRES, in Algorithm 2.7.1) in Section 2.7 and we briefly discuss some few differences between GMRES and BGMRES. In Section 2.10 we provide the final remarks for this chapter.

During all the developments of this chapter, except when explicitly mentioned, we consider exact arithmetic.

Subspaces and Minimum Block Residual

We reproduce the following definitions which can be found, for instance, in [40, (3.1)] for the sake of completeness.

Definition 2.2.1 (Nested Subspaces). Let the sequence of subspace V i ⊂ C n be given for

1 ≤ i ≤ j. If it holds that {0} = V 0 V 1 V 2 ... (2.2.1)
we then say that (2.2.1) is a nested sequence of .

It is important to notice that the sequence of subspaces in Definition 2.2.1 is finite by definition, since dim V j-1 < dim(V j ) ≤ n has to hold for every j ≥ 1, meaning that once we find a k such that V k = C n , the sequence stops and V k+1 is undefined. Also, the number of elements in this sequence is bounded by n.

We define now our main object of study for this chapter: Definition 2.2.2 (Minimum block residual family). A minimum block residual (MBR) method is an iterative method that, at each step j solves

min range( X)⊂Zj   p i=1 Be i -A Xe i 2 2   (2.2.2)
for a given sequence of nested subspaces Z j (cf. Definition 2.2.1).

By definition, methods belonging to the Euclidean minimum block residual family will converge in a finite number of steps (at worst, when Z j = C n ) because the subspace Z j has to grow with j.

Definition 2.2.2 is in fact an attempt to generalize the concept of "minimum residual family" (or MR family) proposed in [START_REF] Eiermann | Geometric aspects in the theory of Krylov subspace methods[END_REF]. Therein, we find a definition analogous to Definition 2.2.2 for the single righthand side case which considers any norm . N in a specific vectorial subspace, i.e. . N does not necessarily have to be a norm in the entire C n . We opt for a simpler approach in Definition 2.2.2, considering only the Euclidean norm. This choice allows us to simplify equation (2.2.2) to

min range( X)⊂Zj B -A X F . (2.2.3)
Here the Frobenius norm is specially suitable because we can consider it as a block vector norm instead of a matrix norm (cf. [64, p.8]). Moreover, if we require X to have minimum Frobenius norm, since A is nonsingular, the problem (2.2.3) has an unique solution.

Next we show Definition 2.2.5 and Definition 2.2.3 which are due to [START_REF] Eiermann | Geometric aspects in the theory of Krylov subspace methods[END_REF] and are just reproduced here for the sake of completeness. We highlight that in [START_REF] Eiermann | Geometric aspects in the theory of Krylov subspace methods[END_REF] it is expected that dim(Z j-1 -Z j ) = 1, ∀j = 2, ..., whereas we expect dim(Z j-1 -Z j ) ≥ 1, ∀j = 2, ... (cf. Definition 2.2.4) due to the block nature of the problem.

Definition 2.2.3 (Correction Subspace). Consider a given MBR method and its sequence of nested subspaces Z j for j = 1, .... We define the s j -dimensional subspace Z j ⊂ C n as the "correction1 subspace". Definition 2.2.4. We define

k j = dim Z j -Z j-1 , k 1 = dim (Z 1 )
, and therefore,

s j = j i=1 k i ,
that is, k j represents how much the correction subspace grew from iteration j -1 to iteration j. Definition 2.2.5 (Residual Approximation Subspace). We denote s j -dimensional subspace W j = AZ j as "approximation subspace". Since A is nonsingular, it also follows that W j-1 ⊂ W j for every j > 1. Since {Z j } j is a nested sequence of subspaces, then {W j } j is also a sequence of nested subspaces.

Let the columns of Zj ∈ C n×sj represent any basis for Z j . Then

Y j = arg min Ȳ ∈C s j ×p B -A Zj Ȳ F , (2.2.4)
defines an approximate solution as X j = Zj Y j . It is a well known result ([60, p.257], for instance) that the solution of (2.2.4) is given by

Y j = (A Zj ) † B. (2.2.5)
We define the block residual of such approximation as

R j = B -AX j = B -A Zj (A Zj ) † B, = (I -A Zj (A Zj ) † )B (2.2.6)
which is clearly an orthogonal projection onto (AZ j ) ⊥ . If we define W j = P Wj B where W j is the approximation subspace according to Definition 2.2.5, we see that

R j = B -W j , (2.2.7)
and in such fashion, we can interpret W j as the approximation of B in W j subspace. For this reason, we call W j "residual approximation subspace" or simply "approximation subspace".

The two following results (Property 2.2.6 and Property 2.2.7) are a generalization of results found in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF], using the concepts of Definition 2.2.3 and Definition 2.2.5. Our contribution in these particular statements is that [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] relies on a specific choice of correction subspace Z j . Having such a result for any nested subspace instead is crucial for our further discussions in Chapter 3.

From equation (2.2.6) and from the fact that B has full rank, we immediately deduce the following property: Property 2.2.6 (Residual Nullity). For any MBR method, it holds that null R j = dim W j ∩ range (B) .

The following property can be trivially deduced from Property 2.2.6 by just using the nonsingularity of A. Property 2.2.7 (Exact Solution Intersection). For any MBR method, it holds that null R j = dim Z j ∩ range (X * ) , where X * is the exact solution of AX = B.

In other words, whenever the nullity of the residual is different from zero, it means that we have found the exact solution of one or more (or a linear combination of) linear systems inside Z j . Several strategies that will be studied later are heavily based on this property of minimum block residual methods.

Because we assumed that Z j is a nested subspace, every Zj can be decomposed in a recursion as

Zj = Zj-1 Zj = Z1 Z2 . . . Zj (2.2.8)
where each Zi ∈ C n×ki , 1 ≤ i ≤ j is a full rank matrix and k i is given in Definition 2.2.4.

Next we define some figures useful for describing how we expand the subspace Z j to Z j+1 .

Definition 2.2.8 (Expansion Subspace). Let S j ⊂ C n be a given subspace of dimension p j + n j for some p j , n j ∈ N such that

n j = dim Z j ∩ S j
and naturally p j = dim S j -n j .

We expand the subspace Z j as

Z j+1 = Z j + S j .
In Section 2.3 we show that most methods perform an expansion of subspace according to Definition 2.2.8, in the sense that it disposes of a subspace to be "added" to Z j , but nothing guarantees that a direct sum holds.

Remark 2.2.9. In a more advanced scenario, instead of disposing of a subspace S j of dimension p j + n j to add to Z j , we dispose of a subspace Kj of dimension p j + n j instead, where

Kj = S j ⊕ P j dim S j = k j+1 + n j , dim P j = d j+1 p j = k j+1 + d j+1 .
These methods use a subspace of dimension k j+1 to update Z j and neglect a subspace of dimension d j+1 (either discarding it or saving it for other purposes). This scenario is going to be discussed in details in Chapter 3. For the purpose of this chapter, we simply assume that d j = 0, ∀j ≥ 1, meaning that k j+1 = p j i.e. we use all the subspace available to expand the correction subspace Z j .

The Block Krylov Subspace

In this section we define the so called block Krylov subspace. Altough many authors (cf. [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF]) use a matrices subspace for that purpose, we choose here to define it as a subspace of C n as in [START_REF] Elbouyahyaoui | Algebraic properties of the block GMRES and block Arnoldi methods[END_REF] : Definition 2.3.1 (Block Krylov Subspace). We define a block Krylov subspace, as

K k (A, B) = range B AB ... A (k-1) B ∈ C n (2.3.1)
This can be rewritten as

K k (A, B) = j i=1 K k (A, Be i ) (2.3.2) = K k (A, Be 1 ) + K k (A, Be 2 ) + ... + K k A, Be p , (2.3.3)
(cf. [44, (2.12)]) where K k (A, Be i ) denotes the k-th Krylov subspace of A with relation to Be i .

Nothing guarantees that a direct sum holds in (2.3.3), therefore these subspaces might have a nonzero intersection -a key property which we explore in Section 3.6 of Chapter 3.

Having in mind what was discussed in the previous section, it is a common practice to set Z j = K j (A, B), giving rise to the so called block Krylov subspace iterative method. Examples of block Krylov iterative methods are block GMRES [START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF] (presenting minimal block residual properties as in Definition 2.2.2), block BiCGStab [START_REF] Guennouni | A block version of BICGSTAB for linear systems with multiple right-hand sides[END_REF], block IDR(s) [START_REF] Du | A block IDR(s) method for nonsymmetric linear systems with multiple right-hand sides[END_REF] (another class of methods, focusing on orthogonal residual properties which we do not address in this thesis) and block QMR [START_REF] Freund | A block QMR algorithm for non-Hermitian linear systems with multiple right-hand sides[END_REF] (presenting a quasiminimal residual property, that is, it minimizes the residual only with respect to a given subspace rather than the entire C n ). This strategy is popular because one can ensure that the exact solution lies inside a block Krylov subspace, as the following corollary shows: Corollary 2.3.2. Let X * be the block solution of AX = B and (i) the degree of the minimum polynomial of A with respect to Be i . Then

range (X * ) ⊂ K (A, B)
where = max{ (i) } p i=1 . We call the block grade of A with respect to B.

Proof. It is a well known result that the exact solution of each AXe i = Be i lies in K (i) (A, Be i ) (see [START_REF] Ipsen | The idea behind Krylov methods[END_REF], for instance). From equation (2.3.3) we have the proof completed.

We refer to [64, §3] for details on the block grade of a matrix with respect to a given block right-hand side, specially to Lemma 5 (where we find an equivalent definition for the block grade) and Theorem 9 (for an equivalent version of Corollary 2.3.2).

Since each (i) is equal or smaller than2 n, it is always advantageous to look for a solution in K (A, B) rather than in C n , but it is specially interesting for the cases in which dim(K (A, B)) is much smaller than n. Also the block Krylov subspaces allow us to derive some properties in the minimum block residual scenario, which we discuss along this chapter.

Using the notation in Definition 2.2.8, we imply from the definition of the block Krylov subspace (2.3.1) that for building such a subspace, we set S j = range(A j B) every iteration j ≥ 1, with Z 1 = range(B). For this specific choice then, according to Definition 2.2.3, for every j we obtain

s j = dim K j (A, B) n j = dim K j (A, B) ∩ range A j B p j = k j+1 = p -n j .
An important result for the single right-hand side case (i.e. p = 1) is that if range A j B ⊂ K j (A, B), then 3 range (X * ) ⊂ K j (A, B) , meaning that the exact solution for the system AX = B is already known and that j ≥ , where is the block grade of A with respect to B. Following our notation, for p = 1, range A j B ⊂ K j (A, B) implies in n j = 1 and thus p j = 0. This is called in the literature happy breakdown or Arnoldi breakdown 4 .

In the multiple right-hand side scenario (i.e. p > 1) an analogous phenomenon is observed. In fact, whenever p -p j > 0, it can be proved that a linear combination of solutions already lies inside K j (A, B) (cf. [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF]). The following proposition formalizes this concept.

Proposition 2.3.3. At each iteration j of a MBR method using block Krylov subspace as correction subspace, it holds that a linear combination of p -p j solutions lie inside K j (A, B).

Proof. We have that

dim range (B) ∩ AK j (A, B) = dim range (B) + dim AK j (A, B) -dim range (B) + AK j (A, B) But range (B) + AK j (A, B) = K j+1 (A, B).
Moreover from the nonsingularity of A we have that

dim AK j (A, B) = dim K j (A, B) therefore, dim range (B) ∩ AK j (A, B) = dim range (B) + dim K j (A, B) -dim K j+1 (A, B) = p + s j -s j+1 = p -k j+1 = p -p j .
Again from the nonsingularity of A, we obtain

p -p j = dim range A -1 B ∩ K j (A, B) = dim range (X * ) ∩ K j (A, B)
where X * denotes the exact solution of AX = B finalizing the proof.

Note that for this choice of Z j , if we always set S j = range(A j B) we have that dim(S j ) = p, thus p -p j = n j for any j. However, in further discussions we show that practical algorithms based on this idea do not always span a subspace S j of dimension p, and thus n j = p -p j for every j, reason why we choose to present Proposition 2.3.3 (and specially Corollary 2.3.4 at the end of this section) with p -p j instead.

To clearly understand Proposition 2.3.3, let X * ∈ C n×(p-pj ) represent a basis for the subspace range (X * ) ∩ K j (A, B).

Since X * can be written as a linear combination of a basis both of range(X * ) and K j (A, B), it follows immediately that there are full-rank complex matrices L 1 and L 2 of suitable dimension such that

X * = Z j L 1 and X * = X * L 2 ,
where Z j ∈ C n×sj is a basis to K j (A, B), and consequently

X * L 2 = Z j L 1 ,
showing that one can write a part of X * as a linear combination of Z j .

Although an analogous of the following corollary can also be found in [103, Corolary 1], we reproduce it here for the sake of completeness. It is a direct implication of Proposition 2.3.3 and Property 2.2.7.

Corollary 2.3.4. At each iteration j of a MBR method using block Krylov subspace as correction subspace, it holds that null R j = p -p j .

Preconditioning the Correction Subspace

A practice even more common than choosing Z j as K j (A, B) is to choose a "preconditioned block Krylov subspace" instead. It consists in choosing a nonsingular preconditioning matrix M ∈ C n×n such that AM presents better numerical properties, to then solve the system

AM M -1 X = AM T = B (2.4.1)
instead. Once T is known, we take X = M T to retrieve the solution of the original system. For solving (2.4.1) one then uses

Z j = M × span B AM B ... (AM ) (j-1) B = M K j (AM , B) (2.4.2)
as correction subspace 5 . An even more general concept is the use of nonlinear variable preconditioners. It consists in applying a (nonlinear) preconditioning operator which depends on the iteration. For a given iteration j and matrix V ∈ C n×p , we represent the application of the preconditioner on V with

Z = M j (V ) (2.4.3)
where Z ∈ C n×p is the preconditioned V . Also, for the sake of simplicity, we always consider that the operator M j (.) is rank-preserving for every j (that is, rank (V ) = rank (Z)) and that it satisfies the following definition.

Definition 2.4.1 (Rank-Preserving Variable Nonlinear Preconditioner). Suppose we dispose of a matrix

V j = V 1 V 2 .
.. V j where each V i ∈ C n×ki for some sequence of values of k i with 1 ≤ i ≤ j, and that we obtain

Z j = M 1 (V 1 ) M 2 (V 2 ) ... M j (V j
) by applying the sequence of (possibly) nonlinear operators M i , 1 ≤ i ≤ j. If rank Z j = rank V j , we say that the sequence of preconditioner operator

M i , 1 ≤ i ≤ j is rank-preserving.
We then establish the following lemma.

Lemma 2.4.2 (Equivalent Preconditioner Matrix). Given V j a full rank matrix, for any sequence of rank-preserving preconditioner operators M i , 1 ≤ i ≤ j, there is at least one nonsingular matrix M j ∈ C n×n such that

Z j = M j V j (2.4.4)
holds.

Proof. One example of such a matrix is

M j = Z j Z ⊥ j V j V ⊥ j -1
where Z ⊥ j (respectively V ⊥ j ) is the orthogonal complement of Z j (respectively V j ) in C n . Lemma 2.4.2 also implies that there is a matrix M i such that

Z i = M i V i
for every 1 ≤ i ≤ j. We prefer the notation on (2.4.4) over the one in (2.4.3) throughout this thesis, but they are equivalent.

Both flexible and fixed preconditioners are relevant and widely applied, and the choice between either is problem dependent. Perhaps the most well known class of fixed preconditioners are the incomplete factorizations [START_REF] Saad | Iterative solution of linear systems in the 20th century[END_REF]Chapter 10]. Concerning variable preconditioners we can exemplify inner-outer iteration methods (see [START_REF] Simoncini | Theory of inexact Krylov subspace methods and applications to scientific computing[END_REF]). Quoting [START_REF] Simoncini | Theory of inexact Krylov subspace methods and applications to scientific computing[END_REF] about variable preconditioners, "one can also consider preconditioners which might improve using information from previous iterations" (see [START_REF] Baglama | Adaptively preconditioned GMRES algorithms[END_REF][START_REF] Erhel | Restarted GMRES preconditioned by deflation[END_REF][START_REF] Joubert | A robust GMRES-based adaptive polynomial preconditioning algorithm for nonsymmetric linear systems[END_REF]). Because the variable preconditioner also covers fixed preconditioner and unpreconditioned case, in this whole thesis we consider that a variable preconditioner is being used unless otherwise noted, and we always suppose that the matrices M i ∈ C n×n , 1 ≤ i ≤ j as well as M j were properly chosen.

The block Krylov correction subspace using a variable preconditioner is thus introduced as follows

Z j = span M 1 B M 2 AM 1 B ... M j A...AM 2 AM 1 B , (2.4.5)
for a given sequence of nonsingular preconditioning matrices M i ∈ C n×n . It is not trivial to write (2.4.5) as a (block) Krylov subspace, but the following theorem shows that if the sequence of preconditioning operators M i is rank-preserving this is always possible.

Theorem 2.4.3. Define the s j -dimensional subspaces

Z j = span M 1 B M 2 AM 1 B ... M j A...AM 2 AM 1 B ,
where the sequence of variable preconditioners M i ∈ C n×n , 1 ≤ i ≤ j is rank-preserving (see Definition 2.4.1). There is always a nonsingular matrix M j ∈ C n×n such that

Z j = M j K j (AM j , B).
Proof. For the theorem to be proved, there should be a matrix M j ∈ C n×n such that the subspace

Z j = M j K j (AM j , B) = M j × span B AM j B . . . (AM j ) j-1 B
equals (2.4.5). One possible case in which this is true is when there is a matrix M j such that all the equalities

M j B = M 1 B M j AM j B = M 2 AM 1 B M j (AM j ) 2 B = M 3 AM 2 AM 1 B . . . M j (AM j ) j-1 B = M j A . . . M 3 AM 2 AM 1 B (2.4.6)
hold. However, substituting the first equality in the subsequent, we can rewrite the requirement as

M j B = M 1 B M j AM 1 B = M 2 AM 1 B M j AM j AM 1 B = M 3 AM 2 AM 1 B . . . M j (AM j ) j-2 AM 1 B = M j A . . . M 3 AM 2 AM 1 B.
Substituting the second equality on the subsequent, and doing so recursively implies that this requirement can be written as

M j B = M 1 B M j AM 1 B = M 2 AM 1 B M j AM 2 AM 1 B = M 3 AM 2 AM 1 B . . . M j A . . . M 3 AM 2 AM 1 B = M j A . . . M 3 AM 2 AM 1 B.
Denoting Sk = AM k-1 . . . AM 1 B, for 1 < k < j and S1 = B we finally have that

M j S1 = M 1 S1 M j S2 = M 2 S2 M j S3 = M 3 S3 . . . M j Sj = M j Sj .
This means that we are looking for a nonsingular matrix M j such that

M j S1 S2 . . . Sj = M 1 S1 M 2 S2 . . . M j Sj . (2.4.7)
Because we assumed that the sequence of variable preconditioners

M i ∈ C n×n is rank-preserving, rank S1 S2 . . . Sj = rank M 1 S1 M 2 S2 . . . M j Sj
allowing us to use Lemma 2.4.2 to show that there is always a nonsingular matrix M j such that (2.4.7) holds, proving the theorem.

The following corollary is another way to state Theorem 2.4.3.

Corollary 2.4.4. Define the s j -dimensional subspaces

Z j = span M 1 B M 2 AM 1 B ..., M j A...AM 2 AM 1 B ,
where the sequence of variable preconditioners M i ∈ C n×n is rank-preserving (see Definition 2.4.1). There is always a nonsingular matrix T j ∈ C n×n such that

Z j = K j (T j A, Z 1 ).
where

Z 1 = M 1 B.
Proof. Analogous to Theorem 2.4.3.

To the best of our knowledge Theorem 2.4.3 (followed by Corollary 2.4.4) is the first demonstration on how to describe subspaces of the form (2.4.5) as a (block) Krylov subspace for both single and multiple right-hand side case, being one of the main theoretical contributions of this chapter of the thesis.

Notice that Corollary 2.4.4 clarifies what was stated in [114, p.27] for the single right-hand side case: "there may not exist any Krylov subspace containing Z j ". In fact, the proof of Theorem 2.4.3 relies on the rank-preserving assumption of the variable preconditioner, and in a general case, we do not guarantee that Z j can be written as a block Krylov subspace. Although theoretically speaking this assumption is rather strong, in practice, a rank-deterioration rarely happens due to the preconditioner application, thus justifying our assumption. Also, it is not trivial to define the concept of block grade (cf. Corollary 2.3.2) for a flexibly preconditioned (block) Krylov subspace because the matrix M j changes every time we expand the subspace Z j . i.e

Z j = M j K j (AM j , B) but Z j+1 ⊂ M j K j+1 (AM j , B).
However, as long as the subspace Z j grows with j, the respective MBR method is convergent (cf. [114, p.27]). In practice, when using a variable preconditioner, we expect to build a variable preconditioned block Krylov subspace such that

range (X * ) ⊂ M k K k (AM k , B) ⊂ K (A, B) where dim(M k K k (AM k , B)) < dim(K (A, B
)) but such inequality holds only if the variable preconditioning matrices M i were properly chosen, and such a choice is highly problem dependent.

We rewrite Proposition 2.3.3 for the flexibly preconditioned case.

Proposition 2.4.5. At each iteration j of a MBR method using

Z j = span M 1 B M 2 AM 1 B ... M j A...AM 2 AM 1 B , (2.4.8)
as correction subspace for a given sequence of rank-preserving variable preconditioning nonsingular matrices M i ∈ C n×n , it holds that a linear combination of p -p j solutions lies inside Z j .

Proof. Thanks to Theorem 2.4.3 we know that Z j = M j K j (AM j , B) for some nonsingular matrix M j ∈ C n×n . Thus

dim range (B) ∩ AM j K j (AM j , B) = dim range (B) + dim AM j K j (AM j , B) -dim range (B) + AM j K j (AM j , B) But range (B) + AM j K j (AM j , B) = K j+1 (AM j , B).
Moreover from the nonsingularity of AM j we have that

dim AM j K j (AM j , B) = dim K j (AM j , B) therefore, dim range (B) ∩ AM j K j (AM j , B) = dim range (B) + dim K j (AM j , B) -dim K j+1 (AM j , B) = p + s j -dim K j+1 (AM j , B) .
To find out the dimension of K j+1 (AM j , B) we use once again the rank-preserving assumption of the variable preconditioner. It holds that

Z j+1 = span M 1 B M 2 AM 1 B ... M j+1 AM j ...AM 1 B
and

K j+1 (AM j , B) = range (B) + AM j K j (AM j , B) = range (B) + AZ j = span B AM 1 B AM 2 AM 1 B ... AM j ...AM 1 B .
Because of the rank-preserving assumption on the variable preconditioner, we conclude that dim(Z j+1 ) = dim(K j+1 (AM j , B)), and as so

dim range (B) ∩ AM j K j (AM j , B) = p + s j -dim K j+1 (AM j , B) = p + s j -s j+1 = p -p j .
Again from the nonsingularity of A, we obtain

p -p j = dim range A -1 B ∩ M j K j (AM j , B) = dim range (X * ) ∩ Z j
where X * denotes the exact solution of AX = B, finalizing the proof.

Although the unpreconditioned case shown in Proposition 2.3.3 can be found in other publications (cf. [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF]), Proposition 2.4.5 is new to the best of our knowledge. This result will be used in Section 3.6.

For a detailed study over variable preconditioners for iterative solvers, we recommend the reading of [107, §9.4], [START_REF] Simoncini | Theory of inexact Krylov subspace methods and applications to scientific computing[END_REF][START_REF] Saad | A flexible inner-outer preconditioned GMRES algorithm[END_REF][START_REF] Carvalho | A flexible generalized conjugate residual method with inner orthogonalization and deflated restarting[END_REF][START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF].

The Block Arnoldi Algorithm

As mentioned previously, looking for an approximate solution inside the (preconditioned) block Krylov subspace is a common strategy. However, the (block) Krylov basis can be very ill-conditioned if it is built naïvely according to its definition (2.3.3) and it is desirable to construct an orthonormal basis to K j (A, B) (or K j (AM j , B)) for stability reasons. We refer to [START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF] for a deep study on the conditioning of Gram-Schmidt-based algorithms for generating orthonormal bases and its stability when considering finite precision arithmetic.

We consider that the reader is familiar with variable preconditioners and preconditioned Arnoldi algorithm (as in [107, p.256]). We introduce now the block flexible Arnoldi method (Algorithm 2.5.1) and the block flexible Arnoldi iteration (Algorithm 2.5.2), which are commonly used not only for generating a stable orthonormal basis for a block Krylov subspace, but also in a number of applications such as solving eigenvalues problems (see [START_REF] Lehoucq | Implementation of an implicitly restarted block Arnoldi method[END_REF][START_REF] Stewart | Matrix Algorithms: Volume 2, Eigensystems[END_REF] for instance).

Remark 2.5.1. Algorithm 2.5.2 is presented such that it will remove linear dependent columns of S (if any; cf. line 5 of Algorithm 2.5.2) ensuring thus that V j+1 has full rank: a rank-revealing QR (RRQR) [START_REF] Björck | Numerical methods for least squares problems[END_REF] algorithm would be used to determine both the deficiency n j and the decomposition SΠ c = QT (with Π c designing a column permutation matrix). In the literature, removing the linear dependent columns of S is called "Arnoldi deflation" [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF].

As discussed later, a deficiency of S characterizes a breakdown in the block Arnoldi procedure. We will show in Section 3.6 that this behaviour is rare in practice because it means that a linear combination of p -p j exact solutions has been found. Thus it is more realistic to consider that the relations n j = 0 and p j = p do hold for every iteration j. Consequently a standard QR decomposition based on modified Gram-Schmidt is then used instead.

Remark 2.5.2. Steps 3 and 4 of Algorithm 2.5.2 amount for the orthogonalization of the basis, which may lack of stability if not performed properly. Indeed in Algorithm 2.5.2 we just present a naïve perspective for the sake of clearness, and an advanced method is suggested for such orthogonalization when implementing this method in practice. Examples cover CGS2 (Classical Gram-Schmidt with reorthogonalization), or BMGS (block Modified Gram-Schmidt) or Ruhe's variant of BMGS [START_REF] Ruhe | Implementation aspects of band Lanczos algorithms for computation of eigenvalues of large sparse symmetric matrices[END_REF]. We refer to [START_REF] Giraud | A robust criterion for the modified Gram-Schmidt algorithm with selective reorthogonalization[END_REF] and [START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF]Chapter 1] for a deep study on the stability of these methods.

Proposition 2.5.3. After j iterations of Algorithm 2.5.1, it holds that

range V j = K j (AM j , B) range Z j = M j K j (AM j , B) (2.5.1)
for some nonsingular matrix M j ∈ C n×n representing the action of the variable preconditioner up to iteration j. Moreover, V j ∈ C n×sj is a full rank orthonormal matrix, and Z j ∈ C n×sj has full rank.

Proof. It is easy to infer from Algorithm 2.5.2 that

V j H j,j-1 = (I -V j-1 V H j-1 )AM j-1 V j-1 (2.5.2)
for every j ≥ 2, and because V j H j,j-1 arises from an economic QR decomposition, it always holds that range(V j H j,j-1 ) = range(V j ).

for every j ≥ 2. From line 1 of Algorithm 2.5.1 we find that range (V 1 ) = range (B).

We prove then that

range V j = range AM j-1 ...AM 1 B -range V j-1 (2.5.3)
for every j ≥ 2. From (2.5.2), we find that

V 2 H 2,1 = (I -V 1 V H 1 )AM 1 V 1 .
and thus

range (V 2 ) = range V 2 H 2,1 = range (I -V 1 V H 1 )AM 1 V 1 = range (I -V 1 V H 1 )AM 1 B = range (AM 1 B) -range (V 1 )
Assuming it is correct for j -1, we find out that

range V j = range V j H j,j-1 = range (I -V j-1 V H j-1 )AM j-1 V j-1 = range (I -V j-1 V H j-1 )AM j-1 ...AM 1 B = range AM j-1 ...AM 1 B -range V j-1
proving (2.5.3) by induction. Using this knowledge for every j show us that

range V j = range V 1 V 2 . . . V j = range (V 1 ) + range (V 2 ) + . . . + range V j = range (B) + range (AM 1 B) + range (AM 2 AM 1 B) + . . . + range AM j-1 ...AM 1 B = span B AM 1 B . . . AM j-1 ...AM 1 B .
In the very same fashion, noticing that Z j = M j V j by definition (and the rank-preserving assumption of the variable preconditioner; see Definition 2.4.1), we obtain that

range Z j = span M 1 B M 2 AM 1 B . . . M j AM j-2 ...AM 1 B .
From Theorem 2.4.3 we know that there is always a nonsingular matrix M j such that

range Z j = span M 1 B M 2 AM 1 B . . . M j AM j-2 ...AM 1 B = M j K j (AM j , B).
To show that range V j = K j (AM j , B) we use the proof of Theorem 2.4.3. To satisfy both equalities in (2.5.1) at once, a possibility is to find a nonsingular matrix M j ∈ C n×n such that all the equalities in (2.4.6) hold as well as

AM j B = AM 1 B AM j AM j B = AM 2 AM 1 B AM j (AM j ) 2 B = AM 3 AM 2 AM 1 B . . . AM j (AM j ) j-2 B = AM j-1 A . . . M 3 AM 2 AM 1 B. (2.5.4)
Using the nonsingularity of A, and multiplying from the left every equation in (2.5.4) by A -1 we verify that all the conditions in (2.5.4) are already contained in (2.4.6), and thus, any nonsingular matrix M j ∈ C n×n satisfying (2.4.6) also satisfies (2.5.4).

To finalize the proof, we highlight that V j is orthonormal and full rank by construction, and that Z j has full rank because of the assumption of a rank-preserving variable preconditioner.

To the best of our knowledge, Proposition 2.5.3 is the first proof that the block flexible Arnoldi algorithm indeed generates a basis for a block Krylov subspace, being this one of the contributions of this thesis. Even considering p = 1, we are unaware of such a demonstration in the flexible case, although for a fixed preconditioner (or unpreconditioned case) this result is well-known. Remark 2.5.4. We recall that we represent the application of the flexible preconditioner M j (.) on V j by M j V j , that is

M j (V j ) = M j V j .
(2.5.5)

However, in a general scenario

M j (B) = M j B. (2.5.6)
In Proposition 2.5.3 we just clarify that there always exists a linear operator M j such that the referred subspaces are Krylov subspaces.

Even though V j is an orthonormal basis to the block Krylov subspace, Z j is not an orthonormal basis to the correction subspace proposed in Section 2.4 though it is considered a reliable and stable basis [START_REF] Saad | Iterative Methods for Sparse Linear Systems[END_REF][START_REF] Simoncini | Flexible inner-outer Krylov subspace methods[END_REF]. We also note that applying the block Arnoldi algorithm is not equivalent to apply p times the Arnoldi algorithm, because the later would generate p orthonormal basis to p different subspaces but these bases need not to be orthonormal among each other. There is an extra computation effort whenever we prefer the block methods. However, block methods can greatly improve the convergence by using information from all subspaces simultaneously. There are also computational gains whenever we are considering a massively parallel computation environment, but we detail this in the end of Chapter 3.

Breakdown in Block Arnoldi

In this subsection we define the generalization of the concept of Arnoldi breakdown, or happy breakdown for block Arnoldi algorithm. We postpone some more generic proofs and concepts to Section 3.6 in Chapter 3, and we present only some basic concepts here.

In line 5, if S has full rank (that is, n j = 0 and p j = p j-1 ), then H j+1,j will be square and nonsingular. In such case, we will have V j+1 = SH -1 j+1,j which resembles the traditional Arnoldi algorithm for the single right-hand side case, where H j+1,j would be in fact ||S|| 2 .

Again considering p = 1, the so called "happy breakdown" or "Arnoldi breakdown" is a situation in which the Arnoldi algorithm can not proceed its execution because range (S) ⊂ range V j . Since steps 3 and 4 of Algorithm 2.5.2 can be summarized as

S = (I -V j V H j )S,
at the beginning of step 5 of Algorithm 2.5.2 we will have S = 0. In the traditional Arnoldi algorithm for single right-hand side, step 5 of Algorithm 2.5.2 is replaced by V j+1 = SH -1 j+1,j , and thus, it generates a division by zero reason why this phenomena is called "breakdown". This is, however, a "happy" breakdown in the sense that, as mentioned previously (see Proposition 2.4.5), whenever range (S) ⊂ range(K j (A, B)) = range(V j ) we can ensure that the solution X * lies inside Z j .

For the multiple right-hand side scenario, we have a similar concept. It is worthy to note that because the QR decomposition always exists, Algorithm 2.5.2 never interrupts its execution, yet we still call such phenomena "breakdown". Definition 2.6.1 (Partial Breakdown). At the j-th iteration of Algorithm 2.5.1, we say that n j partial Arnoldi breakdowns (or simply partial breakdowns) have been detected. Whenever n j = p j-1 , we say that a full Arnoldi breakdown (or simply full breakdown) has been detected. If we use Algorithm 2.5.1 for p = 1 we see that the concept of of breakdown as stated in Definition 2.6.1 also applies to the single right-hand side case, except that any breakdown is a full breakdown. Also, likewise in the single right-hand side case, Algorithm 2.5.1 generates a decomposition which we formalize with the following definition. Definition 2.6.2 (Block Arnoldi Decomposition). Considering the notation in Algorithm 2.5.1 and Algorithm 2.5.2, we refer to

AZ j = V j+1 H j (2.6.1)
as the block Arnoldi relation or block Arnoldi decomposition.

Here we note that the matrix H j is not an upper Hessenberg matrix as in the ordinary Arnoldi decomposition. In case of no breakdown occurrence, H j is block Hessenberg, that is, if no breakdown occurred until iteration j, there is a band of width p of nonzeros below the main diagonal of H j (see Figure 2.1). Whenever a breakdown occurs in the block Arnoldi algorithm, the size of the diagonal block decreases. Figure 2.2 shows a graphical representation of the H j generated by Algorithm 2.5.2 supposing that one breakdown occurred at iteration i and at iteration k.

Block GMRES

In this section we quickly present the Block GMRES (BGMRES) method first introduced by Vital in [START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF]. Analogously to GMRES, BGMRES uses the block Arnoldi method (Algorithm 2.5.2) to generate a block Arnoldi decomposition (Definition 2.6.2). The advantage of using a block Krylov subspace in this case goes beyond ensuring that a solution will be found in a finite number of steps. In fact, the cost for solving the problem (2.2.2) can be greatly decreased thanks to the block Arnoldi relation (2.6.1) .

Defining6

Λ j = Λ j-1 0 pj ×p ∈ C (sj +pj )×p , ∀j ≥ 1,
we can write (2.2.2) as

min range(X)⊂Zj B -AX F = min Y ∈C s j ×p B -AZ j Y F = min Y ∈C s j ×p V j+1 Λ j -V j+1 H j Y F = min Y ∈C s j ×p Λ j -H j Y F
whose solution is given by

Y j = H † j Λ j . (2.7.1)
Since H j ∈ C (sj +pj )×sj and (AZ j ) ∈ C n×sj , as long as (s j + p j ) < n it is more advantageous to solve (2.7.1) than (2.2.5). Finally we set the approximate solution as

X j = X 0 + Z j Y j .
Naturally BGMRES falls into Definition 2.2.2, and shares all the properties mentioned in Section 2.2. For the sake of completion we present in Algorithm 2.7.1 a pseudocode for restarted block flexible GMRES (BFGMRES).

Algorithm 2.7.1: Restarted block flexible GMRES (BFGMRES)

1 Choose an initial guess X 0 ∈ C n×p , a restart parameter m and define a convergence criterion and its scaling matrix;

2 for cycle = 1, . . . , m do 3 Compute the initial true block residual R 0 = B -AX 0 ; 4 Compute the QR decomposition R 0 = V 1 Λ 0 obtaining n 0 = null (R 0 ), p 0 = p -n 0 , V 1 ∈ C n×p0 and Λ 0 ∈ C p0×p ; 5 Define V 1 = V 1 , s 0 = 0 and s 1 = p 0 ; 6 for j = 1, . . . , m do 7
Completion of V j+1 , Z j and H j : Apply Algorithm 2.5.2 to obtain

AZ j = V j+1 H j with V j+1 = V 1 , V 2 , . . . , V j+1 , (2.7.2)
s j and p j , with Z j ∈ C n×sj , V j+1 ∈ C n×(sj +pj ) and H j ∈ C (sj +pj )×sj ;

Convergence Criteria in MBR Methods

In this section we discuss a generalization to the multiple right-hand side scenario of common stopping criteria known for the single right-hand side. In Definition 2.8.1 we propose the partial convergence, a key concept which we explore in later sections and chapters.

The intention of MBR methods in most cases is not to find an exact solution for the problem AX = B, but only an approximation according to a chosen criterion, usually relying on the norm of the scaled residual, defined as

R j D j ,
where D j ∈ C p×p is a nonsingular "scaling matrix ". The choice for D j is application dependent, but common choices are the relative residual criterion

D R j =            1 R 0 e 1 2 1 R 0 e 2 2
. . .

1 R 0 e p 2            , the backward error D B j =         1 A 2 X j e 1 2 + R 0 e 1 2
. . .

1 A 2 X j e p 2 + R 0 e p 2         , or the backward error with respect to A D A j =         1 A 2 X j e 1 2
. . .

1 A 2 X j e p 2         .
The relative residual criterion is constant over j, whereas the backward error and the backward error with respect to A varies with j. For the sake of generality, we always consider a generic matrix D j which may or may not vary at each iteration j.

The goal of the algorithm is then to find a solution X j such that

(B -AX j )D j F = R j D j F ≤ ε (2.8.1)
for a given threshold ε ≥ 0 (notice that if we set ε = 0 we are looking for X j = X * ). Another possibility rather than (2.8.1) is to look for an approximation of the solution X j such that

(B -AX j )D j ψ = R j D j ψ ≤ ε (2.8.2)
but since satisfying (2.8.1) implies satisfying (2.8.2), for the moment we choose the former approach for the sake of simplicity. As mentioned earlier, during the execution of Algorithm 2.7.1 (or any MBR method), in some situations it may be known that we have found the exact solution for a linear combination of right-hand sides, namely when R j is rank deficient. However, we are interested in finding approximations for the solution of a linear combination of right-hand sides rather than the exact solution itself. In fact, according to equation (2.2.2) in Definition 2.2.2, if Be i -A Xe i 2 is large for i = k and small for all i = k, the Frobenius norm of R j will be at least as large as Be k -AXe k 2 , hiding the information that some of the linear systems might have already converged.

Another possibility is to compute the norm of each individual column of the block residual and then use the scaled residual for checking convergence individually for each one of them. However, again it doesn't give any information about the linear combination of the solution. The relative block residual may happen to have two linear dependent columns (therefore R j is rank deficient) and nothing guarantees that the norm of those individual columns will be small.

For this reason we propose a "near-rank deficiency" of the block residual, based on a threshold ε according to the following definition. Definition 2.8.1 (Partial Convergence). If at the end of the j-th iteration of any MBR method there is a full rank orthonormal matrix W ∈ C p×t such that

R j D j W F ≤ t p ε
holds, then we say that t partial convergences have been detected at iteration j.

If t = p (thus ||R j D j W || F = ||R j D j || F ≤ ε)
we say that a full convergence has been detected at iteration j.

Stagnation in BGMRES

In this section we propose a new generalization of the concept of stagnation for multiple right-hand side. This concept will be used later on section Section 3.4. When p = 1, it is commonly said that we have a stagnation in iteration j whenever

R j = R j-1 .
This phenomenon implies a series of other conclusions for the single right-hand side case that are not trivially extended to the multiple right-hand side case. To the best of our knowledge, a generalization of this concept and these properties for multiple right-hand side has not been already addressed. We propose in Definition 2.9.1 a new definition of stagnation for multiple right-hand side scenario, and we dedicate the remaining of this section showing the equivalence of this definition with the single right-hand side case, and showing that some common behaviours associated with the single right-hand side stagnation are also analogous for this definition of stagnation. Definition 2.9.1 (Partial Stagnation). Define

t j = p j -rank (I -V j V H j )R j
Whenever t j > 0 at the end of the j-th iteration of BFGMRES (Algorithm 2.7.1), we say that t j partial stagnations have been detected at iteration j. If t j = p j then we say that a full stagnation has been detected instead.

With Proposition 2.9.3 and Proposition 2.9.4 we show some consequences of the occurrence of partial stagnation according to Definition 2.9.1. Lemma 2.9.2. Consider the j-th iteration of Algorithm 2.7.1 and define

Λ j -H j Y j = Rs j Rp j and Λ j-1 -H j-1 Y j-1 = Rj-1
with Rs j ∈ C sj ×p , Rp j ∈ C p×p , Rj-1 ∈ C sj ×p , and assume that n i = 0, 1 ≤ i ≤ j . If null Rp j = t, then there exists an orthonormal matrix L 1 ∈ C p×t such that

Rs j L 1 = Rj-1 L 1 .
Proof. We write

Rs j Rp j = Λ j-1 0 - H j-1 H j 0 H (j+1,j) Y s j Y p j with Y T j = [(Y s j ) T (Y p j ) T ], Y s j ∈ C sj -p×p , Y p j ∈ C p×p . Then Rs j Rp j = Λ j-1 -H j-1 Y s j 0 p×p - H j Y p j H (j+1,j) Y p j
and thus

Rp j = -H (j+1,j) Y p j .
If null Rp j = t then there is an orthonormal matrix L 1 ∈ C p×t such that

Rp j L 1 = -H (j+1,j) Y p j L 1 = 0.
Because we assumed n i = 0, 1 ≤ i ≤ j, we have that H (j+1,j) is nonsingular, so that

H (j+1,j) Y p j L 1 = 0 ⇐⇒ Y p j L 1 = 0 and consequently Rs j Rp j L 1 = Λ j-1 -H j-1 Y s j 0 p×p L 1 .
Noticing that Y j-1 solves the problem min Y ∈C s j -p×p Λ j-1 -H j-1 Y F and the solution for this minimization is always unique, we deduce that Y s j L 1 = Y j-1 L 1 and

Rs j L 1 = Λ j-1 -H j-1 Y j-1 L 1 = Rj-1 L 1 finalizing the proof.
Proposition 2.9.3. Suppose that each n i = 0 for 1 ≤ i ≤ j. Then, for every iteration j of Algorithm 2.7.1 it holds that

dim range R j ∩ range R j-1 = t j (2.9.1)
where t j is given in Definition 2.9.1.

Proof. Following the notation from Lemma 2.9.2, the following always holds

(I n -V j V H j )R j = (I n -V j V H j ) V j V j+1 Rsj j Rp j = (I n -V j V H j )V j Rsj j + (I n -V j V H j )V j+1 Rp j = V j+1 Rp j ,
(2.9.2) and therefore

t j = null (I n -V j V H j )R j = null V j+1 Rp j = null Rp j .
From Lemma 2.9.2 the proposition is proved.

The immediate consequence of Lemma 2.9.2 and Proposition 2.9.3 is that a stagnation implies that the residual R j is linear dependent with the residual R j-1 , that is, a linear combination of residuals did not change from iteration j -1 to iteration j, establishing the connection between the multiple right-hand side and the single right-hand side case.

The next proposition shows a result which is going to be particular useful in Section 3.4 later on.

Proposition 2.9.4. For every iteration j of Algorithm 2.7.1, it holds that

range V j+1 ⊇ range (I n -V j V H j )R j .
(2.9.3)

If no partial stagnation (see Definition 2.9.1) has occurred, then

range V j+1 = range (I n -V j V H j )R j .
(2.9.4)

Proof. From (2.9.2) in the demonstration of Proposition 2.9.3 we already know that (2.9.3) holds.

If no partial stagnation occurs, then we have that

rank (I n -V j V H j )R j = p j
and since rank V j+1 = p j by definition, we prove (2.9.4).

Conclusions

In this chapter we have studied a generalization of the well known GMRES algorithm for multiple righthand side scenario, the block GMRES. We have developed a theoretical basis related to the subspaces being spanned by BGMRES in the presence of a variable preconditioner. The result found in Theorem 2.4.3 is new to the best of our knowledge even in the single right-hand side case. Some other new properties have been demonstrated (e.g. Proposition 2.5.3 which is a consequence of Theorem 2.4.3). We also determined a generalization of a set of concepts which are common in the single right-hand side scenario, but not globally formalized for the multiple right hands side scenario (as Definition 2.6.1, Definition 2.8.1 and Definition 2.9.1). Among those, we remark the definition of partial stagnation (cf. Definition 2.9.1) which is not trivially deduced from the single right-hand side case. The importance of these definitions will come clear as we advance in a more complex scenario, in Chapter 3.

Chapter 3

Deflation

Introduction

In the previous chapter we have studied Block GMRES (BGMRES) due to Vital [START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF] for solving the problem AX = B, with A ∈ C n×n nonsingular, B, X ∈ C n×p where n p and rank (B) = p. We extended concepts common in the single right-hand side scenario for the block scenario, as partial convergence (when a linear combination of approximate solutions is found rather than the approximate solution of the entire block system) and the partial breakdown (which is basically a linear combination of happy breakdowns). BGMRES has ever since been improved based on the assumption that a subspace of the correction subspace can be discarded, a process called deflation [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF]. It is recognized that to be effective in terms of computational operations, block iterative methods must incorporate a deflation strategy [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF], most notably when a partial convergence is detected.

We briefly summarize now the most common deflation techniques available in the literature. In [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] it is proposed the initial deflation. It consists in performing a block size reduction of the (scaled) initial residual R 0 relying on its (near) rank deficiency. BFGMRESD [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] computes

R 0 = QT (QR decomposition) T D 0 = U + Σ + W H + + U -Σ -W H -+ (singular value decomposition) (3.1.1)
where all the singular values larger or equal than a threshold ε d lie in Σ + and those smaller lie in Σ -, and D 0 is the nonsingular scaling matrix. This gives raise to the low rank approximation of R 0 as

C n×k1 R0 = QU + Σ + W H + .
The algorithm then proceeds with one cycle of nonsingular BGMRES, minimizing R0 -A X rather than R 0 -AX. The cost per iteration is thus reduced since every V j has k 1 columns instead of p. At the end of the said cycle, some manipulations are performed in order to retrieve the approximate solution for the original problem. A truncated variant called BFGMREST [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] allows the user to set a maximum number of columns allowed in R0 thus truncating the block initial residual even if the singular values of the residual are not smaller than ε d . This strategy aims at reducing the memory requirements of the method when many right-hand sides are considered at once. Furthermore, both BFGMRESD and BFGMREST are proposed for variable preconditioner scenario.

The BlMResDefl [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] uses a very similar technique, but relying on a rank-revealing QR decomposition to obtain the decomposition [64, (12.1)]), where Π c is a permutation matrix responsible for reordering the columns of R 0 such that the elements in the diagonal of [(Λ 0 ) T (Λ ∆ 0 ) T ] T are given in nonincreasing order. BlMResDefl then sets the new initial residual to V 1 Λ 0 and proceeds executing one cycle of BGMRES algorithm, as BFGMRESD.

R 0 = V 1 V ∆ 1 Λ 0 Λ ∆ 0 Π H c (cf.
The BlMResDefl [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] proposes not only initial deflation techniques, but also the so called "Arnoldi deflation", already discussed in Remark 2.5.1. It consists of determining which columns of S in line 5 of Algorithm 2.5.2 are linear dependent to ensure that V j+1 is not rank deficient. In case of linear dependency, these columns are removed from V j+1 (characterizing thus the deflation), and the required manipulations are performed over H j . This deflation is reported in [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF] to never become active in practical numerical experiments.

In [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] a deflation technique similar to Arnoldi deflation is proposed for BGMRES-W, therein called inexact (Arnoldi) breakdown. Supposing that no deflation was performed, BGMRES-W deflates whenever S is near rank deficient, basing this choice on the singular values of H j+1,j and a threshold ε d . The deflation incurs some modifications in the block Arnoldi iteration, and the subsequent deflation relies on a submatrix of H j instead of H j+1,j . We refer the reader to [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] for more details. Nevertheless, as the authors observe, BGMRES-W tends to deflate only at the end of the convergence history, and this observation is confirmed in [START_REF] Khabou | Solveur itératif haute performance pour les systèmes linéaires avec seconds membres multiples[END_REF].

In the same publication [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF], BGMRES-R is proposed as an alternative to BGMRES-W relying on the singular values of the (scaled) residual R j every iteration j. Similarly to BFGMRESD, it computes a decomposition as (3.1.1) for R j , and using a submatrix of U + to choose which columns of V j are going to be carried out for the next block Arnoldi iteration, postponing the remaining ones. The postponed columns are used in the block Arnoldi algorithm for orthogonalization purposes only. A FOM variant of BGMRES-W and BGMRES-R can be found in [START_REF] Robbé | Exact and inexact breakdowns in block versions of FOM and GMRES methods[END_REF], and further numerical experiments in [START_REF] Khabou | Solveur itératif haute performance pour les systèmes linéaires avec seconds membres multiples[END_REF].

Besides the aforementioned methods, strategies based on rank-revealing QR-factorizations [START_REF] Businger | Linear least squares solutions by Householder transformations[END_REF] or singular value decomposition [START_REF] Golub | Matrix Computations[END_REF] have been notably proposed both in the Hermitian [START_REF] Nikishin | Variable block CG algorithms for solving large sparse symmetric positive definite linear systems on parallel computers, i: General iterative scheme[END_REF][START_REF] Ruhe | Implementation aspects of band Lanczos algorithms for computation of eigenvalues of large sparse symmetric matrices[END_REF] and non-Hermitian cases [2,[START_REF] Bai | ABLE: an adaptive block Lanczos for non hermitian eigenvalue problems[END_REF][START_REF] Cullum | Two-sided Arnoldi and non-symmetric Lanczos algorithms[END_REF][START_REF] Freund | A block QMR algorithm for non-Hermitian linear systems with multiple right-hand sides[END_REF][START_REF] Loher | Reliable Nonsymmetric Block Lanczos Algorithms[END_REF][START_REF] O'leary | The block conjugate gradient algorithm and related methods[END_REF] for block Lanczos methods. They have been shown to be effective with respect to standard block Krylov subspace methods, but since we are focusing on iterative methods showing a minimal residual property for non-Hermitian problems, we do not focus on the study of these methods.

Variable preconditioning is often required when solving large linear systems. This is notably the case when inexact solutions of the preconditioning system using, e.g., nonlinear smoothers in multigrid [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] or approximate interior solvers in domain decomposition methods [START_REF] Toselli | Domain Decomposition methods -Algorithms and Theory[END_REF]Section 4.3] are considered. The combination of block methods performing deflation at each iteration and variable preconditioning has been rarely addressed in the literature, although the combination of initial deflation with variable preconditioning has been already explored in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF][START_REF] Calandra | Two-level preconditioned Krylov subspace methods for the solution of three-dimensional heterogeneous Helmholtz problems in seismics[END_REF][START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF]. Thus the main purpose of this chapter is to derive a class of flexible minimal block residual methods for non-Hermitian problems that incorporate deflation at each iteration. This chapter is organized as follows. In Section 3.2 we propose a generalization of the concepts established in Chapter 2 aiming at a method able to judiciously choose which (block) Krylov directions are interesting for expanding the correction subspace Z j every iteration j, to then present and re-interpret a block iterative procedure firstly introduced in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] which is able to build an orthonormal basis for the chosen directions only (the "deflated block Arnoldi"). In Section 3.3 a general framework for deflated block Krylov subspace methods is presented. We show that the resulting method (named "deflated minimal block residual" method or DMBR for short) always minimizes the Frobenius norm of the block residual and that the singular values of the scaled block residual are always nonincreasing. In Section 3.4 we propose a criterion for choosing which directions to take into account when expanding Z j at iteration j, which is mostly based on Section 2.9 as well as [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] and [START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF]. Then, in Section 3.6 we show that DMBR using the proposed criterion never breaks down, thus guaranteeing convergence (considering a large enough restart size) along with other properties. In Section 3.5 we use DMBR as a framework to describe existing algorithms as BGMRES, BGMRES-R and BFGMRESD and in which situations these algorithms could be considered as equivalent to DMBR. Then in Section 3.9 we demonstrate the effectiveness of DMBR on three academic illustrations and one real life application, showing that in practical cases, none of these methods are algebraically or numerically equivalent to DMBR. These conclusions are later extended with further experiments in Section 4.6. Finally we draw some conclusions in Section 3.10.

Deflated Block Arnoldi

In this section we present a generalization of the block Arnoldi method (cf. Algorithm 2.5.2 in Section 2.5), which we call here "deflated block Arnoldi", whose focus it to take into account Remark 2.2.9 to then generate a basis of a subspace of smaller dimension.

The deflated block Arnoldi was firstly proposed in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] although the authors do not define a specific name for the method. The main idea is to redefine line 5 of Algorithm 2.5.1 to

S = V j+1 H j+1,j + Q j with range Q j ⊥ range V j+1 (3.2.1)
where Q j is chosen such that it takes into account the numerical rank in the QR factorization. It proceeds orthogonalizing every

Q i against V j+1 , obtaining Qj = (I -V j+1 V H j+1 )[Q 1 ... Q j ]
and generating what is called in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] the inexact block Arnoldi relation 26),( 28)]). To obtain such a relation a series of other modifications is required in the original block Arnoldi algorithm. We refer to [103, §5] and to Subsection 3.5.1 for more details on BGMRES-R algorithm.

AZ j = V j+1 L j + Qj where L j = V H j+1 AZ j (cf. [103, (
We propose in this section a reformulation of the deflated block Arnoldi to fit the discussion of Chapter 2, specifically Section 2.2 (specially Remark 2.2.9) and Section 2.9. Also, in contrast with [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] in our redefinition the criterion for splitting (3.2.1) is generalized and decoupled from the orthonormalization procedure itself. We address possible criterion later in Section 3.4 and Section 3.7.

Letting K j-1 ∈ C n×pj-1 denote a full rank matrix whose range contains all the latest p j-1 Krylov directions available at the end of iteration (j -1) (with 1 ≤ p j-1 ≤ p), the most expensive part of Algorithm 2.5.2 at the j-th iteration -when n is large -lies in the p j-1 applications of the preconditioner (see line 1) and the subsequent p j-1 matrix-vector products (see line 2).

As discussed later in Section 3.4, subspaces of range K j-1 may no longer be needed for ensuring convergence along the iterative procedure. The goal of deflated block Arnoldi is thus to exploit this knowledge spanning a smaller correction subspace, avoiding preconditioner application and matrix-vector products over the uninteresting directions, as well as reducing the orthogonalization cost for the next iteration. Suppose that we are able to judiciously decompose range K j-1 into:

range K j-1 = range V j ⊕ range P j-1 , V j P j-1 H V j P j-1 = I pj-1 , (3.2.2)
where

V j ∈ C n×kj , P j-1 ∈ C n×dj with k j + d j = p j-1 .
For the sake of generality, having in mind the notation established in Section 1.1, we define

V i P i-1 = V i , whenever d i = 0, (3.2.3)
since P i-1 is undefined in such a case, and we consider that d 0 is always equal to zero. At iteration j we consider the k j Krylov directions contained in range(V j ), while leaving aside (or deflating) d j directions contained in range(P j-1 ), performing matrix-vector products and preconditioner applications only over the chosen k j directions of V j . We show the j-th iteration of deflated block Arnoldi method using a variable preconditioner in Algorithm 3.2.1.

As in standard block Arnoldi (cf. Algorithm 2.5.2), Algorithm 3.2.1 orthonormalizes AZ j against V j but additionally against P j-1 also (line 4 and 5 of Algorithm 3.2.1). Here also Remark 2.5.2 is applicable: a naïve orthonormalization algorithm is presented in Algorithm 3.2.1, but a version of block Arnoldi due to Ruhe [START_REF] Ruhe | Implementation aspects of band Lanczos algorithms for computation of eigenvalues of large sparse symmetric matrices[END_REF] or block Householder orthonormalization [START_REF] Baglama | Adaptively preconditioned GMRES algorithms[END_REF][START_REF] Sun | A basis-kernel representation of orthogonal matrices[END_REF] could be used as well. Notice also that whenever d j = 0, using (3.2.3), we find that one iteration of Algorithm 3.2.1 is equivalent to one iteration of Algorithm 2.5.1, having k j = p j-1 , Ĥj = H j and Vj+1 = V j+1 .

In Proposition 3.2.1 we show the flexible Arnoldi relation that is obtained when using the deflated block Arnoldi procedure shown in Algorithm 3.2.1.

Algorithm 3.2.1: Deflated block flexible Arnoldi iteration: completion of

Zj ∈ C n×s j , Vj+1 ∈ C n×(s j +p j ) , Ĥj ∈ C (s j +p j )×s j with Vi, Zi ∈ C n×k i for 1 ≤ i ≤ j, such that ( Vj+1) H Vj+1 = I 1 Receives V j P j-1 ∈ C n×pj-1 with V j ∈ C n×kj and p j-1 = k j + d j where V 1 ... V j P j-1 is orthonormal; 2 Z j = M j V j ; 3 S = AZ j ; 4 H j = V j P j-1
H S, where H j ∈ C (sj-1+pj-1)×kj ;

5 S = S -V j P j-1 H j ; 6 Compute the QR decomposition S = Vj+1 H j+1,j obtaining n j = null (S), Vj+1 ∈ C n×(kj -nj ) and H j+1,j ∈ C (kj -nj )×kj ; 7 Define s j = s j-1 + k j and p j = p j-1 -n j ; 8 Define Z j = Z 1 ... Z j , Vj+1 = V 1 ... P j-1 Vj+1 ; 9 Define Ĥj = H j-1 H j 0 (kj -nj )×sj-1 H j+1,j , or Ĥ1 = H 1 H 2,1 for j = 1 ; Proposition 3.2.1. With the notation of Algorithm 3.2.1, given Z j-1 ∈ C n×sj-1 ,V j ∈ C n×sj , H j-1 ∈ C (sj-1+pj-1)×sj-1 , V j P j-1 ∈ C n×pj-1 and V j ∈ C n×kj , satisfying AZ j-1 = V j P j-1 H j-1 with s j = s j-1 + k j , p j-1 = k j + d j and [V j P j-1 ] H [V j P j-1 ] = I (sj +dj )
, after applying Algorithm 3.2.1 we obtain the block flexible Arnoldi relation

AZ j = Vj+1 Ĥj . (3.2.4)
where Vj+1 is orthonormal.

Proof. We can summarize one iteration of Algorithm 3.2.1 as

Vj+1 H j+1,j = I -V j P j-1 V j P j-1 H AZ j .
Taking H j according to line 4 of Algorithm 3.2.1, the equation above lead us to

AZ j = Vj+1 H j H j+1,j
and this together with the line 1 of Algorithm 3.2.1 yields

AZ j-1 AZ j = V j P j-1 H j-1 Vj+1 H j H j+1,j A Z j-1 Z j = V j P j-1 Vj+1 H j-1 H j 0 (kj -nj )×sj-1 H j+1,j AZ j = Vj+1 Ĥj .
The orthonormality of Vj+1 comes from the fact that we considered V j P j-1 already orthonormal and that Vj+1 was orthonormalized using Classical Gram-Schmidt.

Notice that (3.2.4) is identical to the standard block flexible Arnoldi relation (cf. Definition 2.6.2) except that V j+1 and H j are replaced respectively by Vj+1 and Ĥj . Also, Algorithm 3.2.1 does not address the choice of P j-1 or how to define respectively V j+1 and H j from Vj+1 and Ĥj for its next iteration, which is intimately related to the splitting we mentioned in (3.2.2).

Deflated Minimal Block Residual

We propose in this section the "deflated minimal block residual" method (hence, DMBR), as BGMRES-R [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF], uses the deflated block Arnoldi iteration (cf. Algorithm 2.5.2), to build an orthonormal basis for the approximation subspace. We postpone to Section 3.5 a major comparison between DMBR and BGMRES-R, but briefly mention here that the main difference is that DMBR is able to deflate at the beginning of each iteration, whereas BGMRES-R can deflates only at the end of the iteration. For doing so, DMBR performs an extra step while deflating. This difference between the methods gives raise to a considerably different behaviour as we discuss in Section 3.5 and as we show in practice in Section 3.9.

Also in this section we introduce in our framework a unitary deflation matrix F j whose dimension is yet to be defined. The purpose of this matrix is to represent the action of reorganizing or recombining the columns of a matrix according to a criteria to be established. For instance, letting K be any matrix of proper dimensions, K is said to be undeflated whereas KF j is said to be deflated in the sense that the first (or last) columns of KF j satisfy the chosen criteria (to be discussed in Section 3.4), allowing the method itself to delete these last columns of KF j or to reuse them for a different purpose. Similarly, F H j can also be used to reorganize and recombine the rows of a matrix.

We introduce in Algorithm 3.3.1 a pseudocode for DMBR following the mentioned framework. Therein the deflation is performed on Vj+1 and reflected on the rows of Ĥj (cf. line 9-10 of Algorithm 3.3.1) as in BGMRES-R, but in addition, the rows of Λj are deflated, being this the reason why DMBR is able to deflate at the beginning of each iteration. Also, as in BGMRES-R, because F j is unitary its application do not delete the direction which are not chosen, but simply postpone them. We remark that in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] no unitary deflation matrix as F j is presented for BGMRES-R because the formulation of the methods are different, but as we discuss later in Section 3.5, under certain conditions, both methods are algebraically equivalent.

After applying the unitary deflation operator F j in line 9 of Algorithm 3.3.1 during iteration j, we obtain

AZ j-1 = Vj F j F H j Ĥj-1 = V j P j-1 H j-1 (3.3.3)
which is precisely the relation we need for the j-th iteration of Algorithm 3.2.1 to generate the decomposition (3.2.4). Also, thanks to the fact that F j is unitary, we guarantee that [V j P j-1 ] is still orthonormal.

We show now that regardless of the choice of F j and k j , DMBR minimizes the block residual over Z j .

Lemma 3.3.1. In Algorithm 3.3.1, at the end of the j-th iteration for any 1 ≤ j ≤ m and any chosen unitary matrix F j ∈ C (sj +dj )×(sj +dj ) , we have

R 0 = Vj+1 Λj . Proof. For j = 0 it follows directly from line 4 of Algorithm 3.3.1 that the lemma is correct. Assuming it is true for j = k -1 Vk Λk-1 = R 0 , recalling that every F k is unitary, from Algorithm 3.3.1 we obtain Vk+1 Λk = V k P k-1 Vk+1 Λ k 0 (k k -n k )×p (3.3.4) = V k P k-1 Λ k (3.3.5) = Vk F k F H k Λ k-1 (3.3.6) = R 0 (3.3.7)
proving by induction.

≤ k j ≤ p j-1 , set s j = s j-1 + k j and d j = p j-1 -k j ; 8
Choose the unitary deflation operator F j ∈ C (sj +dj )×(sj +dj ) ;

9 Update V j P j-1 = Vj F j , with V j ∈ C n×sj as the first s j columns of Vj F j ; Update Λ j = F H j Λj-1 with Λ j ∈ C sj ×p and also H j-1 = F H j Ĥj-1 with H j-1 ∈ C sj ×sj (if j > 1);
Completion of Vj+1 , Z j and Ĥj : Apply Algorithm 3.2.1 to obtain

AZ j = Vj+1 Ĥj with Vj+1 = V 1 , . . . , V j , P j-1 , Vj+1 (3.3.1)
as well as the constants p j and n j ;

Set Λj ∈ C (sj +pj )×p as Λj = Λ j 0 (kj -nj )×p ; Set Y j ∈ C sj
F i and 1 ≤ k i ≤ p i-1 , with ≤ i ≤ j. Then Z j Y j solves the problem min range (Z) ⊂ range Z j R 0 -AZ F . (3.3.8)
Proof. We rewrite the problem (3.3.8) as

min Y ∈C s j ×p R 0 -AZ j Y F .
Using Lemma 3.3.1 and (3.3.1) we obtain

R 0 -AZ j Y F = Vj+1 Λj -Vj+1 Ĥj Y F = Λj -Ĥj Y F .
Since Y j is defined as the unique minimal Frobenius norm solution for this problem (see (3.3.2) in Algorithm 3.3.1) and since such solution always exists, the corollary is proven.

Remarking that R j = B -AX j = B -A(X 0 +Z j Y j ) = R 0 -AZ j Y j , Corollary 3.3.2
shows that DMBR method is in fact minimizing the block residual R j over the entire range Z j + range (X 0 ) subspace, regardless of the choice of F j and k j . The following corollary follows directly from Corollary 3.3.2.

Corollary 3.3.3. Let R j be the block residual generated at the j-th iteration of Algorithm 3.3.1 in a given cycle. Then, R j F ≤ R j-1 F .

We can guarantee not only that the Frobenius norm is monotonically decreasing, but also the singular values of the block residual as we show next. Proposition 3.3.4. Let R j be the residual at the end of the j-th iteration of Algorithm 3.3.1 in a given cycle, and suppose that the sequence of preconditioners M i ∈ C n×n is rank-preserving. It holds that

σ i (R j ) ≤ σ i (R j-1 ), 1 ≤ i ≤ p.
Proof. Due to the minimization property (see Corollary 3.3.2) we can write each R i as

R i = Vi+1 (I si -Ĥi Ĥ † i ) Λi = Vi+1 Λi -Vi+1 Ĥi Ĥ † i V H i+1 Vi+1 Λi = R 0 -Vi+1 Ĥi Ĥ † i V H i+1 R 0 = (I n -Vi+1 Ĥi ( Vi+1 Ĥi ) † )R 0 = (I n -AZ i (AZ i ) † )R 0
Consider that we dispose of an orthonormal basis W j to range AZ j where W j can be defined recursively as W j = [W j-1 W j ], W 1 = W 1 . Then using the idempotence property of projectors we obtain

R i = (I n -W i W H i )(I n -W i W H i )R 0 = (I n -W i W H i )(I n -W i-1 W H i-1 -W i W H i )R 0 = (I n -W i W H i )(R i-1 -W i W H i R 0 ) = (I n -W i W H i )R i-1 From [71, Theorem 3.3.16], the proof is finished.
In fact, since Proposition 3.3.4 relies only on the orthogonality with respect to the updated basis, it is in fact applicable to any block method presenting minimal residual properties, which comprises BGMRES, DMBR, BGMRES-R among others. It is not applicable to BFGMRESD or BFGMREST however, because these methods rely on a low rank approximation of the initial residual R 0 which is computed at the beginning of each cycle, characterizing thus a different scenario.

Let R (k) j denote the true block residual obtained at the end of j-th iteration of the k-th cycle of DMBR, where

1 ≤ j ≤ m (that is, the restart size is m). Because R (k+1) 0 = R (k) m , we can ensure also that σ i (R (k+1) 1 ) ≤ σ i (R (k)
m ) meaning that DMBR ensures the monotonically nonincreasing behaviour of the singular values of the true block residual, not only along the iterations, but also along the cycles regardless of the choice of F j and k j , a property also present in BGMRES-R method. This property can be easily extended to the Frobenius norm or the Euclidean norm of the true block residual, and could be extended to any unitarily invariant norm, see [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF]Section 3.3] for a similar proof related to block flexible Krylov subspace methods with deflation performed at restart only.

The nonincreasing behaviour of the singular values will appear as particularly important when determining the unitary deflation operator in Section 3.4. Also, although we can not guarantee that for an arbitrary choice of D j the singular values of R j D j will be also monotonically decreasing, for a fixed D j = D we trivially obtain that Proposition 3.3.4 also holds for R j D.

Choosing the Unitary Deflation Operator

In this section we discuss in detail a criterion for choosing the unitary deflation operator F j , consisting of a reformulation of the so called R-criterion proposed in [103, §5] for BGMRES-R, adapted to fit the discussion in Definition 2.2.8 in Section 2.2 and our previous discussion on the decomposition Equation 3.2.2. We stress that in [103, §5] there is no mention of F j introduced in Section 3.3, and that we use this framework to be able to deduce different possibilities for deflation in block iterative solvers based on BGMRES.

Generalizing Proposition 2.9.4 to DMBR and in view of (3.2.2), we obtain that

range K j = range (I n -V j-1 V H j-1 )R j-1 ∀j > 1
with range (K 1 ) = range (R 0 ). The BFGMRES method uses the whole range K j to expand the correction subspace (that is, it aims at setting range Z j = AM j range K j , reminding that range K j = range V j in this setting when no partial stagnation occurs, cf. Proposition 2.9.4) for iteration j, but as mentioned in Section 3.2 we want to be able to judiciously split range K j .

Recalling that D j-1 is the chosen scaling matrix for iteration (j -1) (cf. Section 2.8), consider that we are at the beginning j-th iteration of Algorithm 3.3.1 and that d j > 0 was already chosen, and that we dispose of the following thin singular value decomposition [60, p.72 §2.5.4]

Λj-1 -Ĥj-1 Y j-1 D j-1 = (3.4.1)
U + U - Σ + 0 0 kj ×(p-pj-1) 0 Σ -0 dj ×(p-pj-1)    W H + W H - W H 0    = (3.4.2) U + Σ + W H + + U -Σ -W H - (3.4.3) with U + ∈ C (sj +dj )×kj , U -∈ C (sj +dj )×dj , Σ + ∈ C kj ×kj ,Σ -∈ C dj ×dj , W + ∈ C p×kj , W 0 ∈ C p×(p-pj-1)
and W -∈ C p×dj where p j-1 = k j + d j , and the singular values are given in nonincreasing order. Knowing that this gives us the thin singular decomposition of the scaled block residual as

R j-1 D j-1 = Vj U + Σ + W H + + U -Σ -W H -,
and denoting

R + j-1 = R j-1 D j-1 W + and R - j-1 = R j-1 D j-1 W -, we then conclude that R + j-1 F = R j-1 D j-1 W + F = Σ + F R - j-1 F = R j-1 D j-1 W -F = Σ -F R j-1 D j-1 W 0 F = 0 If ||Σ -|| F < (p -k j )/p ε where ε is the convergence tolerance, then we have found (p -k j ) partial convergence according to Definition 2.8.1. Namely, X j-1 [W -W 0 ] is an approximate solution of AV = B[W -W 0 ].
In such a case, we no longer need to expand the correction subspace Z j-1 = range Z j-1 targeting further minimization of ||B[W -W 0 ] -AX|| F (which is associated to R - j-1 and the nullspace of the residual) but only to further minimize ||BW + -AX|| F (which is related to R + j-1 ). Inspired by Proposition 2.9.4, the criterion proposed in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] for choosing F j consists thus in splitting range K j as

range V j = range (I n -V j-1 V H j-1 )R + j-1 , range P j-1 = range (I n -V j-1 V H j-1 )R - j-1 , range K j = range (I n -V j-1 V H j-1 )R j-1 = range V j ⊕ range P j-1 .
If we further split the matrices

U + U -=         U + k1 U - k1 U + k2 U - k2 . . . . . . U + kj-1 U - kj-1 U + pj-1 U - pj-1         , with U + ki ∈ C ki×kj ,U - ki ∈ C ki×dj for 1 ≤ i ≤ (j -1), U + pj-1 ∈ C pj-1×kj and U - pj-1 ∈ C pj-1×dj , then R + j-1 = Vj U + Σ + =   (j-1) i=1 V i U + ki + P j-2 Vj U + pj-1   Σ + R - j-1 = Vj U -Σ -=   (j-1) i=1 V i U - ki + P j-2 Vj U - pj-1   Σ -.
This directly drives us to the conclusion that

(I n -V j-1 V H j-1 )R + j-1 = P j-2 Vj U + pj-1 Σ + (3.4.4) (I n -V j-1 V H j-1 )R - j-1 = P j-2 Vj U - pj-1 Σ - (3.4.5)
and thus

range V j = range P j-2 Vj U + pj-1 range P j-1 = range P j-2 Vj U - pj-1 .
Let

F j T = U + pj-1 U - pj-1 be a full QR decomposition of [U + pj-1 U - pj-1 ] ∈ C pj-1×pj-1 , with F j ∈ C pj-1×pj-1
and T ∈ C pj-1×pj-1 . One possible orthonormal basis of range V j (or range P j-1 ) can be obtained by simply setting V j (or P j-1 ) as the first k j (or last d j ) columns of [P j-2 Vj ]F j , i.e.

V j P j-1 = P j-2 Vj F j . (3.4.6)
With this particular choice of V j and P j-2 , a natural choice for F j such that line 9 of Algorithm 3.3.1 holds is

F j = I sj-1 0 0 F j yielding V j-1 P j-2 Vj Vj I sj-1 0 0 F j Fj = V j-1 P j-2 Vj F j , = V j-1 V j P j-1
Naturally, this discussion is valid only for the case in which d j > 0, otherwise no deflation is needed and setting F j = I sj suffices. We finally formalize this criterion for determining F j and choosing k j in Algorithm 3.4.1.

Algorithm 3.4.1: Choosing Fj -Largest Singular Values of Rj-1Dj-1 1 Receive R j-1 D j-1 = V j P j-1 ( Λj-1 -Ĥj-1 Y j-1 )D j-1 and the parameters 1 ≤ k max ≤ p j-1 and ε d ; 2 U ΣW H = ( Λj-1 -Ĥj-1 Y j-1 )D j-1 (Thin SVD) with U ∈ C (sj +dj )×pj-1 , Σ ∈ C pj-1×p and W ∈ C p×p ; 3 Choose kj such that σ l (R j-1 D j-1 ) ≥ 1/p ε d for all 1 ≤ l ≤ kj ; 4 U pj-1 = U (s j-1 + 1 : s j-1 + p j-1 , 1 : p j-1 ), with U pj-1 ∈ C pj-1×pj-1 (that is, the p j-1 last rows of U ); 5 Define F j T = U pj-1 , (full QR Decomposition) with F j ∈ C pj-1×pj-1 ; 6 Set k j = min( kj , k max ); 7 Define F j = I sj-1 0 0 F j ∈ C (sj +dj )×(sj +dj ) ; Remark 3.4.1.
Even if no deflation has occurred until iteration (j-1) (that is, k i = p for 1 ≤ i ≤ (j-1)), Algorithm 3.4.1 builds a matrix F j which is different from the identity in a general scenario. In such a case, we can then force F j = I pj for every iteration until a deflation happens, meaning that we are neglecting the application of F j in line 9 and line 10 since the respective subspaces remain the same. This change aims at saving computational times. Remark 3.4.2. One disadvantage of Algorithm 3.4.1 is that when a partial stagnation occurs (cf. Definition 2.9.1), then the block residual R j-1 lacks of information concerning the last directions, that is

range K j ⊃ range (I n -V j-1 V H j-1 )R j-1 .
When a partial stagnation occurs, then U pj-1 is rank deficient in line 5 of Algorithm 3.4.1. Because Algorithm 3.4.1 builds the F j matrix out of a full QR decomposition, we still can guarantee that range K j = range P j-2 Vj = range V j P j-1 .

(see also Proposition 3.4.3 ahead) even if a stagnation occurs, but we can not guarantee that the whole subspace of range K j associated with R + j-1 (respectively R - j-1 ) lies in range(V j ) (respectively range(P j-1 )). We remark that a stagnation as in Definition 2.9.1 is a rare phenomenon1 and that it has not been observed in our numerical experiments. Algorithm 3.4.1 never breakdowns, even in case of a full stagnation.

The following discussion (Proposition 3.4.3 and Proposition 3.4.4) aims at showing that the subspace dim(Z j ) is monotonically increasing (cf. Corollary 3.4.5). The goal of such a proof is to satisfy the condition previously established in Definition 2.2.1 for nested subspaces.

F i , 1 ≤ i ≤ j, it holds that rank V j P j-1 = rank P j-2 Vj = p j-1 ,
where

P j-2 Vj = Vj if d j-1 = 0 or if j = 1.
Proof. Each Vi has full rank and orthogonal to P i-2 (if defined) by construction. To show that each P i has full rank whenever they are defined, we use a simple induction: P 0 has full rank by construction. If P i-1 has full rank, then P i-1 Vi+1 has full rank, and since F i is unitary, V i+1 P i will be full rank, completing the proof.

F i , 1 ≤ i ≤ j, it holds that rank( Vj+1 ) = rank( V j+1 P j ) = s j + p j .
If additionally the sequence of variable preconditioners M i , 1 ≤ i ≤ j is rank-preserving (cf. Definition 2.4.1), we can ensure that rank Ĥj = rank H j = s j .

Proof. Proposition 3.4.3 already states that rank (V i ) = p i for every 1 ≤ i ≤ j. Noting that each V i is orthogonal to V j by construction with 1 ≤ i ≤ j, we find out that rank(V j ) = j i=1 k i = s j . This together with Proposition 3.4.3 proves that rank( Vj+1 ) = s j + p j .

Supposing that the sequence of variable preconditioning operators M i , 1 ≤ i ≤ j is rank-preserving, we find that rank( Vj+1 Ĥj ) = rank(AZ j ) = s j . Since Vj+1 has full rank, rank( Ĥj ) = s j and the proposition is proven.

Corollary 3.4.5. Assume that j iterations of DMBR (Algorithm 3.3.1) with Algorithm 3.4.1 have been performed in a given cycle, and that a full convergence has not been detected until iteration j. If the sequence of variable preconditioners M i , 1 ≤ i ≤ j is rank-preserving, then {Z i } j i=1 is a sequence of nested subspaces, where each

Z i = range(Z i ), 1 ≤ i ≤ j.
Proof. This corollary is an implication of Proposition 3.4.3, Proposition 3.4.4 and the rank-preserving assumption on the variable preconditioner. Because rank(Z j ) = dim(Z j ) = s j and also because

Z i-1 ⊂ Z i 1 < i ≤ j by construction, we just have to show that s i-1 < s i 1 < i ≤ j. But s i = s i-1 + k i , and k i ≥ 1 (cf. line 6).
Thanks to Corollary 3.4.5 we conclude that DMBR with Algorithm 3.4.1 spans a nested correction subspace (as in Definition 2.2.1) and thus it can indeed be classified as a method belonging to the MBR family of methods (cf. Definition 2.2.2). Before proceeding to the next section we establish one additional relation concerning the correction subspace constructed by DMBR with Algorithm 3.4.1. Proposition 3.4.6 (Block Krylov Subspace Membership). Consider that we have performed j iterations of Algorithm 3.3.1 with Algorithm 3.4.1 for choosing each F i in a given cycle, and that each M i = M , 1 ≤ i ≤ j, where M is an nonsingular square matrix. It holds that

range Vj+1 ⊂ K j+1 (AM -1 , R 0 ). Proof. We have range V1 ⊂ K 1 (AM -1 , R 0 ) = range (R 0 ). Supposing that range Vj ⊂ K j (AM -1 , R 0 ) we deduce that range Vj+1 ⊂ range AM -1 j V j ⊂ AM -1 range V j P j-1 = AM -1 range P j-2 Vj F j-1 ⊂ AM -1 K j (AM -1 , R 0 ) ⊂ K j+1 (AM -1 , R 0 ),
proving the proposition.

Proposition 3.4.6 states that in the case of a fixed preconditioner case, DMBR spans a subspace contained in a block Krylov subspace. However, in the flexible case, we do not guarantee that such a property holds, or any statement analogous to Proposition 2.5.3. For instance, knowing that

range (R 0 ) = range (V 1 ) ⊕ range (P 0 ) range V2 = range (I -V 1 P 0 V 1 P 0 H )AM 1 V 1 it is clear that range V2 ⊆ span R 0 AM 1 R 0 . But if Algorithm 3.4.1 sets V 2 = P 0 and P 1 = V2 , then range V3 ⊆ span R 0 AM 1 R 0 AM 2 R 0 .
We could explore the existence of a matrix M2 ∈ C n×n such that

range V3 ⊆ span R 0 A M2 R 0 .
instead, allowing the characterization of Z j as a subspace of a Krylov subspace, but this issue is beyond the scope of this thesis, and it is subject for future studies. This characterization of the correction subspace spanned by DMBR explains why its behaviour differs from the behaviour of other similar methods, as BFGMRESD and BGMRES-R as well as BGMRES.

In fact, each one of these methods spans a different correction subspace, and after the first cycle there is no guarantee that there is an intersection between the correction subspace spanned by each method, explaining thus why their convergence behaviour tend to be significantly different when compared with equivalent parameters. We explore with more details the difference between these methods in Section 3.5.

Lastly, the following corollary guarantees that if a fixed scaling matrix D is used, then the value of k j is monotonically nonincreasing, not only along the iterations but also along the cycles. Despite of the similarities between DMBR and BGMRES-R, the later cannot guarantee that Corollary 3.4.7 holds along the cycles, but only within one cycle. That happens because BGMRES-R deflates only in the end of each iteration, and as so it cannot choose k 1 < p 0 . We discuss this and other differences between these methods in Subsection 3.5.1, and we analyse the numerical experiments in Section 3.9.

Connections With Existing Methods

In this section we discuss the similarities and disparities between DMBR and other methods published in the literature. We show that most of these methods can be deduced from Algorithm 3.3.1 by simply choosing F j properly. In Subsection 3.5.1 we discuss the connections with BGMRES-R [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF].

Connections with BGMRES-R

We now consider a comparison between DMBR and BGMRES-R [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF]. For the sake of generality we consider a variant with variable preconditioner even though in [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] the algorithm is considered in the case of fixed preconditioner only. Recalling our previous discussion in the beginning of Section 3.2, BGMRES-R generates the inexact (block) Arnoldi relation

AZ j = V j+1 L j + Qj ,
where L j = V H j+1 AZ j . Instead of storing Qj ∈ C n×sj however, BGMRES-R uses a clever algebraic manipulation storing the QR decomposition

Qj = P j G j ,
where P j ∈ C n×dj+1 and G j ∈ C dj+1×sj . Using our notation, without considering major algebraic details, we state that

H j = L j G j .
Moreover, BGMRES-R computes a F j matrix according to Algorithm 3.4.1, and deflates Vj+1 and Ĥj at the end of every iteration. We show in Algorithm 3.5.3 a simplified pseudocode for both methods, highlighting the differences. Notice that in BGMRES-R, because there is no deflation at the beginning of the first iteration, it holds that

F H 2 Λ1 = Λ0 F H 2 0 (p0-n1)×p = Λ0 0 p1×p ,
and thus, for every j we have that

F H j+1 Λj = Λj ,
that is, it is unnecessary to deflate Λj in BGMRES-R. Both methods are thus algebraically equivalent for every cycle in which DMBR chooses k 1 = p 0 , and not equivalent otherwise. The lack of deflation at the beginning of the first iteration of BGMRES-R brings some crucial drawbacks for BGMRES-R, as we highlight next:

• When the deflations happen early in the convergence history. According to our numerical experiments, this behaviour seems to be common among the tested problems. The value of k j tends to quickly decrease in the first cycles, having k m small (often equal to one) at the end of the cycle (cf. Figure 3.1). Considering the extreme case in which k m = 1, BGMRES-R then performs in the following cycle p 0 + m -1 matrix vector and preconditioner applications, whereas DMBR performs only m. Also, notice that in such a case [V j P j+1 ] has 2p 0 +j columns in BGMRES-R and j +p 0 -1 columns in DMBR, meaning that BGMRES-R has a more expensive orthogonalization than DMBR.

• When the restart size m is small. Since in BGMRES-R k 1 = p 0 , a small restart makes the aforementioned behaviour more evident. Deflate:

V j P j-1 = Vj F j ; 9a Deflate: Λ j = F H j Λj-1 ; 10a Deflate: H j-1 = F H j Ĥj
V j+1 P j = Vj+1 F j+1 ; 18a Deflate: H j = F H j+1 Ĥj ; 19a end for BGMRES-R 5b Define k 1 = p 0 and V 1 = V1 ; 6b for j = 1, . . . , m do 7b Choose k j and F j ; 8b Deflate: V j P j-1 = Vj F j ; 9b Deflate: Λ j = F H j Λj-1 ; 10b Deflate: H j-1 = F H j Ĥj
V j+1 P j = Vj+1 F j+1 ; 18b Deflate: H j = F H j+1 Ĥj ; 19b end for 20 Update R 0 = B -AX 0 ; 21 X 0 = X 0 + ZmYm; 22 end for
• When the number of right-hand sides p 0 is large. Same as above.

One of the main novelties of DMBR over BGMRES-R is hence the deflation of Λj , which allows the deflation steps to take place in the beginning of the iteration while still minimizing the norm of the true residual R 0 -AX j . Other novelties we propose are the truncation (that is, setting k max < p 0 in Algorithm 3.4.1) and the flexible preconditioner, as well as decoupling the deflation strategy from the method itself (that is, allowing any unitary F j to be chosen). We refer to Section 3.9 for more details on the numerical experiments and the practical difference between the behaviour of the two methods.

Connections with BFGMRESD

We now consider now a comparison between DMBR and BFGMRESD (as well as BFGMREST) [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF]. Recalling what was mentioned in the beginning of this chapter, BFGMRESD performs a block size reduction of the (scaled) initial residual R 0 relying on its (near) rank deficiency. Rewriting (3.1.1) with our notation, BFGMRESD computes

R 0 = V1 Λ0 (QR decomposition) (3.5.1) Λ0 D 0 = U + Σ + W H + + U -Σ -W H - (singular values decomposition) (3.5.2)
where (3.5.2) has exactly the same dimensions as (3.4.1)-(3.4.3) (assuming H 0 Y 0 = 0 k1×p ) and where D 0 is the chosen nonsingular scaling matrix. Denoting the structures of BFGMRESD with a # superscript, it then proceeds setting

Λ # 1 = Σ + W H + V # 1 = V1 U + .
Notice however that during the first iteration of DMBR using Algorithm 3.4.1 we have that F 1 = F 1 = U + U -and thus

Λ 1 = U + U - H Λ0 = Σ + W H + Σ -W H - D -1 0 V 1 P 0 = V1 U + U -, that is, V 1 = V # 1 and the first k 1 rows of Λ 1 are equal to Λ # 1 D -1 0 .
Therefore, apart from the scaling, we can rewrite BFGMRESD using DMBR framework: P 0 is discarded as well as the last d 1 rows of Λ 1 , and p 0 is defined as k 1 and the deflation happens during the first iteration (or right before the first iteration). We show in Algorithm 3.5.6 a comparison between both methods. Algorithm 3.5.6: Comparison between DMBR and BFGMRESD 1 Choose X 0 , m and a convergence criterion with its scaling matrix;

2 for cycle = 1, . . . , m do 3 Compute R 0 = B -AX 0 ; 4 V1 Λ0 = R 0 (thin QR decomposition); DMBR 5a Choose k 1 and F 1 ; 6a Deflate: V 1 P 0 = V1 F 1 ; 7a Deflate: Λ 1 = F H 1 Λ0 ; 8a Discard P 0 and
AZ j = V j+1 H j ; |; 15b Set Λ j = Λ j 0 (k j -n j )×p ; 16b Solve: min Λ j -H j Y F ; | ; 17b if full convergence detected then break; 18b ; 19b end for 20 Update R 0 = B -AX 0 ; 21 X 0 = X 0 + ZmYm;
22 end for Thus one cycle of DMBR is algebraically equivalent to BFGMRESD only when no deflation occurs (in which case both algorithms are equivalent to BFGMRES). This happens because BFGMRESD generates a different correction subspace, since it discards P 0 and does not orthogonalize V 2 against it.

Another remark is that the truncation of Λ # 1 means in other words that the least squares problem minimizes only (a linear combination of) k 1 columns of the true block residual R j (namely those which did not converge yet) neglect the remaining ones, which is what we are aiming for in a deflated scenario. This is however not true in the case of BFGMREST, which consists of BFGMRESD where k max < p 0 . BFGMREST truncates Λ # 1 even if the singular values of the scaled true residual are not small, meaning that BFGMREST potentially neglect (a linear combination of) columns of R j which did not converge yet.

As demonstrated in Section 3.4, this is not the case for DMBR, which always minimizes the norm of the whole true block residual R j every iteration, regardless of the chosen k j or k max . This behaviour is one of the main contributions of DMBR over BFGMREST for the case of truncated scenario, and has shown to provide a considerable computational gain in our numerical experiments. Naturally, another feature present in DMBR and not in BFGMRESD or BFGMREST is the possibility of deflation every iteration.

Breakdown in DMBR

In this section we continue the study of breakdowns initially mentioned in Section 2.6. We focus on the possibility of occurrence of a breakdown during the execution of DMBR (Algorithm 3.3.1) with Algorithm 3.4.1. We show that the study of breakdowns in DMBR is relevant mainly from the theoretical point of view, since they are associated with the computation of the exact solution (or a linear combination of exact solutions), and the method tends to present a full convergence before the occurrence of any partial breakdown in most practical cases. Because DMBR can be perceived as a generalization of Block GMRES presented in Section 2.7, this section covers both cases.

The concept of (partial) breakdown we use for the deflated scenario is identical to the one used for the undeflated scenario (cf. Definition 2.6.1 in Section 2.6), but we reproduce it here for convenience. Definition 3.6.1 (Partial Breakdown). At the j-th iteration of Algorithm 3.2.1, we say that n j partial Arnoldi breakdowns (or simply partial breakdowns) have been detected. Whenever n j = p j-1 , we say that a full Arnoldi breakdown (or simply full breakdown) has been detected.

The following theorem aims at showing that any breakdown in DMBR with Algorithm 3.4.1 is indeed a "beneficial" breakdown. Theorem 3.6.2. After j iterations of DMBR (Algorithm 3.3.1) with Algorithm 3.4.1 if the sequence of variable preconditioners M i , 1 ≤ i ≤ j is rank-preserving (cf. Definition 2.4.1), then the exact solution of at least p -p j linear combinations of systems is already known.

Proof. Due to the assumption of rank preservation of Z j , we know from Proposition 3.4.4 that rank( Ĥj ) = s j . Using Lemma 3.3.1 and remarking that Y j = Ĥ † j Λj , we find that

R j = Vj+1 Λj -Ĥj Y j = Vj+1 I (sj +pj ) -Ĥj Ĥ † j Λj
Thanks to Proposition 3.4.4, Vj has full rank and we thus conclude that

rank R j = rank I (sj +pj ) -Ĥj Ĥ † j Λj .
It is known that (I (sj +pj ) -Ĥj Ĥ † j ) is the orthogonal projector onto the orthogonal complement of Ĥj , whose dimension is p j . In such a case, the projection of the p columns of Λj onto such subspace could not possibly span a subspace of dimensions larger than p j , and we can ensure that

rank R j = rank I (sj +pj ) -Ĥj Ĥ † j Λj ≤ p j ,
proving that null(R j ) ≥ p -p j . From Property 2.2.7 the proof is completed.

Corollary 3.6.3. For every iteration j of Algorithm 3.3.1 with Algorithm 3.4.1 for choosing each F i , 1 ≤ i ≤ j, it holds that rank R j ≤ p j . Corollary 3.6.4. If a full breakdown has been found in j iterations of Algorithm 3.3.1 with Algorithm 3.4.1 for choosing each F i , 1 ≤ i ≤ j, then range (X * ) ⊂ Z j . Theorem 3.6.2 shows that any occurrence of a breakdown can be considered to be the analogous of the "happy breakdown" for the single right-hand side case. Also it links the occurrence of a partial breakdown with the rank deficiency of the block residual R j .

Remark 3.6.5. Note here that partial breakdowns do not stop the execution of the algorithms, and all the previous properties still hold. Only the full breakdown forces the algorithm to stop since it does not exist 1 ≤ k j ≤ p j-1 = 0 in line 7 of Algorithm 3.3.1.

We have shown that DMBR with Algorithm 3.4.1 will always converge to the exact solution if the restart size is large enough. However, in practice we are not looking for the exact solution (nor can we afford a restart size as large as n or in most cases) but for a full partial convergence (cf. Definition 2.8.1). Unless we are considering ε being close to the machine precision, the partial breakdown phenomena is rather uncommon since the algorithm tends to find an approximation satisfying Definition 2.8.1 before the block true residual shows an exact rank deficiency.

Alternative F j for large p

The objective of this section is to propose a computationally cheaper alternative criterion for choosing F j instead of Algorithm 3.4.1, which preserves some of the previously discussed properties of DMBR. The criterion we propose in this section is purely academic and we were unable to find a real life application in which the use of such a criterion is advised, reason why we briefly discuss this criterion exclusively in this section.

When the assumption n p does not hold, using Algorithm 3.4.1 may be prohibitive due to the singular value decomposition involving ( Λj -Ĥj Y j ), which has dimension (s j + p) × p. It is possible to choose an inferior but cheaper deflation criterion, only assuming that it is possible to perform the QR decomposition in line 4 of Algorithm 3.3.1 (which may be as well prohibitive for large values of p). For the sake of simplicity, in this section we always consider that p j = p.

Suppose that F i = I p , ∀i < j -1 (that is, no deflation has been performed so far) in DMBR. Consider that we dispose of a permutation matrix T j-1 ∈ C p×p such that the scalars

τ i j-1 = R j-1 D j-1 T j-1 e i 2 (3.7.1) = ( Λj-1 -Ĥj-1 Y j-1 )D j-1 T j-1 e i 2 , i = 1, ..., p (3.7.2)
are given in nonincreasing order, that is,

τ 1 j-1 ≥ τ 2 j-1 ≥ ... ≥ τ p j-1 . Defining k j ≤ p such that τ kj j-1 < 1/p ε and d j = p -k j , we split T j-1 = T + j-1 T - j-1 , T + j-1 ∈ C p×kj , T + j-1 ∈ C p×dj and we conclude that R j-1 D j-1 T - j-1 F = p i=kj τ i j-1 2 ≤ d j /p ε,
characterizing a partial convergence (cf. Definition 2.8.1) and thus we can deflate the information associated with R j-1 D j-1 T - j-1 . Since we supposed p i = p, F i = I p , ∀i < j, we can ensure that Vj e i is the direction associated with R j-1 e i and thus the directions we want to keep are Vj T + j-1 and the ones we want to postpone are Vj T - j-1 giving us the relation

V j P j-1 = Vj T j-1
showing thus that

F j = T j-1 and F j = I sj-1 0 0 F j
is the suitable deflation matrix for iteration j according to this criterion. For subsequent iterations, we have to retrieve the original ordering instead. Splitting

F j = F + j F - j , F + j ∈ C p×kj , F + j ∈
C p×dj , we know that P j-1 was reordered in iteration j and corresponds to Vj F - j , whereas Vj+1 corresponds to the update of Vj F + j . Supposing that the QR decomposition in line 6 of Algorithm 3.2.1 does not reorder any information in Vj+1 (which is possible if using a Gram-Schmidtbased QR algorithm without pivoting, for instance) we find out that the matrix with the correct ordering is

P j-1 Vj+1 Tj F H j with Tj = 0 I dj I kj 0
where Tj is a matrix responsible for swapping the position of P j-1 and Vj+1 without modifying the ordering of the columns of each matrix.

Writing the residual R j as

R j = Vj+1 I sj 0 0 Tj F H j I sj 0 0 F j T H j ( Λj -Ĥj Y j )
we want to find the matrix T j such that the scalars

τ i j = R j D j T j e i 2 , (3.7.3)
= ( Λj -Ĥj Y j )D j T j e i 2 (3.7.4)
for i = 1, ..., p are given in nonincreasing order. Having such a matrix, we set F j+1 = Tj F H j T j and F j+1 = I sj 0 0 F j+1 as our deflation matrix. A pseudocode for performing such a choice is depicted in Algorithm 3.7.1. Algorithm 3.7.1: Choosing Fj -Largest Singular Values of Rj-1Dj-1

1 Receive R j-1 D j-1 = V j P j-1 ( Λj-1 -Ĥj-1 Y j-1
)D j-1 and the parameters 1 ≤ k max ≤ p j-1 and ε d ; 2 Obtain the permutation matrix T j-1 such that the sequence

{||( Λj-1 -Ĥj-1 Y j-1 )D j-1 T j-1 e i || 2 } p i=1 is given in nonincreasing order; 3 Choose kj such that ||( Λj-1 -Ĥj-1 Y j-1 )D j-1 T j-1 e l || 2 ≥ ε d for all 1 ≤ l ≤ kj ; 4 Set k j = min( kj , k max ); 5 Set Tj = 0 I dj I kj 0 with k 0 = p and d 0 = 0; 6 F j = Tj F H j-1 T j ; 7 Define F j = I sj-1 0 0 F j ∈ C (sj +dj )×(sj +dj ) ;
We remark that each τ i j can be found by simply ordering the values of T j , which can be done in p log p operations. Also each F j can be stored as a vector of dimension p, because T j , F j-1 and Tj are all permutation matrices. Also, during iteration j we need only F j-1 , T j and the τ i j which means that an extra storage of 3p is needed when compared to BGMRES. Notice that in this case the application of F j+1 does not require any extra operation: it is simply a reordering of columns.

Although this strategy requires nearly no extra cost for computing F j even for values of p close to n, it tends to perform very close to BGMRES in practice, and whenever it is possible, Algorithm 3.4.1 is preferred instead. Therefore we do not focus our attention on Algorithm 3.7.1 but on Algorithm 3.4.1 instead.

Computational Cost and Memory Requirements

We discuss now the computation cost for applying one cycle of DMBR, including the costs associated with the deflated block Arnoldi iteration (cf. Algorithm 3.2.1) and computing the unitary deflation operator (we always suppose here that Algorithm 3.4.1 is used).

We summarize in Table 3.1 the costs occurring during a given cycle of DMBR(m) 2 , excluding matrixvector products and preconditioning operations which are problem dependent. We have included the costs proportional to both the size of the original problem n and the maximal number of right-hand sides p, assuming a QR factorization based on modified Gram-Schmidt and a Golub-Reinsch SVD 3 ; see, e.g, [60, Section 5.4.5] and [70, Appendix C] for further details on operation counts. The total cost of a given cycle is then found to grow as C 1 np 2 + C 2 p 3 + C 3 np and we note that this cost is always nonincreasing along convergence due to block size reduction. Compared to methods including deflation at restart only, additional operations are related to the computations of F 1 , Λ 1 , F j+1 , V j+1 P j , Λ j+1 and H j , operations that behave respectively as p 3 and np 2 . The computation of V j+1 P j is in practice the most expensive one in a given iteration of DMBR(m). Concerning the truncated variant, the computational cost of a cycle will be reduced only when kj > k max since the upper bound on k j+1 will be then active. This situation occurs at the beginning of the convergence due to the nonincreasing behaviour of the singular values of the block residual.

Step

Computational cost

QR factorization of R 0 2np 2 + np Computation of F 1 4p 0 p 2 + 8p 3 + 2p 3 0 Computation of V 1 P 0 2np 2 0 Computation of Λ 1 2p 2 0 p Block Arnoldi procedure 1 C j Computation of Y j 2(s j + p j )s 2 j + ps 2 j Computation of Λj -Ĥj Y j (s j + p j )p + 2(s j + p j )s j p Computation of F j+1 4(s j + p j )p 2 + 8p 3 + 2p 3 j Computation of V j+1 P j 2np 2 j Computation of Λ j+1 2p 2 j p Computation of H j 2p 2 j p Computation of X m np + 2ns m p + s m p Table 3.1: Computational cost of a cycle of DMBR(m) (Algorithm 3.3.1)
. This excludes the cost of matrix-vector operations and preconditioning operations.

2 that is, DMBR with restart parameter equal to m 3 The Golub-Reinsch SVD decomposition R = U ΣV H with R ∈ C m×n requires 4mn 2 + 8n 3 operations when only Σ and V have to be computed.

(4nk i k j + nk j + 4nd j k j ) + 2nk 2 j ).
We do not include in Table 3.1 the cost associated with computing and updating the scaling matrix D j or applying it to the residual R j , as it is dependent of the scaling strategy being used.

Concerning storage proportional to the problem size n, DMBR(m) requires R m , X 0 , X m , V m+1 and Z m respectively leading to a memory requirement of 2ns m + np m + 3np at the end of a given cycle. Since s m varies from cycle to cycle an upper bound of the memory requirement can be given as n(2m+1)p 0 +3np when p 0 linear systems have to be considered at the beginning of a given cycle. We note that the storage is monotonically decreasing along convergence, a feature than can be for instance exploited if dynamic memory allocation is used.

Numerical Experiments

In this section we investigate the numerical behaviour of block flexible Krylov subspace methods including deflation at each iteration on different problems. We start with three academic illustrations, where two of them (namely in Subsection 3.9.1 and Subsection 3.9.2) are meant to reproduce results found previously in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF], and then we extend the numerical experiments to the forward problem associated with wave propagation phenomena where the multiple right-hand side situation frequently occurs. It consists of a challenging realistic application in geophysics, which we describe in more detail in Chapter 4.

Nevertheless, in wave propagation applications normally variable preconditioners are used. Multigrid techniques using Krylov iterative methods both as smoother and coarse grid correction have been used in the current literature and have reported very good performance in massively parallel environment [START_REF] Calandra | Two-level preconditioned Krylov subspace methods for the solution of three-dimensional heterogeneous Helmholtz problems in seismics[END_REF][START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF]. We discuss in depth the preconditioning issue for this problem in Section 4.4, but since the wave propagation our main application in this thesis, we investigate here the behaviour of the block Krylov methods specifically when using variable preconditioners.

Except for the illustration in Subsection 3.9.2, all the right-hand sides correspond to canonical vectors. Thus the block right-hand side B ∈ C n×p is extremely sparse (only one nonzero element per column) and the initial block residual corresponds to a full rank matrix. This also has connections with our geophysical application described in Chapter 4, where the right-hand sides are often sparse.

We compare both BFGMRES-R(m) and DMBR(m) with other preconditioned iterative methods based on flexible BGMRES(m) for the solution of these problems with a zero initial guess (X 0 ) and a small value of the restart parameter m when using a variable preconditioner. The choice of small values of m here is suitable since we are considering mostly inner-outer methods [START_REF] Simoncini | Flexible inner-outer Krylov subspace methods[END_REF]. It is known that for this kind of method, the dimension of the Krylov subspace being built is in fact a combination of the outer and inner subspace.

We use the same stopping criterion for all experiments also. We attempt to find an approximate solution satisfying ||R j D j || ψ ≤ ε, where the scaling matrix is always set as

D -1 j = diag(||Be 1 || 2 , ||Be 2 || 2 , ..., ||Be p || 2 )
for every j, for all the experiments (thus D j is a fixed scaling matrix).

Our goal is to analyse the performance of block Krylov subspace methods including deflation at each iteration versus classical methods discussed in Section 3.5. A primary concern will be to evaluate if DMBR(m) can be efficient when solving problems with multiple right-hand sides both in terms of preconditioner applications and total computational cost.

Poisson Problem

In this experiment we use a Matlab [START_REF][END_REF] implementation of the referred methods and we attempt to reproduce the results for BFGMRESD in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] (see Table 2.6 in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF]). It consists of the two-dimensional Poisson problem with Dirichlet boundary conditions, discretized with a second-order finite differences scheme for a vertexcentred grid, with a mesh-grid equal to 1/128. The coefficient matrix was taken using Matlab's routine gallery('poisson ',128).

In all numerical experiments, the convergence and deflation threshold are set as ε = ε d = 10 -6 . We are interested in analysing the behaviour of the deflation as the number of right-hand sides increases. We start with 5 right-hand sides and proceed by doubling the number of right-hand sides until 160 right-hand sides. All right-hand sides are canonical vectors in this test.

We show in the Tables 3.2 to 3.4 the number of iterations (It), the number of matrix-vector products on a single vector (M V P ) and number of preconditioner applications on a single vector (P r) required for various restarted block flexible Krylov subspace methods performing no deflation (BFGMRES(m)), deflation at the beginning of cycle only (BFGMRESD(m)) and deflation at each iteration (BFGMRES-R(m) and DMBR(m)). We note that all selected methods solve the minimization problem over a subspace of similar maximal dimension (mp). Since the algorithm we propose aims at saving matrix-vector product and preconditioner application, for this numerical experiment we only focus on this factor (cf. Remark 3.9.1 for some comments on the orthogonalization cost). Remark 3.9.1. During iteration j, DMBR orthogonalizes k j vectors against s j + d j vectors, whereas BFGMRESD orthogonalizes k 1 vectors against j × k 1 . Since k j ≤ k 1 and s j ≤ j × k 1 (both due to Corollary 3.4.7), DMBR may actually perform less orthogonalizations per iterations than BFGMRESD.

Therefore, we highlight that although we do not detail the orthogonalization cost for this illustration, the number of orthogonalization steps performed by DMBR is equal or smaller than those performed by BGMRES and BGMRES-R, and in some cases also smaller than those performed by BFGMRESD depending on how early in the cycle the deflation takes place. In Subsection 3.9.3 and Subsection 3.9.4 we show numerical experiments addressing the total computational time, including matrix-vector products, preconditioner applications and orthogonalization.

For the first experiment, in Table 3.2 we use 5 cycles of BGMRES(5) as variable preconditioner, meaning that each preconditioning application involves 25 matrix vector products. We provide in the ρ column the following ratio: ρ(method) = M V P (method(m)) + 25 × P r(method(m)) M V P (DM BR(m)) + 25 × P r(DM BR(m)) .

(3.9.1)

which scales the number of matrix-vector product operations performed with respect to the DMBR method. A value of ρ greater than one indicates that the given block subspace method performs more matrix-vector products than DMBR. We set the restart parameter m = 5 and k max = p. The second experiment, in Table 3.3, we increase the number of iterations per cycle, but reduce the quality of the preconditioner in order to observe the behaviour along several cycles. We use 3 cycles of BGMRES(3) as variable preconditioner (thus, each preconditioning application involves 9 matrix vector products), restart parameter m = 5 and k max = p, and we update the ratio to ρ(method) = M V P (method(m)) + 9 × P r(method(m)) M V P (DM BR(m)) + 9 × P r(DM BR(m)) .

(3.9.2) instead of ρ(method). The third experiment, shown in Table 3.4, considers a more limited memory setting. We set use again 5 cycles of BGMRES(5) as variable preconditioner, but we set k max = 20 while maintaining the other parameters unchanged. We start with p = 40 and proceed by adding 20 right-hand sides until it reaches the 160 limit. Since BGMRES-R cannot limit the memory in such fashion (p 1 is always equal to p in BGMRES-R), we let it aside in this test, and we compare only DMBR with BFMGREST, using ρ(method) from (3.9.1). Table 3.2 reveals that for this particular problem, DMBR performance is clearly superior to BGMRES-R and BFGMRESD, and that the gap between DMBR and the other methods increases with the number of right-hand sides. Notice that, since the restart size is small, BFGMRESD converges performing less matrixvector products and preconditioning applications than BFGMRES-R. This behaviour is expected since restart size 5 and a number of right-hand sides given at once ranging from p = 5 to p = 160. It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner. restart size 15 and a number of right-hand sides given at once ranging from p = 5 to p = 160. It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner. with a number of right-hand sides given at once ranging from p = 40 to p = 160 and using truncation (kmax = 20).

It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner.

BFGMRESD is supposed to benefit from small restart sizes. In Table 3.3, as expected, we observe that BFGMRES-R benefits from the larger restart size and performs considerably less matrix-vector products and preconditioner applications than BFGMRESD. However, DMBR method is still the cheapest in terms of matrix-vector and preconditioner applications. In the more memory constrained test, in Table 3.4, we see that once again the reduction on the number of matrix-vector products of DMBR over BFGMREST is considerable, but this time the difference between the methods does not seem to increase with the number of right-hand sides p.

Convection-Diffusion Problem

We continue Matlab experiments taking into account the matrix-vector products and preconditioner applications cost only. In this subsection we test the behaviour of the deflated methods for a non-Hermitian problem, the convection-diffusion equation given by

-∆u + cu x + du y = g in Ω, u = 1 on ∂Ω
where Ω =]0, 1[ 2 is the interior of the domain and ∂Ω = (0, 1) 2 is the boundary. We take c = d = 256 for our experiments, and discretize the convection-diffusion equation using finite differences in a 5-point Cartesian stencil, once again with mesh size equal to 1/128. We generate a random exact solution satisfying the boundary condition to then obtain the right-hand sides. This experiment is meant to reproduce the results in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF], Table 2.8.

Our setting is very similar to the one in Subsection 3.9.1: first we use 5 cycles of BGMRES(5) as variable preconditioner, ε = ε d = 10 -6 with restart parameter m = 5 and k max = p in Table 3.5. Then in Table 3.6 we use 3 cycles of BGMRES(3) but increase the restart size to m = 15. Finally, in Table 3.7 we consider the limited memory setting with k max = 20 while using 5 cycles of BGMRES [START_REF] Arnoldi | the principle of minimized iterations in the solution of the matrix eigenvalue problem[END_REF] as variable preconditioner and m = 5. We also reproduce in the column ρ the same ratio previously mentioned in (3.9.1) and in the column ρ the ratio in (3.9.2), and although we do not address orthogonalization costs, Remark 3.9.1 is still valid.

In Table 3.7 we see that DMBR method is always able to save matrix-vector product computations by performing deflation every iteration. Observe however that in contrast with Table 3.5 where the number of iterations of each method remains almost unchanged as p increases, in Table 3.7, DMBR performs considerably less iterations than BFMGREST. This speedup has indeed a reason: due to Corollary 3.3.2, we guarantee that DMBR minimizes the entire residual every iteration (regardless of the value of k j ), whereas BFGMREST chooses just a subset of the residual do minimize every cycle (as discussed in Section 3.5). We consider that this is indeed a critical feature of DMBR mainly if p is considerably large (of the order of hundreds).

Also, comparing Table 3.5 with Table 3.6 we see that once again BFGMRES-R profits from long restart sizes whereas BFGMRESD profits from small restart size, but DMBR performs well in both cases. Also, for p = 160 in Table 3.6 only one cycle is performed (all methods converge in 14 iteration) and that BFGMRESD is equivalent to BGMRES and DMBR is equivalent to BFGMRES-R. This behaviour is expected as we explained in Section 3.5.

Complex-valued advection diffusion reaction problem

For this more realistic Matlab experiment, we consider a complex-valued partial differential equation of advection diffusion reaction type in two-dimensions defined on a square domain Ω = [0, 1] 2 . Recently Haber and MacLachlan [START_REF] Haber | A fast method for the Helmholtz equation[END_REF] have proposed a continuous transformation of the acoustic wave equation based on the Rytov decomposition that requires the solution of a partial differential equation that is more amenable to efficient numerical methods than the original indefinite Helmholtz equation. It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner. 5) as variable preconditioner, restart size 15 and a number of right-hand sides given at once ranging from p = 5 to p = 160. It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner. variable preconditioner, with a number of right-hand sides given at once ranging from p = 40 to p = 160 and using truncation (kmax = 20). It denotes the number of iterations, M V P the number of matrix-vector applications on a single vector, P r the number of preconditioner applications on a single vector and ρ a scaled measure of efficiency in terms of number of matrix-vector products performed both by the method and its preconditioner.

Convection-diffusion equation with

Convection-diffusion equation with

with Dirichlet boundary conditions reads as follows:

- ∂ 2 u ∂x 2 - ∂ 2 u ∂y 2 -2iωc(α x ∂u ∂x + α y ∂u ∂y ) + ω 2 (c 2 -κ 2 )u = g s (x) (3.9.3) x = (x, y) ∈ Ω, (3.9.4)
u = 0 on ∂Ω, (3.9.5)

where ω, c, α x , α y , κ are real-valued coefficients. The source term g s (x) = δ(x -x s )e -ic(αxxs+αyys) represents a harmonic point source located at (x s , y s ) in Ω. We consider the pure advection diffusion case (c = κ = 1) and set the following values for the parameters ω = π, α

x = 1/ √ 2 , α y = 1/ √ 2 .
The discrete problem is obtained after second-order finite difference discretization of (3.9.3) with a secondorder upwind scheme for the treatment of the advection terms as in [START_REF] Haber | A fast method for the Helmholtz equation[END_REF]. The right-hand side we choose for this problem correspond to Dirac sources. The inner preconditioner is based on one cycle of non preconditioned GMRES(m) corresponding to pm additional matrix-vector products when considering a linear system with p right-hand sides. Thus flexible outer Krylov subspace methods are required since the preconditioner is then variable. With this simple preconditioner we note that we can derive a purely matrix-free implementation. Both the tolerance and the deflation threshold are set as ε = ε d = 10 -5 in the numerical experiments. Table 3.8 collects the number of outer iterations (It) and number of preconditioner applications on a single vector (P r) required for various restarted block flexible Krylov subspace methods performing no deflation (BFGMRES(m)), deflation at the beginning of cycle only (BFGMRESD(m)) and deflation at each iteration (BFGMRES-R(m) and DMBR(m)) for two different values of the restart parameter (respectively m = 5 and m = 10). We have also included results related to restarted flexible GMRES when solving in sequence the p linear systems independently. We note that all selected methods solve the minimization problem over a subspace of similar maximal dimension (mp). In opposition to the two previous numerical experiments, in this one we consider every floating point operation rather than simply the matrix-vector product performed by both methods. We determine the computational complexity of all algorithms including costs related to QR factorization, singular value decomposition, orthonormalization (as listed in Table 3.1), matrix-vector products and preconditioning and define a measure of efficiency τ as τ (method) = f lops(F GM RES(mp)) f lops(method) .

Thus a value of τ greater than one indicates that the given block subspace method leads to a computational improvement with respect to flexible GMRES applied on the given sequence of linear systems. Table 3.8 reveals that block Krylov subspace methods including deflation either at restart only or at each iteration usually are to be preferred. Indeed those methods always lead to efficiencies τ greater than one. On this application standard block Krylov subspace method (BFGMRES) is not efficient with respect to FGMRES(mp). This highlights the fact that to be effective block subspace methods must incorporate block size reduction or deflation [START_REF] Gutknecht | Block Krylov space methods for linear systems with multiple right-hand sides: An introduction[END_REF]. On the two sets of numerical experiments, whatever the value of the restart parameter m, DMBR(m) always leads to the minimal number of preconditioner applications.

Similarly DMBR(m) also delivers the best efficiency (see bold values in Table 3.8). Finally we note that BFGMRES-R(m) is penalized when the number of outer cycles is large since this method does not include initial deflation at the beginning of the cycle. On this academic problem we have shown the interest of using block Krylov subspace methods that include deflation at each iteration. DMBR(m) is indeed found to be competitive with respect to Krylov subspace methods including deflation at restart only.

Acoustic Full Waveform Inversion

We focus on a specific application in geophysics related to the simulation of wave propagation phenomena in the Earth [START_REF] Virieux | An overview of full waveform inversion in exploration geophysics[END_REF]. Given a three-dimensional physical domain Ω p , the propagation of a wave field in a 2 with a number of right-hand sides given at once ranging from p = 4 to p = 32 for two different values of the restart parameter m = 5 (upper part) and m = 10 (lower part). It denotes the number of iterations, P r the number of preconditioner applications on a single vector and τ a scaled measure of efficiency in terms of computational operations.

heterogeneous medium can be modelled by the Helmholtz equation written in the frequency domain:

- ∂ 2 u ∂x 2 - ∂ 2 u ∂y 2 - ∂ 2 u ∂z 2 - (2πf ) 2 c 2 (x, y, z) u = g s (x), x = (x, y, z) ∈ Ω p . (3.9.6)
u represents the pressure field in the frequency domain, c the variable acoustic-wave velocity in ms -1 , and f the frequency in Hertz. The source term g s (x) = δ(x -x s ) represents a harmonic point source located at (x s , y s , z s ). A popular approach -the Perfectly Matched Layer formulation (PML) [START_REF] Berenger | A perfectly matched layer for the absorption of electromagnetic waves[END_REF][START_REF] Berenger | Three-dimensional perfectly matched layer for absorption of electromagnetic waves[END_REF] has been used in order to obtain a satisfactory near boundary solution, without many artificial reflections.

As in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF] we consider a second-order finite difference discretization of the Helmholtz equation (3.9.6) on an uniform equidistant Cartesian grid of size n x × n y × n z . The same stability condition (12 points per wavelength) relating f the frequency with h the mesh grid size and c(x, y, z) the heterogeneous velocity field has been considered:

f = min (x,y,z)∈Ω h c(x, y, z) 12 h .
In consequence A is a sparse complex matrix which is non-Hermitian and nonsymmetric due to the PML formulation that leads to complex-valued variable coefficients in the partial differential equation [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF]Appendix A]. Due also to their lack of definiteness and symmetry, the resulting linear systems are known to be challenging for iterative methods [START_REF] Erlangga | Advances in iterative methods and preconditioners for the Helmholtz equation[END_REF][START_REF] Ernst | Why it is difficult to solve Helmholtz problems with classical iterative methods[END_REF]. We refer also to Chapter 4 for more details about this problem.

We consider the same perturbed geometric two-level preconditioner presented in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF] that has been shown to be relatively efficient for the solution of three-dimensional heterogeneous Helmholtz problems in geophysics. We refer the reader to [23, Algorithm 5] for a complete description of the geometric preconditioner and to [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] for additional theoretical properties in relation with Krylov subspace methods. Also, later in Subsection 4.4.1 we detail this preconditioner. Since we are in the multiple right-hand sides scenario, we suppose that the perturbed two-grid preconditioner is applied independently once for each right-hand side. Next investigate the performance of the block flexible Krylov methods presented in Section 3.3 on this challenging real-life application. Both the tolerance and the deflation threshold are set as ε = ε d = 10 -5 in the numerical experiments. The source terms correspond to Dirac sources. The numerical results have been obtained on Babel, a Blue Gene/P computer located at IDRIS (PowerPC 450 850 Mhz with 512 MB of memory on each core) using a Fortran 90 implementation with MPI in single precision arithmetic. This code was compiled by the IBM compiler suite with standard compiling options and linked with the vendor BLAS and LAPACK subroutines. with p = 4 to p = 128 right-hand sides given at once. It denotes the number of iterations, P r the number of preconditioner applications on a single vector and T denotes the total computational time in seconds.

As in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF] we consider the velocity field issued from the public domain SEG/EAGE Overthrust model and analyse the performance of the numerical methods at a given frequency f = 3.64 Hz. Both the problem dimension (about 23 million of unknowns) and the maximal number of right-hand sides to be considered [START_REF] Trottenberg | Multigrid[END_REF] correspond to a task that geophysicists typically must face on a daily basis. Thus efficient numerical methods must be then developed for that purpose. In [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF] we have considered block flexible Krylov subspace methods including deflation at restart only on this application for a reduced number of right-hand sides (from 4 to 16). We continue this detailed analysis and investigate the performance of both DMBR(m) and BFGMRES-R(m) with a larger number of right-hand sides. We also consider the variants with truncation in memory (BFGMREST(m, p f )) and with truncation in operations DMBR(m, p f )) with p f set to p/2 in both cases. The number of cores is ranging from 32 for p = 4 to 1024 for p = 128. Since doubling the number of right-hand sides nearly doubles the memory requirement of the block methods, we also multiply the number of cores by a factor of two with respect to the number of right-hand sides. This aims at imposing the same memory constraint on each core for all numerical experiments as in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF]. The maximal memory requested is about 488 Gb for p = 128. Table 3.9 collects in addition to outer iterations (It) and preconditioner applications on a single vector (P r) the computational times in seconds (T ). Among the different strategies DMBR(5) most often delivers the minimal number of preconditioner applications and computational times (see respectively italic and bold values in Table 3.9). This clearly highlights the interest of performing deflation at each iteration both in terms of preconditioner applications and computational operations on this given application. The latter result is especially important since deflating at each iteration induces an additional cost as shown in Table 3.1. DMBR( 5) is thus found to be competitive with respect to methods incorporating deflation at restart only (a gain of up to 15% in terms of computational time is obtained for instance for p = 8) as well as DMBR(5,p/2) (gain of 27% for p = 32 when compared to BFGMREST(5,p/2)). This is a satisfactory improvement since methods including deflation at restart only are already quite efficient in this application as shown in [START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF]. We also note that the improvement over classical block flexible GMRES method is quite large as expected (a gain of up to 61% is obtained for p = 64). Figure 3.1 shows the evolution of k j -the number of Krylov directions effectively considered at iteration j -along convergence for the various block subspace methods in the case of p = 32. Regarding BFGM-RESD(5) and BFGMREST(5,p/2) deflation is performed only at the beginning of each cycle, thus k j is found to be constant in a given cycle. Variations at each iteration can only happen in BFGMRES-R(5) or DMBR [START_REF] Arnoldi | the principle of minimized iterations in the solution of the matrix eigenvalue problem[END_REF]. As expected DMBR(5) enjoys a nonincreasing behaviour for k j along convergence, while peaks occur for BFGMRES-R(5) at the beginning of each cycle. On this example the use of truncation within DMBR(5) tends to delay the start of the decreasing behaviour of k j . After a certain phase deflation is nevertheless active and proves to be useful.

We also remark that the use of truncation techniques in DMBR(m) leads to an efficient method. In certain cases DMBR(5, p/2) is as efficient as DMBR(5) in terms of computational times (see, e.g., the case p = 32 in Table 3.9). This feature is really important in this given application due to the large size of the linear systems. Furthermore DMBR(5, p/2) requires usually less preconditioner applications than BFGMREST(5, p/2). This satisfactory behaviour has indeed a reason: due to Corollary 3.3.2, we guarantee that the truncated variant of DMBR(m) minimizes the entire residual at each iteration (regardless of the value of p j ), whereas BFGMREST(m) chooses just a subset of the residual to be minimized at each cycle. We consider that this is indeed a critical feature of the truncated variant of DMBR(m).

Finally we investigate the convergence properties of a combination of DMBR(m) and BFGMRESD(m).

At the beginning of the convergence history this method (named Combined(m, p s ) in Table 3.9) is fully equivalent to DMBR(m). Then as soon as the number of Krylov directions effectively considered at iteration j (k j ) reaches a given prescribed value (p s ) the method switches to BFGMRESD(m) at the next restart. This mainly aims at reducing the computational cost in the next cycles by performing deflation only at the restart instead of at each iteration. As shown in Table 3.9 this combination leads to further reductions in computational times and is especially appropriate when the number of right-hand sides becomes large on this given application.

Conclusions

We have extended the block restarted flexible GMRES method to a variant that allows the use of deflation (i.e. block size reduction) at each iteration when solving multiple right-hand side problems given at once. The method aims at reducing the cost of each iteration of the block Krylov subspace method by judiciously choosing which information to use and which information to be postponed. We have shown that the Frobenius norm of the block residual is always nonincreasing even along cycles. Furthermore we have justified the choice of the deflation strategy that is based on a nonincreasing behaviour of the singular values of the scaled block residual (which are also nonincreasing along the iterations and cycles).

We have also proposed a variant of the deflated block residual method to be used in a constrained memory environment. Numerical experiments have shown the efficiency of the new method on two different problems issued from wave propagation situations requiring the solution of multiple right-hand side problems. The block flexible method including deflation at each iteration has proven to be efficient in terms of both preconditioner applications and computational operations. It has been found superior to recent block flexible methods including deflation at restart only. This satisfactory behaviour has been observed on an industrial simulation arising in geophysics, where large indefinite linear systems with multiple right-hand sides have been successfully solved in a parallel distributed memory environment. Furthermore reductions in terms of computational times have been obtained by combining methods including deflation at each iteration and deflation at restart only in a second phase. To the best of our knowledge these results consist in one of the first illustrations of the usefulness of block Krylov subspace methods including deflation at each iteration on a realistic three-dimensional application in a parallel distributed memory environment.

To conclude it is worthwhile to note that the theoretical properties of the deflated minimal block residual method hold for any unitary matrix F j+1 . Thus we plan to investigate other possible subspace decompositions in a near future that may lead to further improvements. Finally we note that the analysis proposed in this paper can be extended as well to other block Krylov subspace methods such as block FOM [START_REF] Robbé | Exact and inexact breakdowns in block versions of FOM and GMRES methods[END_REF], block GCRO [START_REF] Yu | A block iterative solver for complex non-hermitian systems applied to large-scale electronic-structure calculations[END_REF], and block simpler GMRES [START_REF] Liu | A simpler block GMRES for nonsymmetric systems with multiple right-hand sides[END_REF].

Chapter 4

Acoustic Full Waveform Inversion

Introduction

In geology and geophysics, the so called "Earth imaging" is a technique used for scanning a delimited subsurface of Earth. The application of Earth imaging techniques is very wide; examples are civil engineering (usually requiring the knowledge of few meters of depth only), landslide analysis of a terrain, or detection and characterization of oil reservoir (often requiring several kilometers of depth) [20, p.15]. Among the strategies for performing such imaging, we highlight here in particular the use of seismic waves. It consists of positioning an acoustic wave propagator (normally called "source") which is able to emit waves at a chosen frequency f through the subsurface. When encountering a layer of reflecting material in the subsurface (as for instance, a salt layer) these waves are refracted and reflected back to the surface. These reflections (sometimes called "echoes") are detected by a special kind of microphone called "geophone" (sometimes called simply "receiver"). See Figure 4.1 for an two-dimensional illustration. A "survey line" is chosen on the surface of the Earth and the experiment is repeated with the source positioned in some of the points along the survey line (about every 25 meters, according to [28, p.1]). With the data gathered by the geophones on the several experiments, and knowing the position of the source and the frequency of the acoustic waves emitted, using mathematical methods, we wish then to find details about the subsurface, as density, velocity of the wave propagation, position and shape of reflective layers among others. We highlight here the search for the so called "velocity model", which consists of a two-dimensional or three-dimensional model of the subsurface estimating the velocity at which the wave propagates in each point of the physical domain. See In this chapter of this thesis, we briefly address a specific mathematical procedure for determining an approximation of the velocity model (it can also be used to approximate other information as density, attenuation or anisotropy of the subsurface), the "Full Waveform Inversion" or FWI for short, which consists basically of an inverse problem. The solution of the linear system arising from this problem is the main motivation for the development of a block iterative solver as discussed in Chapter 2 and Chapter 3. In Section 4.2 we address an overview on the inverse problem without attaining to many details. The importance of Section 4.2 is thus to introduce the forward and the backward problems. Thereafter, we explore details concerning the forward problem, as how to discretize this problem while maintaining certain physical properties (in Section 4.3). In Section 4.4 we address methods for solving the linear problem arising from the discretization of the forward problem, and we address some known issues, as for instance, the difficulties in preconditioning the problem. In Section 4.5 we briefly discuss a software implementation we propose targeting the solution of the full waveform inversion in a massively parallel environment using a object oriented language, and in Section 4.6 we use DMBR, the method proposed in Chapter 3 and implemented into our software, to perform large (up to O(10 9 )) numerical experiments on realistic velocity models at 5Hz and 12Hz, illustrating once more the interest of deflation techniques in a real life application. In Section 4.7 we present the final remarks of this chapter.

The Inverse Problem

In this section we quickly describe the full waveform inversion [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF][START_REF] Sourbier | FWT2D : a massively parallel program for frequency-domain full-waveform tomography of wide-aperture seismic datapart 1: algorithm[END_REF][START_REF] Sourbier | FWT2D : a massively parallel program for frequency-domain full-waveform tomography of wide-aperture seismic datapart 2: numerical examples and scalability analysis[END_REF] which is the main motivation of our work with block iterative solvers presented in Chapter 2 and Chapter 3. Although the full waveform inversion consists of an optimization problem it gives raise to the block linear systems which we aim at solving using the methods we discussed in previous chapters.

Let the real velocity model be denoted by m ( * ) and consider an initial guess m (1) given. The inverse problem consists of an iterative procedure which aims at improving a "synthetic velocity model" m (i) every iteration i, hoping to obtain a model m ( ) as a reliable approximation of m ( * ) in a finite number of steps . It consists of two main steps. In the first one, the so called 1 "forward problem" takes place. The forward problem consists of building a "propagator", a procedure which simulates the propagation of the waves through m (i) when an acoustic wave is ignited at a known position with a known frequency, obtaining the so called wavefield. We recall that during the geophysical experiment, the acoustic waves are ignited on several points of the physical domain along the "survey line", possibly for several different frequencies, and that the geophones gather the data for each test separately, meaning that potentially the forward problem has to solve multiple times per iteration of the inverse problem. Therefore, we represent the set of "source positions" {s j } p j=1 and a set of frequencies {f j } n f j=1 , meaning a total of n f × p solutions have to computed for every iteration i of the inverse problem. We analyse in further sections the multiple sources issue and how to exploit the relations between each case to increase the performance of the forward problem, but we do not address the multiple frequencies in practice, as we explain better in Section 4.3.

Once each wavefield u (i) j for j = 1, ..., n f × p of the forward problem of the i-th iteration is known, it has to be projected onto the position of the geophones by the projector P data : this is done to know the data that the geophones would have gathered if m (i) were the real velocity model. The projected synthetic solution, or "computed data" is then denoted by d (i)

j = P data u (i) j .
The inverse problem compares the computed data with the "observed data", gathered by the geophones on the real experiment, which we denote here by d obs j , for j = 1, ..., n f × p. We consider that the convergence criterion of the inverse problem relies on the cost function [20, p.93]) where S is a diagonal weighting matrix2 and ∆d

C(m (i) ) = 1 2 (n f ×p) j=1 ∆d (i) j † S † S∆d (i) j (4.2.1) (cf.
(i) j = d (i) j -d obs j
is the misfit vector, or residual vector.

The goal of the inverse problem is thus to minimize (4.2.1), characterizing the inverse problem as a weighted least squares problem. However, not only the size of the problem is considerable (the dimension of each m (i) is between O(10 4 ) and O(10 6 ) in a realistic application, cf. [20, p.93]) but it is also very ill-conditioned, consequently making prohibitive the use of global methods. Due to this restriction, local methods are often preferred, considering that an initial guess is known. Algorithm 4.2.1 shows a simple pseudo-code for the FWI using a generic steepest descent algorithm for solving this problem. Remark 4.2.1. Steps 8 to 10 of Algorithm 4.2.1 refer to the choice of the model for the next iteration, and are topic of active research [START_REF] Virieux | An overview of full waveform inversion in exploration geophysics[END_REF]. We briefly mention that some strategies for performing FWI consist of choosing λ models rather than only one, that is, steps 8 to 10 generate m (i+1) k for 1 ≤ k ≤ λ, and during iteration i + 1 the algorithm is ran once for each model, being thus able to effectively search in multiple directions every iteration. This however, comes at a great extra cost, since λ × p × n f forward problems have to be solved for such a case3 .

It is yet to be mentioned how to perform line 7 of Algorithm 4.2.1. As we discuss in Section 4.3, the forward problem is solved by dully discretizing the wave-equation and then solving the linear system

A (i) u (i) j = s (i) j (4.2.2)
where s (i) j represents the position of the source. Considering A (i) given, we obtain the gradient of C(m (i) ) as

∂C(m (i) ) ∂m (i) = G(m (i) ) = n f ×p j=1 R   u (i) j T ∂A (i) ∂m (i) T A (i) -T Pdata S † S∆d (i) j   ,
where Pdata is the operator that projects ∆d (i) j onto the forward problem space and R( * ) is the real part of the referred vector (cf. [20, (2.11)]). We highlight here the term

A (i) -T Pdata S † S∆d (i) j
which is referred to as the back-propagation of the misfit vector. Computing such a vector is equivalent to find the solution of the linear system

A (i) T g (i) = Pdata S † S∆d (i) j (4.2.3)
the so called backward problem. Notice the relation between (4.2.2) and (4.2.3), and that the solution for the later depends on the solution of the former (the solution of the forward problem is needed to compute the right-hand side of the backward problem). When the operator A (i) is symmetric, the only difference between the backward and forward problem lies in the right-hand side.

If (A (i) ) -1 would be known, the solution of the backward problem would take a matrix-vector multiplication operation, otherwise we would have to effectively solve the backward problem once for each frequency and each source position. When using iterative solvers, (A (i) ) -1 is not known and therefore it is extremely important to address the solution of the backward problem. This is subject of future studies.

Discretizing the Forward Problem

In this section we focus on the solution of the forward problem introduced in Section 4.2, and we discuss a proper formulation of this problem as well as some techniques for discretizing and some few computational issues arising when we are formulating the problem to find a numerical solution. Although this is not explicitly mentioned in depth in this section, we always have in mind the formulation for massively parallel environment, as we discuss in more detail in Section 4.4 and Section 4.5.

The Helmholtz Equation

We are interested in determining how the acoustic waves would propagate through the velocity model m (i) inside the bounded parallelepiped domain Ω ⊂ R 3 . Two main approaches are traditionally chosen for finding the solution for such a problem: to consider the problem in the time-domain or in the frequencydomain. Both approaches have their advantages and drawbacks which we do not discuss here in details. We refer to [START_REF] Crase | Nonlinear elastic inversion of land seismic reflection data[END_REF][START_REF] Crase | Robust elastic non-linear waveform inversion: application to real data[END_REF][START_REF] Mora | Inversion = migration + tomography[END_REF][START_REF] Gauthier | Two-dimensional nonlinear inversion of seismic waveforms: numerical results[END_REF][START_REF] Tarantola | Inversion Problem Theory: Methods for data fitting and model parameter estimation[END_REF][START_REF] Tarantola | Inversion of seismic reflection datain the acoustic approximation[END_REF] for details on the time-domain approach and to [START_REF] Sirgue | Efficient waveform inversion and imaging: a strategy for selecting temporal frequencies[END_REF][START_REF] Pratt | Inverse theory applied to multi-source cross-hole tomography. Part II: elastic waveequation method[END_REF][START_REF] Pratt | Inverse theory applied to multi-source cross-hole tomography. Part I: acoustic wave-equation method[END_REF] for details on the frequency-domain approach. Since it is recognized in the literature that the frequencydomain approach is more advantageous when solving the inverse problem, in this thesis we address this approach only.

We use the discussion in [49, §1.2] to deduce the equation suitable for representing the wave propagation through the subsurface in the frequency-domain. We temporarily drop the j index from u j (x) since the discussion here is valid for every j. Consider the time-dependent wave-equation for fluids and solids in the absence of viscosity

∆p = 1 v 2 ∂ 2 p ∂t 2 (4.3.1)
where p(x, t) is the pressure on the point x ∈ R 3 at the instant t ∈ R and v(x) is the propagation speed of compressional waves in the point x. However, we consider that the waves are time-harmonic and that they can be represented as p(x, t) = u(x)exp(-ω w t), (4.3.2) (cf. [49, (1.22)]) where ω w = 2πf is the angular frequency, f ∈ R is the frequency in Hertz, and  = √ -1 . Substituting (4.3.2) into (4.3.1) we then obtain

-∆u(x) -k 2 (x)u(x) = 0 (4.3.3)
where k(x) = 2πf /v(x) is called the wavenumber function and ∆ denotes the Laplacian operator. Introducing the source term g(x) and also assuming that such source term is time-harmonic, we obtain the heterogeneous Helmholtz equation as follows

-∆u(x) -k 2 (x)u(x) = s(x) (4.3.4)
which is in turn time-independent.

Recently it was proposed in [START_REF] Haber | A fast method for the Helmholtz equation[END_REF] a reformulation of the Helmholtz equation using the Rytov decomposition. This reformulation requires the solution of the complex advection-diffusion-reaction equation, which in turn can be solved using efficient multigrid preconditioners. The authors report their approach to be efficient in the two dimensional case. Although these results encourage further research on three dimensional cases, we do not address this situation in this thesis.

Perfectly Matched Layers

Because the domain Ω is supposed to be bounded whereas the geophysical domain comprises the whole Earth, in order to simulate properly the wave propagation phenomena one must add an absorbing boundary condition to the equation, as for instance the perfectly matched layers (PML) [START_REF] Berenger | A perfectly matched layer for the absorption of electromagnetic waves[END_REF][START_REF] Berenger | Three-dimensional perfectly matched layer for absorption of electromagnetic waves[END_REF]. It consists of adding an artificial layer around Ω and modifying the wave-equation only inside these additional layers. For doing that, we consider solving the problem in a larger parallelepiped domain Ω e such that Ω ⊂ Ω e . Denoting the boundary of Ω e by Γ, and Ω pml = Ω e \Ω we then redefine the wave-equation in Ω pml using damping functions such that the value of the pressure of the waves on Γ are always equal to zero (cf. Letting the vector x = (x 1 , x 2 , x 3 ), we use the same damping functions as [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF]:

ξ l (x l ) =                  1 - cos πx l 2l pml if 0 ≤ x l ≤ l pml , 1 if l pml < x l < l l -l pml , 1 - cos π(l l -x l ) 2l pml if l l -l pml ≤ x l ≤ l l . (4.3.5)
for l ∈〚1, 3〛, where l l ∈ R is the length of Ω in the x l direction and l pml denotes the length of the PML layer (which we consider to be uniform in every direction; cf. Figure 4.4). Using Einstein's notation, we then split (4.3.4) as [96, p.103]). The forward problem consists in finding a solution for the problem (4.3.6) in Ω e . Although the homogeneous form of the Helmholtz equation (4.3.3) possesses analytical solutions, this is not the case for the heterogeneous equation (4.3.4), and thus a numerical approximation must be computed.

                   - ∂ 2 ∂(x l ) 2 -k(x) 2 u(x) = s(x) in Ω, - 1 ξ l (x l ) ∂ ∂x l 1 ξ l (x l ) ∂ ∂x l -k(x) 2 u(x) = s(x) in Ω pml \Γ, u(x) = 0 on Γ (4.3.6) (cf.

Discrete Formulation

In the current literature there are several approaches for discretizing (4.3.6) and computing a numerical approximation. Techniques as finite elements and discontinuous Galerkin have been widely applied in the literature. However, in this thesis we opt for a simplified approach, limiting our study to the uniform finite difference techniques, using either 7 or 27 points in the Cartesian grid (cf. Figure 4.5) which are cheap to generate and easily parallelizable.

7-point Cartesian stencil 27-point Cartesian stencil

Consider a uniform Cartesian grid Ω h as the discretization of Ω e for a given distance h ∈ R + between each point such that l l /h = n l ∈ N is the number of points in the direction x l . Then we define the discrete function

u (i,j,k) = u h (ih, jh, kh), where u h = u Ω h with i ∈ 〚1, n 1 〛, j ∈ 〚1, n 2 〛
and k ∈ 〚1, n 3 〛 and analogously for s (i,j,k) , v (i,j,k) and ξ (l,i) . Traditionally, h is chosen as to satisfy the minimal number of points per wavelength n λ ∈ R + , a value which is scheme-dependent. Knowing that a wavelength is defined as the ratio between the velocity of the propagation of the wave and the frequency f , we find that the distance between each points must satisfy

h ≤ v (i,j,k) n λ f , ∀(i, j, k) ∈ Ω h (4.3.7)
where it is normally chosen

h = min (i,j,k)∈Ω h v (i,j,k) n λ f . (4.3.8)
Inequality (4.3.7) is often referred in the literature as stability condition [START_REF] Cohen | Higher-order Numerical Methods for Transient Wave Equations[END_REF] for a second order discretization scheme.

Using this notation, and the second order Taylor expansion, the discrete formulation of (4.3.6) for the

s (i,j,k) =    -ω 2 v 2 (i,j,k) + 1 h 2   1 ξ + (1,i) + 1 ξ - (1,i) + 1 ξ + (2,j) + 1 ξ - (2,j) + 1 ξ + (3,k) + 1 ξ - (3,k)      u (i,j,k) - 1 h 2 1 ξ + (1,i) u (i+1,j,k) - 1 h 2 1 ξ - (1,i) u (i-1,j,k) - 1 h 2 1 ξ + (2,j) u (i,j+1,k) - 1 h 2 1 ξ - (2,j) u (i,j-1,k) - 1 h 2 1 ξ + (3,k) u (i,j,k+1) - 1 h 2 1 ξ - (3,k) u (i,j,k-1) .
(4.3.9) (cf. [96, p.106] for the three-dimensional formulation or [START_REF] Hustedt | Mixed-grid and staggered-grid finite difference methods for frequency-domain acoustic wave modelling[END_REF] for a two-dimensional formulation) where

ξ + (l,i) = 1 2 ξ (l,i) ξ (l,i) + ξ (l,i+1) ξ - (l,i) = 1 2 ξ (l,i) ξ (l,i) + ξ (l,i-1)
for l ∈〚1, 3〛. Defining n = n 1 × n 2 × n 3 and letting u ∈ C n (respectively s ∈ C n ) denote the vectorization of u (i,j,k) (respectively s (i,j,k) ) in lexicographical ordering, we can write (4.3.9) as a linear system Au = s where A ∈ C n×n is a matrix containing the coefficients of the unknowns u whose structure is depicted in Figure 4.7. The resulting matrix A is sparse (has seven diagonals of nonzeros only), in general it is non-Hermitian, and because of the PML formulation (4.3.6) it is also non-symmetric and ill-conditioned depending on k(x). Recalling from Section 4.2 that we have p source positions and for a fixed frequency f , we then obtain that the problem to be solved is

AU = S
where U, S ∈ C n×p . The reason why we cannot consider multiple frequencies is that h depends on the frequency f (cf. (4.3.7)), and thus the dimension of the problem n changes for every frequency. It is beyond the scope of this manuscript to address the multiple frequencies issue, but we refer to [START_REF] Wagner | Application of Padé via Lanczos approximations for efficient multifrequency solution of Helmholtz problems[END_REF][START_REF] Wagner | A Krylov subspace projection method for simultaneous solution of Helmholtz problems at multiple frequencies[END_REF] for recent developments done in order to solve the Helmholtz equation for multiple frequencies, and we limit ourselves to the multi sources scenario.

We mention now some computational advantages of the discrete formulation (4.3.9). Because it is a 7-point stencil, we have to store at most seven nonzeros per row, that is, the storage cost for the discretized operator A is bounded by 7n. Moreover, the only term dependent on v (i,j,k) is the central term u (i,j,k) , meaning that in practice only these central points need to be stored in and all the other values can be computed whenever they are needed, making this scheme even cheaper. Nevertheless in our implementation we opt for storing part of the off diagonal for the sake of avoiding floating point operations. Noticing that each ξ + (li) (or ξ + (li) ) is in fact unidimensional, we store the values of each these functions along a line, totalizing n + 2(n 1 + n 2 + n 3 ) storage for this implementation.

One of the main disadvantages of this scheme lies in the fact that it may not be stable unless we choose a particularly large value for n λ to avoid dispersion errors in the solution, due to the fact that it is a second order scheme. In the literature we usually select 10 ≤ n λ ≤ 12, thus resulting in a small h and consequently a large problem size n.

Advanced Discretization Schemes

We briefly mention now alternative schemes for discretizing (4.3.6) which are available in the literature.

In [START_REF] Harari | Accurate finite difference methods for time-harmonic wave propagation[END_REF] several finite difference discretization schemes are proposed for the discretization of (4.3.4) (instead of (4.3.6)), usually aiming at high accuracy, small dispersion error and anisotropy. The authors present fourth-order accurate schemes on uniform grids. A sixth-order accurate finite difference scheme is proposed in [START_REF] Singer | Sixth order accurate finite difference schemes for the Helmholtz equation[END_REF]. We refer to [START_REF] Gordon | Parallel solution of high frequency Helmholtz equations using high order finite difference schemes[END_REF] for recent numerical experiments with these high-order schemes using the CARP-CG [START_REF] Gordon | CARP-CG: A robust and efficient parallel solver for linear systems, applied to strongly convection dominated pdes[END_REF] iterative solver. In [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF] a scheme consisting of a weighted combination of several rotations of 7-point stencils resulting in a 27-point scheme (cf. Figure 4.8) has been proposed. The weights are chosen such that the dispersion error is minimized for small values of n λ . The numerical experiments in [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF] show that the scheme is considered stable for n λ = 4 in the sense that the dispersion error is still in the acceptable threshold. This means that the discretized operator A when using this scheme is compact: twenty seven diagonals are stored, but the dimension n can be considerably smaller. Although in the presentation the weights are chosen to minimize the dispersion error, a priori one could choose any weighting resulting in different properties of the resulting scheme. Notice that because this scheme is formulated as a combination of second order schemes, its truncation error is also of second order. We refer to [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF][START_REF] Sourbier | FWT2D : a massively parallel program for frequency-domain full-waveform tomography of wide-aperture seismic datapart 1: algorithm[END_REF][START_REF] Sourbier | FWT2D : a massively parallel program for frequency-domain full-waveform tomography of wide-aperture seismic datapart 2: numerical examples and scalability analysis[END_REF] for more numerical experiments using this scheme in two and three dimensions.

Preconditioning the Helmholtz Equation

In the current literature the difficulty in solving the Helmholtz problem is often associated with the wavenumber, which we previously defined as

k(x) = 2πf v(x)
or with its maximal value k max = max x∈Ω k(x). In view of (4.3.8), we find out that k max is inversely proportional h, that is, that larger the k max , the larger must be number of points in the discretized domain n, which for high frequencies can reach the value of O(10 9 ) (see Figure 4.9 for an example with the SEG/EAGE Salt dome velocity model). Not surprisingly, the memory cost for solving the discretized problem arising from the discretization of the Helmholtz equation (4.3.4) or (4.3.6) is often the main bottleneck in this application. In [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF] a sparse multifrontal direct method [4] is employed for both twodimensional and three-dimensional formulations, the largest problem solved being a cut of the SEG/EAGE Overthrust velocity model (cf. Figure 4.2) containing 409 × 109 × 102 points, for 10Hz using the 27-point parsimonious staggered-grid scheme mentioned in Subsection 4.3.4 (cf. Figure 4.8) with n λ = 4 and requiring 450 GB of memory. In [START_REF] Bollhöfer | Algebraic multilevel preconditioner for the solution of the Helmholtz equation in heterogeneous media[END_REF] the same full SEG/EAGE Overthrust velocity model is used, this time with multilevel LDL T factorization preconditioner. The problem size is 409×409×102 with f = 5Hz only, requiring 32 GB of memory. Recent work done in [START_REF] Engquist | Sweeping preconditioner for the helmholtz equation: Moving perfectly matched layers[END_REF] proposes a preconditioner based on LDL T factorization preconditioner, which aims to eliminate the unknowns layer by layer (in 2D) or face by face (in 3D) starting from the boundaries of the domain. This preconditioner is therein called "sweeping preconditioner". This work is later extended in [START_REF] Poulson | A parallel sweeping preconditioner for high frequency heterogeneous 3d helmholtz equations[END_REF] to allow parallelism, being thus able to solve for the SEG/EAGE Overthrust velocity model at 8Hz, with 801 × 801 × 187 grid, using 24 Gb of memory. It was investigated in [START_REF] Haidar | On the parallel scalability of hybrid linear solvers for large 3D problems[END_REF] the behaviour of the algebraic additive Schwarz preconditioner when solving the Helmholtz equation for the SEG/EAGE Overthrust velocity model containing 277 × 277 × 73 points, for f = 7Hz, requiring 150 GB of memory. In general, using a standard sparse direct solver method for solving the Helmholtz problem requires an amount of memory of the order of O(n 2 log n) (cf. [121, (1)]), which can be prohibitively large for high k max . Recent publications [START_REF] Wang | Acoustic inverse scattering via Helmholtz operator factorization and optimization[END_REF][START_REF] Wang | On 3d modeling of seismic wave propagation via a structured parallel multifrontal direct Helmholtz solver[END_REF] attempt to reduce the memory of direct solvers for Helmholtz by using low-rank approximation of sub-block of the coefficients matrix, a method therein called parallel Hierarchically Semi-Separable (HSS) matrix compression. The memory requirement is reported to be almost linear, between O(n) and O(n log n), and the method is used to successfully solve realistic problems, as the SEAM velocity model with grid size 401 × 401 × 201 for up to 20Hz, using 2TB memory. More numerical experiments can be found in [3] and an extension for use with elastic waves in [START_REF] Wang | Massively parallel structured multifrontal solver for time-harmonic elastic waves in 3-d anisotropic media[END_REF].

Since the memory seems to be the main bottleneck when solving the Helmholtz equation for standard sparse direct solvers, the use of iterative solvers becomes a feasible alternative. Krylov iterative solvers are recognized for their good scalability in massively parallel environment which is also one of the main interests when solving the Helmholtz problem due to its large dimension. Moreover, the fact that the discretized Helmholtz operator can be very sparse (e.g. when using 7-point schemes) also encourages the use of Krylov iterative solvers. However, k max is also associated with difficulties in finding Krylov directions to improve the approximate solution when using GMRES method without any preconditioner, culminating in a slow convergence (cf. [52, §2.1]). When using incomplete LU factorizations [START_REF] Saad | Iterative solution of linear systems in the 20th century[END_REF]Chapter 10] as preconditioner, iterative solvers as GMRES [START_REF] Saad | GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems[END_REF], BiCGStab [START_REF] Van Der | BI-CGSTAB: a fast and smoothly converging variant of BI-CG for the solution of nonsmmetric linear systems[END_REF] and QMR [START_REF] Freund | QMR: A quasi-minimal residual method for non-Hermitian linear systems[END_REF] also show a decrease in the convergence behaviour as k max grows [START_REF] Gander | An incomplete LU preconditioner for problems in acoustics[END_REF]. Domain decomposition ( [START_REF] Smith | Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations[END_REF] and specially ). The grid size described here does not take the PML layer into account.

[127, §11.5.2]) techniques used as a preconditioner for GMRES are also reported to be dependent on k max (cf. [52, §2.3]). The study of a preconditioner which is efficient for the Helmholtz equation (and hopefully independent of k max ) is subject of active research.

We highlight here the effort on using geometric multigrid techniques [START_REF] Mccormick | Multigrid methods[END_REF][START_REF] Trottenberg | Multigrid[END_REF] (cf. Figure 4.10) for solving the Helmholtz equation. Multigrid techniques are known to be specially efficient when the problem being solved is symmetric and positive-definite, in which case the eigenvalues are located in the positive real quadrant of the complex plane. However, the eigenvalues of the (unpreconditioned) Helmholtz operator with the PML formulation are spread along the complex plane and depending on k max we may have eigenvalues clustered around the origin, characterizing a very ill-conditioned problem. Another concern of the geometric multigrid approach is that on the coarsest level, the effective n λ might be below the suggested threshold. For instance, using the 7-point stencil the advised n λ is 12, whereas in the scheme presented in Figure 4.10, the coarsest grid will effectively be discretized with 6 points per wavelength. This means that the coarse grid correction might have no physical significance, as is explained in [52, §3.3] (cf. also Figure 9 and Figure 10 in the same publication). The dispersion and phase errors in the coarse level may be enough to invalidate the coarse grid correction, which can in some cases deteriorate the solution instead of improving.

In [START_REF] Elman | A multigrid method enhanced by Krylov subspace iteration for discrete Helmholtz equations[END_REF] it was proposed the use of Krylov subspace methods as FGMRES either as a solver preconditioned by geometric multigrid or as a smoother on intermediate grids. Similar numerical experiments for two-dimensional problems can be found in [START_REF] Duff | Multigrid based preconditioners for the numerical solution of two-dimensional heterogeneous problems in geophysics[END_REF], where sparse multifrontal direct methods are used to solve the coarsest level problem. The authors of [START_REF] Elman | A multigrid method enhanced by Krylov subspace iteration for discrete Helmholtz equations[END_REF] report that Krylov iterative solvers are suitable as smoother on the intermediate levels, and that the convergence seems to be independent on the mesh size h, but still dependent on k max . Most notably, in [START_REF] Elman | A multigrid method enhanced by Krylov subspace iteration for discrete Helmholtz equations[END_REF] the authors report that for some values of k max the coarse grid correction may be significant even when the discretization is too coarse to have physical significance.

Other publications on the subject of geometric multigrid for Helmholtz problem are [START_REF] Erlangga | A robust and efficient iterative method for the numerical solution of the Helmholtz equation[END_REF][START_REF] Erlangga | A novel multigrid based preconditioner for heterogeneous Helmholtz problems[END_REF][START_REF] Kim | Multigrid simulations for high-frequency solutions of the Helmholtz problem in heterogeneous media[END_REF] and on algebraic multigrid [START_REF] Bollhöfer | Algebraic multilevel preconditioner for the solution of the Helmholtz equation in heterogeneous media[END_REF].

In [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] it was then proposed the use of a Krylov subspace not only as a smoother but also at the coarse level of two-level multigrid algorithm. The main idea behind this proposal is that the coarse grid correction does not require high accuracy and an approximate solution suffices. This argument is then supported by rigorous Fourier analysis on two-level multigrid preconditioner therein called perturbed geometric twolevel preconditioner. The author shows that the spectrum of the Helmholtz operator preconditioned by a two-level multigrid with an exact coarse correction is not improved when compared with the perturbed preconditioner with a very loose approximation on the coarse level (cf. [96, §3.4]). We thus conclude that the solver does not benefit from highly accurate coarse corrections. This assumption is also supported by numerical experiments where FGMRES method is used with the two-level perturbed preconditioner, where using a coarse correction relative tolerance of mere 0.6 results in faster convergence than using 10 -12 (cf. [96, p.77], Table 3.7). We address this preconditioner in more details in Subsection 4.4.1.

Another development on preconditioning the Helmholtz operator is the use of the so called complex shifted Laplacian operator [START_REF] Erlangga | A robust and efficient iterative method for the numerical solution of the Helmholtz equation[END_REF][START_REF] Erlangga | A novel multigrid based preconditioner for heterogeneous Helmholtz problems[END_REF]. It consists in using

-∆u(x) -(1 -iβ)k 2 (x)u(x) = g(x)
where β ∈ R instead of (4.3.4) for the multigrid hierarchy levels. The goal of this approach is to obtain an operator which presents a better convergence behaviour when using multigrid method. The complex shifted Laplacian operator has been ever since largely used as a preconditioner for Krylov iterative solvers as GMRES or BiCGStab when solving the Helmholtz equation (cf. [52, p.32] Figure 16 for an analysis of the spectrum of the Helmholtz operator preconditioned by the complex shifted Laplacian operator). Numerical experiments in [START_REF] Erlangga | A novel multigrid based preconditioner for heterogeneous Helmholtz problems[END_REF] show the robustness of the method when applied to realistic geophysical applications at high wavenumbers. However, in [START_REF] Erlangga | Advances in iterative methods and preconditioners for the Helmholtz equation[END_REF][START_REF] Ernst | Why it is difficult to solve Helmholtz problems with classical iterative methods[END_REF] it is shown that the efficiency of the complex shifted Laplacian preconditioner is still linearly dependent on k max . We refer to [START_REF] Bollhöfer | Algebraic multilevel preconditioner for the solution of the Helmholtz equation in heterogeneous media[END_REF][START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF][START_REF] Erlangga | A robust and efficient iterative method for the numerical solution of the Helmholtz equation[END_REF][START_REF] Erlangga | Advances in iterative methods and preconditioners for the Helmholtz equation[END_REF][START_REF] Erlangga | A novel multigrid based preconditioner for heterogeneous Helmholtz problems[END_REF][START_REF] Riyanti | A new iterative solver for the time-harmonic wave equation[END_REF][START_REF] Riyanti | A parallel multigrid-based preconditioner for the 3D heterogeneous high-frequency Helmholtz equation[END_REF][START_REF] Umetani | A multigrid-based shifted Laplacian preconditioner for fourth-order Helmholtz discretization[END_REF] for numerical experiments with the complex shifted Laplacian using low and medium frequency ranges only and more information on how to choose the shift parameter β.

Although the use of complex shifted Laplacian preconditioners is normally advised specially when using multigrid techniques for low or mid frequency cases (about 10Hz), the efficiency of this preconditioner may decrease for high frequencies. In the recent publication [START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF] it was shown that using optimal parameters deduced from Fourier analysis, the perturbed two-level preconditioner proposed in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] shows an increased robustness on heterogeneous problems when compared to the standard approach based on complex shifted Laplacian operator. The performance is further improved when the two-level preconditioner uses a complex shifted Laplacian operator as a preconditioner on the coarse level (cf. [START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF]Table IV]). The numerical experiments in [START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF] confirm the efficiency of their approach when solving for high frequencies.

In this thesis we limit ourselves to the use of the two-level perturbed multigrid proposed in [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF], which we reproduce with more details in Subsection 4.4.1. We focus on the behaviour of the overall method when solving the Helmholtz equation (4.3.6) in the multi sources scenario, using deflation every iteration. We discuss in depth the issues arising from this particular scenario in the next section, and we refer to [START_REF] Calandra | An improved two-grid preconditioner for the solution of three-dimensional Helmholtz problems in heterogeneous media[END_REF] for a recent detailed description of the perturbed preconditioner when using the complex shifted Laplacian operator as preconditioner.

The Perturbed Geometric Two-Level Preconditioner

In this subsection we discuss the perturbed geometric two-level preconditioner [START_REF] Calandra | Two-level preconditioned Krylov subspace methods for the solution of three-dimensional heterogeneous Helmholtz problems in seismics[END_REF][START_REF] Calandra | Flexible variants of block restarted GMRES methods with application to geophysics[END_REF][START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF], briefly mentioned in Section 4.4. Since this is the main preconditioner we will use in our further numerical experiments (as it is the preconditioner used in the experiments in Section 3.9) we provide a more detailed explanation in this section.

Recalling the discussion in Section 4.3, A is the matrix issued from the discretization of the operator (4.3.6) on Ω h . We then define the coarse grid Ω H as the standard geometric coarsening of Ω h in all directions. Without loss of generality, we suppose here that h was chosen such4 that n/8 =: N ∈ N. The discretization of (4.3.6) in Ω H is thus obtained using the same discretization scheme of choice for Ω h , resulting in the operator A H ∈ C N ×N .

We also define the restriction and the interpolation operators

R : C n → C N I : C N → C n .
In all our numerical experiments we use I(.) as the trilinear interpolation, and its adjoint as the restriction operator R(.).

The generic perturbed two-level preconditioner [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] for single right-hand side is depicted in Algorithm 4.4.1. Unless otherwise specified, whenever we apply Algorithm 4.4.1 we use the following parameters

µ = 1, m h = 2, µ H =
= R(V j -AZ µ h ); 3 Solve approximately the coarse problem A H Z H = R H : Apply µ H cycles of FGMRES(m H ) to
A H Z H = R H with initial approximation 0 N ×kj and symmetric Gauss-Seidel as a right preconditioner to obtain the approximation ZH ;

4 Correct the fine grid approximation: Zh = Z µ h + I( ZH ); 5 Polynomial postmoothing: Apply µ cycle(s) of FGMRES(m h ) to AZ = V j with initial approximation Zh and symmetric Gauss-Seidel as a right preconditioner to obtain the approximation Z h ; 6 Define Z j := Z h We highlight that the application of Algorithm 4.4.1 as a preconditioner characterizes a variable preconditioner. Writing Z j = M(V j ) where the nonlinear function M(.) represents the action Algorithm 4.4.1 on V j , we see that the use of the perturbed two-level preconditioner implies the use of a flexible outer method.

Software Implementation

In view of the whole discussion in this chapter, it is clear that many aspects of the full waveform inversion method require future research. Indeed so many different techniques are being applied on different levels of the problem that choosing the best strategy and staying up to date with the most recent advancements is a challenge. Also, due to the vast diversities of knowledge that this problem requires, it would be challenging to write a monolitical software to be used by geophysicists as well as computer scientists working with low level linear algebra optimization. For this reason it is needed a malleable and modular software. We thus discuss now some basic aspects of the software implementation we proposed for solving large sparse linear systems (and thus, being able to solve the forward problem) in a massively parallel environment, using FORTRAN03 object orientation techniques.

In scientific computing community, FORTRAN77 and FORTRAN90 stand as two of the most efficient and popular programming languages. However, both of these languages lack of object orientation tools, as polymorphism and inheritance, which are essential building blocks for modular codes. Since our original proposal was to build a modular software able to be easily changed and adapted to exploit the new techniques being developed in the literature, we opted for using a language which would instead present at least polymorphism and inheritance features.

Among the candidate languages, we highlight here the FORTRAN03, a standard for the FORTRAN compiler published in 2004, but not yet completely implemented as a functional compiler. Instead, specific compilers implementing a limited set of FORTRAN03 standards have been developed, as ifort (Intel FORTRAN Compiler), gfortran (GNU FORTRAN compiler), pgi (The Portland Group, Inc.), among others. Nevertheless, the basic object orientation features are supported by most of these compilers.

The choice of FORTRAN03 over other object oriented languages makes sense when considering a software using only the most basic object orientation concepts, specially since we are targeting a community that is already used to FORTRAN77 and FORTRAN90 softwares, making the transition between procedural programming to object oriented programming more smooth and comfortable, in contrast with other languages which could require a deep knowledge on object orientation and software engineering.

We describe basically the advantage of using polymorphism and inheritance in Figure 4.11. Just as an example, consider we have a type C_Solver and that the objects C_BFGMRES, C_BFGMRESD, and C_DMBR which inherit from C_Solver. But more than that, the object C_FBGMRES requires another object of the type C_Solver to be used as preconditioner. This means that any of the objects C_BFGMRESD, and C_DMBR could be used as preconditioner, including C_BFGMRES itself. Also, if in any moment it is implemented an object of the type C_BiCGStab who inherits from C_Solver, then C_BiCGStab could be used as a preconditioner for C_BFGMRES without any modification in the code. We do not attain to details of the code in this subsection, but we refer to Appendix A for a detailed user guide of the libraries we implemented using FORTRAN03 and object orientation. We highlight that we compared the code written in FORTRAN03 with the prototypes implemented in FORTRAN90 used in the numerical experiments in Subsection 3.9.3 and Subsection 3.9.4. The FORTRAN03 version showed a marginal speedup of 1% to 3.5% with respect to the prototype code, thus guaranteeing that the addition of the object orientation did not bring any slow down to our code.

Numerical Experiments

This section is dedicated to numerical experiments related to acoustic full waveform inversion method problem, using the software we developed using FORTRAN03. The goal of this section is two fold: to show that the software we developed is suitable for solving large scale problems (approximately O(10 9 )) in a massively parallel environment, and to reinforce the interest in using the deflation techniques proposed in Chapter 3 in the acoustic full waveform inversion context.

Forward Problem: Smoothed SEG/EAGE Salt Dome

The numerical experiments performed in Section 3.9 concern the acoustic wave propagation phenomena in frequency domain through the academic velocity field SEG/EAGE Overthrust, which is a realistic model, reflecting the real properties of the subsurface of Earth. However, for the purpose of the inverse problem, the approximated velocity model could present considerably different properties (especially during early iterations, or during the computation of an initial guess). For this reason, it turns out to be interesting to investigate the behaviour of deflation in a scenario where the velocity model is rather an approximation of a realistic model.

In this subsection we present numerical experiments related to the velocity model known as SEG/EAGE Salt Dome (already shown in Figure 4.9). This is a particularly challenging problem to be solved using iterative solvers due to a discontinuity in the velocity model (representing a dome of salt in the subsurface of Earth), which abruptly increases the velocity of propagation of the compressional waves (cf. Figure 4.12 for a graphical representation of the interior of this velocity model). We also perform perturbations on this velocity model, applying different degrees of compression techniques and obtaining smoothed approximated versions of this model (cf. The experiments performed in this section aim at investigating how much the behaviour of the deflation of techniques proposed in Chapter 3 change as the velocity field turns smoother. All experiments were performed on a Bullx B510 computer located at CERFACS (two Intel Sandy Bridge with 2.6 Ghz and 32 Gb of memory per node). In all experiments we used DMBR(5) preconditioned by perturbed two-level method (cf. Subsection 4.4.1). The frequency was set to 5Hz, and using one length of PML and 12 points per wavelength, the resulting discretized domain contained 624 × 616 × 240 points5 being thus a problem of dimension 9.2 × 10 7 . We used 16 source positions equally spaced along a line (from point (285, 309, 38) to point (340, 309, 38) ) on the surface of the domain, and the memory cost per source point in this experiment is 11.8 Gb per right-hand side, plus the storage required to store the coefficients matrix A. The total cost including A and all right-hand sides is 189.6 Gb. The experiment was performed with 128 cores. Table 4.1 shows the results in number of iterations, number of preconditioner applications and total computational time for the test. We can verify that, as the velocity field becomes smoother, fewer iterations and fewer preconditioner applications are needed for finding an approximate solution satisfying the relative residual stopping criterion (for = 10 -5 ). Figure 4.17 shows the history of k j along the convergence of DMBR( 5) for each velocity model. As we can see in , the history of k j for each velocity model is indeed very similar. For instance, Smoothed×3 velocity model shows a faster convergence in number of iterations for DMBR(5) when compared to the original SEG/EAGE salt dome velocity model (therein called Smoothed×0), but it starts deflating in an earlier iteration. The curves of evolution of k j are relatively similar for all the velocity models. We consider this result very satisfactory and it encourages further research on the behaviour of deflation in the inverse problem application. 4.1) for each SEG/EAGE Salt dome velocity field and its respective smoothed versions.

Forward Problem: Mid Frequency Case

In this subsection we quickly expose the solution we obtained when using our libraries to solve the Helmholtz equation for mid frequencies. The goal of this experiment is to illustrate that the software was designed for solving at higher frequencies as long as enough memory resources are available.

As in the previous section, we limit ourselves to the use of two-level techniques as described in Subsection 4.4.1, but here we use FGMRES(5) as outer solver rather than DMBR [START_REF] Arnoldi | the principle of minimized iterations in the solution of the matrix eigenvalue problem[END_REF]. The reason for this choice is that we perform this experiment using only one right-hand side. As the previous experiments, the following experiments was performed on a Bullx B510 computer located at CERFACS (two Intel Sandy Bridge with 2.6 Ghz and 32 Gb of memory per node): using the SEG/EAGE salt dome velocity field, we solve the Helmholtz equation for 12Hz, and using one length of PML and 12 points per wavelength, the resulting discretized domain contained 1376 × 1376 × 480 points being thus a problem of dimension 9.1 × 10 8 . The only source was positioned at the very center of the domain. We used 128 cores, and the memory cost for this experiment was 141.7 Gb and with a computational time of 2, 012 seconds (approximately 33min) to find an approximate solution satisfying the relative residual smaller than 10 -5 .

Figure 4.18 show a graphical representation of SEG/EAGE Salt Dome along with the wavefield obtained after solving the Helmholtz equation in this setting.

Conclusions

In this chapter we provided a wide overview of several aspects of the full waveform inversion problem. We focused on the challenges arising in the solution of the forward problem, and we highlight that there is no generic optimal solution, as the best method for solving the forward problem depends on characteristics of the problem, as for instance, the wavenumber (thus also the frequency), the size of the geophysical domain, among others. With this in mind, we motivate the use of a flexible object oriented code using MPI and FORTRAN03 which can be easily adapted to use new strategies or to choose which strategy is the best depending on the problem parameters. We propose a first version of this code, which implements some finite difference discretization schemes as well as some iterative solvers, as BFGMRESD [START_REF] Langou | Iterative methods for solving linear systems with multiple right hand sides[END_REF], BGMRES-R [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] and DMBR presented in Chapter 3 as well as some preconditioners based on multigrid techniques, as the perturbed two level preconditioner [START_REF] Pinel | A Perturbed Two-level Preconditioner for the Solution of Three-Dimensional Heterogeneous Helmholtz problems with Applications to Geophysics[END_REF] detailed in Subsection 4.4.1. Tests performed with this software has shown a satisfactory level of flexibility and further solvers, preconditioners and discretization schemes will be implemented in our future work. Appendix A contains a basic user guide with more detailed description of the code with examples and description of key functions, types and classes. We have shown a practical use of the code with numerical experiments performed at 5Hz and 12Hz using the academical velocity field SEG/EAGE salt dome, a challenging and realistic model. With this experiment we illustrated further the behaviour of the deflation techniques proposed in Chapter 3 when applied to the forward problem arising from the acoustic full waveform inversion problem, showing the interest in using DMBR method in real life applications. Flexible GCRO-DR

Foreword

In this chapter we present the flexible generalized conjugate residual method with inner orthogonalization and deflated restarting (FGCRO-DR) method. It consists of an iterative solver able to recycle subspace information of one cycle to accelerate the convergence of the following cycle, a technique initially proposed for the solution of sequence of linear systems, that is, when we have the sequence of systems

Ax (i) = b (i) , with each x (i) , b (i) ∈ C n for 1 ≤ i ≤ p
, where b (i) may depend on some x (j) for 1 ≤ j < i. In such a scenario, the use of block methods as proposed in Chapter 2 and 3 is not possible. The development of this method has been greatly inspired by the application described in Chapter 4, since subspace recycling techniques could be used to accelerate the convergence when the problem presents multiple left-hand sides, that is

A (i) X = B, with X, B ∈ C n×p , A (i) ∈ C n×n 1 ≤ i ≤ λ,
a situation which could arise as well from the Earth imaging scenario (cf. Remark 4.2.1). We refer to [1,[START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] for more details on the use of recycling techniques for the multiple left-hand side case.

In this chapter we introduce the FGCRO-DR method for single right hand sides scenario and discuss its main properties, focusing on the comparison between FGCRO-DR and FGMRES-DR [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF], a closely related method which is unable to recycle subspace information. We demonstrate that, in spite of the similarities between both methods, when a variable preconditioner is used they are only equivalent if a collinearity condition holds (cf. Theorem 5.2.11) We let as future work the investigation of the efficiency of FGCRO-DR recycling techniques when solving the forward and backward problem or when solving for multiple models simultaneously. Another important issue we do not address in this manuscript is the generalization of FGCRO-DR for the block scenario and the potential combination of recycling technique with block techniques, which is also subject of our future research.

A Flexible Generalized Conjugate Residual Method with Inner Orthogonalization and Deflated Restarting

The title as well as the contents of this section corresponds to [START_REF] Carvalho | A flexible generalized conjugate residual method with inner orthogonalization and deflated restarting[END_REF], joint work with Luiz Mariano Carvalho, Serge Gratton and Xavier Vasseur.

Introduction

In recent years, several authors studied inner-outer Krylov subspace methods that allow variable preconditioning for the iterative solution of large sparse linear systems of equations. One of the first papers describing a subspace method with variable preconditioning is due to Axelsson and Vassilevski who proposed the Generalized Conjugate Gradient method [START_REF] Axelsson | A black box generalized conjugate gradient solver with inner iterations and variable-step preconditioning[END_REF]. See also [START_REF] Axelsson | Iterative solution methods[END_REF]Section 12.3] for additional references. Since then, numerous methods have been proposed to address the symmetric, non-symmetric or non-Hermitian cases; these include Flexible Conjugate Gradient [START_REF] Notay | Flexible conjugate gradients[END_REF], Flexible GMRES (FGMRES) [START_REF] Saad | A flexible inner-outer preconditioned GMRES algorithm[END_REF], Flexible QMR [START_REF] Szyld | FQMR: A flexible quasi-minimal residual method with inexact preconditioning[END_REF] and GMRESR [START_REF] Van Der Vorst | GMRESR: a family of nested GMRES methods[END_REF] among others. This class of methods is required when preconditioning with a different (possibly nonlinear) operator at each iteration of a subspace method is considered. This notably occurs when adaptive preconditioners using information obtained from previous iterations [START_REF] Baglama | Adaptively preconditioned GMRES algorithms[END_REF][START_REF] Erhel | Restarted GMRES preconditioned by deflation[END_REF] are used or when inexact solutions of the preconditioning system using e.g. adaptive cycling strategy in multigrid [START_REF] Notay | Recursive Krylov-based multigrid cycles[END_REF] or approximate interior solvers in domain decomposition methods [START_REF] Toselli | Domain Decomposition methods -Algorithms and Theory[END_REF]Section 4.3] are considered. The latter situation is frequent when solving very large systems of linear equations resulting from the discretization of partial differential equations in three dimensions. Thus flexible Krylov subspace methods have gained a considerable interest in the past years and are subject to both theoretical and numerical studies [START_REF] Simoncini | Flexible inner-outer Krylov subspace methods[END_REF]. We refer the reader to [START_REF] Simoncini | Recent computational developments in Krylov subspace methods for linear systems[END_REF]Section 10] for additional comments on flexible methods.

When non variable preconditioning is considered, the full GMRES method [START_REF] Saad | GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems[END_REF] is often chosen for the solution of non-symmetric or non-Hermitian linear systems because of its robustness and its minimum residual norm property [START_REF] Saad | Iterative Methods for Sparse Linear Systems[END_REF]. Nevertheless to control both the memory requirements and the computational cost of the orthogonalization scheme, restarted GMRES is preferred; it corresponds to a scheme where the maximal dimension of the approximation subspace is fixed. It means in practice that the orthonormal basis built is thrown away at the end of the cycle. Since some information is discarded at the restart, the convergence may stagnate and is expected to be slower compared to full GMRES. Nevertheless to retain the convergence rate a number of techniques have been proposed; they fall in the class of augmented and deflated methods; see e.g. [START_REF] Baker | A technique for accelerating the convergence of restarted GMRES[END_REF][START_REF] De Sturler | Truncation strategies for optimal Krylov subspace methods[END_REF][START_REF] Eiermann | Analysis of acceleration strategies for restarted minimal residual methods[END_REF][START_REF] Morgan | A restarted GMRES method augmented with eigenvectors[END_REF][START_REF] Saad | Analysis of augmented Krylov subspace methods[END_REF]. Deflated methods compute spectral information at a restart and use this information to improve the convergence of the subspace method. One of the most recent procedure based on a deflation approach is GMRES with deflated restarting (GMRES-DR) [START_REF] Morgan | GMRES with deflated restarting[END_REF]. This method reduces to restarted GMRES when no deflation is applied, but may provide a much faster convergence than restarted GMRES for well chosen deflation spaces as described in [START_REF] Morgan | GMRES with deflated restarting[END_REF].

Quite recently a new minimum residual norm subspace method based on GMRES allowing deflated restarting and variable preconditioning has been proposed in [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF]. It mainly attempted to combine the numerical features of GMRES with deflated restarting and the flexibility property of FGMRES. Numerical experiments in [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF] have shown the efficiency of Flexible GMRES with deflated restarting (FGMRES-DR) on both academic and industrial examples. In this section we study a new minimum residual norm subspace method based on the Generalized Conjugate Method with inner Orthogonalization (GCRO) [START_REF] De Sturler | Nested Krylov methods based on GCR[END_REF] allowing deflated restarting and variable preconditioning. It is named Flexible Generalized Conjugate Residual Method with Inner Orthogonalization and Deflated Restarting (FGCRO-DR) and can be viewed as an extension of GCRO-DR [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] to the case of variable preconditioning. A major advantage of FGCRO-DR over FGMRES-DR is its ability to solve sequence of linear systems (where both the left and right-hand sides could change) through recycling [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF]. In [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] Parks et al. mentioned that GCRO-DR and GMRES-DR were algebraically equivalent i.e. both methods produce the same iterates in exact arithmetic when solving the same given linear system starting from the same initial guess. When variable preconditioning is considered, it seems therefore natural to ask whether FGCRO-DR and FGMRES-DR could be also algebraically equivalent. We address this question in this section and the main theoretical developments that are proposed will help us to answer this question. The main contributions of the section are then twofold. First we prove that FGCRO-DR and FGMRES-DR can be considered as algebraically equivalent if a collinearity condition between two certain vectors is satisfied at each cycle. When considering non variable preconditioning, these theoretical developments will also allow us to show the algebraic equivalence between GCRO-DR and GMRES-DR that was stated without proof in [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF]. Secondly we carefully analyze the computational cost of FGCRO-DR and show that the proposed method is nearly as expensive as FGMRES-DR in terms of operations per cycle. Furthermore it is explained how to include subspace recycling into FGCRO-DR and numerical experiments are reported showing the efficiency of FGCRO-DR. This section is organized as follows. In Subsection 5.2.2 we introduce the general background of this study. We briefly recall the main properties of FGMRES-DR and then introduce the FGCRO-DR method both from a mathematical and algorithmic points of view. Subsection 5.2.7 is mainly devoted to the analysis of both flexible methods. Therein we show that both methods can be algebraically equivalent in the flexible case if a certain collinearity condition is satisfied at each cycle. In Subsection 5.2.11 we compare FGCRO-DR and FGMRES-DR in terms of computational operations per cycle and storage and discuss the solution of sequences of linear systems through subspace recycling. Finally we draw some conclusions and perspectives in Subsection 5.2.15.

Flexible Krylov methods with restarting

General setting

Notation Throughout this section we denote by . the Euclidean norm, I k ∈ C k×k the identity matrix of dimension k and 0 i×j ∈ C i×j the zero rectangular matrix with i rows and j columns. Given N ∈ C n×m Π N ⊥ = I n -N N † will represent the orthogonal projector onto range (N )

⊥ , where the superscript † refers to the Moore-Penrose inverse. Finally given

Z m = [z 1 , • • • , z m ] ∈ C n×m , we will usually decompose Z m into two submatrices defined as Z k = [z 1 , • • • , z k ] ∈ C n×k and Z m-k = [z k+1 , • • • , z m ] ∈ C n×(m-k) .
Setting We focus on minimum residual norm based subspace methods that allow flexible preconditioning for the iterative solution of

Ax = b, A ∈ C n×n , x, b ∈ C n (5.2.1)
given an initial vector x 0 ∈ C n . In this section A is supposed to be nonsingular. Flexible methods refer to a class of methods where the preconditioner is allowed to vary at each iteration. We refer the reader to e.g. [START_REF] Simoncini | Recent computational developments in Krylov subspace methods for linear systems[END_REF] for a general introduction on Krylov subspace methods and to [START_REF] Simoncini | Recent computational developments in Krylov subspace methods for linear systems[END_REF]Section 10] and [START_REF] Saad | Iterative Methods for Sparse Linear Systems[END_REF]Section 9.4] for a review on flexible methods. The minimum residual norm GMRES method [START_REF] Saad | GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems[END_REF] has been extended by Saad [START_REF] Saad | A flexible inner-outer preconditioned GMRES algorithm[END_REF] to allow variable preconditioning. The resulting algorithm known as FGMRES(m) relies on the Arnoldi relation

AZ m = V m+1 Hm , (5.2.2)
where Z m ∈ C n×m , V m+1 ∈ C n×(m+1) has orthonormal columns and Hm ∈ C (m+1)×m is upper Hessenberg.

We denote by M j the preconditioning operator at iteration j and remark that M j may be a nonlinear preconditioning function. We will then denote by M j (v) the action of M j on a vector v. In (5.2.2), the columns of V m+1 form an orthonormal basis of the subspace spanned by the following vectors

{r 0 , Az 1 , • • • , Az m } with r 0 = b -Ax 0 whereas Z m = [z 1 , • • • , z m ] and V m = [v 1 , • • • , v m ] are related by Z m = [M 1 (v 1 ), • • • , M m (v m )] with v 1 = r 0 r 0 .
At the end of the cycle an approximate solution x m ∈ C n is then found by minimizing the residual norm r 0 -AZ m y over the space x 0 + range (Z m ). Thus we obtain that

x m = x 0 + Z m y * ,
where y * is the solution of the following least-squares problem of size (m + 1) × m y * = argmin y∈C m r 0 -AZ m y = argmin y∈C m r 0 e 1 -Hm y ,

where e 1 is the first canonical vector of C m+1 . Flexible subspace methods with restarting are based on a procedure where the construction of the subspace is stopped after a certain number of steps (denoted by m in this section with m < n). The method is then restarted mainly to control both the memory requirements and the cost of the orthogonalization scheme. In FGMRES(m) the restarting consists in taking as an initial guess the last iterate of the cycle (x m ).

The main focus of this section is to present minimum residual norm subspace methods with deflated restarting that allow flexible preconditioning. Deflated restarting aims at determining an approximation subspace of dimension m as a direct sum of two subspaces of smaller dimension, where one of these subspaces will contain relevant spectral information that will be kept for the next cycle. We refer the reader to e.g. [START_REF] Saad | Analysis of augmented Krylov subspace methods[END_REF] and [START_REF] Simoncini | Recent computational developments in Krylov subspace methods for linear systems[END_REF]Section 9] for a review on augmented and deflated methods. Flexible methods with deflated restarting will notably satisfy the following flexible Arnoldi relation

AZ m = V m+1 Hm with V H m+1 V m+1 = I m+1 , (5.2.3)
where Hm ∈ C (m+1)×m is not necessarily of upper Hessenberg form. In this section we call this relation a flexible Arnoldi-like relation due to its similarity to relation (5.2.2).

Stagnation and breakdown

We refer the reader to [START_REF] Simoncini | Flexible inner-outer Krylov subspace methods[END_REF]Section 6] for general comments and a detailed discussion on the possibility of both breakdown and stagnation in flexible inner-outer Krylov subspace methods. Although important, these issues are not addressed in this section and we assume that no breakdown occurs in the inner-outer subspace methods that will be proposed.

Flexible GMRES with deflated restarting

A number of techniques have been proposed to compute spectral information at a restart and use this information to improve the convergence rate of the Krylov subspace methods; see, e.g., [START_REF] Morgan | A restarted GMRES method augmented with eigenvectors[END_REF][START_REF] Morgan | Implicitly restarted GMRES and Arnoldi methods for nonsymmetric systems of equations[END_REF][START_REF] Morgan | GMRES with deflated restarting[END_REF][START_REF] Saad | Analysis of augmented Krylov subspace methods[END_REF].

These techniques have been exclusively developed in the case of a fixed preconditioner. Among others GMRES-DR is one of those methods. It focuses on removing (or deflating) the eigenvalues of smallest magnitude. A full subspace of dimension k, k < m (and not only the approximate solution with minimum residual norm) is now retained at the restart and the success of this approach has been demonstrated on many academic examples [START_REF] Morgan | A restarted GMRES method augmented with eigenvectors[END_REF]. Approximations of eigenvalues of smallest magnitude are obtained by computing harmonic Ritz pairs of A with respect to a certain subspace [START_REF] Morgan | GMRES with deflated restarting[END_REF]. We present here a definition of a harmonic Ritz pair equivalent to the one introduced in [START_REF] Paige | Approximate solutions and eigenvalue bounds from Krylov subspaces[END_REF][START_REF] Sleijpen | A Jacobi-Davidson iteration method for linear eigenvalue problems[END_REF]; it will be of key importance when defining appropriate deflation strategies. We call y a harmonic Ritz vector associated with the harmonic Ritz value θ.

As in the case of fixed preconditioning, deflated restarting may also improve the convergence rate of flexible subspace methods. In [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF] a deflated restarting procedure has been proposed for the FGMRES algorithm. The i-th cycle of the resulting algorithm called FGMRES-DR is now briefly described and we denote by r

AZ m P k = V m+1 P k 0 1×k , c -Hm y * diag(λ 1 , . . . , λ k ) α 1×k , (5.2.4
)

AZ m P k = [V m P k , r (i-1) 0 ] diag(λ 1 , . . . , λ k ) α 1×k , (5.2.5)
where r

(i-1) 0 = V m+1 (c -Hm y * ) and α 1×k = [α 1 , . . . , α k ] ∈ C 1×k .
Next, the QR factorization of the (m + 1) × (k + 1) matrix appearing on the right-hand side of relation (5.2.4) is performed as

P k 0 1×k , c -Hm y * = QR (5.2.6)
where Q ∈ C (m+1)×(k+1) has orthonormal columns and R ∈ C (k+1)×(k+1) is upper triangular. We write the matrix Q obtained in relation (5.2.6) as

Q = Q m×k 0 1×k , ρ ρ , (5.2.7)

Flexible GCRO with deflated restarting

GCRO-DR [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] -a combination of GMRES-DR and GCRO -is a Krylov subspace method that allows deflated restarting and subspace recycling simultaneously. This latter feature is particularly interesting when solving sequences of linear systems with possibly different left or right-hand sides. As pointed out in [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF], GCRO-DR is attractive because any subspace may be recycled. In this section we restrict the presentation to the case of a single linear system as proposed in (5.2.1).

GCRO and GCRO-DR belong to the family of inner-outer methods [START_REF] Axelsson | Iterative solution methods[END_REF]Ch. 12] where the outer iteration is based on GCR, a minimum residual norm method proposed by Eisenstat, Elman and Schultz [START_REF] Eisenstat | Variational iterative methods for nonsymmetric system of linear equations[END_REF]. To this end GCR maintains a correction subspace spanned by range (Z m ) and an approximation subspace spanned by range (V m ), where Z m , V m ∈ C n×m satisfy

A Z m = V m , V H m V m = I m .
The optimal solution of the minimization problem min b -Ax over the subspace x 0 + range (Z m ) is then found as

x m = x 0 + Z m V H m r 0 . Consequently r m = b -A x m satisfies r m = r 0 -V m V H m r 0 = Π V ⊥ m r 0 , r m ⊥ range (V m ) .
In [START_REF] De Sturler | Nested Krylov methods based on GCR[END_REF] de Sturler proposed an improvement to GMRESR [START_REF] Van Der Vorst | GMRESR: a family of nested GMRES methods[END_REF], an inner-outer method based on GCR in the outer part and GMRES in the inner part. He suggested that the inner iteration takes place in a subspace orthogonal to the outer Krylov subspace. In this inner iteration the projected residual equation

(I n -V m V H m )Az =
r m is solved only approximately. If a minimum residual norm subspace method is used in the inner iteration to solve this projected residual linear system, the residual over both the inner and outer subspaces are minimized. This leads to the GCRO (Generalized Conjugate Residual method with inner Orthogonalization) Krylov subspace method [START_REF] De Sturler | Nested Krylov methods based on GCR[END_REF]. Numerical experiments [START_REF] De Sturler | Nested Krylov methods based on GCR[END_REF] indicate that the resulting method may perform better than other inner-outer methods (without orthogonalization) in some cases.

The GCRO method with deflated restarting (named GCRO-DR) based on harmonic Ritz value information has been proposed in [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF]. An approximate invariant subspace is used for deflation following closely the GMRES-DR method. We refer the reader to [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] for a description of this method, algorithms and implementation details. We present now a new variant of GCRO-DR that allows flexible preconditioning by explaining the different steps occurring during the i-th cycle. Again we denote by r (i-1) 0 = b -Ax (i-1) 0 , V m+1 , Hm and Z m the residual and matrices obtained at the end of the (i -1)-th cycle.

We suppose that a flexible Arnoldi-like relation of type (5.2.3) holds. As in Subsection 5.2.4 an important point is to specify which harmonic Ritz information is selected. Given a certain matrix W m ∈ C n×m to be specified later on, such as range (W m ) = range (V m ), the deflation procedure relies on the use of

k harmonic Ritz vectors Y k = W m P k of AZ m W †
m with respect to range (W m ), where Y k ∈ C n×k and P k ∈ C m×k . W m will notably satisfy a property detailed in Lemma 5.2.8 and we point out that the calculation of W † m is not needed in the practical implementation of the algorithm (see Section 5.2.12). In Lemma 5.2.4 we detail a useful relation satisfied by the harmonic Ritz vectors.

AZ m P k = [W m P k , r (i-1) 0 ] diag(θ 1 , . . . , θ k ) β 1×k , (5.2.16)
where r

(i-1) 0 = V m+1 (c -Hm y * ) and β 1×k = [β 1 , . . . , β k ] ∈ C 1×k .
Proof. According to Definition 5.2.1, the harmonic residual vectors AZ m W † m W m p j -θ j W m p j and the residual vector r (i-1) 0 = V m+1 (c -Hm y * ) all belong to a subspace of dimension m + 1 (spanned by the columns of V m+1 ) and are orthogonal to the same subspace of dimension m (spanned by the columns of AZ m subspace of range (V m+1 )), so they must be collinear. Consequently there exist k coefficients noted β j ∈ C with 1 ≤ j ≤ k such that ∀j ∈ {1, . . . , k} AZ m p j -θ j W m p j = β j r (i-1) 0 .

(5.2.17)

Setting β 1×k = [β 1 , . . . , β k ] ∈ C 1×k , the collinearity expression (5.2.17) can be written in matrix form as

AZ m P k = [W m P k , r (i-1) 0 ] diag(θ 1 , . . . , θ k ) β 1×k .
Due to the flexible Arnoldi-like relation (5.2.3), relation (5.2.16) can be also expressed as

V m+1 Hm P k = [W m P k , r (i-1) 0 ] diag(θ 1 , . . . , θ k ) β 1×k . ( 5
P k = QR with Q ∈ C (m+1)×k and R ∈ C k×k .
Proposition 5.2.5. In flexible GCRO with deflated restarting, the relation

AZ k = V k with V H k V k = I k holds at the i-th cycle with matrices Z k , V k ∈ C n×k defined as Z k = Z m P k R -1 , V k = V m+1 Q,
where V m+1 and Z m refer to matrices obtained at the end of the (i -1)-th cycle. In addition V H k r (i-1) 0 = 0 holds during the i-th cycle.

Proof. By using information related to the QR factorization of Hm P k and the flexible Arnoldi relation (5.2.3) exclusively, we obtain

A Z k = AZ m P k R -1 , = V m+1 Hm P k R -1 , = V m+1 Q, = V k . Since both V m+1 and Q have orthonormal columns, V k satisfies V H k V k = I k . Finally since r (i-1) 0
is the optimum residual at the i -1-th cycle, i.e. (AZ m ) H r (i-1) 0 = 0 we obtain

P H k (AZ m ) H r (i-1) 0 = 0, (V m+1 Hm P k ) H r (i-1) 0 = 0, R H V H k r (i-1) 0 = 0.
This finally shows that V H k r (i-1) 0 = 0 since R is supposed to be nonsingular.

To complement the subspaces, the inner iteration is based on the approximate solution of

(I n -V k V H k )Az = (I n -V k V H k )r (i-1) 0 = r (i-1) 0 ,
where the last equality is due to Proposition 5.2.5. For that purpose FGCRO-DR then carries out m -k steps of the Arnoldi method with flexible preconditioning leading to

(I n -V k V H k ) A [z k+1 , • • • , z m ] = [v k+1 , • • • , v m+1 ] Hm-k (I n -V k V H k ) A Z m-k = V m-k+1 Hm-k with v k+1 = r (i-1) 0 /||r (i-1) 0
. At the end of the cycle this gives the flexible Arnoldi-like relation

A [Z k , Z m-k ] = [V k , V m-k+1 ] I k V H k A Z m-k 0 m-k+1×k
Hm-k m+1) and Hm ∈ C (m+1)×m . At the end of the i-th cycle, an approximate solution x + range(Z m ), the corresponding residual being r

A Z m = V m+1 Hm , where Z m ∈ C n×m , V m+1 ∈ C n×(
(i) 0 = b -Ax (i) 0 , with r (i) 0 ∈ range(V m+1 ).

Algorithms

Details of the FGCRO-DR method are given in Algorithm 5.2.1, where Matlab-like notations are adopted (for instance in step 7b, Q(1 : m, 1 : k) denotes the submatrix made of the first m rows and first k columns of matrix Q noted Q m×k in equation (5.2.7)). For the sake of completeness the FGMRES-DR algorithm has been also described with notations chosen as close as possible to FGCRO-DR to make a code comparison as easy as possible. Concerning Algorithm 5.2.1 we make the following comments:

• As discussed later the computation of W † m in step 5a is not required thanks to the definition of the harmonic Ritz pair (see Definition 5.2.1).

• As pointed out by Morgan [START_REF] Morgan | GMRES with deflated restarting[END_REF] and Parks et al. [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF] we might have to adjust k during the algorithm to include both the real and imaginary parts of complex eigenvectors.

• In steps 10a and 10b M (i) j denotes the possibly nonlinear preconditioning operator at iteration j during the i-th cycle.

Analysis of FGMRES-DR and FGCRO-DR

We compare now the flexible variants of GMRES-DR and GCRO-DR introduced respectively in Subsection 5.2.4 and 5.2.5 . In the following we use the superscript to denote quantities related to the FGMRES-DR algorithm e.g. Y k denotes the set of harmonic Ritz vectors computed in the FGMRES-DR algorithm. When analyzing both algorithms we will suppose that identical preconditioning operators are used in steps 10a and 10b i.e.

∀i, ∀j ∈ {k

+ 1, • • • , m}, M (i) j (.) = M (i) j (.) . ( 5
0) 0 = x 0 + Zmy * , r (0) 0 = b -Ax (0) 0 = V m+1 (c -Hmy * ), W m = V m 4: while r (i) 0 > b × tol do i ← i + 1 FGCRO-DR 5a: (
Compute k harmonic Ritz vectors of AZmW † m with respect to range (Wm) and store them in

Y k . Define P k such that Y k = W m P k . 6a: Q R = Hm P k 7a: W k = W m P k R -1 8a: V k = V m+1 Q 9a: Z k = Z m P k R -1 10a: Apply m -k flexible preconditioned Arnoldi steps with (In -V k V H k )A and v k+1 = r (i-1) 0 / r (i-1) 0 such that (In -V k V H k )A z k+1 , . . . , zm = v k+1 , . . . , v m+1 Hm-k with z j = M (i) j (v j ) 11a: Set Hm = I k V H k AZ m-k 0 m-k+1×k Hm-k yielding A z 1 , . . . , zm = v 1 , . . . , v m+1 Hm and define W m = W k Vm(1 : n, k + 1 : m) FGMRES-DR 5b:
Compute k harmonic Ritz vectors of AZmV H m with respect to range (Vm) and store them in

Y k . Define P k such that Y k = V m P k . 6b: QR = P k 0 1×k c -Hmy * 7b: Hk = Q H HmQ( 1 : m , 1 : k) 8b: V k+1 = V m+1 Q 9b: Z k = ZmQ( 1 : m , 1 : k) 10b:
Apply m -k flexible preconditioned Arnoldi steps with A and v k+1 while maintaining orthogonality to 13:

V k such that A z k+1 , . . . , zm = v k+1 , . . . , v m+1 Hm-k with z j = M (i) j (v j ) and V H m+1 V m+1 = I m+1 11b
x (i) 0 = x (i-1) 0 + Zmy * 14: r (i) 0 = b -Ax (i) 0
15: end while

Equivalent preconditioning matrix

Definition 5.2.6. Equivalent preconditioning matrix. Suppose that

V p = [v 1 , • • • , v p ] ∈ C n×p and Z p = [M 1 (v 1 ), • • • , M p (v p )] ∈ C n×p
obtained during a cycle of a flexible method with (standard or deflated) restarting (with 1 ≤ p ≤ m < n) are both of full rank i.e. rank V p = rank Z p = p. We will then denote by M Vp ∈ C n×n a nonsingular equivalent preconditioning matrix defined as

Z p def = M Vp V p .
(5.2.21) Such a matrix represents the action of the nonlinear operators M j on the set of vectors v j (with j = 1, • • • , p). It can be chosen e.g. as

M Vp = [Z p Z p ][V p V p ] -1
where Z p (respectively V p ) denotes an orthogonal complement of Z p (respectively V p ) in C n .

A numerical illustration

In this section we intend to illustrate the results shown in Section 5.2.10 and 5.2.10 on a simple numerical example. We consider a real symmetric positive definite matrix A = Q D Q T of size 200 with Q orthonormal and D diagonal with entries ranging from 10 -4 to 1. The spectrum of A contains eigenvalues of small magnitude1 and consequently the use of deflation techniques should improve the convergence rate of Krylov subspace methods if the harmonic Ritz values of smallest modulus are taken into account.

In this experiment we consider a polynomial preconditioner represented by two iterations of unpreconditioned GMRES for the solution of Ax = b with b given by b = Ae Ae 2 (e ∈ R 200 denoting the vector with all components equal to one) starting from a zero initial guess. Figure 5.1 shows the histories of convergence of various flexible methods minimizing over a subspace of same dimension i.e. respectively FGMRES [START_REF] Baker | An efficient block variant of GMRES[END_REF], FGMRES-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF], FGCRO-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] and full flexible GMRES with such a variable preconditioner. Flexible methods with deflated restarting are found to be efficient since they are close to the full flexible GMRES method in terms of performances. We also remark that the convergence histories of FGCRO-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] and FGMRES-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] are different. According to Corollary 5.2.12 we compute the scalar product of v k+2 and v k+2 (which are both vectors of unit norm) to determine the cosinus of the angle between these two vectors. The values are reported in Table 5.1 for the first five cycles. With such a variable preconditioner it is found that the methods are not equivalent in the first cycle already since the collinearity condition between v k+2 and v k+2 is not fulfilled. The situation is similar during the next cycles and it explains why different convergence histories for FGMRES-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] and FGCRO-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] observed in Figure 5.1 are obtained in such a case. As expected from Section 5.2.10, if a fixed right preconditioner is used, the convergence histories of GMRES-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF] and GCRO-DR [START_REF] Baker | An efficient block variant of GMRES[END_REF][START_REF] Axelsson | Iterative solution methods[END_REF]) are found to be exactly similar (results not shown here). In such a case v k+2 and v k+2 fulfill the collinearity condition; this is confirmed in Table 5.1 when a diagonal preconditioning is used.

Cost of a cycle

We summarize in Table 5.2 the main computational costs associated with each generic cycle of FGMRES-DR(m, k) and FGCRO-DR(m, k). In FGCRO-DR(m, k), an Arnoldi method based on the modified Gram-Schmidt procedure has been assumed 2 . We have only included the costs proportional to the size of the original problem n which is supposed to be much greater than m and k. We denote respectively by op A and op M the floating point operation counts for the matrix-vector product and the preconditioner application.

= (m -k)(opM + opA) + n(2(m + 1)k + 1 + 2mk + (m - k)(2m + 2k + 6)). Computation of FGMRES-DR(m, k) FGCRO-DR(m, k) Vm(:, 1 : k + 1) 2n(m + 1)(k + 1) 2n(m + 1)k + n Zm(:, 1 : k) 2nmk 2nmk
Vm(:, k + 2 : m + 1) (m -k)op A + n(m -k)(2m + 2k + 5) (m -k)op A + n(m -k)(2m + 2k + 6) Zm(:, k + 1 : m) (m -k)op M (m -k)op M Total cost C + n (m + k + 1) C
The generalized eigenvalue problem in FGCRO-DR(m, k) has been ignored in Table 5.2 since it can be performed at a cost independent of n as outlined in Section 5.2.12. Furthermore the computation of c (required at step 12 of Algorithm 5.2.1) has not been included in Table 5.2 since in both methods it can be obtained at a cost independent of n (see Proposition 3 in [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF] for FGMRES-DR). From Table 5.2 we deduce that FGCRO-DR(m, k) requires slightly less operations per cycle than FGMRES-DR(m, k).

Storage requirements

We only consider the storage proportional to the size of the original problem n. Similarly as in FGMRES-DR(m, k) (see [58, Section 3.2.2]), if the matrix multiplications V m+1 Q and Z m P k R -1 at steps 8a and 9a of Algorithm 5.2.1 are performed in place (i.e. respectivel overwriting V k and Z k y), FGCRO-DR(m, k) only requires the storage of Z m and V m+1 which corresponds to (2m + 1) vectors of length n. The same storage cost is needed in FGMRES-DR(m, k) as detailed in [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF].

Solution of sequence of linear systems

As advocated in [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF], GCRO-DR(m, k) is suited for the solution of a sequence of slowly changing linear systems defined as A l x l = b l where both the matrix A l ∈ C n×n and the right-hand side b l ∈ C n change from one system to the next, and the linear systems may typically not be available simultaneously. Next, we analyze how subspace recycling can be used in FGCRO-DR(m, k). We suppose that FGCRO-DR(m, k) has been applied for the solution of a given linear system (indexed by s -1) in this sequence and that appropriate subspaces to be recycled Z s-1 k and W s-1 k have been selected during a given cycle. As explained in Proposition 5.2.5, the relations

A s-1 Z s-1 k = V s-1 k with V s-1 k H V s-1 k = I k and range(W s-1 k ) = range(V s-1 k
) are then supposed to hold. Proposition 5.2.14 details how to consider subspace recycling in the initial phase of FGCRO-DR(m, k), when solving the new linear system A s x s = b s with x 0 as an initial guess. Proposition 5.2.14. Suppose that Z s-1 k and W s-1 k are defined from solving a previous linear system A s-1 x s-1 = b s-1 with FGCRO-DR(m, k) and that A s x s = b s is the new linear system to be solved. In the initial phase of flexible GCRO with deflated restarting and subspace recycling, the relation

A s Z s k = V s k with V s k H V s k = I k holds with matrices V s k , Z s k ∈ C n×k defined as V s k = Q, Z s k = Z s-1 k R -1 with QR = A s Z s-1 k where Q ∈ C n×k has orthonormal columns and R ∈ C k×k is upper triangular. In addition we define W s k ∈ C n×k as W s k = W s-1 k R -1 .
Proof. By using information related to the reduced QR factorization of A s Z s-1 k and respectively the relation

A s-1 Z s-1 k = V s-1 k , we easily obtain A s Z s k = A s Z s-1 k R -1 = Q, = V s k . Since Q has orthonormal columns, V s k satisfies V s k H V s k = I k . Finally W s k = W s-1 k R -1
is imposed to make sure that the relation shown in Lemma 5.2.8 will hold at the end of the initial phase of FGCRO-DR(m, k) with subspace recycling. are defined from solving a previous linear system

A s-1 x s-1 = b s-1 and satisfy A s-1 Z s-1 k = V s-1 k with V s-1 k H V s-1 k = I k and range(W s-1 k ) = range(V s-1 k ); 2 r 0 = b s -A s x 0 ; 3 Q R = A s Z s-1 k ; 4 V s k = Q; 5 Z s k = Z s-1 k R -1 ; 6 W s k = W s-1 k R -1 ; 7 x (0) 0 = x 0 + Z s k V s k H r 0 ; 8 r (0) 0 = r 0 -V s k V s k H r 0 ; 9 Apply m -k flexible preconditioned Arnoldi steps with (I n -V s k V s k H )A s and v s k+1 = r (0) 0 / r (0) 0 such that (I n -V s k V s k H )A s z s k+1 , . . . , z s m = v s k+1 , . . . , v s m+1 Hm-k with z s j = M (i) j (v s j ); 10 d * = arg min d∈Z s m r (0) 0 -A s d , x (1) 0
= x (0) 0 + d * , r (1)
0 = b s -A s x (1) 0 ; 11 W s m = W s k V s m (1 : n, k + 1 : m) ;
In the case of a sequence where only the right-hand sides are changing, we note that the reduced QR factorization (step 2 in Algorithm 5.2.2) is not required. The complete construction of the initial generation of subspaces V s m+1 , Z s m , W s m is sketched in Algorithm 5.2.2. Once V s m+1 , Z s m and W s m have been obtained, the main cycle of FGCRO-DR(m, k) (lines 4 to 15 of Algorithm 5.2.1) can be applied straightforwardly. Subspace recycling can thus be easily used in FGCRO-DR(m, k) to solve sequences of linear systems.

A numerical illustration

As a numerical illustration we consider sequences of linear systems arising from the finite difference discretization of multidimensional elliptic partial differential equations (isotropic Laplace operator) posed on the [0, 1] d hypercube with homogeneous Dirichlet boundary conditions. These sequences correspond to situations where only the right-hand sides are changing for a given dimension d. An efficient solution method is of primary interest in certain applications related to e.g. financial engineering, molecular biology or quantum dynamics [START_REF] Beylkin | Algorithms for numerical analysis in high dimensions[END_REF][START_REF] Zubair | Multigrid for high dimensional elliptic partial differential equations on nonequidistant grids[END_REF]. In the numerical experiments reported here (performed in Matlab) we have used second order finite difference discretization schemes leading to sparse matrices with at most 2d + 1 nonzero elements per row. We analyze the performances of various flexible methods used with four iterations of unpreconditioned GMRES as a preconditioner. This polynomial preconditioner is a variable nonlinear function which thus requires a flexible Krylov subspace method as an outer method [START_REF] Simoncini | Theory of inexact Krylov subspace methods and applications to scientific computing[END_REF]. Table 5.3 collects the number of matrix-vector products of some flexible methods minimizing over a subspace of same dimension respectively i.e. FGMRES [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF], FGMRES-DR [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF][START_REF] Baker | An efficient block variant of GMRES[END_REF], FGCRO-DR [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF][START_REF] Baker | An efficient block variant of GMRES[END_REF] and FGCRO-DR(20,10) with subspace recycling. Using deflation helps to improve the convergence rate of flexible GMRES in this application since a reduction of approximately 20% to 25% in terms of matrix-vector products is obtained for FGMRES-DR(20,10) independently of the dimension d. FGCRO-DR(20,10) leads to numbers of matrix-vector products which are similar to FGMRES-DR [START_REF] Brossier | Imagerie Sismique à Deux Dimensions des Milieux Visco-Élastiques par Inversion des Formes d'ondes: développements méthodologiques et application[END_REF][START_REF] Baker | An efficient block variant of GMRES[END_REF] although the convergence histories are found to be different. Finally using both deflation and recycling in FGCRO-DR leads to a significant decrease in terms of matrix-vector products. A reduction in the range of 40% to 45% is indeed obtained versus another flexible Krylov subspace method with deflated restarting (FGMRES-DR(m, k)). This can be considered as a primary advantage over FGMRES-DR(m, k) since FGMRES-DR(m, k) does not allow subspace recycling. It nicely extends to the flexible setting the advantage of GCRO-DR versus GMRES-DR previously illustrated in [START_REF] Parks | Recycling Krylov subspaces for sequences of linear systems[END_REF]. We note that the resulting method is factorization free and mostly relies on matrix-vector products, a nice feature if distributed memory platforms are targeted to address numerical problems of larger size in higher dimension.

Conclusion and perspectives

In this section we have studied a new minimum residual norm subspace method with deflated restarting that allows flexible preconditioning based on the GCRO subspace method. The resulting method named FGCRO-DR has been presented together with FGMRES-DR, a recently proposed algorithm of the same family but based on the GMRES subspace method. A theoretical comparison analysis of both algorithms has been performed in Subsection 5.2.7, where Theorem 5.2.11 -the main result of this section -proves the algebraic equivalence of FGMRES-DR and FGCRO-DR if a certain collinearity condition holds at each cycle. Corollary 5.2.13 has also proved that GMRES-DR and GCRO-DR are algebraically equivalent when a fixed right preconditioner is used. Furthermore we have carefully analyzed the computational cost of a given cycle of FGCRO-DR and have shown that FGCRO-DR is nearly as expensive as FGMRES-DR in terms of operations. FGCRO-DR offers the additional advantage to be suitable for the solution of sequences of slowly changing linear systems (where both the matrix and right-hand side can change) through subspace recycling.

In [START_REF] Carvalho | A flexible generalized conjugate residual method with inner orthogonalization and deflated restarting[END_REF] variants of FGCRO-DR have been proposed which only differ in the formulation of the projected generalized eigenvalue problem. As future work we plan to investigate the numerical properties of these variants on realistic problems of large size for both single and multiple left or right-hand side situations. Of interest are applications related to e.g. steady or unsteady simulations of nonlinear equations [START_REF] Carpenter | A general algorithm for reusing Krylov subspace information[END_REF] or stochastic finite element methods [START_REF] Eiermann | Computational aspects of the stochastic finite element method[END_REF][START_REF] Ullmann | Solution Strategies for Stochastic Finite Element Discretizations[END_REF] in three dimensions where variable preconditioning using approximate solvers has to be usually considered. We also note that when all right-hand sides are available simultaneously and when the matrix is fixed, block subspace methods may be also suitable. Thus a perspective could be to propose a block variant of FGCRO-DR.

Chapter 6

Conclusions

The earth imaging problem related to geophysics consists of one of the most difficult problems in today's high performance computing community. The challenges are spread on several domains as geophysics, optimization, numerical analysis, among others. In this thesis we focused on aspects concerning the forward problem arising from the full waveform inversion technique, most notably the multi source scenario. We focused on the solution of very challenging large sparse non-Hermitian linear systems arising from the discretization of the three-dimensional Helmholtz equation. Due to the large size of the problem we chose then the use of Krylov iterative solvers which are known for optimal memory control and good scalability in massively parallel environment, but require flexible preconditioners based on multigrid techniques to be efficient in terms of computational cost. The goals of this thesis were originally the following: develop a flexible block Krylov iterative solver able to efficiently handle the multiple right-hand sides situation, investigate subspace recycling possibilities for accelerating the convergence when multiple models are used, and the implementation of a modular software using object orientation paradigms (specially polymorphism) for solving the proposed problem using the method we developed in a massively parallel environment .

Regarding the development of a flexible block Krylov iterative solver, we focused on a GMRES based method due its minimal residual property and monotonically decreasing behaviour of the norm of the residual. In a first moment we extended the needed theory on flexible Krylov solvers for the multiple right-hand sides in order to obtain a detailed and unified vision on this scenario. The conclusions we drew from this extension greatly influenced and facilitated the understanding of the multiple right-hand sides case, to later on exploit deflation techniques. Deflation techniques are recognized to be necessary when using Krylov solvers for multiple right-hand sides. We were then able to extend the deflation techniques already available in the literature and finally propose a variant of a BGMRES based block Krylov subspace method able to perform deflation at the beginning of every iteration based on information associated with the singular values of the scaled residual, a method we named in this thesis deflated minimal block residual, or DMBR. To justify the new variant, we performed numerical experiments on both small academical cases and on large real life application. Most notably, we compared DMBR with other methods known to be efficient for solving the forward problem arising from the full waveform inversion application. In this comparison using the SEG/EAGE Overthrust velocity field, we observed a speedup in terms of computational time of up to 30%, for low frequency cases using hundreds of right-hand sides.

We implemented an object oriented code using FORTRAN03 targeting performance in a massively parallel environment using standard MPI partitioning, as well as being as close as possible to FORTRAN90 programming. This was achieved by using only the most basic FORTRAN03 object orientation features, which are ultimately encapsulated and thus not imposing any software engineering knowledge upon the final user. The code is flexible enough to accommodate changes without requiring rewriting of current routines. The numerical experiments in this chapter show not only the interest of using such a software in large scale real life application, but we also reinforced the conclusion that deflation techniques are essential when using block Krylov solvers for the multisources scenario.

Later on we investigated a family of flexible methods performing deflated restart for the single right-103 hand side case, a technique to accelerate the convergence of inner-outer iterative methods using harmonic Ritz values. We have discussed the similarities between the recently proposed FGMRES-DR with the method we then proposed, FGCRO-DR. We show that although in the fixed preconditioner setting both methods are algebraically equivalent, in the flexible setting this is not true unless a collinearity condition holds for every iteration, a phenomena unlike to happen in practice. We verified with numerical experiments the different convergence history of the two methods. Later on we showed that when using FGCRO-DR, it is possible to recycle information when solving sequence of linear systems, a feature not present in FGMRES-DR. We performed numerical experiments on simple academic problems showing that in this case the gain obtained through subspace recycling can be large. Therefore, we consider the enhancements we proposed for the iterative solver are significant for the solution of the forward problem arising from the full waveform inversion optimization problem. However, some further investigation could bring an even more significant improvement. Whereas the strategy for handling multiple right-hand sides has shown great performance for hundreds of right-hand sides, in the earth imaging scenario often it is required the solution of tens of thousands of right-hand sides. For solving problems with such a large number of right-hand sides, we consider in our future research to investigate the combination of recycling strategies with deflation strategies, that is, partitioning the tens of thousands of right-hand sides into "clusters" of hundreds of right-hand sides, solving each cluster using block strategies and then using recycling strategies to recycle information from the solution of one cluster to improve the convergence of the solution of the next cluster of right-hand sides.

Another possibility for our future research is the use of block and subspace recycling strategies inside the preconditioner. For the moment, only our solver uses block and deflation strategies, whereas the preconditioner performs independent iterations of GMRES for each right-hand side. It was not yet investigated in the literature the use of block and deflation techniques in the coarse level or smoother of the geometric multigrid. Likewise, we are unaware of any result related to the use of block and deflation techniques in the geometric multigrid cycle preconditioner when using complex shifted Laplacian operator, a result that could be indeed very interesting for the inverse problem. Similarly, subspace recycling could greatly speed up the solution on coarse levels, which remains the most expensive part of the preconditioner.

Additionally, in our future research we would like to consider advanced high-order accurate schemes and discontinuous Galerkin discretization techniques as well as other PDE formulations of the forward problem (e.g. visco-elastic equation). Notably the recent reformulation of the Helmholtz equation using the Rytov decomposition seems attractive to us, as our academical numerical experiments involving the solution of the advection-diffusion equation showed the interest of using deflation techniques for this scenario. Since the convergence of iterative solvers is highly dependent on the numerical properties of the coefficient matrix being used, it is of our interest to investigate the behaviour of both recycling strategies and block deflation strategies in this scenario. In the same sense, although in this thesis we investigate the behaviour of block Krylov iterative solvers when using a two-level multigrid preconditioner, we would like to investigate in the near future the deflation and recycling behaviour for Helmholtz equation when using non-multigrid based preconditioners, such as the sweeping preconditioner, CARP-CG, or low-rank approximation techniques as the characteristics of the properties of the preconditioned Krylov subspace depends intrinsically on the matrix and (variable) preconditioner particularities.

We then extend C_Matrix to a type C_CSR, a very common matrix format. In this extension we finally add the necessary information for storing the matrix: an array of nonzero elements, array of index of first nonzero element of each row, etc. We then implement the matrix_vector procedure for performing a matrix-vector product using these structures.

Aside, we could extend C_Matrix to another type, for instance, C_HarwellBoeing. Similarly we add the structures for storing the matrix in this new type, and we implement the matrix-vector routine for this data type format. Now that we have both these types implemented, it would make a lot of sense to pass to GMRES a type C_Matrix instead of C_CSR or C_HarwellBoeing. The point is that GMRES does not need to know how the matrix-vector product is performed, it just needs to know it can be performed and which arguments it needs to pass to the matrix_vector routine.

This was the main idea behind the libraries and modules we proposed. The iterative solver receives a class which acts like a matrix, and then uses it to perform matrix vector products. Additionally, the solver itself is a class, and it uses another solver class as preconditioner. For instance, BGMRES can be preconditioned by BGMRES itself.

The following sections are dedicated to the specification and description of the libraries and their basic usage.

Notice that since every T_Error has a pointer to another T_Error, it is possible to create a complex stack of errors showing where the error originally occurred. As we are going to mention later, the function Print_Error will print every error in the stack recursively. The error id, message, and everything else is purely optional. However, the following advices should be kept in mind, as following always the same standards is normally a good idea: 6 digits rule : Create error IDs with exactly 6 digits. Probably no one has a code with more than 999.999 errors possibilities; Better to keep this limit! Unique ID : Do not use the same ID for different circumstances, even if the error message is the same. If your error message will be print on the screen and the error ID is unique, we can search the ID in the source code; if it is not unique, we will have to figure out which part of the code threw that error.

Precompiler : In the libraries we write, the error handling is done in a separate file. For instance m_diag7pts.err contain the error handling of m_diag7pts.f90. This is because when reading m_diag7pts.f90 we are usually not interested in reading every single possibility of error in a given function. To clean the code, we decided to use:

IF (this.NE.that) ERROR_012345

where ERROR_012345 is a macro defined in m_diag7pts.err. Note that since the IF is still declared inside m_diag7pts.f90, whoever reads this file will know that if "this" is not equal to "that", an error will be thrown; and if the reader is interested in knowing more about this error, he can easily access m_diag7pts.err and read the full handling of this situation.

Severe Error: By default, the errors are printed only if they happen in the master process -a good idea if you don't want to read "Failed to initialize" once for every processor. However, some errors are quite severe and/or can happen in individual processes. For these errors, just put a negative ID number. These are treated as "severe errors" and can be printed by any process, not necessarily the master.

Fatal: Not every error aborts the run. Some of them might be even expected. For the error abort the run, the variable IsFatal Errors that should normally be fatal: -Dimension Mismatch: something should be bigger or smaller than it is; if this error was detected, then it is avoiding a segmentation fault. The normal strategy is to print the dimensions found, the ones expected and abort the run immediately. -Not enough memory: another one that prevents a disaster. This error should always be fatal as after trying to allocate something the code usually expects to use it.

Module M_Class

The most primitive classes that is implemented in this and the subsequent libraries. Every single class we implemented extends this one (or extends a class which extends this one) and must follow the shape established here.

! ----------------------------------------------------------------------!

Module M_Standard27pts

This module contains the definition of a standard implementation of a 27 point stencil operator. So far it has been tested with Operto et. al discretization scheme [START_REF] Operto | 3d finitedifference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: A feasibility study[END_REF].

TYPE, EXTENDS(C_StencilCollection) C_Standard27pts

Description: This is the class for storing the the almost matrix-free 7 point stencil operator, tuned for using with the Helmholtz operator with PML boundary conditions.

Important Components:

COMPLEX(27,:,:,:) A For each point (i, j, k), A(:, i, j, k) contains the coefficient of the central point plus the coefficients of its 26 neighbours. Important Procedures:

Apply(y,yLen,x,xLen,p) IMPLEMENTED Implements the deferred procedure inherited from C_Operator In this library we implement several iterative solvers as well as preconditioners. Thanks to polymorphism, every solver implemented here can also be used as a preconditioner.

! ----------------------------------------------------------------------! Suppose that the following is known: ! MyID: the rank of current MPI process ! CommHandler: the communication handler ! MemAvailable: current memory available ! fill_matrix(): a subroutine which receives any C_Operator ! and fills it coefficients with some values ! ------------------------------------------------------------------

Module M_Solver

This module contains the definition of the abstract C_Solver class. This is the class which is going to be used to create iterative solver classes and routines later.

TYPE, ABSTRACT, EXTENDS(C_Class) C_Solver

Description:

The solver class has to be as flexible as possible to accommodate every possible future implementation, of an iterative solver, therefore only the strictly generic entities are declared here. For the sake of generality also, the preconditioner is as well considered as a solver in the sense that they also come from a C_Solver class. In a bigger picture, solvers can be seen as a function; they receive a right-hand side b, an initial guess x, and a coefficient matrix A to then compute x = f (x, b), which is supposedly an approximation of A -1 b. So far, nothing prevents the use of this class for implementing direct solvers, but this has never been the original intention.

Important Components:

C_Operator POINTER MyOperator This object defines several internal values for the solver. For instance, the allocation of the vectors V j in BFGMRES are dependant of MyOperator%LocalLen and MyOpera-tor%CommLen INTEGER Maxp

Maximum number of right-hand sides that this solver is going to deal with at once. Used to compute the maximum size of some structures before allocating it.

LOGICAL IsVariable

Every object which inherits from C_Solver should be considered as a flexible solver by default. Set this flag to .FALSE. in a specific solver if this is not true, since sometimes it is possible to optimize memory cost and operations when using fixed preconditioner. Important Procedures:

Apply(x,xLen,b,bLen,p) DEFERRED As in the C_Operator type, this is the core of the C_Solver class. It simply starts the solver execution.

Recalling the inheritance that we discussed earlier, since C_Solver extends C_Class, it also possesses the component IsVerbose. This flag is particularly useful for the solver, although it is also present in any C_Operator, for instance. If it is active, the solver will print the history of the relative residual on the screen, for instance. Also, because C_Solver extends C_Class, it also possesses the Initialize() function (which was deferred in C_Class and is also deferred in C_Solver).

Module M_BFGMRES

This module contains the first implementation of a C_Solver, the C_BFGMRES.

Notice that C_BFGMRES, as the C_Diag7pts and C_Standard27pts implement both Initialize() and Destroy() functions from C_Class. Likewise, in this module some extra functions for handling the new TYPE, ABSTRACT, EXTENDS(C_Solver) C_BFGMRES

Description:

The first solver we implement. It consists of BFGMRES method proposed by Vital [START_REF] Vital | Étude de Quelques Méthodes de Résolution de Problèmes Linéaires de Grande Taille Sur Multiprocesseur[END_REF]. It requires some extra parameters to be allocated, like the tolerance and the restart size. The stopping criterion of this solver is fixed: it always stops when the . ψ of the relative residual is smaller than the tolerance, or when the maximum number of cycles has been performed.

Important Components:

C_Solver POINTER MyPC This object defines the preconditioner to be used by this C_BFGMRES. Notice that it could be C_BFGMRES itself, since C_BFGMRES is also a C_Solver. If no preconditioner is going to be used, one can use the C_NoSolver which we describe later in the module M_NoSolver.

INTEGER MaxCycles

Maximum number of cycles to be performed. The solver stops and returns its final approximation of the solution even if it did not converge INTEGER MaxzDim BFGMRES will restart whenever the number of iterations reach MaxRestartSize or whenever the dimension of the subspace Z j reaches MaxzDim, whichever comes first. INTEGER MaxRestartSize BFGMRES will restart whenever the number of iterations reach MaxRestartSize or whenever the dimension of the subspace Z j reaches MaxzDim, whichever comes first.

REAL Tol

Convergence threshold for BFGMRES. It will return whenever the . ψ of the relative residual is smaller than Tol or MaxCycles have been performed. The first one is used when no preconditioner is desired, the second one uses one iteration of local symmetric Guass-Seidel iteration (to be covered later with more details in the M_GaussSeidel) and the later uses 10 cycles of unpreconditioned BGMRES(5) as preconditioner. Alternatively, instead of providing the O_PCToCreate, one can provide a pointer to a C_Solver, which is then going to be used as preconditioner. In Figure A.9 we illustrate how to use this situation.

! ----------------------------------------------------------------------

! Suppose that the following is known: ! MemAvailable: current memory available ! Matrix: a C_Operator with the coefficient matrix already ready for use ! vecx: a (xlen,p) vector containing the initial guess ! vecb: a (blen,p) vector containing the right-hand side ! ------------------------------------------------------------------ ! --------------------------------------------------------------------- --------------------------------------------------------------------

! ----------------------------------------------------------------------! Suppose that the following is known: ! MemAvailable: current memory available ! Matrix: a C_Operator with the coefficient matrix already ready for use ! vecx: a (xlen,p) vector containing the initial guess ! vecb: a (blen,p) vector containing the right-hand side ! ------------------------------------------------------------------

3. 1

 1 Computational cost of a cycle of DMBR(m) . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Numerical experiment: Poisson problem with 5 cycles of BGMRES(5) as variable preconditioner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Numerical experiment: Poisson problem with 3 cycles of BGMRES(3) as variable preconditioner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Numerical experiment: Poisson problem with 5 cycles of BGMRES(5) as variable preconditioner and k max = 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Numerical experiment: convection-diffusion problem with 5 cycles of BGMRES(5) as variable preconditioner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Numerical experiment: convection-diffusion problem with 3 cycles of BGMRES(3) as variable preconditioner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Numerical experiment: convection-diffusion problem with 5 cycles of BGMRES(5) as variable preconditioner and k max = 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Numerical experiment: two-dimensional complex-valued advection diffusion problem . . . . 3.9 Numerical experiment: acoustic full waveform inversion . . . . . . . . . . . . . . . . . . . . 4.1 Numerical experiment: SEG/EAGE Salt Dome smoothing . . . . . . . . . . . . . . . . . . . 5.1 Scalar product v T k+2 v k+2 during the first five cycles of FGCRO-DR . . . . . . . . . . . . . . 5.2 Computational cost of a generic cycle of FGMRES-DR and FGCRO-DR . . . . . . . . . . . 5.3 Numerical experiment: Solution of a d-dimensional elliptic partial differential equation problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Chapter 1

2 andX

 2 * as the exact solution for AX = B. We use I k ∈ C k×k to denote the identity matrix of dimension k and 0 i×j ∈ C i×j the zero rectangular matrix with i rows and j columns. The superscript H denotes the transpose conjugate operation. Given a vector d ∈ C k with components d i , D = diag(d 1 . . . d k ) is the diagonal matrix D ∈ C k×k such that D ii = d i . If C ∈ C k×l we denote the singular values of C by σ 1 (C) ≥ • • • ≥ σ min(k,l) (C) ≥ 0.

Figure 2 . 1 :

 21 Figure 2.1: Representation of a block lower Hessenberg matrix with a band of p elements in its lower off-diagonal.

Figure 2 . 2 :

 22 Figure 2.2: Representation of Hj generated by Algorithm 2.5.1 after the occurrence of a partial breakdowns at iteration i and anther in iteration k.

Algorithm 3 . 3 . 1 : 7 Choose the scalar 1

 33171 Restarted Flexible DMBR 1 Choose an initial guess X 0 ∈ C n×p , a restart parameter m and define a convergence criterion and its scaling matrix; 2 for cycle = 1, . . . , m do 3 Compute the initial block residual R 0 = B -AX 0 ; 4 V1 Λ0 = R 0 (thin QR decomposition) and determine p 0 = rank (R 0 ), with V1 ∈ C n×p0 and Λ0 ∈ C p0×p ; 5 Define s 0 = 0 and H 0 Y 0 = 0 p0×p ; 6 for j = 1, . . . , m do

  ×p as the unique minimal Frobenius norm solution to the problem min Y ∈C s j ×p Λj -Ĥj Y F (3.3.2) ; if full convergence detected (see Definition 2.8.1) then break; ; end for X 0 = X 0 + Z m Y m ; end for Thanks to Lemma 3.3.1 we can write the following proposition. Corollary 3.3.2 (Block Residual Minimization). Consider that j iterations of Algorithm 3.3.1 have been carried out in a given cycle for any chosen sequence of unitary matrices

Proposition 3 . 4 . 3 .

 343 After j iterations of Algorithm 3.3.1 with Algorithm 3.4.1 for choosing each

Proposition 3 . 4 . 4 .

 344 After j iterations of Algorithm 3.3.1 with Algorithm 3.4.1 for choosing each

Corollary 3 . 4 . 7 .

 347 For every iteration j > 1 of DMBR (Algorithm 3.3.1) with Algorithm 3.4.1, if D i = D for every 1 ≤ i ≤ j for a given square matrix D ∈ C p×p , it holds that k j ≥ k j-1 . This property also holds along the cycles of DMBR.Proof. This corollary is an implication of the choice of each k j in Algorithm 3.4.1 and Proposition 3.3.4.

Algorithm 3 . 5 . 3 :

 353 Comparison between DMBR and BGMRES-R1 Choose X 0 , m and a convergence criterion with its scaling matrix; 2 for cycle = 1, . . . , m do 3 Compute R 0 = B -AX 0 ; 4 V1 Λ0 = R 0 (thin QR decomposition); DMBR 5a Define k 1 = p 0 and V 1 = V1 ; 6a for j = 1, . . . , m do 7a Choose k j and F j ; 8a

1

  Algorithm 3.2.1: the block Arnoldi method based on modified Gram-Schmidt requiresm j=1 j i=1 (4nk i k j + nk j + 4nd j k j ) operations (line 4 to 5) plus m j=1 2nk 2 j operations for the QR decomposition of S (line 6). Thus C j =

Figure 3 . 1 :

 31 Figure 3.1: Acoustic full waveform inversion (SEG/EAGE Overthrust model). Case of p = 32. Evolution of kj versus iterations for p = 32 in BFGMRES(5), BFGMRESD(5) (top, left part), BFGMRES-R(5) (top, right part), DMBR(5) (bottom, left part) and truncated variants (BFGMREST(5,p/2), DMBR(5,p/2)) (bottom, right part).

Figure 4 . 1 :

 41 Figure 4.1: A source for acoustic waves s propagates waves (represented in green) through the subsurface. When meeting a reflective layer (in grey) these waves are reflected back (represented in blue) to the surface, and are detected by the geophones g.

  Figure 4.2 for an example of a three-dimensional velocity model.

Figure 4 . 2 :

 42 Figure 4.2: Graphical representation of the three-dimensional academic velocity model SEG/EAGE Overthrust generated using Paraview [69]. The image on the left represents the whole velocity model, and the image on the right represents a cut of the original velocity model showing its interior.

Figure 4 .

 4 3 shows an illustration of the difference between the data gathered by the geophones in the real experiment (left) and the simulation performed by the propagator on the synthetic velocity model (right).

Figure 4 . 3 :

 43 Figure 4.3: On the left: acoustic waves propagated by a source s are reflected by a reflective layer (in grey) and are detected by the geophones g. On the right: the propagator simulates the behaviour of the waves through the synthetic velocity model. Since the information reaching the geophones is considerably different from the observed data, the inverse problem should decide to update the synthetic velocity model in order to obtain a better approximation.

  Figure 4.4 for a two-dimensional graphical representation).

Figure 4 . 4 :

 44 Figure 4.4: A graphical representation of the PML in two-dimensions. Left: a slice of the SEG/EAGE Salt dome velocity model, defining Ω. Right: the wave propagation on the extended domain Ωe containing the PML, with Ω being represented as the red box.

Figure 4 . 5 :

 45 Figure 4.5: Graphical representation of three-dimensional uniform finite difference Cartesian stencils.

Figure 4 . 7 :

 47 Figure 4.7: Pattern of the coefficient matrix arising from the discrete formulation of the Helmholtz equation with PML in a Cartesian uniform grid with a 7-point stencil (4.3.9)

Figure 4 . 8 :

 48 Figure 4.8: Image taken from [93]: combination of several 7-point stencil resulting in a 27-point parsimonious staggered-grid scheme.

Figure 4 . 9 :

 49 Figure 4.9: The academic SEG/EAGE Salt dome velocity model and the value of h related to f , considering that n λ = 12. Notice that if f = 10Hz, n = O(10 9). The grid size described here does not take the PML layer into account.

Figure 4 . 10 :

 410 Figure 4.10: Graphical representation of a basic V -cycle geometrical multigrid. Other advanced multigrid schemes can be deduced from the basic V -cycle.

2

 2 10, and m H = 10. (4.4.1) Algorithm 4.4.1: Perturbed geometric two-level cycle to obtain the approximation Zj ∈ C n×k j to the system AZ = Vj for a given fixed right-hand side Vj ∈ C n×k j with the zero initial guess 1 Polynomial presmoothing: apply µ cycle(s) of FGMRES(m h ) to AZ = V j with initial approximation 0 n×kj and symmetric Gauss-Seidel as a right preconditioner to obtain the approximation Z µ h ; Restrict the fine level residual: R H

Figure 4 . 11 :

 411 Figure 4.11: Basic representation of polymorphism in inheritance. Inheritance guarantees that every object of the type C_Child1, C_Child2 and C_Child3 contains every component (e.g. variables, pointers, etc) that is contained in C_P arent. Polymorphism guarantees that every object of the type C_Child1, C_Child2 and C_Child3 can be treated as a C_P arent type also. For instance, a subroutine or function which requires an object of the type C_P arent as an argument, could receive also a C_Child1 or C_Child3 object instead without requiring any kind of modification in the code.

Figure 4 .

 4 13 to Figure 4.16 for graphical representations of the smoothed velocity field).

Figure 4 . 12 :

 412 Figure 4.12: Graphical representation of the interior of the three-dimensional academic velocity generated using Paraview [69]. The image on the left represents a plane cut at y = 308. The image on the right shows only the points of the domain which have velocity equal or higher than 4, 000 m/s, delineating the structure of the dome of salt.

Figure 4 .

 4 13 to Figure 4.16 (left side) show a graphical representation of SEG/EAGE Salt Dome and its different smoothed versions, along with the wavefield obtained after solving the Helmholtz equation for its respective velocity model.

Figure 4 . 13 :

 413 Figure 4.13: Original SEG/EAGE Salt dome velocity field, and its respective wavefield obtained from the solution of the Helmholtz equation for 5Hz. This shows the solution for the 8-th right-hand side.

Figure 4 . 14 :

 414 Figure 4.14: Smoothed×1 version of SEG/EAGE Salt dome velocity field, and its respective wavefield obtained from the solution of the Helmholtz for 5Hz equation. This shows the solution for the 8-th right-hand side.

Figure 4 . 15 :

 415 Figure 4.15: Smoothed×2 version of SEG/EAGE Salt dome velocity field, and its respective wavefield obtained from the solution of the Helmholtz for 5Hz equation. This shows the solution for the 8-th right-hand side.

Figure 4 . 16 :

 416 Figure 4.16: Smoothed×3 version of SEG/EAGE Salt dome velocity field, and its respective wavefield obtained from the solution of the Helmholtz for 5Hz equation. This shows the solution for the 8-th right-hand side.

Figure 4 . 17 :

 417 Figure 4.17: Evolution of kj along the iterations of DMBR(5) preconditioned by a two-level perturbed multigrid V-cycle (cf. Table 4.1) for each SEG/EAGE Salt dome velocity field and its respective smoothed versions.

Figure 4 . 18 :

 418 Figure 4.18: SEG/EAGE Salt dome velocity field, and its respective wavefield obtained from the solution of the Helmholtz equation for 12Hz. This experiment required 141.7 Gb of memory

Abstract

  This work is concerned with the development and study of a minimum residual norm subspace method based on the Generalized Conjugate Residual method with inner Orthogonalization (GCRO) method that allows flexible preconditioning and deflated restarting for the solution of non-symmetric or non-Hermitian linear systems. First we recall the main features of Flexible Generalized Minimum Residual with deflated restarting (FGMRES-DR), a recently proposed algorithm of the same family but based on the GMRES method. Next we introduce the new inner-outer subspace method named FGCRO-DR. A theoretical comparison of both algorithms is then made in the case of flexible preconditioning. It is proved that FGCRO-DR and FGMRES-DR are algebraically equivalent if a collinearity condition is satisfied. While being nearly as expensive as FGMRES-DR in terms of computational operations per cycle, FGCRO-DR offers the additional advantage to be suitable for the solution of sequences of slowly changing linear systems (where both the matrix and right-hand side can change) through subspace recycling. Numerical experiments on the solution of multidimensional elliptic partial differential equations show the efficiency of FGCRO-DR when solving sequences of linear systems.

Definition 5 . 2 . 1 .

 521 Harmonic Ritz pair. Consider a subspace U of C n . Given B ∈ C n×n , θ ∈ C and y ∈ U, (θ, y) is a harmonic Ritz pair of B with respect to U if and only if By -θ y ⊥ B U or equivalently, for the canonical scalar product, ∀w ∈ range (B U) w H (By -θ y) = 0.

,

  V m+1 , Hm and Z m the residual and matrices obtained at the end of the (i -1)-th cycle. Based on the Arnoldi-like relation (5.2.3), the deflation procedure proposed in [58, Proposition 1] relies on the use of k harmonic Ritz vectors Y k = V m P k of AZ m V H m with respect to range (V m ), where Y k ∈ C n×k and P k ∈ C m×k . In Lemma 5.2.2 shown in [58, Lemma 3.1], we recall a useful relation satisfied by the harmonic Ritz vectors. Lemma 5.2.2. In flexible GMRES with deflated restarting, the harmonic Ritz vectors are given by Y k = V m P k with corresponding harmonic Ritz values λ k . P k ∈ C m×k satisfies the following relation:

Lemma 5 . 2 . 4 .

 524 In flexible GCRO with deflated restarting, the harmonic Ritz vectors are given by Y k = W m P k with corresponding harmonic Ritz values θ k . The matrix P k = [p 1 , • • • , p k ] ∈ C m×k satisfies the following relation:

  n is then found by minimizing the residual norm b -A(x (i-1) 0 + Z m y) over the space x (i-1) 0

.2. 20 ) 5 . 2 . 1 :

 20521 Algorithm Flexible GCRO-DR(m, k) and Flexible GMRES-DR(m, k) 1: choose m, k, tol and x 0 2: r 0 = b -Ax 0 , β = r 0 , v 1 = r 0 /β, c = βe 1 , i ← 0 3: Apply FGMRES(m) to obtain Hm, Zm, V m+1 such that AZm = V m+1 Hm, y * = arg min y∈C m c -Hmy , x

1

 1 , . . . , zm = v 1 , . . . , v m+1 Hm 12: y * = arg min y∈C m c -Hm y with c = V H m+1 r (i-1) 0

Figure 5 . 1 :

 51 Figure 5.1: Convergence histories of different flexible methods applied to Ax = b where A ∈ R 200×200 is symmetric positive definite with some eigenvalues of small magnitude.

2

  In FGCRO-DR(m, k) (step 10a of Algorithm 5.2.1) the action of (In -V k V H k ) operations. In FGMRES-DR(m, k) (step 10b of Algorithm 5.2.1) the Arnoldi method based on modified Gram-Schmidt requires m j=k+1 j i=1 (4n) operations due to maintaining orthogonality to V k whereas norm computation and normalization cost m j=k+1 (3n) operations.

Algorithm 5 . 2 . 2 :1

 522 Initial generation of V s m+1 , Z s m and W s m when subspace recycling is used to solve A s x s = b s Suppose that V s-1 k , Z s-1 k and W s-1 k

!!Figure A. 6 :

 6 Figure A.6: Example of usage of C_Diag7pts, using polymorphism

Figure A. 7 :

 7 Figure A.7: Example of usage of C_Standard27pts, using polymorphism. This example is identical to Figure A.6, except that here we call New_Standard27pts instead of New_Diag7pts

  Important Procedures: Apply(x,xLen,b,bLen,p) IMPLEMENTED Implements the deferred procedure inherited from C_Solver Initialize(MemAvailable) IMPLEMENTED Implements the deferred procedure inherited from C_Class Destroy() IMPLEMENTED Implements the deferred procedure inherited from C_Class class are provided. MODULE M_BFGMRES most important functions: New_BMGRES() An interface accepting multiple types of parameter set. See Figure A.8 for usage. Most notably, we highlight here the existence of the parameter O_PCToCreate. It can receive, for the moment, one of the following values: NoSolver, GaussSeidel or BFGMRES.

Figure A. 8 :

 8 Figure A.8: Example of usage of C_BFGMRES, using no preconditioner. There is a strong similarity between this and Figure A.6 and Figure A.7 examples

Figure A. 9 :

 9 Figure A.9: Example of usage of C_BFGMRES, using a customized C_BFGMRES as preconditioner.

  -! Because this example is perfectly analogous to C_BFGMRES examples ! We just show here an example of use of the interface New_DMBR ! -

Figure A. 11 :!

 11 Figure A.11: Example of usage of C_GeoMultigrid for creating a variant of the two-level perturbed geometric multigrid preconditioner proposed in [96] (cf. Algorithm 4.4.1). In this variant we use BFGMRES instead of FGMRES for the smoothing and coarse correction steps.

Figure A. 12 :

 12 Figure A.12: Example of usage of C_GeoMultigrid for creating a three-level multigrid. Here we we the example in Figure A.11 to build the two bottom levels, and use 15 cycles of unpreconditioned BGMRES(7) as smoother on the finest level.

  last d 1 rows of Λ 1 , and set p 0 = k 1 ;

	15a	Set Λj =	Λ j 0 (k j -n j )×p	;

9a for j = 1, . . . , m do 10a Choose k j and F j ; 11a Deflate: Vj Pj-

1 = Vj Fj ; 12a Deflate: Λ j = F H j Λj-1 ; 13a Deflate: H j-1 = F H j Ĥj-1 ; 14a Apply Algorithm 3.2.1 obtaining AZ j = Vj+1 Ĥj ; 16a Solve: min Λj -Ĥj Y F ; 17a if full convergence detected then break; 18a ; 19a end for BFGMRESD 5b Choose k 1 and F 1 ; 6b Deflate: V 1 P 0 = V1 F 1 ; 7b Deflate: Λ 1 = F H 1 Λ0 ; 8b Discard P 0 and last d 1 rows of Λ 1 , and set p 0 = k 1 ; 9b for j = 1, . . . , m do 10b Choose k j and F j ; 11b Deflate: Vj Pj-1 = Vj Fj ; 12b Deflate: Λ j = F H j Λj-1 ; 13b Deflate: H j-1 = F H j Ĥj-1 ; 14b Apply Algorithm 2.5.1 obtaining

Table 3 . 2

 32

	Poisson equation -Grid : 128 × 128
		m = 5, no truncation (k max = p)
		p = 5			p = 10
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 18	115	90	2.75 19	240	190 4.08
	BFGMRES-R 22	77	47	1.45 24	139	79	1.72
	BFGMRESD 22	72	42	1.30 23	133	73	1.60
	DMBR 23	62	32	1	25	105	45	1
		p = 20			p = 40
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 16	420	320 4.46 15	760	600 5.13
	BFGMRES-R 25	260	140 1.98 26	578	298 2.60
	BFGMRESD 27	272	132 1.88 31	566	246 2.17
	DMBR 26	208	68	1	27	389	109	1
		p = 80			p = 160
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 14	1440	1120 5.65 11	2400	1760 5.35
	BFGMRES-R 26	1127	567 2.92 27	2199	1079 3.35
	BFGMRESD 29	1019	459 2.38 28	1983	863 2.70
	DMBR 28	742	182	1	28	1417	297	1

: Poisson equation discretized with h = 1/128 with 5 cycles of BGMRES(5) as variable preconditioner,

Table 3 . 3

 33

		Poisson equation -Grid : 128 × 128
		m = 15, no truncation (k max = p)
		p = 5			p = 10
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 49	275	245	2.99	52	590	520	4.67
	BFGMRES-R 55	132	97	1.21	60	221	151	1.39
	BFGMRESD 52	137	102	1.27	67	247	167	1.54
	DMBR 57	115	80	1	60	177	107	1
		p = 20			p = 40
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 48	1080	960	5.59	38	1720	1520	5.63
	BFGMRES-R 65	429	269	1.62	67	791	471	1.82
	BFGMRESD 72	472	292	1.76	73	883	523	2.02
	DMBR 68	321	161	1	70	568	248	1
		p = 80			p = 160
	It M V P	P r	ρ	It M V P	P r	ρ
	It	MVP	PC	Ratio	It	MVP	PC	Ratio
	BFGMRES 29	2640	2320	5.23	27	4960	4320	6.02
	BFGMRES-R 65	1498	858	2.02	69	2844	1564	2.29
	BFGMRESD 70	1610	970	2.27	75	3235	1795	2.63
	DMBR 67	1039	399	1	69	1907	627	1

: Poisson equation discretized with h = 1/128 with 3 cycles of BGMRES(3) as variable preconditioner,

Table 3 . 4 :

 34 Poisson equation discretized with h = 1/128 with 5 cycles of BGMRES(5) as variable preconditioner,

		p = 40			p = 80
	It M V P P r	ρ	It M V P	P r	ρ
	BFGMREST 26	531	251 2.06 27	1066	506 1.98
	DMBR 27	397	117	1	26	806	246	1
		p = 120			p = 160
	It M V P P r	ρ	It M V P	P r	ρ
	BFGMREST 40	1880	800 1.94 52	2952	1032 1.80
	DMBR 31	1359	399	1	32	1846	566	1

Poisson equation -Grid : 128 × 128 m = 5, truncation with k max = 20

Table 3 . 5

 35 This equation Convection-diffusion equation with c = d = 256 -Grid : 128 × 128 m = 5, no truncation (k max = p)

		p = 5			p = 10
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 30	185	150 1.45 29	360		1.57
	BFGMRES-R 34	156	116 1.12 32	293		1.16
	BFGMREST 33	153	113 1.10 30	265		1.06
	DMBR 32	143	103	1	31	263		1
		p = 20			p = 40
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 28	700	560 1.85 20	1000		1.74
	BFGMRES-R 30	493	353 1.17 27	867		1.29
	BFGMREST 29	474	334 1.11 30	795		1.13
	DMBR 29	441	301	1	28	734		1
		p = 80			p = 160
	It M V P	P r	ρ	It M V P	P r	ρ
	BFGMRES 16	1680	1280 1.75 11	2400	1760 1.60
	BFGMRES-R 22	1428	948 1.30 22	2539	1579 1.45
	BFGMREST 27	1407	847 1.17 22	2397	1437 1.32
	DMBR 23	1204	724	1	23	2041	1081	1

: Convection-diffusion equation, with h = 1/128 and c = d = 256, with 5 cycles of BGMRES(5) as variable preconditioner, restart size 5 and a number of right-hand sides given at once ranging from p = 5 to p = 160.

Table 3 .

 3 c = d = 256 -Grid : 128 × 128

			m = 15, no truncation (k max = p)
			p = 5			p = 10
		It	M V P	P r	ρ	It	M V P	P r	ρ
	BFGMRES	90	485	450	1.38	89	960	890	1.61
	BFGMRES-R 108	390	345	1.06	92	659	579	1.05
	BFGMRESD 102	382	342	1.05	102	692	612	1.11
	DMBR 103	365	325	1	92	631	551	1
			p = 20			p = 40
		It	M V P	P r	ρ	It	M V P	P r	ρ
	BFGMRES	72	1560	1440	1.65	45	1960	1800	1.52
	BFGMRES-R 104	1105	945	1.09	62	1519	1279	1.09
	BFGMRESD	98	1158	998	1.15	69	1554	1314	1.12
	DMBR	97	1028	868	1	65	1416 MVP	PC	Ratio
	BFGMRES	26	2320	2080	1.64	14	2560	2240	1.24
	BFGMRES-R	45	1699	1379	1.10	14	2116	1796	1.00
	BFGMRESD	60	2035	1635	1.30	14	2560	2240	1.24
	DMBR	46	1651	1251	1	14	2116	1796	1

6: Convection-diffusion equation, with h = 1/128 and c = d = 256, with 3 cycles of BGMRES(

Table 3 . 7 :

 37 Convection-diffusion equation, with h = 1/128 and c = d = 256, with 5 cycles of BGMRES(5) as

		p = 40			p = 80
	It M V P	P r	ρ	It	M V P	P r	ρ
	BFGMREST 40	1015	655 1.24	60	2240	1200 1.30
	DMBR 33	847	527	1	46	1800	920	1
		p = 120			p = 160
	It M V P	P r	ρ	It	M V P	P r	ρ
	BFGMREST 85	3860	1700 1.37 105	5620	2100 1.34
	DMBR 62	2920	1240	1	78	4280	1560	1

c = d = 256 -Grid : 128 × 128 m = 5, truncation with k max = 20

Table 3 .

 3

	Complex-valued advection diffusion problem -Grid : 128 × 128
			m = 5
			p = 4			p = 8
	Method	It	Pr	τ	It	Pr	τ
	FGMRES(5p)	75	75	1.00 155 155	1.00
	BFGMRES(5)	43 172 0.41	39 312	0.37
	BFGMRESD(5)	50	80	0.95	45 100	1.33
	BFGMRES-R(5)	39	80	0.90	39 127	0.96
	DMBR(5)	44	67 1.06 43	87	1.38
			p = 16			p = 32
	Method	It	Pr	τ	It	Pr	τ
	FGMRES(5p)	315 315 1.00 635 635	1.00
	BFGMRES(5)	26 416 0.40	17 544	0.39
	BFGMRESD(5)	49 150 1.27	45 235	0.97
	BFGMRES-R(5)	39 214 0.85	40 386	0.62
	DMBR(5)	45 121 1.42 43 181	1.14
			m = 10
			p = 4			p = 8
	Method	It	Pr	τ	It	Pr	τ
	FGMRES(10p)	75	75	1.00 155 155	1.00
	BFGMRES(10)	22	88	0.57	20 160	0.57
	BFGMRESD(10)	22	63	0.96	24 104	0.96
	BFGMRES-R(10) 23	51	1.43	23	78	1.43
	DMBR(10)	23	46 1.69 23	65	1.69
			p = 16			p = 32
	Method	It	Pr	τ	It	Pr	τ
	FGMRES(10p)	315 315 1.00 635 635	1.00
	BFGMRES(10)	17 272 0.47	16 512	0.31
	BFGMRESD(10)	26 186 0.70	30 350	0.43
	BFGMRES-R(10) 23 126 1.32	22 216	1.00
	DMBR(10)	23	98 1.63 22 156	1.29

8: Two-dimensional complex-valued advection diffusion problem. Case of h = 1/128, ω = π, αx = 1/ √ 2 , αy = 1/ √

Table 3 . 9 :

 39 Acoustic full waveform inversion -Grid : 433 × 433 × 126 Acoustic full waveform inversion (SEG/EAGE Overthrust model). Case of f = 3.64 Hz (h = 50 m),

			p = 4			p = 8			p = 16
	Method	It	Pr	T	It	Pr	T	It	Pr	T
	FGMRES(5p)	56	56	624	112	112	629	224	224	665
	BFGMRES(5)	14	56	622	14	112	631	14	224	668
	BFGMRESD(5)	14	43	489	15	70	401	15	120	371
	BFGMRES-R(5)	16	44	503	16	74	431	16	134	417
	DMBR(5)	16	39	452	16	57	339	18	102	328
	BFGMREST(5,p/2)	24	48	542	23	80	447	20	140	410
	DMBR(5,p/2)	16	40	459	15	68	392	17	124	384
	Combined(5,p/2)	15	41	471	15	62	359	15	103	323
	Combined(5,p/4)	18	41	474	15	59	346	15	102	320
			p = 32			p = 64			p = 128
	Method	It	Pr	T	It	Pr	T	It	Pr	T
	FGMRES(5p)	434 434	670 1152 1152 925 2531 2531 1187
	BFGMRES(5)	14	448	713	18	1152 962	19	2432 1187
	BFGMRESD(5)	15	225	371	20	490	422	25	1015	509
	BFGMRES-R(5)	18	283	466	25	618	537	28	1489	762
	DMBR(5)	19	181 316	25	413	375	28	915	497
	BFGMREST(5,p/2)	20	255	396	25	550	444	28	1125	524
	DMBR(5,p/2)	16	189	310	24	444	396	29	976	523
	Combined(5,p/2)	15	184 305	20	409	348	25	899	442
	Combined(5,p/4)	20	191	320	20	398	342	25	898	448

Table 4 .

 4

	Velocity Field	It	Pr	T
	Salt Dome	35 365 1330
	Smoothed×1 Salt Dome	33 337 1238
	Smoothed×2 Salt Dome	28 285 1056
	Smoothed×3 Salt Dome	23 246	921

1: Experiment at f = 5 Hz with p = 16 source points. The method used was DMBR(5) preconditioned by perturbed two-level preconditioner (cf. Subsection 4.4.1). It denotes the number of iterations, P r the number of preconditioner applications on a single vector and T denotes the total computational time in seconds. 128 cores were used in this experiment, and the memory cost is 11.8 Gb per right-hand side, with a total of 189.6 Gb Chapter 5

  .2.18)If required, β 1×k can be deduced from (5.2.18) by(c -Hm y * ) H ( Hm P k -V H m+1 W m P k diag(θ 1 , . . . , θ k )) = (c -Hm y * ) H (c -Hm y * )β 1×k . (5.2.19) Next, the QR factorization of the (m + 1) × k matrix Hm P k appearing in relation (5.2.18) is performed as Hm

Table 5 . 2 :

 52 Computational cost of a generic cycle of FGMRES-DR(m, k) and FGCRO-DR(m, k). C represents the total cost of FGCRO-DR(m, k) and corresponds to C

Table 5 . 3 :

 53 Solution of a d-dimensional elliptic partial differential equation problem on a 16 d grid with homogeneous Dirichlet boundary conditions (d = 2, • • • , 5). Total number of matrix-vector products (#M vp) required to solve a sequence of twelve linear systems with different flexible methods. The variable preconditioner is based on four iterations of unpreconditioned GMRES. The stopping criterion corresponds to a reduction of six orders of magnitude of the normalized residual in the Euclidean norm. Harmonic Ritz values of smallest modulus have been considered when deflating.

	Grid	16 2	16 3	16 4	16 5
	Problem size (n)	(225)	(3375) (50625) (759375)
	Method	#M vp #M vp #M vp	#M vp
	FGMRES(20)	972	1176	1272	1128
	FGMRES-DR(20,10)	732	948	1020	876
	FGCRO-DR(20,10) (no recycling)	732	948	1020	876
	FGCRO-DR(20,10) (with recycling)	457	541	547	529

(vecx,vecy)

  ! Now we initialize the C_Operator we just created. ! Remember that this C_Operator is ALSO a C_Standard27pts!! CALL matrix%Initialize(MemAvailable) ! Now it is initialized and allocated. We can fill it with values CALL fill_matrix(matrix) ! Now we can use it; with apply() function for instance CALL matrix%Apply(vecy,ylen,vecx,xlen,p,0,err) ! We are done using this C_Operator. We just destroy it and finish CALL matrix%Destroy() DEALLOCATE

				----
	(...)
	CLASS(C_Operator), POINTER	::	matrix
	TYPE(T_Error)	::	err
	COMPLEX, ALLOCATABLE	::	vecx(:,:),vecy(:,:)
	INTEGER			::	xlen,ylen,nlocx,nlocy,nlocz,p
	nlocx	=	10
	nlocy	=	11
	nlocz	=	12
	p	=	1
	xlen	=	nlocx*nlocy*nlocz
	ylen	=	(nlocx+2)*(nlocy+2)*(nlocz+2)
	matrix => New_Standard27pts(	MyID,	&
				'New Test Matrix',	&
				nlocx,	&
				nlocy,	&
				nlocz,	&
				CommHandler)

ALLOCATE(vecx(xlen,p)) ALLOCATE(vecy(ylen

,p)) vecx = CMPLX(1.0,1.0) vecy = CMPLX(0.0,0.0) ! This creates the C_Operator as a C_Standard27pts. It uses polymorphism.

  ! Now we can use it; with apply() function. We set it to verbose ! just to make sure to see some convergence information solver%IsVerbose = .TRUE. ! We can also make the preconditioner verbose; in that case we ! see both convergence information being printed preconditioner%IsVerbose = .TRUE. CALL solver%Apply(vecb,blen,vecx,xlen,p,0,err) ! We are done. We just destroy everything. Alternatively we could ! use the DestroyAll procedure which destroys recursively every pointer.

						----
	(...)
	CLASS(C_Solver), POINTER	::	solver, preconditioner
	TYPE(T_Error)	::	err
	! We create a C_BFGMRES to be used as a preconditioner
	! Only 5 cycles of BGMRES(3) are going to be used
	preconditioner => New_BFGMRES(	Name			=	'This is a Preconditioner',	&
			MyOperator		=	matrix,	&
			MaxCycles		=	5,	&
			MaxzDim			=	3*p,	&
			MaxRestartSize	=	3,	&
			Maxp			=	p,	&
			Tol			=	0.0,	&
			O_PCToCreate		=	'NoSolver' )
	! And now we create the solver...
	solver => New_BFGMRES(	Name		=	'My testing BFGMRES',	&
		MyOperator	=	matrix,	&
		MaxCycles	=	50,	&
		MaxzDim	=	10*p,	&
		MaxRestartSize	=	10,	&
		Maxp		=	p,	&
		Tol		=	1e-6,	&
		MyPC		=	preconditioner )
	! We can initialize the preconditioner and then the solver
	! or initialize only the solver, as it will recursively initialize
	! the preconditioner if needed
	CALL preconditioner%Initialize(MemAvailable)
	CALL solver%Initialize(MemAvailable)
	CALL preconditioner%Destroy()
	CALL solver%Destroy()

  -! This is is valid. Since Maxkj was not passed it is set to Maxp, ! DeflationTol is set to Tol and Deflation=F_DMBR_DEFLATION_STD Example of usage of C_DMBR. It is analogous to examples in Figure A.8 and Figure A.9

	solver => New_DMBR( ! We could alternatively pass all the parameters Name = 'My testing DMBR', MyOperator = matrix, MaxCycles = 50, MaxzDim = 10*p, MaxRestartSize = 10, Maxp = p, Tol = 1e-6, O_PCToCreate = 'NoSolver' ) solver => New_DMBR( Name = 'My testing DMBR', MyOperator = matrix, MaxCycles = 50, MaxzDim = 10*p, MaxRestartSize = 10, Maxp = p, Tol = 1e-6, DeflationTol = 1e-8, Maxkj = p/2, Deflation = F_DMBR_DEFLATION_STD, & & & & & & & MyPC = preconditioner ) CLASS(C_Operator), POINTER, INTENT(INOUT) :: FineOP, CoarseOP INTEGER , INTENT(IN) :: p TYPE(C_Class), POINTER :: self, Smoother, Coarse & & & & & & & & & & ! Create the Smoother and the Coarse solver. We choose BFGMRES for both. Smoother => New_BFGMRES( Name = 'SmootherBGMRES', & MyOperator = FineOP, & MaxCycles = 2, & MaxzDim = 2*p, & MaxRestartSize = 2, & Maxp = p, & Tol = 0.0, & O_PCToCreate = 'GaussSeidel' ) Coarse => New_BFGMRES( Name = 'CoarseBGMRES', & MyOperator = CoarseOP, & MaxCycles = 10, & MaxzDim = 5*p, & MaxRestartSize = 5, & Maxp = p, & Tol = 0.0, & O_PCToCreate = 'GaussSeidel' ) ! Finally, creates the C_GeoMultigrid itself self => New_GeoMultigrid( Name = 'BlockPerturbed_TwoGrid', & MaxIt = 1, & Smoother = Smoother, & Coarse = Coarse ) Figure A.10: FUNCTION BlockPerturbed_TwoGrid(FineOP, CoarseOP, p) RESULT(self) END FUNCTION BlockPerturbed_TwoGrid

Following the notation in Eiermann and Ernst[START_REF] Eiermann | Geometric aspects in the theory of Krylov subspace methods[END_REF], if we have an initial guess X 0 , we try to find a correction C j ∈ C n×p where range C j ⊂ Z j such that X j = X 0 + C j minimizes the residual.

this result can be found, for instance, in[START_REF] Horn | Matrix Analysis[END_REF] p.142, Theorem

3.3.1]3 this can be found, for instance, in[START_REF] Ipsen | The idea behind Krylov methods[END_REF]

the origin of the name is not related to Krylov subspaces but to the well known Arnoldi algorithm (which can be found in[START_REF] Arnoldi | the principle of minimized iterations in the solution of the matrix eigenvalue problem[END_REF] or [107, §6.12], for instance) which generates a stable basis for the Krylov subspace. We study the block version of this algorithm and the occurrence of breakdowns in more details in Section 2.5.

because range (T ) ⊂ K (AM , B) according to Corollary 2.3.2, we know that range (X) ⊂ M K (AM , B); thus M K (AM , B) is a more suitable correction subspace than K (AM , B)

Define H j = H j-1 H j 0 pj ×sj-1 H j+1,j, orH 1 = H 1 H 2,1 if j = 1;

The matrix Λ 0 ∈ C p 0 ×p is obtained from the first line of Algorithm 2.5.2

a more common case would be a near stagnation, where the rank of (I -V j-1 V H j-1 )R j-1 is computed using a threshold ε t . We do not address this situation in this thesis

sometimes called "direct problem".

We do not address the choice of S in this thesis, and we simply consider it given. Moreover, in many cases a regularization term is also added to the misfit function, but we do not address this situation here.

as we mention shortly, the forward problem, along with the backward problem (whenever it is present) are the most expensive part of Algorithm

4.2.1.

or n/4 =: N ∈ N for the two-dimensional case.

Since we set the processor's geometry to 8 × 4 × 4 we are obligated to allow hx = hy due to round off, generating thus a different number of points in direction x and y.

The eigenvalues of A are logarithmically spaced (10 -4 , 10 -3 , 10 -2 ) and linearly distributed between 0.02 and 1 with step 1/200.

Acknowledgments

We would like to thank the referees and the associate editor for their careful readings and valuable suggestions that helped us to improve our manuscript significantly. We also thank Iain S. Duff and Xavier Pinel for fruitful discussions and comments. The participation of the first author in the initial preparation of this article was guaranteed thanks to grant CNPq-473420/2007-4, coordinated by Professor Nelson Maculan. The first author would like to acknowledge the warm welcome he received at CERFACS, in the Parallel Algorithms Team, during his sabbatical leave where the initial work was completed.

List of Algorithms

Update model m (i+1) = m (i) + α (i) δm (i) ; where Q m×k ∈ C m×k and ρ ∈ C m+1 is defined as

(5.2.8)

Proposition 5.2.3. In flexible GMRES with deflated restarting, the flexible Arnoldi relation

V H k+1 V k+1 = I k+1 , (5.2.10)

holds at the i-th cycle with matrices Z k , V k ∈ C n×k and Hk ∈ C (k+1)×k defined as

.12)

.14)

where V m+1 , Z m and Hm refer to matrices obtained at the end of the (i -1)-th cycle.

Proof. Relations (5.2.9), (5.2.10), (5.2.12), (5.2.13) and (5.2.14) have been shown in [58, Proposition 2].

Respectively, from relations (5.2.13) and (5.2.6), we deduce ] = range (V k+1 ) since R is supposed to be nonsingular.

FGMRES-DR then carries out m -k Arnoldi steps with flexible preconditioning and starting vector v k+1 while maintaining orthogonality to V k leading to

We note that Hm-k ∈ C (m-k+1)×(m-k) is upper Hessenberg. At the end of the i-th cycle this gives the flexible Arnoldi-like relation

Hm-k where V m+1 ∈ C n×(m+1) , Hm ∈ C (m+1)×m and B k×m-k ∈ C k×(m-k) results from the orthogonalization of [v k+2 , • • • , v m+1 ] against V k+1 . We note that Hm is no more upper Hessenberg due to the leading dense (k + 1) × k submatrix Hk . At the end of the i-th cycle, an approximate solution x (i) 0 ∈ C n is then found by minimizing the residual norm b -A(x (i-1) 0 + Z m y) over the space x (i-1) 0 + range(Z m ), the corresponding residual being r (i) 0 = b -Ax (i) 0 , with r (i) 0 ∈ range(V m+1 ). We refer the reader to [START_REF] Giraud | Flexible GMRES with deflated restarting[END_REF] for the complete derivation of the method and numerical experiments showing the efficiency of FGMRES-DR on both academic and industrial examples. Vm the equivalent preconditioning matrices used in the initialization phase of both algorithms (step 3 in Algorithm 5.2.1). With this notation we remark that the following relations hold

(5.2.22)

We first analyze the relation between Z m and V m .

Lemma 5.2.7. At the end of the i-th cycle of the FGMRES-DR method Z m and V m satisfy

(5.2.23)

Proof. The initialization phase leads to the relation

Vm V m . We suppose that at the end of the i -1th cycle the following relation holds:

Vm V m . At step 9b of the i-th cycle Z k is defined as

The next lemma details a relation between Z m and W m that is satisfied in the FGCRO-DR method. Lemma 5.2.8. At the end of the i-th cycle of the FGCRO-DR method Z m and W m satisfy

(5.2.24)

Proof. The initialization phase leads to the relation

Wm W m . We suppose that at the end of the i -1th cycle the following relation holds:

Wm W m . At step 9a of the i-th cycle Z k is defined as

Lemma 5.2.7 and 5.2.8 show that Z m , V m , Z m and W m satisfy relations that will play a central role in Subsection 5.2.10. We investigate next the relation between Z m and V m . Lemma 5.2.9. At the end of the i-th cycle of the FGCRO-DR method Z m and V m satisfy

(5.2.25)

Proof. We use the relation AZ k = V k satisfied in the FGCRO-DR method shown in Proposition 5.2.5. The proof is then completed since

We conclude this section by presenting a technical lemma related to the FGMRES-DR method. CHAPTER 5. FLEXIBLE GCRO-DR Lemma 5.2.10. During the i-th cycle of the FGMRES-DR method, v k+1 satisfies the following relation

where r

denotes the residual obtained at the end of the (i -1)-th cycle.

Proof. Using Proposition 5.2.3 and relation (5.2.8) we obtain

.

Since V m Q m×k has orthonormal columns this last expression now becomes

.

Since Q m×k is the orthogonal factor of the QR decomposition of P k , we obtain

Since from Lemma 5.2.4 Y k = V m P k , the proof is then completed.

Analysis of the FGMRES-DR and FGCRO-DR methods

Lemma 5.2.8 has already described an important property satisfied by W m in the FGCRO-DR method proposed in Algorithm 5.2.1. We will analyze further the relation between the FGMRES-DR and FGCRO-DR methods. The next theorem states that the two flexible methods generate the same iterates in exact arithmetic under some conditions involving notably two vectors.

Theorem 5.2.11. We denote by r

0 the residual obtained at the end of the i-th cycle of the FGCRO-DR method (see step 14 of Algorithm 5.2.1). We suppose that Definition 5.2.6 holds and that the same equivalent preconditioning matrix is obtained at the end of the i-th cycle of both FGCRO-DR and FGMRES-DR algorithms i.e. M (i)

Vm . Under this assumption the harmonic Ritz vectors Y k and Y k can be chosen equal during the i + 1-th cycle. If in addition there exists a real-valued positive coefficient

in the FGCRO-DR algorithm, then both algorithms generate the same iterates in exact arithmetic and

.29)

with

.31)

where Q ∈ C (k+1)×(k+1) is a unitary matrix and X ∈ C (k+1)×(k+1) is a nonsingular triangular matrix.

Proof. The whole proof is performed in three parts assuming that we analyze the i + 1-th cycle of each algorithm. Suppose that at the beginning of the i + 1-th cycle (step 4) there exist a unitary matrix Q ∈ C (k+1)×(k+1) and a nonsingular matrix X ∈ C (k+1)×(k+1) such that the following relations hold

(5.2.34)

(5.2.35)

We will then prove the existence of a unitary matrix Q ∈ C (k+1)×(k+1) and of a nonsingular matrix X ∈ C (k+1)×(k+1) such that at the end of the i + 1-th cycle

Regarding FGCRO-DR we assume that at the beginning of the i + 1-th cycle (step 4)

(5.2.40)

We will also prove that relation (

0 . We will denote r 0 this residual for ease of notation.

Part I -Steps 5a and 5b

In this part, we prove that we can choose

FGCRO-DR Let y j = W m p j be the j-th column of Y k . Since y j is a harmonic Ritz vector of AZ m W † m with respect to range (W m ), the following relation holds (see Definition (5.2.1))

From Lemma 5.2.8 and relation (5.2.42) we deduce

Vm Z m Xp j ) = 0, where we have used explicitly the assumption on the equivalent preconditioning matrix obtained at the end of the i-th cycle i.e. M (i)

Vm . Next, the application of Lemma 5.2.7 leads to

Since X is nonsingular the last equality proves that V m Xp j is a harmonic Ritz vector of A Z m V H m with respect to range V m associated to the Ritz value θ j . From relations (5.2.41) and (5.2.43) we deduce that the harmonic Ritz vectors can be chosen to be equal and correspond to the same harmonic Ritz values. In this case they notably satisfy the following equality ∀j ∈ {1, • • • , k}, V m Xp j = W m p j i.e. p j = Xp j .

(5.2.44)

We will then denote by Y = Y k = Y k the k harmonic Ritz vectors computed in either FGCRO-DR or FGMRES-DR. We assume that the harmonic Ritz values θ j (1 ≤ j ≤ k) are non zero.

Part IIa -Steps 6a to 10a, 6b to 10b We show that at the end of steps 10a and 10b the following

] . This result will help us to prove the existence of the matrix Q introduced in relation (5.2.36).

FGCRO-DR Since AZ m P k = V k R (Proposition 5.2.5), we deduce from Lemma 5.2.4

This relation leads to the following result

] .

(5.2.46)

] can be written as, using

].

If T ∈ C (k+1)×(k+1) denotes the following triangular matrix

due to relation (5.2.47), Z k+1 T can be written as

].

(5.2.51)

FGMRES-DR Similarly from Lemma 5.2.7, Z k+1 can be expressed as

where k+1) denotes the following triangular matrix

Z k+1 T can be expressed as ]. We can further improve this result by showing the following equality

k+1 (r

(5.2.54)

k+1 (r

Using the assumption on the equivalent preconditioning matrix obtained at the end of the i-th cycle i.e. M (i)

Vm we have

(5.2.56)

CHAPTER 5. FLEXIBLE GCRO-DR

The fact that identical (possibly nonlinear) preconditioning operators are used in steps 10a and 10b of Algorithm 5.2.1 (see relation (5.2.20)) allows us to write

).

(5.2.57) Relations (5.2.56) and (5.2.57) finally show the relation (5.2.53). Consequently from relations (5.2.51), (5.2.52) and (5.2.53) we deduce that there exists a nonsingular matrix X ∈ C (k+1)×(k+1) such that

This proves the relation proposed in equation (5.2.37). Since T and T are both triangular, we note that X = T T -1 is also triangular.

Part IIIa -Steps 10a and 10b

We first show that v k+2 = v k+2 by expressing these two quantities in function of r

). Since from Proposition 5.2.5

commute and from Part IIa of the proof, the following expression can be derived

).

(5.2.59)

FGMRES-DR

The following expression for v k+2 = vk+2 /|| vk+2 || is obtained using Lemma 5.2.10

).

(5.2.60)

Due to the assumption (5.2.27) of Theorem 5.2.11 we deduce from (5.2.59) and (5.2.60) that vk+2 = η i+1 vk+2 with η i+1 positive and therefore v k+2 = v k+2 .

Part IIIb -Steps 10a and 10b

In this part we continue the analysis of the Arnoldi procedure with flexible preconditioning and show that v k+2+j = v k+2+j for j = 1, . . . , m -k -1.

For the case j = 1, we introduce vk+3 and vk+3 such that

The application of the Arnoldi procedure in both algorithms leads to

Thus from Parts II and IIIa of the proof we obtain that v k+3 and v k+3 are equal. The proof can then be completed by induction.

Results from Parts II and III justify the relation (5.2.38)

Consequently from Lemma 5.2.7, Lemma 5.2.9 and relation (5.2.20) we deduce the relation (5.2.39). This finally shows the main relations (5.2.28) and (5.2.29) of Theorem 5.2.11 that are satisfied at the end of the i + 1-th cycle.

First consequence of Theorem 5.2.11 Corollary 5.2.12. If the same flexible preconditioning operators are used in both Arnoldi procedures (steps 10a and 10b of Algorithm 5.2.1) and if at each cycle i there exists a real-valued positive coefficient

or equivalently (from relations (5.2.59) and (5.2.60)) such that vk+2 = η i vk+2 , FGCRO-DR and FGMRES-DR are algebraically equivalent.

Proof. We have already emphasized that M (0)

Vm in relation (5.2.22). In Theorem 5.2.11 we have analyzed the i+1-th cycle of both algorithms assuming that M (i)

Vm . First we have proved in Part IIb the relation (5.2.53) and secondly, respectively in Parts IIIa and IIIb, that

Consequently the same equivalent preconditioner matrix is obtained at the end of the i + 1-th cycle i.e. M (i+1)

Wm and M (i+1) Vm can be chosen equal. We deduce that FGCRO-DR and FGMRES-DR are algebraically equivalent.

About GCRO-DR and GMRES-DR

We propose a second consequence of Theorem 5.2.11 analyzed now with a fixed preconditioning matrix M .

(5.2.61)

Due to (5.2.61) and Part IIIa of Theorem 5.2.11 we deduce the following development

By induction it is possible to deduce the rest of the proof regarding vk+j , j > 2. Using range

(5.2.62)

A straightforward reformulation of Lemma 5. Fixed right preconditioner 1.00 1.00 1.00 1.00 1.00

Further features of FGCRO-DR(m, k)

In this section we first compare FGCRO-DR(m, k) with FGMRES-DR(m, k) presented in Algorithm 5.2.1 from both a computational and storage point of view. Then we detail how subspace recycling can be used in FGCRO-DR(m, k) when solving a sequence of linear systems.

Computational cost

We first analyze the computational cost related to the generalized eigenvalue problem to deduce harmonic Ritz information and then detail the total cost of the proposed method.

Harmonic Ritz information

The generalized eigenvalue problem (5.2.41) can be also written as

can be decomposed at the end of the cycle as

where the structure of the (k + 1) × (k + 1) block V H k+1 W k+1 is as follows

where the superscript is related to the cycle index. Thus storing the (m + 1)

(5.2.65)

Due to Proposition 5.2.5 and the definition of v k+1 , we have v H k+1 V k = 0. Thus we finally obtain that

where β 1×k is obtained from relation (5.2.19) which does only involve projected quantities. This allows us to deduce v H k+1 W k at a cost independent of n. From this development we draw two important consequences from a computational point of view. First, (V H m+1 W m ) (i) can be obtained recursively at a cost that is independent of the problem size n. Secondly storing W m (that would represent m additional vectors of size n) is not mandatory, only V H m+1 W m -matrix of size (m + 1) × m -is required.

Appendix A

User Guide

Introduction

We describe now with some more details the library we implemented using FORTRAN03 and MPI for solving the forward problem arising from the acoustic full waveform inversion method. These libraries were written targeting compatibility with Depth Imaging and Velocity Analysis (or DIVA), a proprietary software maintained by TOTAL. However, a priori they could be used for any high performance computing purpose, not necessarily related to acoustic full waveform inversion. As it was discussed in Section 4.6, one of the main concerns was to produce routines to be used by programmers who are used to FORTRAN90 instructions, but would produce extra modularity and code reuse due to object orientation, reason why we choose the FORTRAN03 language. The current source code is compatible with Intel Fortran Compiler version 12.1.5 20120612.

In this chapter we thus discuss basic aspects of the technical knowledge necessary to make the transition between procedural to object oriented programming, to then show with few details how to use the libraries we implemented. We suppose that the reader is familiar with both FORTRAN90 and MPI use. In the current version most of the routines associated with MPI are hidden from the final user, but we mention that only the standard data partition is used.

This appendix is organized as follows. First we introduce in the Section FORTRAN03 Basic Guidelines the features from FORTRAN03 that are not contained in FORTRAN90 standard. In Section Polymorphism and Inheritance we explain these two key concepts. Once they are established, we describe the libraries we proposed to be used by DIVA software: the libEina, libOperator and libSolver. We proposed a fourth library called libDiscretizer but unlike the other libraries we propose, libDiscretizer is not independent of DIVA software, and for this reason we do not address it here. Finally, in Section Conclusions we propose the final remarks of this appendix.

We highlight that this is not a complete reference guide, it instead contains an introductory material for using the libraries we proposed.

FORTRAN03 Basic Guidelines

In this section we briefly discuss aspects of FORTRAN03 programming which are not contained in FOR-TRAN90 standard. Since the purpose is to have as few changes as possible (the FORTRAN90 syntax is compatible with FORTRAN03), we just use a limited set of extra features from FORTRAN03, namely the keywords EXTENDS, ABSTRACT, PASS and NOPASS, DEFERRED and CLASS, which we describe next.

105

EXTENDS Keyword

Defines a new type as an extension of another type. This is the basic building block for inheritance and polymorphism. In Figure A.1, both C_ChildType1 and C_ChildType2 have all the variables declared inside the C_ParentType, but they also have their own exclusive variables. Notice that both C_ChildType1 and C_ChildType2 could be extended to a third generation type. In addition, the variables child1 (respectively, child2) has multiple types, that is, it is a C_ParentType and a C_ChildType1 (respectively, C_ChildType2) at the same time.

ABSTRACT Keyword

Used when defining a type. If a type is defined as being ABSTRACT, then it cannot be used for any purpose except being extended to another type using the EXTENDS keyword. In -----------Example 1 ------------TYPE -----------Example 2 ------------TYPE A priori the usefulness of ABSTRACT may not be clear, but it is essential to the use of DEFERRED PROCEDURES. Just as an example, consider that an abstract type is just an interface for a type and not a type itself.

0) ! ------------Example 3 ------------! This example does not even compile TYPE(C_AbsParentType

PASS and NOPASS Keyword

As in FORTRAN90, FORTRAN03 allows the use of pointer to functions to be set inside types using the keyword PROCEDURE. A PROCEDURE acts exactly like a variable belonging to the type it is defined, but it also requires the CALL keyword (in case it is a subroutine) and the referred arguments. However, in FORTRAN03 it was introduced the possibility of using the PASS and NOPASS keywords. If PASS is active, it will send the type itself as one of the arguments of the function, reducing thus the number of arguments that need to be passed when calling the procedure.

DEFERRED Keyword

Pretty much like the ABSTRACT keyword, the DEFERRED keyword defines a procedure as an interface. The deferred procedure contains nothing but a name and the type and structure of the types it refers to. It must be declared inside an ABSTRACT INTERFACE statement. It is important to notice that a DEFERRED procedure can be defined only inside an abstract type, otherwise a compilation error is thrown. In Figure A.4 we show an example of declaration of a deferred procedure. To be able to use a deferred procedure, it has to be implemented before. In Figure A.5 we show two types that extend C_Class3, and each implement procedure4 in a different manner. Both methods are valid and both methods could coexist.

CLASS Keyword

Finally we refer to the CLASS declaration. We define the arguments that a specific function or subroutine receives, an argument declared as a CLASS(C_Class) can be of type C_Class or any other type that extends C_Class using the EXTENDS keyword. Although not mentioned, we used the CLASS declaration ! ----------Example of Use ----------TYPE ! ----------Example of Use ----------TYPE

Polymorphism and Inheritance

We quickly address these two terms which we are going to use from now on. In the previous section we already showed examples of these two features of FORTRAN03, although we did not address them specifically.

Inheritance stands for the ability to draw a hierarchy of types, in which each type possesses all the procedures and variables of the types above them in the hierarchy. For instance, in Figure A.1, we say that the type C_ChildType1 inherits the variables Name1, IProperty1 and IProperty2 from C_ParentType. In Figure A.5, C_ChildType3 inherits no only Name3, RProperty3 from C_Class3, but also the procedures procedure3 and procedure4 (although the later is not implemented).

Polymorphism stands for the ability of admitting multiple different types to be used for a specific purpose. In Empiric discussion Consider that we have a function called GMRES implemented. GMRES needs, among other things, a matrix in order to apply the Arnoldi procedure. In practice there are many ways to store a matrix, specially when a massively parallel software is being used. Some of these formats exploit special structures of the matrix, some others aim at avoiding communication between nodes and etc.

However, regardless on how the matrix is stored, some attributes of a matrix always exist. For instance, every matrix has a number of rows, number of nonzeros, etc. Every matrix also should have a procedure for performing matrix-vector product. This procedure might be very different from the traditional matrixvector routine, depending on how the matrix is stored. Nevertheless, it is not important for GMRES function to know how the matrix-vector product works, but instead, to receive the result of a matrixvector product routine.

In this scenario, we could create an abstract type C_Matrix. This abstract type would contain some basic information, as the number of rows, the number of the MPI rank, communication handler, etc. This abstract type would contain a DEFERRED PROCEDURE called matrix_vector which would receive nothing but the vector that needs to be multiplied and the vector in which we want to store the product (as in Figure A.4).

libEina Basic Documentation

We now describe the main classes and types defined in the libEina library.

Modules M_Eina, M_OptimizationFlag and M_Topology

We quickly describe the following modules without attaining to details, as their knowledge is not crucial for using the main routines of this and the next libraries we describe.

M_Eina: This module contains a series of simple routines. Normally they are really simple and are meant to improve the readability of the code. No crucial functions or types to be mentioned, as this module is mainly used internally.

M_OptimizationFlag: Contains the specification of a special kind of flag, the OFLAG. It is used in some functions to tweak some very specific optimization options. We do not cover this in this user guide, since it is an advanced topic.

M_Topology: Contains the specification of some constants used in stencil definition.

Module M_Error

Contains the error handler type (T_Error) definition and some few functions to control it. This type is meant to be used mainly with the subroutines Print_Error, Print_Warning and Print_Bug which we discuss later in the description of the module M_Class.

MODULE M_Error most important functions:

New_Error(id, msg) a function that returns a type T_Error with the passed ID and message.

No_Error() a function that returns a type T_Error containing no error at all Unexpected_Error() a function that returns a type T_Error containing an error without any specific description

We describe now the T_Error type contained in M_Error module.

TYPE T_Error

Description:

The error handling type. Used to print errors, warning and to throw back the error to the caller. Important Components:

INTEGER ID A UNIQUE error ID CHARACTER*512 orig

A string (normally optional) containing the name of the function that generated this error.

CHARACTER*1024 msg

A string (normally mandatory) containing the error message concerning this error T_Error POINTER stack This error might have been caused by another error -literally. So it would be a good idea do attach it such that the called will have the full information regarding the causer of the problem. LOGICAL IsFatal Logical flag telling if we should immediately abort the run due to this error

TYPE, ABSTRACT C_Class

Description:

The abstract class, to be inherited by every other class.

Important Components: INTEGER MyID

Rank of the MPI process INTEGER CommHandler Integer containing the Communicator Handler for MPI routines INTEGER MemUsed

Contains the size in bytes that this object is allocating from the memory at the moment.

INTEGER MemNeeded

Contains the size in bytes that this object will take from the memory if initialized (allocated).

LOGICAL IsVerbose

Tells if this object should print informations. Important Procedures:

Print_Info() It is a "hello world" subroutine. It prints some basic info regarding this C_Class on the screen. A generic one is provided here, but it should normally be overridden inside the extended class for additional information.

Print_Error()

Useful for printing errors on the screen. If the error is too severe (number below zero) the message is printed for every process regardless if MyID=0 or not. Destroy DEFERRED Deallocates current object. Do NOT destroy pointers (since they may be used somewhere else) DestroyAll() Like Destroy, but also destroys every pointer recursively. This might break your code if not used wisely. Initialize(MemAvailable) DEFERRED This subroutine considers that all parameters necessary for performing the allocation of the object and all its structures are already set, and that the allocation can proceed. It receives MemAvailable as parameter, which is the maximum memory that the user has available for allocating. It is updated after the allocation of everything.

libOperator Basic Documentation

This library contains the modules for manipulating different operators (or matrices) and also the interpolation and restriction routines used by the geometric multigrid algorithm.

Module M_Operator

This module contains the definition of the abstract C_Operator class. This is the class which is going to be used to create the matrices later. Most modules and routines which require a coefficients matrix will include this module (and not the ones extending the C_Operator).

TYPE, ABSTRACT EXTENDS(C_Class) C_Operator

Description:

C_Operator is an abstract class able to accommodate any kind of function the function y = f (x), given x and y of proper dimensions. We dispose of a certain number of procedures inside this class, but the core procedure is apply(), which does exactly what was specified before: apply the function f on x. In fact, in the apply routine, the variable y is supposed to be INTENT(INOUT), so this function can be performing in fact a y = f (x, y). Nevertheless, we keep this class simplified.

Always have in mind that the purpose is to have this as a parallel operator, that is, to be used in MPI application, therefore several parameters regarding MPI are present here.

Module M_StencilCollection

Extends the C_Operator to the C_StencilCollection class, which is also abstract. In this class, however, we already have some assumptions, as for instance, that the referred operator is coming from a finite difference discretization scheme (for the moment, it always assumes it is a three-dimensional discretization).

TYPE, ABSTRACT, EXTENDS(C_Operator) C_StencilCollection

Description: This is an extension of C_Operator which adds some basic structures associated with finite differences discretization three-dimensional discretization schemes.

Module M_Diag7pts

This module contains the definition of an almost matrix-free implementation of a 7 point stencil operator.

It is specifically tuned for the Helmholtz operator, discretized with PML boundary condition. We use a special characteristic of such discretized operators and store only the main diagonal, using an economic technique for storing off-diagonal elements in small vectors.

TYPE, EXTENDS(C_StencilCollection) C_Diag7pts

Description: This is the class for storing the the almost matrix-free 7 point stencil operator, tuned for using with the Helmholtz operator with PML boundary conditions.

MODULE M_Standar27pts most important functions:

New_Standard27pts(MyID, Name, o_nlocx, o_nlocy, o_nlocz, CommHandler)

Returns an object of the type C_Standard27pts.

Module M_Transformation, M_FullInterpolation and M_FullRestriction

We quickly describe the following modules without attaining to details, as their knowledge is not crucial for using the main routines of this and the next libraries we describe.

M_Transformation: Contains the abstract type used for interpolations and restrictions. This is basically meant to accommodate any rectangular matrix. It is used by the C_GeoMultigrid class which we describe later.

M_FullInterpolation: Contains the full weighted interpolation object. Used by C_GeoMultigrid.

M_FullRestriction: Contains the full weighted restriction object. Used by C_GeoMultigrid.

Modules M_NoSolver, M_GaussSeidel, M_FGMRES and M_LinAlg

We do not advise the user to deal directly with any of these modules, and for this reason we quickly address them here before proceeding to the discussion of other modules.

M_NoSolver Internally, every C_Solver like C_BFGMRES need a pointer to another C_Solver to use as preconditioner, even in the unpreconditioned case, because it calls the procedure precondi-tioner%Apply(). This module fills this gap, providing a "solver" that simply copies the input vector b to vector x whenever the Apply() procedure is called. Although one can actually use C_NoSolver, the current implementation of libSolver automatically creates one whenever it is needed such that the user do not have to mind this issue.

M_GaussSeidel Provides a C_Solver, meant to be used as preconditioner for another solver. It is intrinsically dependent on the structure of the C_Operator being used. Because of that, the actual routines for performing it are currently implemented inside the respective operator. Both C_Diag7pts and C_Standard27pts have an implementation of the local symmetric Guass-Seidel algorithm. The C_GaussSeidel class calls this routine directly from the C_Operator, acting as some sort of interface. This means that this module in fact need to be updated whenever a new C_Operator is added to lib-Operator library, which is undesirable. As C_NoSolver, there is no need for the user to create its own C_GaussSeidel as the routines provided in libSolver create it automatically if needed. An important information about this C_Solver is that it contains a component MaxIt, meaning that several iterations of the local symmetric Gauss-Seidel algorithm could be used as preconditioner. However, libSolver uses by default only one iteration.

M_FGMRES This module is analogous to M_BFGMRES. Noticeable differences are the absence of the MaxzDim component and the replacement of O_PCToCreate='BFGMRES' by O_PCToCreate='FGMRES'. This module executes the single right-hand side FGMRES algorithm for each one of the p right-hand side and thus it is non-optimal. It is often advised to use M_BFGMRES instead.

M_LinAlg Provides important linear algebra routines implemented in parallel tailored to be used with M_Solver. We mention here the Get_Norm routine for computing the Euclidean norm of each column of a block vector, the CGS2 which implements the classical Gram-Schmidt algorithm with reorthogonalization, and QR_CGS2 which computes the QR decomposition of a matrix using the CGS2 function. Since these are low level routines, we do not address them deeply here.

Module M_DMBR

This module contains the main iterative solver we propose, the C_DMBR.

In an overall, the module M_DMBR is analogous to the module M_BFGMRES.

Description:

The DMBR solver, performing deflation at the beginning of every iteration. It contains all the components of C_BFGMRES plus some extra parameters for handling deflation.

Important Components:

C_Solver POINTER MyPC This object defines the preconditioner to be used by this C_DMBR. Notice that it could be C_DMBR itself or C_BFMGRES, for instance.

INTEGER MaxCycles

Maximum number of cycles to be performed. The solver stops and returns its final approximation of the solution even if it did not converge INTEGER MaxzDim DMBR will restart whenever the number of iterations reach MaxRestartSize or whenever the dimension of the subspace Z j reaches MaxzDim, whichever comes first. INTEGER MaxRestartSize DMBR will restart whenever the number of iterations reach MaxRestartSize or whenever the dimension of the subspace Z j reaches MaxzDim, whichever comes first.

REAL Tol

Convergence threshold for DMBR. It will return whenever the . ψ of the relative residual is smaller than Tol or Max-Cycles have been performed.

REAL DeflationTol

Deflation threshold for DMBR. Used to determine the value of k j in the beginning of every iteration.

INTEGER Maxkj

Used to enforce a maximum value for k j even if no small singular values were detected.

INTEGER Deflation

Flag used to set the kind of deflation. F_DMBR_DEFLATION_NONE does not deflate (equivalent to C_BFGMRES), F_DMBR_DEFLATION_IT deflates only at the end of each iteration (equivalent to [START_REF] Robbé | Exact and inexact breakdowns in the block GMRES method[END_REF] R-criterion algorithm), F_DMBR_DEFLATION_CY deflates at the beginning of each cycle.

Module M_GeoMultigrid

The last module belonging to libSolver, which implements a V cycle of geometric multigrid algorithm. Due to the polymorphism, however, this could be used to implement several levels of multigrid TYPE, ABSTRACT, EXTENDS(C_Solver) C_GeoMultigrid

Description:

The C_GeoMultigrid, meant to be used as a preconditioner. A priori it is nothing but a two-level V cycle of geometric multigrid, but it could be any arbitrary geometric multigrid due to polymorphism, as we explain later. As M_DMBR and M_BFGMRES, this module also provides some interfaces for dealing with internal values. However, they require the knowledge of the interpolation and restriction routines. Since we are trying to attain a basic aspects, we stick only with the simplest form, which also assumes that PreSmoother=PostSmoother.

MODULE M_GeoMultigrid most important functions:

New_GeoMultigrid(Name,MaxIt, Maxp,Smoother,Coarse)

Returns a C_GeoMultigrid ready to be initialized and used as a preconditioner. The Smoother passed is used as both PreSmoother and PostSmoother.

Conclusions

In this User Guide we provided the basic tools for making the transition between FORTRAN90 procedural programming to FORTRAN03 object oriented programming. We explained the most essential concepts and then we introduced the libraries we implemented using only the most basic concepts. These libraries could be used, a priori, for kind of application using MPI routines. Although the results we obtained with this library were satisfactory, we aim at improving the number of options available, as adding new solvers, preconditioners and matrix storage formats.