Mike Fisher

Xavier Vasseur

Fadela Kabeche

Benjamin François

Kamer Kaya

Zafer Zeren

Kut Tuǧçe

Rabia Zeren

Santiago Özerkmen

Alcalde S Martinez

Gürol

Didem Akçay

Yasemin Bolat

Melike Gürün

Sezin Selçuk

Anthony T Weaver

Andy M Moore

During this work we have developed collaboration with

Keywords: assimilation de données, approche duale, optimisation, précondionnement, méthode des gradients data assimilation, dual approach, optimization, preconditioning, conjugate-gradients, Lanczos method, trust-region methods . . . . . . . 3.1.4 Preconditioning varying systems . . . . . . . . . . . . . . .

, qui assure la convergence globale des points critiques du premier ordre sous des hypothèses faibles. Nous considérons la seconde de ces approches dans cette thèse. En outre, compte tenu de la grande échelle de ce problème, nous proposons ici d'utiliser un algorithme de région de confiance particulier s'appuyant sur la méthode du gradient conjugué tronqué de Steihaug-Toint pour la résolution approchée du sous-problème (Conn, Gould and Toint, 2000, p. 133-139) La résolution de ce sous-problème dans un espace à n dimensions (par CG ou Lanczos) est considérée comme l'approche primale. Comme alternative, une réduction significative du coût de calcul est possible en réécrivant l'approximation quadratique dans l'espace à m dimensions associé aux observations. Ceci est important pour les applications à grande échelle telles que celles quotidiennement traitées dans les systèmes de prévisions météorologiques.

Cette approche, qui effectue la minimisation de l'espace à m dimensions à l'aide CG ou de ces variantes, est considérée comme l'approche duale.

that n is large compared to m. These problems are encountered for instance when three-dimensional fields are estimated from physical observations, as is the case in data assimilation in Earth system models [START_REF] Daley | Atmospheric data analysis[END_REF][START_REF] Kalnay | Atmospheric modeling, data assimilation and predictability[END_REF]. In meteorological applications for example, the estimation field in the assimilation procedure is the initial state of the dynamical system, which is then integrated forward in time to produce a weather forecast.

A widely used algorithm in this context is the Gauss-Newton (GN) method, known in the data assimilation community under the name of incremental four dimensional variational data assimilation (Incremental 4D-Var) (Courtier, Thépaut and Hollingsworth, 1994). The GN method relies on the approximate solution of a sequence of linear least-squares problems in which the nonlinear least-squares cost function is approximated by a quadratic function in the neighbourhood of the current nonlinear iterate. However, it is well known that this simple variant of the Gauss-Newton algorithm does not ensure a monotonic decrease of the cost function and that convergence is not guaranteed. Removing this difficulty is typically achieved by using a line-search [START_REF] Dennis | Numerical methods for unconstrained optimization and nonlinear equations[END_REF] or trust-region [START_REF] Conn | Trust-region methods[END_REF] strategy, which ensures global convergence to first order critical points under mild assumptions. We consider the second of these approaches in this thesis. Moreover, taking into consideration the large-scale nature of the problem, we propose here to use a particular trust-region algorithm relying on the Steihaug-Toint truncated conjugate-gradient method for the approximate solution of the subproblem (Conn, Gould and Toint, 2000, pp. 133-139).

Solving this subproblem in the n-dimensional space (by CG or Lanczos) is referred to as the primal approach. Alternatively, a significant reduction in the computational cost is possible by rewriting the quadratic approximation in the m-dimensional space associated with the observations. This is important for large-scale applications such as those solved daily in weather prediction systems. This approach, which performs the minimization in the m-dimensional space using CG (or Lanczos) or variants thereof, is referred to as the dual approach for reasons that will be discussed in Chapter 4.

The first proposed dual approach (Da Silva et al., 1995;Cohn et al., 1998;Courtier, 1997), known as the Physical-space Statistical Analysis System (PSAS) in the data assimilation community, starts by solving the corresponding dual cost function in IR m by a standard preconditioned CG (PCG), and then recovers the step in IR n through multiplication by an n × m matrix. Technically, the algorithm consists of recurrence formulas involving m-vectors instead of nvectors. However, the use of PSAS can be unduly costly as it was noticed that the linear least-squares cost function does not monotonically decrease along the nonlinear iterations when applying standard termination criteria (El Akkroui, Gauthier, Pellerin and Buis, 2008).

Another dual approach has been proposed by Gratton and Tshimanga (2009) and is known as the Restricted Preconditioned Conjugate Gradient (RPCG) method. It generates the same iterates in exact arithmetic as those generated by the primal approach, again using recursion formula involving m-vectors.

The main interest of RPCG is that it results in significant reduction of both memory and computational costs while maintaining the desired convergence property, in contrast with the PSAS algorithm. The relation between these two dual approaches and the question of deriving efficient preconditioners (Gratton, Sartenaer and Tshimanga, 2011) -essential when large-scale problems are considered -was not addressed in Gratton and Tshimanga (2009).

The main motivation for this thesis is to address these open issues. In particular, we are interested in designing preconditioning techniques and a trust-region globalization which maintain the one-to-one correspondance between primal and dual iterates, thereby offering a cost-effective computation in a globally convergent algorithm.

List of Tables

3.1 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 . . . . . 3.2 A summary of the required matrix-vector products for each approach used for the solution of the k-th linear system (3.1) with LMPs within Algorithms 3.1, 3.2, and 3.5 under the assumption that P 0 = B . . . . . 3.3 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where preconditioning is achieved by using the quasi-Newton LMPs. . . . . . 5.1 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with Lanczos-type algorithms . . . . . . . . . . . . . . . . . . . . . 5.2 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs.

Approaches (E), (F) and (G) are defined in Section 5.1. . . . . . . . . . 5.3 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the Ritz LMPs. Approaches (E), (F) and (G) are defined in Section 5.1. . . . . . . . . . . . . . . . . 5.4 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the spectral LMPs. Approaches (E), (F) and (G) are defined in Section 5.1. . . . . . . . . . . . . . . . . 5.5 A summary of the characteristics of Algorithms 3.2 and 4.4. Approach (E) is defined in Section 3.1. . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs. Approach (F) is defined in Section 5.1. . . . . . . . . . . . . . . . . . . . . 5.7 A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs. Approaches (F) and (H) are defined in Section 5.1 and 5.

List of Figures

Introduction and motivation

This thesis investigates the conjugate-gradient (CG) method and Lanczos method for the solution of under-determined nonlinear least-squares problems regularized by a quadratic penalty term. Such problems often result from a maximum likelihood approach, and involve a set of m physical observations and n unknowns that are estimated by nonlinear regression. We suppose here that n is large compared to m. These problems are encountered for instance when three-dimensional fields are estimated from physical observations, as is the case in data assimilation in Earth system models [START_REF] Daley | Atmospheric data analysis[END_REF][START_REF] Kalnay | Atmospheric modeling, data assimilation and predictability[END_REF]. In meteorological applications for example, the estimation field in the assimilation procedure is the initial state of the dynamical system, which is then integrated forward in time to produce a weather forecast.

A widely used algorithm in this context is the Gauss-Newton (GN) method, known in the data assimilation community under the name of incremental four dimensional variational data assimilation (Incremental 4D-Var) (Courtier, Thépaut and Hollingsworth, 1994). The GN method relies on the approximate solution of a sequence of linear leastsquares problems in which the nonlinear least-squares cost function is approximated by a quadratic function in the neighbourhood of the current nonlinear iterate. However, it is well known that this simple variant of the Gauss-Newton algorithm does not ensure a monotonic decrease of the cost function and that convergence is not guaranteed. Removing this difficulty is typically achieved by using a line-search [START_REF] Dennis | Numerical methods for unconstrained optimization and nonlinear equations[END_REF] or trust-region [START_REF] Conn | Trust-region methods[END_REF] strategy, which ensures global convergence to first order critical points under mild assumptions. We consider the second of these approaches in this thesis. Moreover, taking into consideration the large-scale nature of the problem, we propose here to use a particular trust-region algorithm relying on the Steihaug-Toint truncated conjugate-gradient method for the approximate solution of the subproblem (Conn, Gould and Toint, 2000, pp. 133-139).

Solving this subproblem in the n-dimensional space (by CG or Lanczos) is referred to as the primal approach. Alternatively, a significant reduction in the computational cost is possible by rewriting the quadratic approximation in the m-dimensional space associated with the observations. This is important for large-scale applications such as those solved daily in weather prediction systems. This approach, which performs the minimization in the m-dimensional space using CG (or Lanczos) or variants thereof, is referred to as the dual approach for reasons that will be discussed in Chapter 4.

The first proposed dual approach (Da Silva et al., 1995;Cohn et al., 1998;Courtier, 1997), known as the Physical-space Statistical Analysis System (PSAS) in the data assimilation community, starts by solving the corresponding dual cost function in IR m by a standard preconditioned CG (PCG), and then recovers the step in IR n through multiplication by an n × m matrix. Technically, the algorithm consists of recurrence formulas involving m-vectors instead of n-vectors. However, the use of PSAS can be unduly costly as it was noticed that the linear least-squares cost function does not monotonically decrease along the nonlinear iterations when applying standard termination criteria (El Akkroui, Gauthier, Pellerin and Buis, 2008).

Another dual approach has been proposed by Gratton and Tshimanga (2009) and is known as the Restricted Preconditioned Conjugate Gradient (RPCG) method. It generates the same iterates in exact arithmetic as those generated by the primal approach, again using recursion formula involving m-vectors. The main interest of RPCG is that it results in significant reduction of both memory and computational costs while maintaining the desired convergence property, in contrast with the PSAS algorithm. The relation between these two dual approaches and the question of deriving efficient preconditioners (Gratton, Sartenaer and Tshimanga, 2011) -essential when large-scale problems are considered -was not addressed in Gratton and Tshimanga (2009).

The main motivation for this thesis is to address these open issues. In particular, we are interested in designing preconditioning techniques and a trust-region globalization which maintain the one-to-one correspondance between primal and dual iterates, thereby offering a cost-effective computation in a globally convergent algorithm.

The outline of the thesis is as follows.

Chapter 2 gives the background material that will be useful for the following chapters. It consists of the general formulation of the nonlinear least-squares problem and explains the solution methods for these kind of problems, in particular the GN method and its variants. CG and Lanczos methods that are used for the solution of the linear systems arising in a GN method are then explained. Preconditioning is discussed, with particular emphasis on the class of Limited Memory Preconditioners (LMPs) (Gratton, Sartenaer and Tshimanga, 2011). The chapter is concluded by explaining the dual approach which will be exploited in Chapter 4.

Chapter 3 is concerned with the solution of the subproblem using the CG method. It focuses on preconditioning of a sequence of symmetric and positive definite linear systems with multiple right-hand sides, As = b k , by using directions generated during the solution of the previous linear system. Each linear system is assumed to be a member of a convergent sequence. Different ways of preconditioning using LMPs are introduced in the sense that each approach requires different forms of the preconditioner and may avoid expensive matrix-vector products (when the linear system is large scale). These approaches are later analysed when used for solving a sequence of linear systems

Contributions

The main contributions in this thesis are

• to design a general LMP that can be used with RPCG and RPLanczos with the aim of accelerating the convergence of the dual algorithm as well as keeping the convergence properties of the primal approach (Chapter 4),

• to derive particular LMPs for RPCG and RPLanczos such as the quasi-Newton LMP, the Ritz-LMP and the spectral LMP (Chapter 4),

• to design a globally convergent algorithm in dual space where the preconditioning is performed with a LMP (Chapter 6),

• to derive an alternative PCG algorithm (PCG Inverse Free algorithm (PCGIF)) by exploiting the structure of the linear system in primal space that can avoid expensive matrix-vector products (Chapter 3),

• to design a general LMP and particular LMPs for the PCGIF Algorithm (Chapter 3),

• to derive the Lanczos version of the PCGIF Algorithm where the preconditioning is achieved by the LMPs (Chapter 5),

CHAPTER 2 Background material

This chapter consists of fundamental information that will be a reference for the following chapters. First, the problem formulation for the regularized nonlinear least-squares problem is explained, starting from a general formulation of the nonlinear least-squares problem. After defining the problem, solution methodologies are discussed taking into consideration situations where the dimension of the problem is large. We focus on the Gauss-Newton method as a solution algorithm, in which one solves a sequence of linear least-squares subproblems where each member of the sequence is a local quadratic approximation of the original nonlinear least-squares problem. The Gauss-Newton method can be improved in terms of its convergence behaviour by using line-search or trust-region strategies that we also outline in this chapter.

The solution of the linear least-squares subproblems arising in a Gauss-Newton iteration can be found by solving the corresponding linear systems. We consider wellknown Krylov subspace methods, in particular the conjugate gradients and the Lanczos methods to solve those linear systems. Lanczos methods are applicable when the system (Hessian) matrix is large and symmetric, whereas conjugate gradients are applicable when the Hessian is large, symmetric and positive definite (which is the case for the application in this study). Both methods are explained in detail and the related algorithms are provided.

It is well-known that when using iterative methods, preconditioning is critical in order to accelerate the convergence. We summarize a class of Limited Memory Preconditioners (LMPs) to precondition the linear systems when solved with a Lanczos or conjugate gradient method. These preconditioners construct an approximation to the inverse Hessian matrix by using information from preceding iterations.

We conclude the chapter by introducing the dual subproblem of the original linear least-squares subproblem which will be exploited in the coming chapters.

The nonlinear least-squares problem

Nonlinear least-squares problems arise most commonly from data fitting and optimal control problems that take place in various applications such as physical, chemical and aerospace applications. The main idea is to find the best model fit to the observed data in the sense that the sum of square errors between the observed data and the model prediction is minimized.

In other words, let us assume that we observe a phenomenon at time t i , i = 1, 2, . . . , m and have access to a (possibly noisy) data set (y i , t i ). Suppose also that a prediction model y = H(x, t), is available, where H is nonlinear in x ∈ IR n , that provide a theoretical value of y knowing a state vector x. Our aim is to find an estimate of the parameter vector x that minimizes the sum of square errors of the residual component

d i = H(x, t i ) -y i ,
preferably in suitable norms taking care of the (statistical) knowledge on the noise affecting the data (Tarantola, 2005, p. 64). The residual components of d i : IR n → IR can be assembled into a residual vector (2.1)

Minimizing the sum of square errors of the residual vector d(x) leads to the nonlinear least-squares problem:

min x∈IR n f (x) = 1 2 d(x) T R -1 d(x) = 1 2 d(x) 2 R -1 , (2.2)
where the m × m symmetric positive-definite matrix R is an estimate of the dataerror covariance matrix. When m ≥ n, this nonlinear least-squares problem is called overdetermined. Note that the nonlinearity of the problem is due to the nonlinearity of the model H in x. If the model H is linear in x, then the minimization problem (2.2) is called a linear least-squares problem.

The overdetermined nonlinear least-squares problem (problem (2.2) with m ≥ n) is typically solved by well-known line-search (Dennis and Schnabel, 1983, p. 227) and trust-region strategies [START_REF] Conn | Trust-region methods[END_REF] which are based on the Newton and quasi-Newton approaches with modifications that consider the special structure of the cost function f (x) and of its derivatives (Nocedal and Wright, 2006, p. 247).

The regularized nonlinear least-squares problem

When the solution of nonlinear least-squares problem (2.2) exists but is not unique, the problem has to be regularized. This may happen for instance when m < n.

A possible solution technique is formulated by adding a regularization term to the original problem, as in the well-known Tikhonov regularization (Björck, 1996, p. 101). This regularization leads to the following weighted regularized problem min

x∈IR n f (x) = 1 2 d(x) 2 R -1 + 1 2 τ 2 x -x c 2 B -1 , (2.3)
where the center vector x c is an application-dependent vector. The vector x c can be defined from some approximation of the solution or chosen as a zero vector [START_REF] Eriksson | Regularization methods for nonlinear least squares problems. part i: Exactly rank-deficiency[END_REF]. The n × n symmetric positive-definite matrix B is an estimate of the center vector-error covariance matrix, and τ > 0 is the regularization parameter. The problem (2.3) is also called a damped nonlinear least-squares problem (Björck, 1996, p. 101).

We can reformulate the regularized problem (2.3) as min

x∈IR n f (x) = 1 2 d(x) v(x) 2 W = 1 2 u(x) 2 W , (2.4)
where the W -norm is defined by the (m + n) × (m + n) symmetric positive-definite block matrix

W = R -1 B -1 , v : IR n → IR n is defined as v(x) = [v 1 (x), . . . , v n (x)] T , v i (x) = τ (x i -(x c ) i ), i = 1, .
. . , n and u : IR n → IR m+n is defined as u(x) = [d 1 (x), . . . , d m (x), v(x)] T . As a result, the overdetermined nonlinear least-squares problem can be written as in (2.2) and the methods for overdetermined nonlinear least-squares problem can be applied to solve the problem. Before giving details on the solution methods, we first calculate the first and second order derivatives of the cost function in (2.4) which are sometimes needed by solution methods.

Calculating the derivatives

The problem (2.4) can be viewed as a special case of an unconstrained optimization problem which requires for its solution the computation of the cost function f (x) and sometimes of its derivative ∇f (x) and its Hessian ∇ 2 f (x). We start this section by deriving first and second order derivatives of the cost function f (x) given in (2.4).

Let us start with the first-derivative matrix of u(x) which is simply the Jacobian matrix J(x) ∈ IR (m+n)×n , defined by

J(x) =                     ∂d 1 (x)
∂x n                     =               ∇u 1 (x) T . . . ∇u m (x) T ∇u m+1 (x) T . . . ∇u m+n (x) T               =               H 1 . . . H m e T 1 . . . e T n              
, where e i is the i-th column of the n×n identity matrix I n and for simplicity we consider τ = 1. Then Jacobian matrix can be written in a compact form as

J(x) = H I n , (2.5)
where H = [H T 1 , H T 2 , . . . , H T m ] T ∈ IR m×n . So the first derivative of f (x) defined in (2.4) can be expressed as follows, ∇f (x) = m+n j=1 W T j u(x)∇u j (x) = J(x) T W u(x)

= H T R -1 d(x) + B -1 (x -x c ), (2.6)
where W j is the jth column vector of W matrix and d(x) is defined in (2.1). The Hessian matrix of the component u j (x) is given by

∇ 2 u j (x) =        ∂ 2 u j (x) ∂ 2 x 1 . . . ∂ 2 u j (x) ∂x 1 ∂x n . . . . . . . . . ∂ 2 u j (x) ∂x n ∂x 1 . . . ∂ 2 u j (x) ∂ 2 x n       
.

Note that ∇ 2 u j (x) = 0 for j = m+1, . . . , n since u j (x) is linear in x for j = m + 1, . . . , m + n.

Then, the second derivative of f (x) is calculated as

∇ 2 f (x) = m+n j=1 ∇u j (x)W T j J(x) + m+n j=1 W T j u(x) ∇ 2 u j (x) = J(x) T W J(x) + m j=1 W T j u(x) ∇ 2 u j (x) = (B -1 + H T R -1 H) + S(x), (2.7)
where S(x) = m j=1 W T j u(x) ∇ 2 u j (x) includes second order derivatives in ∇ 2 f (x).

Solving the regularized problem

It is very common to find the solution of the nonlinear least-squares problem of the form (2.4) by Newton's method. Starting from an initial vector x 0 , Newton's method generates a sequence of iterates {x k }, k = 0, 1, . . . , K. This sequence is generated by computing the search direction s based on the quadratic model m k

m k (s) = f (x k ) + s T ∇f (x k ) + 1 2 s T ∇ 2 f (x k )s. (2.8)
This model is the Taylor approximation of the function f (x) in the neighborhood of a given iterate x k . Assuming that ∇ 2 f (x k ) is positive definite, the minimizer of the quadratic function (2.8) is given by setting the derivative of m k (s) to zero, i.e.

s k = -(∇ 2 f (x k )) -1 ∇f (x k ).
(2.9) Thus, the next iterate is updated according to

x k+1 = x k + s k .
(2.10)

Replacing the objective function f (x), the gradient ∇f (x k ) and Hessian ∇ 2 f (x k ) by (2.4), (2.6) and (2.7) respectively in (2.8) leads to

m k (s) = 1 2 u T k W u k + s T (H T k R -1 d k + B -1 (x k -x c )) + 1 2 s T (B -1 + H T k R -1 H k + S k )s,
(2.11) where u k = u(x k ), d k = d(x k ), S k = S(x k ) and H k = [∇u 1 (x k ) T , . . . , ∇u m (x k ) T ] T . Similarly, replacing ∇f (x k ) and ∇ 2 f (x k ) in (2.9) gives

s k = (B -1 + H T k R -1 H k + S k ) -1 (B -1 (x c -x k ) -H T k R -1 d k ).
(2.12) Therefore, from (2.10) and (2.12), the minimizer of the regularized problem (2.4) in the neighborhood of x k is given as follows

x k+1 = x k + (B -1 + H T k R -1 H k + S k ) -1 (B -1 (x c -x k ) -H T k R -1 d k ).
In (2.7) usually matrix vector products with H are easily available. However, this is not the case for S(x) which includes second order derivatives that are usually not available. As mentioned in (Dennis and Schnabel, 1983, p. 218), [p. 263] (Nocedal and Wright, 2006) there are different iterative approaches to handle this issue. For a zero-residual problem, i.e. S(x * ) = 0 where x * is the solution, or a small-residual problem, i.e. S(x * ) is small relative to J(x * ) T W J(x * ), S(x) can be omitted. It leads to the Gauss-Newton algorithm. As it can be easily seen, the Gauss-Newton method convergence depends on the problem nonlinearity. For a zero-residual problem the Gauss-Newton method is locally q-quadratically convergent whereas for the problems that are not very nonlinear (small-residual problem), the method is locally q-linearly convergent (Dennis and Schnabel, 1983, p. 225). For a large-residual problem, i.e. S(x * ) is large relative to J(x * ) T W J(x * ), the method may not even be locally convergent (Björck, 1996, p.343). The Gauss-Newton method convergence also depends on the starting vector x 0 . In particular, it is not necessarily globally convergent, i.e. a sequence of iterates {x k } obtained by Gauss Newton method starting from an arbitrary initial vector x 0 may not converge to a local minimizer of f (x). However global convergence can be obtained by introducing a line search or trust region approachs.

For a large-residual nonlinear problem, another alternative can be to use nonlinear optimization algorithms directly. However, in this approach the special structure of the nonlinear least-squares problem is ignored and the convergence rate of algorithm may be inferior to Gauss-Newton method for small or zero-residual problems.

A hybrid method can be another option to handle the problem. The main idea in this approach is to consider the Gauss-Newton method if the problem is a zero-residual or small residual problem, otherwise consider the approaches that handles a large residual problem. For details on hybrid methods, we refer to the references (Nocedal and Wright, 2006, p. 263-264), (Björck, 1996, p. 348-351).

We focus on the Gauss-Newton type methods throughout this study, since our application problem arises from operational data assimilation, which is a large-scale problem with a small-residual, and it is well-known that the Gauss-Newton algorithm is an efficient solver for this kind of problem.

The Gauss-Newton method

The Gauss-Newton method consists of solving a sequence of linearized least-squares problems

min s 1 2 H k s + d k 2 R -1 + 1 2 s + x k -x c 2 B -1 , (2.13)
where J, u and W are defined in Section 2.2. The cost function in (2.13) is a quadratic approximation of the nonlinear cost function f (x) given in (2.4) in the neighborhood of a given iterate x k . On each iteration k, this quadratic cost function is minimized to determine a step s k+1 starting from the iterate x k . This step is then used to update the current iterate from

x k+1 = x k + s k . (2.14)
Assuming that the Jacobian matrix J k (at iteration k) has full column rank, the solution to the problem (2.13) at the k-th iterate is given as

s k = (B -1 + H T k R -1 H k ) -1 (B -1 (x c -x k ) -H T k R -1 d k ).
(2.15)

The matrices H k , B and R appearing in (2.15) may be so large that they can't be stored explicitly, and the information they contain may only be available through matrixvector products. In this case it is natural to solve the linear system in (2.15) by Krylov subspace methods (Saad, 1996), whose sole access to the system matrix is via such products. Therefore, the Gauss-Newton algorithm first applies this method to the linear system (2.16) where B -1 + H T k R -1 H k is symmetric and positive definite and updates the iterate as in (2.14). The main loop of the Gauss-Newton method that generates the iterates {x k } is called the outer loop. The iterative loop of solving the linear system (2.16) by an iterative Krylov subspace method is called the inner loop since it is nested within the outer loop of the Gauss-Newton algorithm.

(B -1 + H T k R -1 H k )s = B -1 (x c -x k ) -H T k R -1 d k ,
We next explain variants of the Gauss-Newton method by including line search or trust region strategies.

The damped Gauss-Newton method

The damped Gauss-Newton method (Björck, 1996, p. 343), (Dennis and Schnabel, 1983, p. 227) uses a line search strategy in a Gauss-Newton method. The key idea in a line search strategy is to control the step length along the descent direction by ensuring that it gives a sufficient decrease in the objective function f (x), measured by some conditions such as the Armijo and Wolfe conditions (Nocedal and Wright, 2006, p. 21,33,34).

It can be shown that whenever J k has full column rank and the gradient ∇f (x k ) is nonzero, the Gauss-Newton step s k is in a descent direction (Nocedal and Wright, 2006, p. 254) for the cost function f (x) given in (2.4). This suggests to improve the Gauss-Newton algorithm by using a line search strategy. A line search strategy in a Gauss-Newton method gives rise to the damped Gauss-Newton method which generates the iterates as follows:

x k+1 = x k + α k s k ,
where α k > 0 is the step length. The selection of α k is critical in this method. Ideally, it is selected as the solution of the problem min

α u(x k + αs k ) 2 W .
However in general it may be expensive to find the minimizer of this univariate function.

Instead, an inexact line search method can be used, provided that the step sufficiently reduces the function f (x) at minimal cost. For more details on the line search strategy we refer to (Nocedal and Wright, 2006, Chapter 3), (Björck, 1996, p. 344-346) and (Dennis and Schnabel, 1983, p. 116-129).

As expected from the property of the Gauss-Newton method, the damped Gauss-Newton method is locally convergent on zero and small residual problems. Additionally, it is locally convergent on large-residual problems or very nonlinear problems. It is not always globally convergent (Dennis and Schnabel, 1983, p. 227), (Björck, 1996, p. 346) and the rate of convergence may still be slow on large-residual problems.

Another issue is that when J(x) is rank deficient, the damped Gauss-Newton method is not well defined (it is not possible to apply a line search strategy) since the Gauss Newton direction may not be a descent direction. This problem can be overcome with the so-called trust region approach which is an alternative strategy to improve the Gauss-Newton algorithm. In the next section we give the details on the Gauss-Newton methods that uses a trust-region strategy.

The Levenberg-Marquart method

Improving the convergence properties of the Gauss-Newton method in a trust-region framework take its origins from [START_REF] Levenberg | A method for the solution of certain problems in least squares[END_REF], [START_REF] Morrison | Methods for nonlinear least squares problems and convergence proofs[END_REF] and [START_REF] Marquardt | An algorithm for least-squares estimation of nonlinear parameters[END_REF]. This method is named the Levenberg-Morrison-Marquardt method in Conn, [START_REF] Conn | Trust-region methods[END_REF] because of their independent contributions.

At first, [START_REF] Levenberg | A method for the solution of certain problems in least squares[END_REF] suggested to use a damping procedure in the Gauss-Newton method by adding a multiple of the identity to the Hessian matrix. Afterwards, [START_REF] Morrison | Methods for nonlinear least squares problems and convergence proofs[END_REF] and [START_REF] Marquardt | An algorithm for least-squares estimation of nonlinear parameters[END_REF] independently showed that there is a link between damping the Hessian and reducing the length of the step length. They also recognised that minimizing a model with a damped Hessian is equivalent to minimizing the original model in a restricted area defined by the damping parameter (Conn, Gould and Toint, 2000, p.8).

On each iteration k, the Levenberg-Marquardt method solves the subproblem min

s 1 2 u k + J k s 2 W + 1 2 µ k s 2 2 ,
(2.17)

to compute the step where µ k ≥ 0 is the parameter controlling the size of s k [p. 346] [START_REF] Björck | Numerical methods for least squares problems[END_REF]. With this formulation s k is well-defined also in the case where J(x) is rank-deficient. The problem (2.17) can be reformulated as a linear least-squares problem as follows:

min s 1 2 W 1/2 J k µ k I n s + W 1/2 u k 0 2 2 .
This formulation allows us again to use linear least-squares algorithms.

The local convergence behaviour of the Levenberg-Marquardt method is similar to that of the Gauss-Newton method (Dennis and Schnabel, 1983, Theorem 10.2.6). The method may be slowly locally convergent if the problem has a large-residual or is very nonlinear. Global convergence property of this method can be proved by reformulating the problem (2.17) as a trust-region problem (Nocedal and Wright, 2006, p. 261) whose global convergence is well-known from the trust-region theory.

We next explain the Gauss-Newton algorithm with a trust-region approach which can be considered as a generalization of the Levenberg-Marquardt method.

The Gauss-Newton method with a trust-region approach

The Gauss-Newton method equipped with a trust-region approach solves at each iteration the linear least-squares problem defined by (2.13) subject to a quadratic constraint, yielding min

s 1 2 u k + J k s 2 W , (2.18) subject to s 2 k ≤ ∆ k
where • k is an iteration dependent norm, and ∆ k is the radius of the trust region,

B k = {x ∈ IR n | x -x k k ≤ ∆ k } ,
which is the region where we believe that the objective function of the regularized nonlinear-least squares problem (2.4) is adequately approximated by 1 2 u k + J k s 2 W . After (possibly approximately) solving the subproblem (2.18), the step s k is accepted or rejected and the trust-region radius is updated accordingly. The acceptance of the trial point and the change in the trust-region radius are decided by considering the ratio

ρ k = f (x k ) -f (x k + s k ) m k (x k ) -m k (x k + s k ) , (2.19)
where f is the objective function given in (2.3) and m k is its quadratic approximation which is defined in (2.18). This ratio of achieved to predicted reductions gives an indication of the quadratic model's quality. If it is larger than a given small positive constant, the step is accepted and the trust-region radius possibly enlarged, while, if it is too small or negative, the step is rejected and the trust-region radius decreased. This approach can be given by Algorithm 2.1 proposed by (Conn, Gould and Toint, 2000, p. 116) in which we define the subproblem by (2.18). We name this algorithm basic Gauss-Newton trust-region algorithm (BGNTR).

Up to now, we presented variants of Gauss-Newton method that can be used to solve a regularized nonlinear-least squares problem. We next give details on how to calculate the Gauss-Newton step s k by applying Krylov subspace methods to the linear system (2.16).

Solving the linearized subproblem

We pointed out that solving the regularized nonlinear least-squares problem (2.4) with a Gauss-Newton method consists of solving linear systems in the form of (2.16) sequentially. In this section we explain how to adapt the Krylov subspace methods in this context.

Let us define

A = B -1 + H T k R -1 H k , and b = B -1 (x c -x k ) -H T k R -1 d k .
Then, the linear system (2.16) can be rewritten simply in the form

As = b.
(2.20)

Algorithm 2.1: Basic Gauss-Newton trust region algorithm 1 Initialization: Choose an initial point x 0 and an initial trust region radius ∆ 0 .

Choose the constants η 1 , η 2 , γ 1 , γ 2 that satisfy

0 < η 1 ≤ η 2 < 1 and 0 < γ 1 ≤ γ 2 < 1
Set k = 0 and calculate the function value at the initial, f (x 0 ).

2 Norm definition: Define • k .

3 Calculation of s k+1 approximately: Solve the subproblem (2.18).

4 Acceptance of the trial point: Compute the ratio

ρ k = f (x k ) -f (x k + s k+1 ) m k (x k ) -m k (x k + s k+1 ) , If ρ k ≥ η 1 , x k+1 = x k + s k+1 ; otherwise, x k+1 = x k .
5 Trust-region radius update:

Set ∆ k+1 ∈    [∆ k , ∞) if ρ k ≥ η 2 , [γ 2 ∆ k , ∆ k ) if ρ k ∈ [η 1 , η 2 ), [γ 1 ∆ k , γ 2 ∆ k ) if ρ k < η 1 ,
Increment k by 1 and go to step 2. Krylov subspace methods search for an approximate solution for a linear system in the form of (2.20) in a subspace s 0 + K l (A, r 0 ) where s 0 is the initial guess,

K l (A, r 0 ) = span r 0 , Ar 0 , A 2 r 0 , . . . , A l-1 r 0 , (2.21)
is the Krylov subspace of dimension l and r 0 = b -A s 0 is the initial residual. Note that the subspaces given by (2.21) are nested, i.e. K l ⊆ K l+1 .

To uniquely define the l-th iterate (s l ), several Krylov subspace methods impose the Petrov-Galerkin condition (Saad, 1996, p.144)

r l ⊥ L l (A, r 0 ), (2.22)
where L l (A, r 0 ) is a l-dimensional subspace. The condition (2.22) is called a Galerkin condition when L l (A, r 0 ) = K l (A, r 0 ), and leads to the Full Orthogonalization Method (FOM) which minimizes the residual in the A -1 norm, i.e. r A -1 . Choosing L l (A, r 0 ) = AK l (A, r 0 ) gives the Generalized Minimum RESidual method (GMRES) which finds the approximation s l by minimizing the residual in Euclidean norm, i.e. r 2 . If A is symmetric, then FOM and GMRES can be optimized to reduce the memory needs of the methods. These algorithms are known as the Lanczos method and the MINimum RESidual method (MINRES), respectively. It is also well known that Conjugate Gradient (CG) is mathematically equivalent to the Lanczos(-Cholesky) method when A is symmetric and positive definite (Saad, 1996, p.176).

In this study, we are interested in the Lanczos and the CG method which are well-known and widely used for solving large-scale and symmetric positive definite linear systems. Therefore, in the rest of the thesis we concentrate on these two Krylov subspace methods.

The conjugate gradient method (CG)

The linear Conjugate Gradients (CG) was proposed by [START_REF] Hestenes | Methods of conjugate gradients for solving linear systems[END_REF] and is among the most useful iterative techniques for solving large-scale symmetric positive definite linear systems. Actually, CG was originally considered as a direct method since in exact arithmetic the method converges in at most n iterations, n being the order of the linear system matrix. Later, it was shown that in finite arithmetic CG does not terminate at most n iterations [START_REF] Engeli | Refined iterative methods for computation of the solution and the eigenvalues of self-adjoint bondary value problems. (mitteilungen aus dem institut für angewandte mathematik der eth zürich, nr. 8.) 107 s. basel/stuttgart 1959. birkhäuser verlag. preis brosch. dm 17[END_REF], and it was noticed that the convergence of the method depends on the distribution of the eigenvalues of the linear system matrix [START_REF] Kaniel | Estimates for some computational techniques in linear algebra[END_REF]. This leads to the idea of transforming or preconditioning the linear system in a way that the eigenvalue distribution becomes more favorable and the convergence improves significantly. The CG method is used as an iterative method by [START_REF] Reid | On the method of conjugate gradients for the solution of large sparse linear equations[END_REF] who renewed attention to this algorithm. As a result, CG is recognized as a useful iterative method for symmetric positive definite linear systems [START_REF] Saad | Iterative solution of linear systems in the 20th century[END_REF].

CG solves the linear system of equations As = b where A is an n × n symmetric and positive definite matrix. Solving the linear system is equivalent to minimizing the quadratic problem [p. 102] (Nocedal and Wright, 2006)

min φ(s) = 1 2 s T As -b T s. (2.23) Problem (2.23) can be reformulated as min φ(s) = 1 2 s T As -b T s = 1 2 (s -s * ) T A(s -s * ) - 1 2 b T s * ,
where s * = A -1 b is the solution. Defining the A-norm which is also referred to as the energy norm to be x

2 A = x T Ax leads to min 1 2 s -s * 2 A - 1 2 b 2 A -1 .
Since the second term of this problem is constant, we can see that minimizing the quadratic problem (2.23) is also equivalent to minimizing the energy norm of the error, i.e. min 1 2 ss * 2 A .

(2.24)

The CG method minimizes the quadratic function φ(s) or equivalently solves the linear system As = b by generating the iterates {s i } i=0,1,...,l as a linear combination of the initial point s 0 ∈ IR n and the search directions {p i } i=0,1,...,l-1

s i+1 = s i + α i p i ,
where α i is the one-dimensional minimizer of the quadratic function φ(•) in the search direction p i which is computed as (Nocedal and Wright, 2006, p. 103)

α i = r T i p i p T i Ap i .
The new search direction is generated using only the previous direction, i.e.

p i = r i -β i p i-1
where r i is the residual and the scalar β i is chosen ensuring that p i and p i-1 are conjugate with respect to A, i.e.

p T i Ap j = 0, for all i = j.

Defining a new direction using only the previous one is a crucial property of the method in terms of its practical implementation.

The pseudo-code for the CG method is given in Algorithm 2.2.

Algorithm 2.2: CG Algorithm

1 r 0 = b -As 0 2 ρ 0 = r T 0 r 0 3 p 0 = r 0 4 for i = 0, 1, . . . , l -1 do 5 q i = Ap i 6 α i = ρ i /q T i p i 7 s i+1 = s i + α i p i 8 r i+1 = r i -α i q i 9
Check convergence and stop if desired accuracy is reached

10 ρ i+1 = r T i+1 r i+1 11 β i+1 = ρ i+1 /ρ i 12 p i+1 = r i+1 + β i+1 p i 13 end
Note that, Algorithm 2.2 minimizes the quadratic problem over the set s 0 + span{p 0 , p 1 , . . . , p l-1 } which is the same as s 0 + span{r 0 , Ar 0 , . . . , A l-1 r 0 } (Golub and Van Loan, 1996, p.523), where l is the number of the performed successful iterations.

The Lanczos method

The Lanczos method was proposed initially by [START_REF] Lanczos | An iteration method for the solution of the eigenvalue problem of linear differential and integral operators[END_REF] to compute the eigenvalues of a symmetric matrix. The "Lanczos idea" is later applied to solve large sparse linear equation and least squares problem [START_REF] Saunders | Solution of sparse rectangular systems using LSQR and CRAIG[END_REF]Saad, 1996). The results on the equivalence between the CG and Lanczos method for the solution of symmetric and positive definite linear systems can be found in Paige and Saunders (1975), [START_REF] Cullum | The Lanczos phenomenon-an interpretation based upon conjugate gradient optimization[END_REF], Golub and Van Loan (1996, p. 528), Saad (1996, p. 176-182) and Meurant (2006, p. 46-49).

As mentioned in Section 2.6, when the Lanczos method is used for the solution of a linear system in the form As = b, the method searches for an approximate solution s l in a subspace s 0 + K l (A, r 0 ) where s 0 is the initial guess, r 0 = b -A s 0 is the initial residual, and the Krylov subspace K l (A, r 0 ) is given by (2.21). It imposes the Galerkin condition r l ⊥ K l (A, r 0 ), which is equivalent to

V T l (b -As l ) = 0, (2.25)
where

V l = [v 1 , v 2 , ..., v l
] is an n × l matrix whose column vectors form an orthonormal basis for K l (A, r 0 ). The basis vectors are known as Lanczos vectors and are constructed using the symmetric Lanczos algorithm (Saad, 1996, p. 174-175). The Lanczos vectors are by construction orthonormal vectors, i.e. v T j v i = 0 for i = j and v T j v i = 1 for i = j. As a result, the approximate solution using the Krylov subspace K l (A, r 0 ) is given by

s l = s 0 + V l y l , (2.26)
where y l is derived as follows.

Multiplying the expression (2.26) with the matrix A from the left and using the resulting expression in the Galerkin condition (2.25) gives that

V T l (b -As 0 -AV l y l ) = 0, V T l (r 0 -AV l y l ) = 0, V T l AV l y l = V T l r 0 .
(2.27)

Choosing v 1 = r 0 / r 0 2 and defining a l-dimensional vector e 1 = [1, 0, ..., 0] T we obtain that

V T l r 0 = V T l r 0 2 v 1 = r 0 2 e 1 .
(2.28)

In (2.28) we have used the orthonormality property of the Lanczos vectors. Using the expressions (2.27) and (2.28) the vector y l is given by (Saad, 1996, p. 152-153)

y l = T -1 l ( r 0 2 e 1 ), (2.29)
where T l = V T l AV l . The matrix T l can be interpreted as a matrix representation of the projection of A onto the Krylov subspace K l (A, r 0 ).

The pseudo-code for the Lanczos method is given in Algorithm 2.3.

Algorithm 2.3: Lanczos Algorithm for Linear Systems

1 r 0 = b -As 0 2 β 0 = r 0 2 3 β 1 = 0 4 v 0 = 0 5 v 1 = r 0 /β 0 6 V 1 = v 1 7 for i = 1, ..., l do 8 w i = Av i -β i v i-1 9 α i = w T i v i 10 w i := w i -α i v i 11 β i+1 = w i 2 12 v i+1 = w i /β i+1 13 V i := [V i , v i+1 ] 14 (T i ) i,i = α i 15 if i > 1 then 16 (T i ) i-1,i = (T i ) i,i-1 = β i 17 end 18
Check convergence and stop if desired accuracy is reached

19 end 20 y l = T -1 l (β 0 e 1 ) 21 s l = s 0 + V l y l

From Lanczos to CG

In this section we provide the formulas that allow one to calculate the CG iterates from the Lanczos iterates. First we derive the formula for the residual r i = b -As i , i = 1, . . . , l. Replacing the expression (2.26) in the residual gives

r i = b -A(s 0 + V i y i ) = r 0 -AV i y i .
(2.30)

Using the following relation (Saad, 1996, p.148) (2.31) where V i , T i , β i+1 , v i+1 are defined in Algorithm 2.3 and e i is the i-th column of the i × i identity matrix, we obtain from (2.30) that

AV i = V i T i + β i+1 v i+1 e T i ,
r i = r 0 -V i T i y i -β i+1 v i+1 e T i y i .
Using the relation (2.29) and the definition v 1 = r 0 / r 0 2 (line 5 of Algorithm 2.3), the formula for the residual becomes

r i = r 0 -V i r 0 2 e 1 -β i+1 v i+1 e T i y i , = r 0 -r 0 2 v 1 -β i+1 v i+1 e T i y i , = -β i+1 v i+1 e T i y i .
(2.32)

The scalar α i and the vector q i can be derived from the lines 7 and 8 of the CG Algorithm respectively. The resulting algorithm to compute the iterates of CG Algorithm 2.2 from those of Lanczos Algorithm 2.3 is given by Algorithm 2.4. In Algorithm 2.4, the vector v i+1 is the (i + 1)-th column of the matrix V l and (T l ) i+1,i refers to the (i + 1, i)-th entry of the matrix T l .

Algorithm 2.4: Lanczos2CG

1 Given l, r 0 , s 0 , T l , β l+1 , V l from Algorithm 2.3 2 (T l ) l+1,l = β l+1 3 β 0 = r 0 2 4 p 0 = r 0 5 ρ = β 2 0 6 for i = 1, . . . , l do 7 y i = T -1 i (β 0 e 1 ) 8 r i = -(T l ) i+1,i v i+1 e T i y i 9 ρ i = r T i r i 10 β i = ρ i /ρ i-1 11 p i = r i + β i p i-1 12 s i = s 0 + V i y i 13 α i = (s i -s i-1 ) 2 / p i-1 2 14 q i = -(r i -r i-1 )/α i 15 end

Eigenspectrum estimates

During the minimization process, information on the eigenspectrum of the matrix A can be useful for constructing preconditioners. We explain in this section how to estimate these eigenpairs from the Lanczos and CG coefficients.

The eigenpairs of the matrix A can be estimated by calculating the eigenpairs of the tridiagonal matrix T l generated in Lanczos Algorithm 2.3. Let us denote the eigenvalues of the matrix T l by θ i , i = 1, . . . , l and corresponding eigenvectors by u i , i.e.

T l u i = θ i u i .

(2.33)

The pair given by (θ i , u i ) where

u i = V l u i , (2.34)
is known as Ritz pair and may be considered as approximates of the eigenpair of the matrix A. The eigenvalues θ i are known as the Ritz values and the approximated eigenvectors u i are known as the Ritz vectors (Meurant, 2006, p. 8).

From the theoretical equivalence of Lanczos and CG, it is possible to construct the matrices T l and V l from the coefficients of the CG Algorithm 2.2, and hence to obtain approximate eigenvalues of the matrix A. In particular, denoting the entries of the tridiagonal matrix T l by η i , γ i and η i+1 , i.e.

T l =         γ 1 η 2 η 2 γ 2 η 3 • • • η l-1 γ l-1 η l η l γ l        
, these entries can be obtained from Algorithm 2.2 using the relations (Saad, 1996, p.181-182)

γ i+1 = 1 αi if i = 0, 1 αi + βi-1 αi-1 if i > 0, (2.35) η i+1 = β i-1 α i-1 . (2.36)
The columns of the matrix V l can be generated by (Golub and Van Loan, 1996, p. 528)

v i+1 = (-1) i r i r i 2 .
(2.37)

Preconditioning

It is well-known that a preconditioner is desirable when applying an iterative method to accelerate the convergence. A preconditioner typically transforms the linear system into one that has more favorable spectral properties, since these properties are related with the method convergence rate. A good preconditioner should be cheap to construct and apply [START_REF] Benzi | Preconditioning techniques for large linear systems: A survey[END_REF]. With a good preconditioner the cost of constructing and applying the preconditioner should be less than the gain in computational cost [START_REF] Benzi | Preconditioning techniques for large linear systems: A survey[END_REF], (Barrett et al., 1994, p.35). We recommend the references [START_REF] Benzi | Preconditioning techniques for large linear systems: A survey[END_REF][START_REF] Saad | Iterative solution of linear systems in the 20th century[END_REF] for more information on a wide class of preconditioners. When using CG or the Lanczos method, a suitable preconditioner P can be used in the Preconditioned CG (PCG) and Preconditioned Lanczos (PLanczos) algorithms (Golub and Van Loan, 1996;Nour-Omid et al., 1988;Axelsson, 1996;Chan et al., 1999). These algorithms are given respectively by Algorithm 2.5 and Algorithm 2.6.

In Algorithm 2.6 the norm is defined as x P = √ x T P x. The algorithm that generates the iterates of PCG from PLanczos is given by Algorithm 2.7 for further reference. Note that, in Algorithm 2.7, the vector v i+1 is the (i + 1)-th column of the matrix V l and (T l ) i+1,i refers to the (i + 1, i)-th entry of the matrix T l .

Algorithm 2.5: PCG Algorithm

1 r 0 = b -As 0 2 z 0 = P r 0 3 ρ 0 = r T 0 z 0 4 p 0 = z 0 5 for i = 0, 1, . . . , l -1 do 6 q i = Ap i 7 α i = ρ i /q T i p i 8 s i+1 = s i + α i p i 9 r i+1 = r i -α i q i z i+1 = P r i+1
Check convergence and stop if desired accuracy is reached

ρ i+1 = r T i+1 z i+1 β i+1 = ρ i+1 /ρ i p i+1 = z i+1 + β i+1 p i end Algorithm 2.6: PLanczos 1 r 0 = b -As 0 2 z 0 = P r 0 3 β 0 = r 0 P 4 β 1 = 0 5 v 0 = 0 6 v 1 = r 0 /β 0 7 z 1 = z 0 /β 0 8 V 1 = v 1 9 for i = 1, . . . , l do w i = Az i -β i v i-1 α i = w T i z i w i := w i -α i v i z i := P w i β i+1 = w i P v i+1 = w i /β i+1 z i+1 := z i /β i+1 V i := [V i , v i+1 ] (T i ) i,i = α i if i > 1 then (T i ) i-1,i = (T i ) i,i-1 = β i end
Check convergence and stop if desired accuracy is reached

end y l = T -1 l (β 0 e 1 ) s l = s 0 + P V l y l Algorithm 2.7: PLanczos2PCG 1 Given P, r 0 , s 0 , T l , β l+1 , V l , l from Algorithm 2.6 2 (T l ) l+1,l = β l+1 3 z 0 = P r 0 4 β 0 = r T 0 z 0 5 p 0 = z 0 6 ρ = β 2 0 7 for i = 1, . . . , l do 8 y i = T -1 i (β 0 e 1 ) 9 r i = -(T l ) i+1,i v i+1 e T i y i 10 z i = P r i 11 ρ i = r T i z i 12 β i = ρ i /ρ i-1 13 p i = z i + β i p i-1 14 s i = s 0 + P V i y i 15 α i = (s i -s i-1 ) 2 / p i-1 2 16 q i = -(r i -r i-1 )/α i 17 end

Preconditioning with Limited Memory Preconditioners

In this study we consider a class of Limited Memory Preconditioners (LMPs) that is designed for solving systems of equations with multiple right hand sides [START_REF] Morales | Automatic preconditioning by limited memory quasinewton updating[END_REF]Gratton, Sartenaer and Tshimanga, 2011). The general definition of the LMP is given as (Gratton, Sartenaer and Tshimanga, 2011, Definition 2.1):

Definition 2.1 Let A and M be symmetric and positive definite matrices of order n and assume that S is any n by matrix of rank , with ≤ n. The symmetric matrix

P = [I n -S(S T AS) -1 S T A]M [I n -AS(S T AS) -1 S T ] + S(S T AS) -1 S T (2.38)
is called the Limited Memory Preconditioner (LMP).

In the formula (2.38), the matrix M is considered to cluster most eigenvalues at 1 with relatively few outliers and called first-level preconditioner (Gratton, Sartenaer and Tshimanga, 2011). Whereas, the preconditioner P is called second-level preconditioner and is used to capture the directions which slows down the CG method (or the Lanczos method) that the first-level preconditioner M is leaving out (Gratton, Sartenaer and Tshimanga, 2011). The main properties of the preconditioner P can be summarized as (see (Gratton, Sartenaer and Tshimanga, 2011) and [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF])

• if = n in Definition 2.1, then P = A -1 ,
• the LMP preconditioner is symmetric and positive definite,

• the LMP preconditioner clusters at least eigenvalues of the matrix P A at 1 and does not expand the remaining part of the spectrum,

• the LMP can be applied as an operator in order to calculate matrix-vector multiplications,

• the LMP is cheap to apply.

Under the assumption that the columns of S ∈ IR n× are conjugate with respect to the matrix A, i.e. s T i As j = 0 for i = j, and setting M = P 0 the preconditioner (2.38) results in a simplified formula (Gratton, Sartenaer and Tshimanga, 2011) given by

P l = I n - l i=1 s i s T i s T i As i A P 0 I n - l i=1 As i s T i s T i As i + l i=0 s i s T i s T i As i . (2.39)
From the formula (2.39) or (2.38) three particular instances of the LMPs: quasi-Newton LMP, spectral LMP and Ritz LMP are derived by using the relevant information obtained from a CG (or Lanczos) run while solving a linear system involving A.

The quasi-Newton LMP is constructed by using l successive (linearly independent) search directions, p i , i = 1, . . . , l obtained during a CG (or Lanczos) run. Replacing s i by p i in the formula (2.39) results in the quasi-Newton preconditioner inductively given as (Gratton, Sartenaer and Tshimanga, 2011)

P l = I n - p l p T l A p T l Ap l P l-1 I n - Ap l p T l p T l Ap l + p l p T l p T l Ap l . (2.40)
This preconditioner amounts to the preconditioner proposed by [START_REF] Morales | Automatic preconditioning by limited memory quasinewton updating[END_REF].

The Ritz LMP uses the linearly independent Ritz vectors u i , i = 1, . . . , l and the Ritz values θ i generated during a CG (or Lanczos) run. Choosing S = [u 1 , . . . , u l ] and M = I n in the formula (2.38) leads to the Ritz LMP matrix given as (Gratton, Sartenaer and Tshimanga, 2011)

P l = I n + S(Θ -1 -I l )S T -Sωv T l+1 -v l+1 ω T S T + Sωω T S T (2.41)
where Θ = diag(θ i ) and ω = (ω 1 , . . . , ω l ) T , with

ω i = (e T l u i )β l+1 θ i , for i = 1, . . . , l.
(2.42)

The vector u i in (2.42) denotes the eigenvector of the tridiagonal matrix T l and β l+1 is defined in Lanczos Algorithm 2.3. The spectral LMP is generated from the independent unit eigenvectors υ i of A and corresponding eigenvalues λ i . Choosing S = [υ 1 , . . . , υ l ] and M = I n in the formula (2.38) leads to the spectral LMP matrix given as (Gratton, Sartenaer and Tshimanga, 2011)

P = I n -S(Λ -1 -I l )S T (2.43)
where Λ = diag(λ i ). The spectral-LMP amounts to the well-known preconditioner in operational data assimilation systems proposed by [START_REF] Fisher | Minimization algorithms for variational data assimilation[END_REF].

It is shown in [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF] that the spectral LMP generated by Ritz information behaves like the Ritz LMP when the the error on the Ritz pair is small. It is also noted in Gratton, Sartenaer and Tshimanga (2011, Theorem 4.6) that when all the available information (search directions and Ritz vectors) are used while constructing the quasi-Newton LMP and the Ritz LMP, these LMPs span the same space and hence, they are mathematically equivalent. For more details on comparisons of the mentioned preconditioners, we recommend the references (Gratton et al., 2011;Tshimanga et al., 2008;[START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF].

We summarized above the LMPs and their effects when used for preconditioning systems of equations with multiple right hand sides. It is also important to understand how to use LMPs in the context of nonlinear least-squares problem. Since the local quadratic approximations of the nonlinear least-squares problem change, the solution of the problem with an iterative method (like the Gauss-Newton method) requires solving a sequence of slightly varying systems. In this case, the LMP formula (2.38) would still be applicable provided that the new matrix A is used. Note that, for this preconditioner to be considered effective, the cost of constructing and applying the preconditioner should be less than the gain obtained.

CG relation with LSQR and Generalized LSQR

In the previous sections, we focus on solving the linearized system (2.16)

(B -1 + H T R -1 H) s = B -1 (x c -x) -H T R -1 d,
(2.44) (where we have dropped the index k for simplicity) by using a CG or Lanczos method. This linear system represents the normal equations

(J T W J)s = -J T W u
where the matrices J and W and the vector u are defined in Section 2.3. As mentioned before this normal equation is associated with the linear least-squares problem (2.13):

min s 1 2 Js + u 2 W (2.45)
or equivalently given as

min s 1 2 Hs + d 2 R -1 + 1 2 s + x -x c 2 B -1 .
(2.46)

The overdetermined least-squares problem (2.45) can be alternatively solved by using the well-known LSQR method [START_REF] Paige | LSQR: an algorithm for sparse linear equations and sparse least squares[END_REF][START_REF] Saunders | Solution of sparse rectangular systems using LSQR and CRAIG[END_REF], which is based on the Lanczos bidiagonalization procedure (Golub and Van Loan, 1996, p. 495).

When considering the equivalent regularized least-squares problem (2.46), the LSQR method could not be applied directly since it requires a special structure. Defining a variable δ x = B -1/2 (s + xx c ) and substituting it into (2.46), the problem can also be stated in this special structure as min δ x

1 2

H I n δ x - c 0 2 2 , (2.47)
where H = R -1/2 HB 1/2 is a m-by-n matrix with m < n and c = R -1/2 H(xx c ) -R -1/2 d. It is noted in (Saunders, 1995, Result 8) that since H T H + I n is symmetric and positive definite, the LSQR method generates mathematically equivalent iterates δ x i to those of CG on the linear system

( H T H + I n ) δ x = H T c.
(2.48)

This linear system can be written as:

(B 1/2 H T R -1 HB 1/2 + I) δ x = B 1/2 H T R -1 (H(x -x c ) -d).
(2.49)

Note that, at convergence the solution of the linear system (2.44) and the solution of the linear system (2.49) are related by the expression

s k = x c -x k + B 1/2 δ x k (
where k is the index for the outer loop of the Gauss-Newton method).

The problem structure for solving a regularized least squares problem with LSQR can be generalized by using the a non-standard norm [START_REF] Arioli | Iterative methods for symmetric quasi-definite linear systems[END_REF]. This problem can then be solved by using the generalized LSQR (G-LSQR) method proposed by [START_REF] Arioli | Iterative methods for symmetric quasi-definite linear systems[END_REF]. G-LSQR generates mathematically equivalent iterates δx i to those of applying PCG on the linear system

(B -1 + H T R -1 H) δx = H T R -1 (H(x -x c ) -d),
(2.50)

with the preconditioner matrix B. Accordingly, it also generates the same iterates as those of CG applied on the linear system (2.49) with δx i = B 1/2 δ x i . Note also that, at convergence the solution of the linear system (2.44) and the solution of the linear system (2.50) are related with

s k = x c -x k + δx k .
LSQR is numerically more reliable than CG when the system matrix is ill-conditioned and many iterations are performed (Björck, 1996, p. 288, 307-309), [START_REF] Paige | LSQR: an algorithm for sparse linear equations and sparse least squares[END_REF]. However it requires extra storage which may not be afordable for large scale problems. LSQR can be preferable to CG when memory is not a problem and high accuracy is expected.

The dual problem

The original quadratic subproblem (2.45), considered the primal problem, can be converted into an alternative problem called the dual problem. The dual problem can be used in practice for sensitivity analysis, or to design algorithms for solving the primal problem (Nocedal and Wright, 2006, p. 343).

Let us rewrite the convex quadratic subproblem (2.45) as a constrained problem min r,s

1 2 r 2 W subject to Js + u -r = 0 (2.51)
which can, in turn, be solved using duality theory. The dual objective function for the primal objective function in (2.51) can be defined as (Nocedal and Wright, 2006, p.343-349)

q(λ) = inf r,s L(r, s, λ) def = inf r,s 1 2 r T W r -λ T (Js + u -r), (2.52)
where q : IR m+n → IR and λ ∈ IR m+n is a Lagrange multiplier vector. The infimum is achieved when

∇ r L(r, s, λ) = W r + λ = 0, ∇ s L(r, s, λ) = J T λ = 0,
which yields that r = -W -1 λ. We may therefore substitute r and use the equation J T λ = 0 in the expression (2.52) and obtain the dual objective explicitly as follows:

q(λ) = - 1 2 λ T W -1 λ -λ T u.
Then the dual problem for the constrained primal problem (2.51) is given by max λ q(λ).

(2.53)

Note that the cost function of the dual problem (2.53) is quadratic and the dual problem has no constraints. This dual problem is defined on a space of the same dimension as the primal problem. We will see in Chapter 4 that there may be different dual approaches based on the structure of the primal problem, which allows us to define the dual problem on a space of smaller dimension than the primal problem, that will provide computationally more efficient algorithms.

The relation between the constrained primal and dual problem can be given from the weak duality theorem (Nocedal and Wright, 2006, Theorem 12.11), i.e. for any r feasible for (2.51) and any λ ∈ IR m+n , we have q(λ) ≤ f (r) where, since f does not depend on s in this particular case, f (r) = 1 2 r 2 W . From this theorem one can observe that the optimal value, λ * , of the dual problem (2.53) gives a lower bound on the optimal value, r * , for the constrained primal problem (2.51) (Nocedal and Wright, 2006, p. 345).

If q(λ * ) = f (r * ), it is said that there is no duality gap and if q(λ * ) < f (r * ) it is said that there is a duality gap (Bertsekas, 1995, Section 5.1.2). Since the constrained primal problem has a convex cost function with a linear constraint, from strong duality theorem (Bertsekas, 1995, Proposition 5.1.2), there is no duality gap between the constrained primal and dual problems given by (2.51) and (2.53) respectively.

CHAPTER 3

Solving the subproblem with the conjugate-gradient method This chapter considers applying the preconditioned conjugate-gradients to a sequence of symmetric and positive definite linear systems of the form

A k s = b k , where A k = B -1 + H T k R -1
H k and b k are defined in (2.16), and where preconditioning is achieved by using the Limited Memory Preconditioners (LMPs) described in Section 2.6.6. These linear systems arise from a sequence of linear least-squares subproblems (2.13), of a Gauss-Newton (GN) algorithm with each member of the sequence being the least-squares problem (2.3). We recall that the matrices B and R appearing in the linear system are the symmetric and positive definite error-covariance matrices.

The LMP for the system matrix A k is obtained by using directions generated during solving the linear system at (k -1)-th outer loop of the GN algorithm. This way of preconditioning improves a preconditioner at each outer loop and is known as the warmstart preconditioning. We first focus on warm-start preconditioning of a sequence of linear systems with multiple right-hand sides, As = b k , by using directions generated during solving the previous linear system. Note that, each linear system is part of a sequence, which is supposed to produce a convergent sequence of solutions.

We present different ways of preconditioning the linear system As = b k with a warm-start preconditioner. The first approach is to use a split preconditioning with a standard conjugate gradient method (CG Algorithm 2.2). This approach requires the square-root factorization of the preconditioner, i.e. P = F F T . In a particular case when P = B, this approach avoids the matrix-vector products with B -1 under the assumption that B 1/2 is available. The second alternative is to use PCG Algorithm 2.5 where the preconditioner P can be directly used without requiring any square-root factorization.

Under the assumption that the B -1 P operator is available, which is possible as shown in Section 3.1, we propose a variant of conjugate-gradients as a third alternative which uses right-symmetric-preconditioning that both avoids the matrix-vector products with B -1 and the square-root factorization of the matrix B. This alternative can be preferable especially for large scale problems since calculating the inverse or a factorization of B, as required by the two first approaches, may not be affordable in terms of computational cost.

Starting from these three types of preconditioning, we present three algorithms solving a sequence of linear systems with multiple right hand-sides in which the inner minimization is performed with the corresponding conjugate gradients approach using the general formulations of the corresponding LMPs. These algorithms generate a mathematically equivalent sequence of iterates {s k }. We continue the chapter by discussing how to use particular LMPs such as the quasi-Newton LMP, Ritz LMP and spectral LMP, for each approach.

We later analyze three variants of conjugate gradients when used for solving a sequence of linear systems in the form of A k s = b k . We conclude the chapter by providing inexpensive formulae to compute the quadratic cost function (2.13) and the norm of its gradient, which are useful quantities to stop inner iterations in a nonlinear solver.

Preconditioning the conjugate-gradient method with LMPs

In this section we are dealing with applying conjugate gradients to the systems of equations with multiple right hand sides,

(B -1 + H T R -1 H) s = B -1 (x c -x k ) -H T R -1 d k , (3.1) of the form of As = b k ,
with a warm-start LMP. Each linear system is part of a sequence and is not isolated.

It is assumed that this sequence produces a convergent sequence of solutions.

We next present different approaches on preconditioning of the linear system (3.1) with a symmetric and positive definite LMP defined by (2.38),

P = [I n -S(S T AS) -1 S T A]M [I n -AS(S T AS) -1 S T ] + S(S T AS) -1 S T ,
in the sense that each approach requires different forms of the preconditioner and may avoid expensive matrix-vector products (when the linear system is of large-scale). We recall that the preconditioner P is often called second-level preconditioner for the matrix A and used to capture the directions which slow down the CG method that the firstlevel preconditioner M is leaving out.

While presenting different approaches, we first explain the approach over a single linear system. We later explain how to use this method for solving a convergent sequence of linear systems. In this case using a warm-start technique improves the preconditioner along the sequence.

We start with the approach that uses a split preconditioning with the standard conjugate gradient (Algorithm 2.2) as explained below.

Preconditioning with CG Algorithm 2.2 Solving the linear system (3.1) with a LMP P formulated by (2.38) by using CG Algorithm 2.2 requires the factored form of the LMP. We recall Theorem 3.9 of Gratton, Sartenaer and Tshimanga (2011) that provides this factorization.

Let M = LL T with L T ∈ IR n×n , and S T AS = G T G with G ∈ IR × . Then the LMP matrix of P can be factored as P = F F T with F given by

F = L -SG -1 G -T S T AL + SG -1 X -T S T L -T (3.2)
where

S T L -T L -1 S = X T X, with X ∈ IR × , is any factorization of S T L -T L -1 S.
Using this factored form of the LMP P , Algorithm 2.2 solves the preconditioned system

F T AF s = F T b k , (3.3)
for the preconditioned variable s. The approximate solution s is then obtained from the relation s = F s.

Let us now explain in detail how to solve a convergent sequence of linear systems (3.1) by using CG Algorithm 2.2 with a warm-start LMP formulated by (3.2). Assume that we choose an initial guess x 1 . Assume also that the factorization of the initial preconditioner

P 0 = F 0 F T 0 (3.4)
is available. Then for k = 1, CG Algorithm solves the preconditioned system

F T 0 AF 0 A1 s = F T 0 b 1 b1 , (3.5)
for the variable s and recovers the approximate solution from the relation s 1 = F 0 s 1 . This approximate solution is then used to update the current iterate from x 2 = x 1 + s 1 as explained in Section 2.5.

While performing the CG algorithm on the linear system (3.5), the information can be stored to obtain the factored form of the LMP for the system matrix A 1 , i.e. Y 1 = F 1 F T 1 , with F 1 is constructed from (3.2). Note that while constructing F 1 from (3.2), M 0 is taken as an identity matrix of order n since P 0 = F 0 F T 0 is already acting as a first-level preconditioner.

Using the preconditioner F 1 for k = 2, CG Algorithm solves the preconditioned system

F T 1 (F T 0 AF 0 )F 1 A2 s = F T 1 F T 0 b 2 b2 , (3.6)
for the variable s and recover the approximate solution from s 2 = F 0 F 1 s 2 . This approximate solution is then used to update the current iterate from x 3 = x 2 + s 2 .

While performing the CG algorithm on the linear system (3.6) the information can be stored to obtain the factored form of the LMP for the system matrix A 2 , i.e. Y 2 = F 2 F T 2 , with F 2 is constructed from (3.2). Note that while constructing F 2 from (3.2), M 1 is taken again as an identity matrix of order n since in this case the matrix F 1 F 0 F T 0 F T 1 can be considered as an implicit first-level preconditioner. The same strategy can be carried over for later steps. Using the preconditioner F k , for the (k + 1)-th linear system CG Algorithm solves the preconditioned system

F T k (F T k-1 . . . F T 1 F T 0 AF 0 F 1 . . . F k-1 )F k A k+1 s = F T k F T k-1 . . . F T 1 F T 0 b k+1 b k+1 , (3.7)
for the variable s and recover the approximate solution from

s k+1 = F 0 F 1 . . . F k-1 F k s k+1 .
This approximate solution is then used to update the current iterate from x k+2 = x k+1 + s k+1 . Algorithm 3.1 outlines this implementation.

Algorithm 3.1: CG for solving a convergent sequence of linear systems with multiple right-hand sides 1 Initialization: Choose an initial preconditioner P 0 and factorize the preconditioner such that P 0 = F 0 F T 0 . Choose an initial vector x 1 . 2 Calculate preconditioned right hand side bk :

bk = F T k-1 . . . F T 1 F T 0 [B -1 (x c -x k ) -H T R -1 d k ]
3 Perform inner loop: Solve the preconditioned linear system

A k s = b k ,
with CG Algorithm 2.2. During CG extract and save the information to precondition the next linear system with F k where F k is constructed from (3.2).

4 Recover the approximate solution:

s k = F 0 F 1 . . . F k-1 s k
5 Update the iterate:

x k+1 = x k + s k
6 Increment k by 1 and go to step 2.

This algorithm requires the factored form of the LMP, and assumes that the factorization of the initial preconditioner, i.e. P 0 = F 0 F T 0 , is available. When solving the k-th linear system in Algorithm 3.1, CG Algorithm 2.2 requires a single matrix-vector product with each of the operators B -1 , H, H T , R -1 , F i and F T i with i = 0, . . . k -1. Note that the cost of the matrix-vector products with the limited memory preconditioners F i and F T i , for i > 0 is not expensive [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF]. When there is no initial preconditioner available, the initial preconditioner can be chosen as P 0 = B since it is known that this preconditioning clusters most eigenvalues at 1. Assuming that B = B 1/2 B 1/2 is available, it can be easily seen from (3.7) that matrix vector products with B -1 can be avoided by choosing

F 0 = B 1/2 , i.e. F T 0 AF 0 = I n + B 1/2 H T R -1 HB 1/2
. This can reduce the computational cost since the matrix B is not diagonal in general and therefore the matrix-vector products with B -1 may be expensive especially for large-scale problems. As a result, in this special case, when solving the k-th linear system in Algorithm 3.1, the CG Algorithm requires matrix-vector products with each of the operators B 1/2 , H, H T , R -1 , F i , and

F T i with i = 1, . . . k -1.
We continue this section by explaining another way of preconditioning a sequence of linear systems (3.1) with PCG Algorithm 2.5.

Preconditioning with PCG Algorithm 2.5

The main difference in solving the linear system (3.1) with PCG Algorithm 2.5 rather than CG Algorithm 2.2 with a preconditioner is that PCG Algorithm 2.5 does not require the factored form of the preconditioner. This can be algorithmically significant when the preconditioner factorization is not available or expensive to compute.

Therefore, another way of preconditioning can be performed by applying Algorithm 2.5 to linear system (3.1) of the form of As = b k with a LMP (2.38).

Let us now explain in detail how to solve a convergent sequence of linear systems (3.1) by using PCG Algorithm 2.5 with a warm-start LMP formulated by (2.38). Assume that we choose an initial guess x 1 . Then for k = 1, PCG Algorithm solves the system A s = b 1 , (3.8) for the variable s by using an initial preconditioner P 0 . The current iterate is updated by using the approximate solution s 1 from x 2 = x 1 + s 1 .

While performing the PCG algorithm on the linear system (3.8), the information can be stored to obtain the LMP P 1 for the system matrix A by using the formula (2.38). This preconditioner can then be used to precondition the second linear system. Note that while constructing P 1 from (2.38), the first level preconditioner M 0 is taken as the initial preconditioner, i.e. M 0 = P 0 .

Using the preconditioner P 1 for k = 2, PCG Algorithm solves the linear system

A s = b 2 , (3.9)
for the variable s. The current iterate is updated by using the approximate solution s 2 from x 3 = x 2 + s 2 .

While performing the PCG algorithm on the linear system (3.9) the information can be stored to obtain the LMP P 2 for the system matrix A by using the formula (2.38). Note that while constructing P 2 from (2.38), the first level preconditioner M 1 is taken as the preconditioner used for k = 2, i.e. M 1 = P 1 .

The same strategy can be carried over for later steps. Using the preconditioner P k , for the (k + 1)-th linear system PCG Algorithm solves the preconditioned system

A s = b k+1 , (3.10)
for the variable s. The current iterate is updated by using the approximate solution s k+1 from x k+2 = x k+1 + s k+1 . Algorithm 3.2 outlines this implementation.

Algorithm 3.2: PCG for solving a convergent sequence of linear systems with multiple right-hand sides 1 Initialization: Choose an initial preconditioner P 0 , and an initial vector x 1 .

2 Calculate right hand side b k :

b k = B -1 (x c -x k ) -H T R -1 d k
3 Perform inner loop: Using the preconditioner P k-1 solve the linear system:

As = b k
with PCG Algorithm 2.5. During PCG extract and save the information to precondition the next linear system with an LMP P k formulated by (2.38).

4 Update the iterate:

x k+1 = x k + s k
5 Increment k by 1 and go to step 2.

Note that, this algorithm does not require the factorization of the preconditioner.

When solving the k-th linear system in Algorithm 3.2, PCG Algorithm 2.5 requires a single matrix-vector product with each of the operators B -1 , H, H T , R -1 and P k-1 . Note that the cost of matrix-vector products with the limited memory preconditioner P k , k > 0 is not expensive [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF]Gratton, Sartenaer and Tshimanga, 2011).

As with Algorithm 3.1, when there is no initial preconditioner available, the initial preconditioner can be chosen as P 0 = B. In this special case, for the first linear system of Algorithm 3.2 during PCG the matrix vector products with B -1 can be avoided (Derber and Rosati, 1989) as explained below.

During PCG Algorithm 2.5, the system matrix A = B -1 + H T R -1 H is applied to the search direction p i (see line 6 of Algorithm 2.5). Let us define an additional vector p i defined by p i = B -1 p i . Taking P = B and multiplying line 14 of Algorithm 2.5 by B -1 , we obtain that (3.11) where we used the fact that r i = P -1 z i = B -1 z i , for i > 0. Therefore, the matrixvector products with B -1 can be avoided by introducing the vector p i . As a result, in this special case (P 0 = B) for the first linear system of Algorithm 3.2, the PCG Algorithm requires a single matrix-vector product with each of the operators H, H T , R -1 and B.

p i = r 0 if i = 0, r i + β i p i-1 if i > 0,
Note that, when the same preconditioning P k = B is applied for the linear systems in sequence within Algorithm 3.2, matrix-vector products with B -1 can also be avoided. However, using the LMP formulated by (2.38) does not allow us to use the vector p i for the matrix-vector products of B -1 p i since r i = P -1 z i = B -1 z i .

Is there any other way of preconditioning the linear system (3.1) with the LMP given by (2.38) that avoids matrix-vector products with B -1 as well as the factorization of the preconditioner? In the next part, we answer this question.

Preconditioning with PCG Inverse Free Algorithm

We can apply the symmetric and positive definite matrix P formulated by (2.38) as a right symmetric preconditioner while solving the linear system (3.1). This leads to solving the linear system

(B -1 P + H T R -1 HP ) v = B -1 (x c -x k ) -H T R -1 d k ,
(3.12) equipped with the P -inner product for the variable v. The approximate solution s is then obtained from s = P v.

Let us now assume that the matrix B -1 P is available and define this matrix by C = B -1 P . Substituting the relation P = BC into the linear system (3.12) gives that (3.13) which suggests solving the linear system

(C + H T R -1 HBC) v = B -1 (x c -x k ) -H T R -1 d k ,
(I n + H T R -1 HB) s = B -1 (x c -x k ) -H T R -1 d k , (3.14)
equipped with the B-inner product for the variable s by using the preconditioner matrix C. Note that, the system matrix in (3.14) is symmetric with respect to the B-inner product. The approximate solution is recovered from s = Bs. This way of minimization with conjugate gradients results in PCG Inverse Free (PCGIF) Algorithm 3.3 that avoids matrix-vector multiplication with B -1 . It can be easily shown that the vectors defined in PCG Algorithm 2.5 when applied to the linear system (3.1) with a preconditioner P are related with the vectors defined Algorithm 3.3: PCGIF Algorithm (version 1)

1 r 0 = b -ABs 0 2 z 0 = Cr 0 3 p 0 = z 0 4 ρ 0 = r T 0 Bz 0 5 for i = 0, 1, . . . , l -1 do 6 q i = (p i + H T R -1 HBp i ) 7 α i = ρ i /q T i Bp i 8 s i+1 = s i + α i p i 9 r i+1 = r i -α i q i 10 z i+1 = Cr i+1 11
Check convergence and stop if desired accuracy is reached

12 ρ i+1 = r T i+1 Bz i+1 13 β i+1 = ρ i+1 /ρ i+1 14 p i+1 = z i+1 + β i+1 p i 15 end
in PCGIF Algorithm with the following relations

r i = r i z i = Bz i q i = q i s i = Bs i p i = Bp i                (i ≥ 0) (3.15)
under the assumption that B -1 P is available. Using the relation p i = Bp i and Aconjugacy property of p i we also have the property

p T i BABp j = 0 for i = j, (3.16)
for further reference. Algorithm 3.3 can be transformed into a cheaper form by introducing additional vectors, reducing its cost per loop to a single matrix-vector product with B. More precisely, consider p and z defined by

z i = Bz i and p i = Bp i .
(3.17)

for i ≥ 0. If we multiply lines 3 and 14 of Algorithm 3.3 by the matrix B, we get

p i = z 0 if i = 0, z i + β i p i-1 if i > 0.
(3.18)

Substituting these definitions into Algorithm 3.3, yields the final version of PCGIF Algorithm (Algorithm 3.4).

Algorithm 3.4: PCGIF Algorithm

1 r 0 = b -ABs 0 2 z 0 = Cr 0 3 p 0 = z 0 4 z 0 = Bz 0 5 ρ 0 = r T 0 z 0 6 p 0 = z 0 7 for i = 0, 1, . . . , l -1 do 8 q i = (p i + H T R -1 Hp i ) 9 α i = ρ i /q T i p i 10 s i+1 = s i + α i p i 11 r i+1 = r i -α i q i 12 z i+1 = Cr i+1 13 z i+1 = Bz i+1 14
Check convergence and stop if desired accuracy is reached

15 ρ i+1 = r T i+1 z i+1 16 β i+1 = ρ i+1 /ρ i+1 17 p i+1 = z i+1 + β i+1 p i 18 p i+1 = z i+1 + β i+1 p i 19 end
Now, the question is whether it is possible to obtain the matrix C = B -1 P as the form of a limited memory type (or in other words find a limited memory type preconditioner C such that P = BC) where P is an LMP formulated by (2.38). The following lemma provides a general formula that allows one to obtain such a matrix C. Lemma 3.1 Let BA and BM be symmetric and positive definite matrices of order n. Assume also that S is any n by l matrix of rank l, with l ≤ n. Then the n-by-n matrix given by

C = [I n -S(S T BA S) -1 S T BA] M [I n -A S(S T BA S) -1 S T B] + S(S T BA S) -1 S T B (3.19
) is symmetric with respect to the B-inner product and BC is positive definite.

Suppose also that M = BM . If we denote S = BS and A = AB then the LMP matrix P given by (2.38) and the matrix C defined by (3.19) satisfy P = BC.

Proof. Multiplying C on the left by B gives that

BC = [B-BS(S T BAS) -1 S T BA]M [I n -AS(S T BAS) -1 S T B]+BS(S T BAS) -1 S T B.
(3.20) Using the assumption that BA is symmetric, we have

BC =[I n -BS(S T BA S) -1 S T A T ] BM [I n -A S(S T BA S) -1 S T B] + BS(S T BA S) -1 S T B.
Similarly using the symmetry of BM and BA we obtain that

BC = [I n -BS(S T BA S) -1 S T A T ] M T [B -BA S(S T BA S) -1 S T B] +BS(S T BA S) -1 S T B, BC = [I n -BS(S T BA S) -1 S T A T ] M T [I n -A T B S(S T BA S) -1 S T ]B +BS(S T BA S) -1 S T B, = C T B,
which proves symmetry of C with respect to the B-inner product.

Since BA is symmetric and positive definite and S has full column rank, the matrix S T BA S is also symmetric and positive definite (Golub and Van Loan, 1996, Theorem 4.2.1). Therefore, the matrix S T BA S has an inverse and it can be factored such that S T BA S = (S T BA S) 1/2 (S T BA S) 1/2 . Defining the matrix V = S(S T BA S) -1/2 and substituting it in equation (3.20), we obtain that

BC = [B -BV V T BA] M [I n -AV V T B] + BV V T B.
(3.21)

Let us define symmetric and positive definite matrices as M = BM and A = AB -1 , and the matrix W = BV with W T AW = I l , I l is the identity matrix of order l. Substituting these matrices in (3.21) gives that

BC = [I n -BV V T BA] M [I n -ABV V T B] + BV V T B = [I n -W W T A] M [I n -AW W T ] + W W T . (3.22)
In Lemma 3.3 of the reference (Gratton, Sartenaer and Tshimanga, 2011), it is proved that a matrix having the form of (3.22) is positive definite under the assumptions that A and M are symmetric and positive definite and W T AW = I l . Applying the same results proves that BC is a positive definite matrix.

Using the relations M = BM , S = BS and A = AB given in the lemma, we deduce from (2.38) that

P = [I n -BS(S T BABS) -1 S T BA]BM [I n -ABS(S T BABS) -1 S T B] +BS(S T BABS) -1 S T B = [B -BS(S T BA S) -1 S T BA] M [I n -A S(S T BA S) -1 S T B] +BS(S T BA S) -1 S T B = BC
which completes the proof.

2

The formula (3.19) given in Lemma 3.1 can be rewritten inductively if S ∈ IR n× be A-conjugate with respect to the B-inner product. The next lemma provides such form of the preconditioner (3.19).

Lemma 3.2 Let BA and BM be n × n symmetric positive definite matrices. Assume also that S ∈ IR n× be A-conjugate with respect to the B-inner product. Then the matrix C defined by (3.19) can be written as

C = I n - i=1 s i s T i s T i BA s i BA M I n - i=1 A s i s T i s T i BA s i B + i=0 s i s T i s T i BA s i B, (3.23)
which can also be written inductively as

C = I n - s s T BA s T BA s C -1 I n - A s s T B s T BA s + s s T B s T BA s . (3.24)
Proof. From the A-conjugacy of the matrix S with respect to the B-inner product, S T BA S becomes a diagonal matrix. Then, S(S T BA S) -1 S T can be written as

S(S T BA S) -1 S T = i=1 s i s T i s T i BA s i . ( 3
BC = I n - Bs s T BAB -1 s T BAB -1 Bs BC -1 I n - AB -1 Bs s T B s T BAB -1 Bs + Bs s T B s T BAB -1 Bs = B - Bs s T BA s T BA s C -1 I n - A s s T B s T BA s + Bs s T B s T BA s = B I n - s s T BA s T BA s C -1 I n - A s s T B s T BA s + s s T B s T BA s ,
from which the desired result is obtained. 2

Lemma 3.1 and 3.2 suggest to construct a limited memory preconditioner matrix C for the system matrix A = AB of the linear system (3.14) that can be used during PCGIF Algorithm 3.4. Therefore, another alternative to precondition the linear system (3.1) is that applying PCGIF Algorithm to the linear system (3.14) of the form of A s = b k (3.27)

A equipped with the B-inner product for the variable s by using the limited memory preconditioner C formulated by (3.19). The approximate solution is obtained from s = Bs.

We are left with the task of integrating this approach when solving a convergent sequence of linear systems with multiple right-hand sides. Assume that we choose the initial guess x 1 . Assume also that the initial preconditioner C 0 such that

C 0 = B -1 P 0 (3.28)
is available. Then for k = 1, PCGIF Algorithm 3.4 solves the preconditioned system

A s = b 1 (3.29)
for the variable s by using the preconditioner C 0 and recovers the approximate solution from the relation s 1 = B s 1 . This approximate solution is then used to update the current iterate from x 2 = x 1 + s 1 .

While performing the PCGIF algorithm on the linear system (3.29), the information can be stored to obtain the LMP C 1 for the system matrix A by using the formula (3.19). This preconditioner can then be used to precondition the second linear system. Note that while constructing C 1 from (3.19), M 0 is taken as the initial preconditioner matrix, i.e. M 0 = C 0 . From Lemma 3.1 we ensure that P 1 = BC 1 where P 1 is the preconditioner for the second linear system of Algorithm 3.2 assuming that S 0 = BS 0 .

Using the preconditioner C 1 for k = 2, PCGIF solves the preconditioned system

A s = b 2 (3.30)
for the variable s and recover the approximate solution from s 2 = B s 2 . This approximate solution is then used to update the current iterate from

x 3 = x 2 + s 2 .
As it is done for k = 1, while performing the PCGIF algorithm on the linear system (3.30) the information can be stored to obtain the LMP C 2 for the system matrix A by using the formula (3.19). Note that while constructing C 2 from (3.19), M 1 is taken as the preconditioner used for k = 2, M 1 = C 1 . From Lemma 3.1 we ensure that P 2 = BC 2 where P 2 is the preconditioner for the third linear system in Algorithm 3.2 assuming that S 1 = BS 1 .

The same strategy can be carried over for later steps. Using the preconditioner C k , for the (k + 1)-th linear system PCGIF Algorithm solves the preconditioned system

A s = b k+1 (3.31)
for the variable s and recover the approximate solution from s k+1 = Bs k+1 . This approximate solution is then used to update the current iterate from x k+2 = x k+1 + s k+1 . Algorithm 3.5 outlines this implementation. In Algorithm 3.5, choosing x 1 = x c the term B -1 (x cx k ) can be calculated from the following equation

B -1 (x c -x k ) = - k-1 j=1 s j (3.32)
Algorithm 3.5: PCGIF for solving a convergent sequence of linear systems with multiple right-hand sides 1 Initialization: Assume that C 0 = B -1 P 0 is available. Choose an initial vector x 1 .

2 Calculate right hand side b k :

b k = B -1 (x c -x k ) -H T R -1 d k .
3 Perform inner loop: Using the preconditioner C k-1 solve the linear system:

A s = b k
with PCGIF Algorithm 3.4. During PCGIF extract and save the information to precondition the next linear system with an LMP C k formulated by (3.19).

4 Recover the approximate solution:

s k = Bs k
5 Update the iterate:

x k+1 = x k + s k
6 Increment k by 1 and go to step 2.

where we used the relations s k = Bs k and x k+1 = x k + s k . Algorithm 3.5 requires neither the factorization of the preconditioner nor matrixvector products with B -1 under the assumption that C 0 = B -1 P 0 is available.

When solving the k-th linear system in Algorithm 3.5, PCGIF Algorithm 3.4 requires a single matrix-vector product with each of the operators H, H T , R -1 , B and C k-1 .

Summary

We have presented different conjugate gradient algorithms for the solution of the linear system (3.1) with the LMP given by (2.38) in the context of solving a convergent sequence of linear systems with multiple right-hand sides. These algorithms differ in the assumption they made on the preconditioner form and matrix-vector products. They are listed below:

• Approach (A): Apply CG Algorithm 2.2 to the linear system (3.7) of the form of A s = b k with a preconditioner constructed from (3.2),

• Approach (B): Apply PCG Algorithm 2.5 to the linear system (3.1) of the form of As = b k with a preconditioner constructed from (2.38),

• Approach (C): Apply PCGIF Algorithm 3.4 to the linear system (3.14) of the

form of A s = b k with a preconditioner constructed from (3.19).
The main properties of these approaches in terms of preconditioning can be listed as follows:

• The first approach (A) requires the factorization of the preconditioner. When used for solving a sequence of linear systems the factorization of initial preconditioner needs to be available.

• The second approach (B) does not require any factorization of the preconditioner.

• The third approach (C) requires neither the factorization of the preconditioner nor the matrix-vector products with B -1 under the assumption that B -1 P operator is available.

These three approaches are presented by Algorithms 3.1, 3.2 and 3.5 whose characteristics are listed in Table 3.1.

Algorithm Inner min.

LMP formula Assumption

3.1 App. (A) F k+1 given by (3.2) with M k = I n F 0 is available 3.2 App. (B) P k+1 given by (2.38) with M k = P k 3.5 App. (C) C k+1 given by (3.19) with M k = C k B -1 P 0 is available and S k = BS k
Table 3.1: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5

When solving the k-th linear system, the iterates of Algorithm 3.1 are related to the iterates of Algorithm 3.2 with

s i = F 0 F 1 . . . F k-1 s i . (3.33)
where i is the index for the inner loop. We can rewrite the linear system (3.7) as

P 1/2 k AP 1/2 k = P 1/2 k b k+1 (3.34)
where

P 1/2 k = F 0 . . . F k .
From this system it can be observed that the relation between the preconditioner Y k = F k F T k used during Algorithm 3.1 and the preconditioner P k used during Algorithm 3.2 can be given as

P k = P 1/2 k-1 Y k P 1/2 k-1 . (3.35)
When solving the k-th linear system, the iterates of PCGIF Algorithm 3.4 are related to the iterates of PCG Algorithm 2.5 with (3.36) under the assumption that P k-1 = BC k-1 . Note that, from Lemma 3.1 assuming that P 0 = BC 0 ensures that P k = BC k for k > 0, when choosing the first level preconditioners as given in Table 3.1 and S k = BS k .

s i = Bs i ,
Let us now consider the particular case that the initial preconditioner when solving a sequence of linear systems is chosen as P 0 = B. Then C 0 = B -1 P 0 = I n is available. In this special case, we summarize the required matrix-vector products for each approach, (A), (B) and (C) for the solution of the k-th linear system within Algorithms 3.1, 3.2, and3.5 in Table 3.2.

Approach matrix-vector products Assumption

(A) H, H T , R -1 , B 1/2 , F i , F T i for i = 1, . . . k -1 F 0 = B 1/2 is available (B) for k = 1 → H, H T , R -1 and B P 0 = B for k > 1 → H, H T , R -1 , B -1 and P k (C) H, H T , R -1 , B and C k B -1 P 0 is available
Table 3.2: A summary of the required matrix-vector products for each approach used for the solution of the k-th linear system (3.1) with LMPs within Algorithms 3.1, 3.2, and 3.5 under the assumption that P 0 = B As a result, we know that Algorithms 3.1 and 3.2 generate a mathematically equivalent sequence of iterates {s k } since it is well-known that CG Algorithm 2.2 and PCG Algorithm 2.5 are equivalent algorithms (Golub and Van Loan, 1996). Algorithms 3.2 and 3.5 whose characteristics are listed in Table 3.1 also generate a mathematically equivalent sequence of iterates {s k } which can be easily observed from the relations (3.15). The choice of the algorithm depends on the availability of the required form of the preconditioner, operators and their matrix-vector product cost.

We next explain how to construct and implement particular LMPs: the quasi-Newton LMP, the Ritz LMP and the spectral LMP in the context of the given algorithms. This part will include for each preconditioner a definition of the preconditioner, its properties, an extraction methodology of the required information from the corresponding conjugate gradient algorithm, and memory and flops requirements for the application of the preconditioner.

Preconditioning with the quasi-Newton LMP

We consider in this section the use of quasi-Newton LMPs (Gratton, Sartenaer and Tshimanga, 2011) formulated by the equation (2.40) which are derived from the inverse Hessian approximations using the Limited Memory BFGS (LBFGS) updating formula [START_REF] Morales | Automatic preconditioning by limited memory quasinewton updating[END_REF]. We adapt the quasi-Newton LMP formula given by (2.40) that can be used during Algorithms 3.1, 3.2 and 3.5.

We start with explaining how to construct a quasi-Newton LMP within Algorithm 3.2 in which the linear system in sequence is solved by Approach (B). Note that the index (k -1) for the preconditioner used for the k-th linear system in sequence within these algorithms is dropped for the sake of simplicity throughout this section.

Approach (B) with the quasi-Newton LMP

Approach (B) applies PCG Algorithm 2.5 to the linear system (3.1) of the form of As = b k . Choosing the search directions p i , i = 0, . . . , j from PCG Algorithm in the formula (2.40), the quasi-Newton LMP P j+1 for A = B -1 + H T R -1 H for the k-th linear system can be constructed from

P j+1 = (I n -τ j p j q T j )P j (I n -τ j q j p T j ) + τ j p j p T j , (3.37)
where p j , τ j = 1/(q T j p j ) and q j = Ap j are obtained during PCG applied to the previous linear system. The initial preconditioner matrix (which can be interpreted as the firstlevel preconditioner) for the formula (3.37) is chosen as the preconditioner used for the previous linear system.

In (3.37) the j "secant pairs" consisting of descent directions p i and associated changes in gradient q i can be chosen from different strategies. For instance, the use of the j last pairs is proposed in (Nocedal and Wright, 2006, p. 177), while a uniform sampling across all generated pairs is proposed in [START_REF] Morales | Automatic preconditioning by limited memory quasinewton updating[END_REF]. A remarkable feature of the update (3.37) is that the matrix ∆P j defined by ∆P j = P j+1 -P j is the solution to the following minimization problem (Nocedal and Wright, 2006, pp. 138-140):

min ∆Pj W 1/2 ∆P j W 1/2 F (3.38)
subject to ∆P j = ∆P T j , P j+1 q j = p j , (3.39)

where W is any symmetric positive definite matrix satisfying W p j = q j . Note that W 1/2 ∆P j W 1/2 F = ∆P j W , where • W is a weighted Frobenius norm of weight W . The solution of problem (3.38)-(3.39) can be computed in close form and is given by ∆P j = W -1 q j (p j -P j q j ) T + (p j -P j q j )q T j W -1 q T j W -1 q j -q T j (p j -P j q j )W -1 q j q T j W -1 (q T j W -1 q j ) 2 .

(3.40)

Substituting the expressions W p j = q j and τ j = 1/(q T j p j ) into (3.40) gives that ∆P j = p j (p j -P j q j ) T + (p j -P j q j )p T j q T j p j -q T j (p j -P j q j )p j p T j (q T j p j ) 2 = τ j p j (p j -P j q j ) T + τ j (p j -P j q j )p T jτ 2 j q T j (p j -P j q j )p j p T j = τ j p j p T jτ j p j q T j P j + τ j p j p T jτ j P j q j p T jτ 2 j (q T j p j )p j p T j + τ 2 j q T j P j q j p j p T j = τ j p j p T jτ j p j q T j P j + τ j p j p T jτ j P j q j p T jτ j p j p T j + τ 2 j q T j P j q j p j p T j = τ j p j p T jτ j p j q T j P jτ j P j q j p T j + τ 2 j q T j P j q j p j p T j , from which it can be seen that P j + ∆P j gives (3.37). We now explain how to construct a quasi-Newton LMP within Algorithm 3.1 in which the linear system in sequence is solved by Approach (A).

Approach (A) with the quasi-Newton LMP

Approach (A) applies CG Algorithm 2.2 to the linear system (3.7) of the form of A k s = b k within Algorithm 3.1. Choosing the search directions p i , i = 0, . . . , j from CG Algorithm in the formula (2.40), the quasi-Newton LMP Y j+1 for A k in the k-th linear system can be constructed from

Y j+1 = (I n -τ j p j q T j )Y j (I n -τ j q j p T j ) + τ j p j p T j , (3.41)
where p j , τ j = 1/( q T j p j ) and q j = A k-1 p j are obtained during CG applied to the previous linear system. Note that, the initial preconditioning matrix is chosen as the identity matrix of order n.

Remember that while applying the quasi-Newton LMP Y j+1 during CG Algorithm 2.2, the square-root factorization of Y j+1 is required. This factored form Y j+1 = F j+1 F T j+1 can be calculated from (Gratton, Sartenaer and Tshimanga, 2011)

F j+1 = I n -p j τ j q T j + √ τ j r T j r j 2 F j , (3.42)
where r j is the residual defined in CG Algorithm. Note that for the first linear system, the factorization of the preconditioner is assumed to be available. We are left with the task of deriving the quasi-Newton LMP for Approach (C) which applies PCGIF Algorithm 3.4 on the linear system (3.14) of the form of A s = b k within Algorithm 3.5.

Approach (C) with the quasi-Newton LMP

Let us assume that P 0 = BC 0 , where P 0 is the initial preconditioner used for Algorithm 3.2 and C 0 is the initial preconditioner used for Algorithm 3.5. Assume also that for the k-th linear system P k-1 is constructed from (2.38) and C k-1 is constructed from (3.19) where the pairs are chosen such that S k-1 = BS k-1 . Under these assumptions from Lemma 3.1, the relation P k = BC k naturally hold (see Section 3.1).

When using the conjugate-gradient method, the pairs can be chosen as search directions. Let us now assume that for the k-th linear system P k-1 is constructed from (2.38) by using the search directions p i , i = 0, . . . , j that are obtained during PCG Algorithm 2.5 and C k-1 is constructed from (3.19) by using the search directions p i , i = 0, . . . , j that are obtained during PCGIF Algorithm 3.4 while solving the (k-1)th linear system. From the relation p i = Bp i given in (3.15), the assumption S k-1 = BS k-1 is satisfied. Using the same strategy as explained in Section 3.1, it can be observed that the relation P k = BC k hold (where the preconditioners P k and C k are constructed from the search directions obtained from PCG Algorithm and PCGIF Algorithm respectively when solving the k-th linear system).

We can also observe from (3.16) that when solving the (k + 1) linear system in Algorithm 3.5 the search directions p i generated by PCGIF Algorithm 3.4 are A(= AB)conjugate with respect to the B-inner product. Using this property from Lemma 3.2, the LMP given by (3.19) can be written as

C j+1 = I n - p j p T j BA p T j BA p j C j I n - A p j p T j B p T j BA p j + p j p T j B p T j BA p j .
Using this formula the quasi-Newton LMP for the k-th linear system in Algorithm 3.5 can be constructed from

C j+1 = I n -τ j p j q T j B C j I n -τ j q j p T j + τ j p j p T j (3.43)
where p i = Bp i , q i = Ap i and τ i = 1/(q T i p i ) are obtained during PCGIF applied to the previous linear system. The initial matrix to construct the preconditioner (3.43) for the k-th linear system is chosen as the preconditioner that is used for the (k -1)-th linear system.

Note that, from Lemma 3.1 the preconditioner (3.43) is symmetric with respect to the B-inner product and BC j+1 is positive definite. Using the relations from (3.15), it can be easily shown that the preconditioner P j+1 formulated by (3.37) and the preconditioner formulated by (3.43) satisfies the relation P j+1 = BC j+1 .

In the formula (3.43) the vectors p j , q j and the scalar τ j are by-products of the PCGIF Algorithm, however the quantity Bq i is not a direct by-product of the algorithm. Therefore it may seem at first that constructing the preconditioner C requires additional matrix-vector products with matrix B. However, the matrix-vector product Bq j can be obtained also as a by-product of PCGIF Algorithm as explained below.

Introducing a new vector l such that l i = Br i , suggests computing the vector z i defined in Algorithm 3.4 from the relation

z i = Bz i = BCr i = C T Br i = C T l i ,
where we used the symmetry of the preconditioner C with respect to the B inner product, i.e. BC = C T B. Multiplying line 11 of Algorithm 3.4 by B gives that

l i+1 = l i -α i Bq i
from which the formula to compute the matrix-vector product Bq i can be obtained as

Bq i = (l i -l i+1 )/α i .
Defining t i = Bq i and using all the relations Algorithm 3.4 can be transformed into Algorithm 3.6 on the following page.

As a result it is possible to find the corresponding quasi-Newton LMPs for variants of conjugate gradients (Approaches (A), (B) and (C)) such that when they are used within the corresponding algorithms ( Algorithms 3.1, 3.2 and3.5), these algo-Algorithm 3.6: PCGIF Algorithm (version for quasi-Newton LMP)

1 r 0 = b -As 0 2 l 0 = Br 0 3 z 0 = Cr 0 4 p 0 = z 0 5 z 0 = C T l 0 6 ρ 0 = r T 0 z 0 7 p 0 = z 0 8 for i = 0, 1, ..., l -1 do 9 q i = (p i + H T R -1 Hp i ) 10 α i = ρ i /q T i p i 11 s i+1 = s i + α i p i 12 r i+1 = r i -α i q i 13 l i+1 = Br i+1 14 t i = (l i -l i+1 )/α i 15 z i+1 = Cr i+1 16 z i+1 = C T l i+1 17
Check convergence and stop if desired accuracy is reached

18 ρ i+1 = r T i+1 z i+1 19 β i+1 = ρ i+1 /ρ i 20 p i+1 = z i+1 + β i+1 p i 21 p i+1 = z i+1 + β i+1 p i 22 end
rithms generate a mathematically equivalent sequence of iterates {s k }. We showed that such preconditioners can be constructed from vectors which are by-products of the corresponding algorithms (Algorithms 2.2, 2.5 and 3.4).

We summarize the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with the quasi-Newton LMP in Table 3.3. In this table we define another approach for PCGIF Algorithm when used with the quasi-Newton LMP : Approach (D): Apply PCGIF Algorithm 3.6 on the linear system (3.14) of the form of A s = b k with a preconditioner constructed from (3.43)

Algorithm

Inner min. Quasi-Newton LMP formula Assumption

Approach (A)

F j given by (3.42) F 0 is available 3.2 Approach (B) P j given by (3.37)

Approach (D)

C j given by (3.43) B -1 P 0 is available Table 3.3: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where preconditioning is achieved by using the quasi-Newton LMPs.

We now want to discuss the implementation issues for each quasi-Newton LMP. For Approach (A) the matrix-vector products with the quasi-Newton preconditioner F j and its transpose F T j can be calculated recursively [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF]. In addition to matrix-vector products with the initial matrices, F 0 and F T 0 , these two recursions require in total (8j + 3)n flops. For Approach (B) matrix-vector products with P j can be calculated using the L-BFGS two-loop recursion (Nocedal and Wright, 2006, p.178, Algorithm 7.4). In addition to matrix-vector products with the initial matrix P 0 , this recursion requires in total 8jn flops. For Approach (D), matrix-vector products with C j and C T j can be calculated by adapting the L-BFGS two-loop recursion which results in Algorithm 3.7 and 3.8. Each algorithm requires 8jn flops and matrix-vector products with C 0 and C T 0 respectively. We summarize the cost for applying the preconditioner and memory requirements in Table 3.4.

Algorithm 3.7: Compute z= Cr

1 Given r set v ← r 2 for i = j : -1 : 1 do 3 η i ← τ i p T i v 4 v ← v -η i q i 5 end 6 z ← C 0 v 7 for i = 1 : j do 8 ζ ← τ i t T i z 9 z ← z + p i (η i -ζ) 10 end Algorithm 3.8: Compute z = C T l 1 Given l set v ← l 2 for i = j : -1 : 1 do 3 η i ← τ i p T i v 4 v ← v -η i t i 5 end 6 z ← C T 0 v 7 for i = 1 : j do 8 ζ ← τ i q T i z 9 z ← z + p i (η i -ζ)
Approach LMP Reference Memory req. Flops req. mat-vec prod. From Table 3.4, we can see that applying C and C T require more memory and flops comparing the others. However, for large scale problems considering Approach (D) avoids both factorization of B and inverse of B under the assumption that B -1 P is available, 16jn flops become very small comparing the cost for matrix-vector multiplications with B -1 and B 1/2 .

Preconditioning with the Ritz LMP

We now consider preconditioning a linear system by using spectral information in particular Ritz pairs, which is shown to be an effective preconditioning technique for variants of CG or Lanczos methods [START_REF] Fisher | Minimization algorithms for variational data assimilation[END_REF], (Gratton, Sartenaer and Tshimanga, 2011) and [START_REF] Rey | A rayleigh-ritz preconditioner for the iterative solution to large scale nonlinear problems[END_REF]. In this section we focus on the Ritz LMPs (Gratton, Sartenaer and Tshimanga, 2011) formulated by the equation (2.41) and adapt this preconditioner that it can be used during Algorithms 3.1, 3.2 and 3.5.

Let us first recall the definition of the Ritz pair (Parlett, 1980):

Definition 3.1 A scalar θ is called a Ritz value of A with respect to a subspace L if there exists a nonzero vector u ∈ L, called Ritz vector, such that (Auθu) ⊥ L, where orthogonality is considered with respect to the canonical inner product. The pair (θ, u) is called a Ritz pair of A with respect to L. If θ is a Ritz value, its multiplicity is the maximum number of independent vectors u i such that (θ, u i ) is a Ritz pair of A with respect to L.

We want to construct such Ritz pairs to construct a Ritz LMP from the information that is obtained during variants of the CG algorithm (Approaches (A), (B) and (C)). Since the spectral information is not directly available from a CG run, we first explain how to extract these Ritz pairs from the iterates of the each algorithm and later we define the Ritz-LMP.

We start with explaining how to use a Ritz LMP within Algorithm 3.1 in which the linear system in sequence is solved by Approach (A).

Approach (A) with the Ritz LMP

Let us consider applying CG Algorithm 2.2 to the linear system (3.7) of the form of A k s = b k within Algorithm 3.1. Assume that CG Algorithm performed successive inner iterations. Then the approximate solution s calculated by CG Algorithm lies in a subspace s 0 + K ( A k , r 0 ) where s 0 is the initial guess and r 0 is the initial residual. As explained in Section 2.6.4, the orthonormal basis for the Krylov subspace K ( A k , r 0 ) can be constructed from the column vectors v i , i = 1, . . . , . These vectors are extracted from

v i+1 = (-1) i r i r i 2 , (3.44)
where r i is the residual given in Algorithm 2.2. The projected matrix

T = V T A k V (3.45)
where V = [ v 1 , . . . , v l ] can also be computed during CG Algorithm as explained in Section 2.6.4. Therefore, the Ritz pairs (θ i , u i ), i = 1, . . . , of the matrix A k with respect to subspace K ( A k , r 0 ) can be computed by using the eigenpairs (θ i , ūi ) of the matrix T as follows:

(θ i , u i ) = (θ i , V ūi ). (3.46)
Choosing these Ritz pairs (θ i , u i ), i = 1, . . . , obtained from CG Algorithm in the formula (2.41), the Ritz LMP for the matrix A k in k-th linear system can be generated from (Gratton, Sartenaer and Tshimanga, 2011)

Y k-1 = I n + U (Θ -1 -I ) U T -U ω v T +1 -v +1 ω T U T + U ωω T U T , (3.47)
where U = [ u 1 , . . . , u ], Θ = diag(θ i ), and ω = (ω 1 , ..., ω ) T , with

ω i = (e T u i )β +1 θ i , for i = 1, ..., .
(3.48)

The vector e is defined as the -th column of the identity matrix I of order . In formula (3.47), the Lanczos vector v +1 can be computed from (3.44) for i = and the scalar β +1 can be obtained from formula (2.36) where η +1 = β +1 . While applying the preconditioner during Algorithm 3. (Gratton, Sartenaer and Tshimanga, 2011)

1 the factorization of Y k-1 is required. This factored form Y k-1 = F k-1 F T k-1 can be calculated from
F k-1 = I n + U (Θ -1/2 -I ) U T -U ω v T +1 . (3.49)
Note that for the first linear system, the factorization of P 0 = F 0 F T 0 is assumed to be available. Algorithm 3.9 summarizes how to built the Ritz-LMP (3.49).

We now explain how to construct a Ritz LMP within Algorithm 3.2 in which the linear system in sequence is solved by Approach (B).

Approach (B) with the Ritz LMP

Let us consider applying PCG Algorithm 2.5 to the linear system (3.1) of the form of As = b k within Algorithm 3.2. In order to define the Ritz-LMP for the matrix A, we use the relations of the information obtained from CG Algorithm 2.2 and PCG Algorithm 2.5 as follows.

Algorithm 3.9: Construct the factored form of the Ritz-LMP for the matrix A k andβ +1 from CG Algorithm 2.2 2 Calculate the eigenpairs (θ i , ūi ), i = 1, . . . , of the tridiagonal matrix T :

1 Given V , T , v +1
T Ū = Θ Ū where Θ = diag(θ i ) and Ū = [ū 1 , . . . , ū ].
3 Generate the vector ω = (ω 1 , ..., ω ) T with ω i calculated from the formula (3.48)

4 Calculate the Ritz pairs (θ i , u i ) of the matrix A k : (Θ, U ) = (Θ, V l Ū ) where U = [ u 1 , . . . , u ].
5 Construct the factored form of the Ritz-LMP from the formula (3.49)

From the relation (3.35), preconditioning the linear system

A k s = b k given by (3.7) with the preconditioner Y k = F k F T k during CG Algorithm 2.
2 is equivalent to preconditioning the linear system As = b k given by (3.1) with the preconditioner P k . Therefore, the preliminary Ritz-LMP P k-1 for the matrix A can be derived by substituting the Ritz-LMP formula Y k-1 given by (3.47) into (3.35) which gives that

P k-1 = P 1/2 k-2 Y k-1 P 1/2 k-2 = P k-2 + P 1/2 k-2 U (Θ -1 -I ) U T P 1/2 k-2 -P 1/2 k-2 U ω v T l+1 P 1/2 k-2 -P 1/2 k-2 v l+1 ω T U T P 1/2 k-2 + P 1/2 k-2 U ωω T U T P 1/2 k-2 . (3.50)
In this formula, the Lanczos vectors and the spectral information obtained from CG Algorithm 2.2 is required. We next explain by Lemma 3.3 and 3.4 how to extract this information from PCG Algorithm 2.5, and later reformulate this LMP in terms of the vectors and spectral information obtained from PCG Algorithm.

Lemma 3.3 Let v i , i = 1, . . . , be Lanczos vectors generated from (3.44) during CG Algorithm 2.2 applied to the preconditioned linear system A s = b where A = P 1/2 AP 1/2 , s = P -1/2 s and b = P 1/2 b, and P is symmetric and positive definite. Then the Lanczos vectors v i , i = 1, . . . , generated from the formula

v i+1 = (-1) i r i r i P (3.51)
during PCG Algorithm 2.5 applied to the linear system As = b with the preconditioner P satisfies the relation

v i = P 1/2 v i . (3.52)
The Lanczos vectors v i , i = 1, . . . , construct an orthonormal basis with respect to the P -inner product for the Krylov subspace K (AP, r 0 ). The projected matrix given by (3.45) can be written in terms of the matrix V = [v 1 , . . . , v ] as

T = V T P AP V (3.53)
which defines an orthogonal projection of AP with respect to the P -inner product onto the Krylov subspace K (AP, r 0 ). Moreover, the vectors defined by

g i = P v i (3.54)
constructs an orthonormal basis with respect to the P -1 -inner product for the Krylov subspace K (P A, P r 0 ). The projected matrix given by (3.45) can be written in terms of the matrix G = [g 1 , . . . , g ] as

T = G T AG (3.55)
which defines an orthogonal projection of P A with respect to the P -1 -inner product onto the Krylov subspace K (P A, P r 0 ).

Proof. Using the relations given in the Lemma it can be easily shown that the residuals r i = b -As i and r i = b i -A s i generated by Algorithm 2.5 and Algorithm 2.2 satisfy the following relation (Golub and Van Loan, 1996, p. 533)

r i = P 1/2 r i ,
with i = 0, . . . , l -1. Substituting this relation into (3.44) gives that

v i+1 = (-1) i r i r i 2 = (-1) i P 1/2 r i r i P = P 1/2 (-1) i r i r i P = P 1/2 v i+1
which proves the first part of the Lemma. The Lanczos vectors v i+1 construct an orthonormal basis for the Krylov subspace K ( A, r 0 ) (Saad, 1996, p.147, 152). From orthonormal vectors v i+1 and the relation

v i+1 = P 1/2 v i+1 , we have V T P V = I .
Using this equation with the definition (3.54) yields that G T P -1 G = I . We can write the Krylov subspace K ( A, r 0 ) as

K ( A, r 0 ) = span{ r 0 , A r 0 , . . . , A l-1 r 0 } = span{P 1/2 r 0 , P 1/2 AP 1/2 P 1/2 r 0 , . . . , (P 1/2 AP 1/2 ) l-1 P 1/2 r 0 } = P 1/2 span{r 0 , AP r 0 , . . . , (AP ) l-1 r 0 } = P 1/2 K (AP, r 0 ). (3.56)
Then, from the relations (3.52) and (3.56), the column vectors of the matrix V form an orthonormal basis with respect to the P -inner product for the Krylov subspace K (AP, r 0 ).

Finally, T matrix generated from (3.45) can be rewritten as

T = V T A V = V T P AP V (3.57)
where we used the relation A = P 1/2 AP 1/2 and V = P 1/2 V . We now show that g i vectors form an orthornormal basis for K (P A, P r 0 ). Multiplying the subspace K (AP, r 0 ) spanned by the vectors v i from the right with P gives P K (AP, r 0 ) = span{P r 0 , P AP r 0 , . . . , P (AP ) -1 r 0 } = span{P r 0 , (P A)P r 0 , . . . , (P A) -1 P r 0 } = K (P A, P r 0 ).

(3.58)

From this relation and (3.54), the columns of the matrix G form an orthonormal basis with respect to the P -1 -inner product for the Krylov subspace K (P A, r 0 ). Using the relations (3.57) and G = P V , the matrix T (3.45) can be written as

T = V T P AP V = G T AG
which completes the proof. 2

Lemma 3.4 Suppose that (θ i , u i ), i = 1, . . . , be the Ritz pairs of the matrix A with respect to the subspace K ( A, r 0 ), i.e. (θ i , u i ) = (θ i , V ūi ). The matrix V consists of column vectors v i , i = 1, . . . , defined in Lemma 3.3, and ūi are the eigenvectors of the matrix T given by (3.45). Let's define the vectors u i such that

u i = V ūi (3.59) where V = [v 1 , . . . , v
] is a matrix whose column vectors are defined in Lemma 3.3. Then the vectors u i satisfy the relation

u i = P 1/2 u i (3.60)
and the pairs (θ i , u i ) define the Ritz pairs of the matrix of AP with respect to the subspace K (AP, r 0 ).

Let us define the vectors x i such that

x i = G ūi (3.61)
where G = [g 1 , . . . , g ] is a matrix whose column vectors are defined in Lemma 3.3. Then the vectors x i satisfy the relation

x i = P u i (3.62)
and the pairs (θ i , x i ) define the Ritz pairs of the matrix P A with respect to the subspace K (P A, P r 0 ).

Proof. Using the relation (3.52), we can write the vectors u i as

u i = V ūi = P 1/2 V ūi
from which it can be seen that the vectors defined by (3.59) satisfies the relation (3.60).

From the definition of the lemma (θ i , ūi ) are the eigenpairs of the tridiagonal matrix T l given by (3.45). Defining Θ = diag(θ i ) and Ū = (u 1 , . . . , u ), we have

Ū T T Ū = Θ.
(3.63)

From Lemma 3.3 the matrix T can be written as T = V T P AP V where V has orthonormal columns with respect to the P -inner product, i.e. V T P V = I that span K (AP, r 0 ). Then, the pairs (θ i , u i ) = (θ i , V u i ) define the Ritz pairs of the matrix AP with respect to the subspace K (AP, r 0 ) (Golub and Van Loan, 1996, p.402, 474-475).

We now prove the second part of the lemma. Using the relation (3.54), it can be easily seen that G = P V . Substituting this relation into the definition (3.61) yields that

x i = G ūi = P V ūi = P u i
where we use the definition (3.59). From Lemma 3.3 the matrix T can be written as T = G T P -1 P AG where G has orthonormal columns with respect to the P -1 -inner product, i.e. G T P -1 G = I that span K (P A, P r 0 ). Then, the pairs (θ i , x i ) = (θ i , G u i ) define the Ritz pairs of the matrix P A with respect to the subspace K (P A, P r 0 ) (Golub and Van Loan, 1996, p.402, 474-475).

2

We now use the vectors and relations given by Lemma 3.3 and Lemma 3.4 to reformulate the LMP given by (3.50) in the sense that all the information can be obtained directly from PCG Algorithm 2.5.

Let us define the column matrix U = [u 1 , . . . , u ] whose columns are defined in Lemma 3.4. Substituting the relations (3.60) for i = 1, . . . , and (3.52) for i = + 1, into (3.50) we obtain the formula

P k-1 = P k-2 + P k-2 U (Θ -1 -I )U T P k-2 -P k-2 U ωv T l+1 P k-2 (3.64) -P k-2 v l+1 ω T U T P k-2 + P k-2 U ωω T U T P k-2 ,
where the vector v l+1 is defined by (3.51) for i = l. In order to obtain the vector ω and the matrices U and Θ in the formula (3.64), the matrices V and T defined in Lemma 3.3 and the scalar β +1 needs to be generated and stored. The matrix V is generated from the column vectors defined by (3.51) for i = 0, . . . , -1. The matrix T is obtained as explained in Section 2.6.4 and the scalar β +1 can be obtained from formula (2.36) where η +1 = β +1 . Once we have all the required information from PCG Algorithm 2.5, Algorithm 3.10 summarizes how to built the the Ritz-LMP (3.64).

Algorithm 3.10: Construct the Ritz-LMP for the matrix A (version 1)

1 Given P k-2 , V , T , v +1 and β +1 from PCG Algorithm 2.5
2 Calculate the eigenpairs (θ i , ūi ), i = 1, . . . , of the tridiagonal matrix T :

T Ū = Θ Ū where Θ = diag(θ i ) and Ū = [ū 1 , . . . , ū ].
3 Generate the vector ω = (ω 1 , ..., ω ) T with ω i calculate from the formula (3.48)

4 Calculate the Ritz pairs (θ i , u i ) of the matrix AP :

(Θ, U ) = (Θ, V l Ū )
where U = [u 1 , . . . , u ].

5 Construct the Ritz-LMP from the formula (3.64)

The Ritz-LMP P k-1 given by (3.64) can be written in a more compact form as explained below.

Let us define a matrix X = [x 1 , . . . , x ] whose column vectors are defined in Lemma 3.4. Substituting the relations (3.62) for i = 1, . . . , , we obtain that

P k-1 = P k-2 + X(Θ -1 -I )X T -Xωg T l+1 -g l+1 ω T X T + Xωω T X T (3.65)
where the vector g l+1 is defined by (3.54) for i = l + 1. Alternatively, this formula can be used to construct the preconditioner P k-1 . In this case to obtain the matrix X, the matrix G defined in Lemma 3.3 needs to be generated and stored. Once we have all the required information from PCG Algorithm 2.5, Algorithm 3.11 summarizes how to built the Ritz-LMP (3.65).

In this section we are left with the task of deriving the Ritz LMP for Approach (C).

Approach (C) with the Ritz LMP

We now consider applying PCGIF Algorithm 3.4 to the linear system (3.14) of the form of A s = b k within Algorithm 3.5.

We suppose that P 0 = BC 0 where P 0 is the initial preconditioner for Algorithm 3.2 and C 0 is the initial preconditioner for Algorithm 3.5. Our aim is to find the preconditioner C k-1 for the k-th linear system of Algorithm 3.5 such that P k-1 = BC k-1 where P k-1 is the Ritz-LMP constructed from (3.64) and used for the k-th linear system of Algorithm 3.2.

Assume that while solving the (k -1)-th linear systems in Algorithm 3.2 and 3.5

Algorithm 3.11: Construct the Ritz-LMP for the matrix A (version 2)

1 Given P k-2 , G , T , g +1 and β +1 from PCG Algorithm 2.5

2 Calculate the eigenpairs (θ i , ūi ), i = 1, . . . , of the tridiagonal matrix T :

T Ū = Θ Ū where Θ = diag(θ i ) and Ū = [ū 1 , . . . , ū ].
3 Generate the vector ω = (ω 1 , ..., ω ) T with ω i calculate from the formula (3.48)

4 Calculate the Ritz pairs (θ i , x i ) of the matrix P A:

(Θ, X) = (Θ, G l Ū )
where X = [x 1 , . . . , x ].

5 Construct the Ritz-LMP from the formula (3.65) the relation P k-2 = BC k-2 holds. Substituting this relation into (3.64) gives that

P k-1 = BC k-2 + BC k-2 U (Θ -1 -I )U T C T k-2 B -BC k-2 U ωv T l+1 C T k-2 B -BC k-2 v l+1 ω T U T C T k-2 B + BC k-2 U ωω T U T C T k-2 B = B[C k-2 + C k-2 U (Θ -1 -I )U T C T k-2 B -C k-2 U ωv T l+1 C T k-2 B -C k-2 v l+1 ω T U T C T k-2 B + C k-2 U ωω T U T C T k-2 B],
from which we can write the preliminary formula for C k-1 as

C k-1 = C k-2 + C k-2 U (Θ -1 -I )U T C T k-2 B -C k-2 U ωv T l+1 C T k-2 B (3.66) -C k-2 v l+1 ω T U T C T k-2 B + C k-2 U ωω T U T C T k-2 B.
This formula requires the information from PCG Algorithm 2.5. Similar to the derivation of the preconditioner P k-1 , we first explain by Lemma 3.5 and 3.6 how to extract the required information from PCGIF Algorithm 3.4 and later reformulate the LMP (3.66) in terms of vectors and matrices that are directly available from PCGIF Algorithm.

Lemma 3.5 Let v i , i = 1, . . . , l be Lanczos vectors generated from (3.51) during PCG Algorithm 2.5 applied to the preconditioned linear system As = b with a symmetric and positive definite matrix P . Then the Lanczos vectors v i , i = 1, . . . , l generated from the formula

v i+1 = (-1) i z i r T i z i (3.67)
during PCGIF Algorithm 3.4 applied to the linear system ABs = b with the preconditioner C such that P = BC satisfy the relation

v i = Cv i . (3.68)
Moreover, the vectors defined by

g i = Bv i , (3.69)
coincide with those given in Lemma 3.3.

Proof. Under the assumption that P = BC, the relation between the residuals of Algorithms 2.5 and 3.4 is given as

r i = r i .
Multiplying (3.51) from left by C and substituting the relation between the residuals gives the desired result v i = Cv i . Substituting the relations P = BC and v i = Cv i in (3.54) we obtain that

g i = BCv i = Bv i which completes the proof. 2 Lemma 3.6 Let (θ i , u i ), i = 1, .
. . , l be the Ritz pairs of the matrix AP with respect to the subspace K (AP, r 0 ) defined in Lemma 3.4. Let us define the vectors u i such that

u i = V ūi (3.70)
where V = [v 1 , . . . , v ] and its column vectors are defined in Lemma 3.5. Then the vectors u i satisfy the relation We now use the vectors and relations given by Lemma 3.5 and Lemma 3.6 to reformulate the LMP given by (3.66) in the sense that all the information can be obtained directly from PCGIF Algorithm 3.4.

u i = Cu i . ( 3
Let us define the matrix U = [u 1 , . . . , u ] whose columns are defined in Lemma 3.6. Substituting the relations (3.71) for i = 1, . . . , into (3.66) gives that

C k-1 = C k-2 + U (Θ -1 -I )U T B -U ωv T l+1 B -v l+1 ω T U T B + U ωω T U T B, (3.72)
where the vector v l+1 is defined by (3.68) for i = + 1. The formula (3.72) requires matrix-vector multiplications with B which may appear computationally costly. However, we can avoid this matrix-vector multiplications by observing that

Bu i = BCu i = P u i = x i .
where x i is defined in Lemma 3.4. Substituting this relation and the vector g l+1 defined by (3.69) for i = l + 1 into (3.72) yields that

C k-1 = C k-2 + U (Θ -1 -I )X T -U ωg T l+1 -v l+1 ω T X T + U ωω T X T . (3.73)
In order to obtain the vector ω and the matrices U , X and Θ in the formula (3.73), the matrices V ,G and T and the scalar β +1 needs to be generated and stored. The column vectors of the matrix V and G = [g 1 , . . . , g ] are defined in Lemma 3.5. The matrix T is obtained as explained in Section 2.6.4 and the scalar β +1 can be obtained from formula (2.36) where η +1 = β +1 . Once we have all the required information from PCGIF Algorithm 3.4, Algorithm 3.12 summarizes how to built the the Ritz-LMP (3.73).

Algorithm 3.12: Construct the Ritz-LMP for the matrix

A 1 Given C k-2 , V , G , T , v +1 , g +1 and β +1 from PCGIF Algorithm 3.4
2 Calculate the eigenpairs (θ i , ūi ), i = 1, . . . , of the tridiagonal matrix T :

T Ū = Θ Ū where Θ = diag(θ i ) and Ū = [ū 1 , . . . , ū ].
3 Generate the vector ω = (ω 1 , ..., ω ) T with ω i calculate from the formula (3.48)

4 Generate the matrix U = [u 1 , . . . , u ] whose column vectors are calculated from

u i = V l ūi
5 Generate the matrix X = [x 1 , . . . , x ] whose column vectors are calculated from

x i = G l ūi 6 Construct the Ritz-LMP from the formula (3.73)
As a result it is possible to find the corresponding Ritz LMPs for variants of conjugate gradients (Approaches (A), (B) and (C)) such that when they are used within the corresponding algorithms (Algorithms 3.1, 3.2 and 3.5), these algorithms generate a mathematically equivalent sequence of iterates {s k }. We showed that these preconditioners can be constructed from vectors which are by-products of the corresponding algorithms (Algorithms 2.2, 2.5 and 3.4).

Table 3.5 summarizes the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with the Ritz LMP and Table 3.6 summarizes the cost of applying the preconditioner and memory requirements.

Using the information from Table 3.6 together with the cost of B 1/2 and B -1 , it can be decided which algorithm is more efficient in terms of the computational cost and which is to be chosen to solve the sequence of linear systems (3.1).

Preconditioning with the spectral LMP

In this section, we recall the spectral LMP (2.43) and adapt this preconditioner that can be used during Algorithms 3.1, 3.2 and 3.5. We start with explaining how to use a spectral LMP within Algorithm 3.1 in which the linear system in sequence is solved by Approach (A).

Approach (A) with the spectral LMP

Let us consider applying CG Algorithm 2.2 to the linear system (3.7) of the form of A k s = b k within Algorithm 3.1. Let us denote the exact eigenpairs of the system matrix A k-1 in the (k -1)-th linear system by (λ i , υi ), i.e.

A k-1 υi = υi λ i .

(3.74)

Then as explained in Section 2.6.5 by using the eigenpairs (λ i , υi ), the preconditioner Y k-1 that approximates the inverse of A k-1 can be calculated as

Y k-1 = I n -U (Λ -1 -I l ) U T , (3.75) where U = [ υ 1 , ..., υ l ], Λ = diag(λ i ).
While applying the preconditioner during Algorithm 2.2 the factorization of Y k-1 is required. This factored form Y k-1 = F k-1 F T k-1 can be calculated from (Gratton, Sartenaer and Tshimanga, 2011)

F k-1 = I n -U (Λ -1/2 -I l ) U T .
(3.76)

We now explain how to construct a spectral LMP within Algorithm 3.2 in which the linear system in sequence is solved by Approach (B).

Approach (B) with the spectral LMP

Let us consider applying PCG Algorithm 2.5 to the linear system (3.1) of the form of As = b k within Algorithm 3.2. The spectral LMP P k-1 can be derived from (3.35) and (3.75) as

P k-1 = P 1/2 k-2 Y k-1 P 1/2 k-2 = P k-2 + P 1/2 k-2 U (Λ -1 -I l ) U T P 1/2 k-2 . (3.77) Using the relation A k-1 = P 1/2 k-2 AP 1/2
k-2 from (3.34) and the equation (3.74), it can be easily seen that the pairs (λ i , x i ) where x i is defined by

x i = P 1/2 k-2 υ i ,
are the eigenpairs of the matrix P k-2 A. Defining X = [x 1 , . . . , x ], the spectral LMP formula (3.77) can be rewritten in a more compact form as

P k-1 = P k-2 + X(Λ -1 -I l )X T .
(3.78)

In this section we are left with the task of deriving the spectral LMP for Algorithm 3.5.

Approach (C) with the spectral LMP

We consider applying PCGIF Algorithm 3.4 to the linear system (3.14) of the form of AB s = b k within Algorithm 3.5.

We suppose that P 0 = BC 0 where P 0 is the initial preconditioner for Algorithm 3.2 and C 0 is the initial preconditioner for Algorithm 3.5. Our aim is to find the preconditioner C k-1 for the k-th linear system in Algorithm 3.5 such that P k-1 = BC k-1 where P k-1 is the spectral LMP constructed from (3.78) and used for the k-th linear system in Algorithm 3.2.

Assume that when solving the (k -1)-th linear system in Algorithm 3.2 and 3.5, the relation P k-2 = BC k-2 holds. Let us define a matrix U = [u 1 , . . . , u ] whose column vectors are defined by

u i = B -1 x i . (3.79)
Substituting this relation into (3.78) and using the relation

P k-2 = BC k-2 gives that P k-1 = BC k-2 + BU (Λ -1 -I l )U T B = B [C k-2 + U (Λ -1 -I l )U T B],
from which the preconditioner C k-1 can be written as

C k-1 = C k-2 + U (Λ -1 -I l )X T . (3.80)
As a result it is possible to find the corresponding spectral LMPs for variants of conjugate gradients (Approaches (A), (B) and (C)) such that when they are used within the corresponding Algorithms 3.1, 3.2 and 3.5, these algorithms generate a mathematically equivalent sequence of iterates {s k }.

We summarize the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with the spectral LMP in Table 3.7, and the cost of applying the spectral preconditioner with its memory requirements in Table 3 The spectral LMP requires to compute the eigenpairs of the system matrix which can not be afordable for large-scale systems. For example, [START_REF] Fisher | Minimization algorithms for variational data assimilation[END_REF] propose to replace the exact eigenpairs in the formula (3.75) by the Ritz pairs called as the inexact spectral LMP in Tshimanga, Gratton, Weaver and Sartenaer (2008). In this study, they compared two LMPs and concluded that the Ritz LMP is a general and stabilized version of the inexact spectral LMP.

Preconditioning varying systems

As mentioned before, the solution of the nonlinear least-squares problem (2.4) is found by solving a sequence of slowly varying linear systems (2.16) in the form of

A k s = b k .
(3.81)

We want to understand if it is possible to precondition also these varying linear systems by using the previous information. From

A k = B -1 + H T k R -1 H k , we assume that B
and R are constant and H k is changing along the outer loop k due to the linearization of the nonlinear model H(x k ). Let us assume that the LMP P k-1 which is an inverse approximation of the linear system matrix A k-1 is inherited to precondition the linear system A k s = b k at the k-th outer loop of the Gauss-Newton method. Then CG applies on the preconditioned system P (Golub and Van Loan, 1996, Corollary 8.1.6) we have

1/2 k-1 A k P 1/2 k-1 s = P 1/2 k-1 b k . (3.82) Defining A k = A k-1 + E k , from
|λ i (P 1/2 k-1 A k P 1/2 k-1 ) -λ i (P 1/2 k-1 A k-1 P 1/2 k-1 )| ≤ P 1/2 k-1 E k P 1/2 k-1 2 (3.83)
for i = 1 : n. This corollary gives us the information on the condition number of the perturbed matrix P

1/2 k-1 A k P 1/2
k-1 as follows. Let us denote the eigenvalues of the matrix P

1/2 k-1 A k-1 P 1/2
k-1 by λ i and the eigenvalues of the perturbed matrix P

1/2 k-1 A k P 1/2 k-1 by λ i .
For each eigenvalue it follows from the inequality (3.83) that

| λ i -λ i | ≤ P k-1 2 E k 2 , -P k-1 2 E k 2 ≤ λ i -λ i ≤ P k-1 2 E k 2 , λ i -P k-1 2 E k 2 ≤ λ i ≤ λ i + P k-1 2 E k 2 ,
Using this equality for the minimum and maximum eigenvalues gives that

λ min -P k-1 2 E k 2 ≤ λ min ≤ λ min + P k-1 2 E k 2 ,
(3.84)

λ max -P k-1 2 E k 2 ≤ λ max ≤ λ max + P k-1 2 E k 2 . (3.85)
Assuming that (λ min -P k-1 2 E k 2 ) > 0, we can obtain from inequalities (3.84) and (3.85) that

λ max -P k-1 2 E k 2 λ min + P k-1 2 E k 2 ≤ λ max λ min ≤ λ max + P k-1 2 E k 2 λ min -P k-1 2 E k 2 . (3.86)
The inequality (3.86), therefore gives a bound on the condition number of the preconditioned perturbed matrix. Then, provided that E k 2 is small enough, the inherited preconditioner can be still used as a second level preconditioner if (3.86) is satisfactory. Note that, [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF]) also analyzes the effect of using inherited preconditioner to precondition the new system when E k is a first-order perturbation of the matrix A k and provide first-order bounds for the perturbed preconditioned matrix (Tshimanga, 2007, Theorem 2.5.1).

If the analysis described above is not ensuring that P k-1 is a good preconditioner for A k , we have to build a preconditioner P k using the LMP formula (2.38) with the new matrix A k . In this case, considering the directions S inherited from the (k -1)th outer loop, a preconditioner is obtained with the new matrix A k which has the good properties (e.g. clustering most eigenvalues at 1) that any LMP preconditioner has. This generation however requires matrix-vector products with the new matrix A k and is interesting if the preconditioning effect balances the cost of building the preconditioner.

Convergence issues

The convergence properties of CG are well known and for the detail information we refer to the references Nocedal and Wright (2006), [START_REF] Meurant | The Lanczos and conjugate-gradient algorithms[END_REF] and Axelsson (1996). When PCG is used with a class of LMPs, the convergence properties are provided in Gratton, Sartenaer and Tshimanga (2011) and [START_REF] Tshimanga | On a class of limited memory preconditioners for large-scale nonlinear least-squares problems (with application to variational ocean data assimilation[END_REF].

In this section we provide inexpensive formulas for computing the necessary information on each iteration from the recurrence relations of PCG for monitoring the convergence of the minimization.

Monitoring convergence

The values of the quadratic cost function and the norm of the cost function gradient are important for monitoring the convergence of the minimization. In addition to the total value of the quadratic cost function Q(s) given in (2.46), the relative contributions to (Q) from the background term (Q b ) and observation term (Q o ) may provide additional useful diagnostic information. Inexpensive formulae for computing all these quantities on each iteration from the recurrence relations of PCG and PCGIF (Algorithms 2.5 and 3.4) are derived in this section. Note that for the sake of simplicity, the initial guess is chosen as s 0 = 0 for all derivations.

Quadratic cost function

Using a Taylor series expansion about the initial guess s 0 = 0, for PCG the quadratic cost function Q[s i ] in (2.46) can be expressed as (Gratton and Tshimanga, 2009)

Q[s i ] = Q[0] - 1 2 s T i r 0 , (3.87)
where i = 0, 1, ..., l -1, l is the number of iterations performed during PCG and

Q[0] = f (x k ) = 1 2 d T R -1 d + 1 2 (x c -x k ) T B -1 (x c -x k ).
The Q b term can be calculated as

Q b [s i ] = 1 2 (s i + x k -x c ) T B -1 (s i + x k -x c ) = 1 2 s T i B -1 s i + 1 2 (x k -x c ) T B -1 (x k -x c ). (3.88)
The Q o term can then be calculated from

Q o [s i ] = Q[s i ] -Q b [s i ]. (3.89)
For PCGIF the same formula can be used for the quadratic cost function value

Q[s i ] and Q o [s i ], on the other hand Q b [s i ] can alternatively calculated from Q b [s i ] = 1 2 s T i f i + 1 2 (x -x c ) T B -1 (x -x c )
where f i = B -1 s i can be computed without the need to apply B -1 by including an additional recurrence relation in Algorithm 3.4 as follows:

f i = 0 if i = 0 f i-1 + α i-1 p i-1 if i > 0
Taking x 1 = x c and using the vector f i , the term B -1 (x kx c ) can also be calculated from

B -1 (x k -x c ) = k-1 j=1 B -1 s k = k j=1 f (j) l-1
where f (j) l-1 denotes the last iteration (i = l -1) of PCGIF at the j-th outer loop of Gauss-Newton method.

Norm of the quadratic cost function gradient

For PCG, the gradient norm is given by

∇Q[s i ] P = r i P = r T i z i
where the vectors z i and r i are defined in Algorithm 2.5. The gradient norm for PCGIF can be calculated equivalently from the same formula.

CHAPTER 4 Conjugate gradients in dual space

In this chapter, the least-squares subproblems (2.46) solved at each iteration of the GN method is rewritten as a quadratic minimization problem subject to linear equality constraints. Contrary to the primal constrained problem introduced in Section 2.7, the structure of the subproblem (2.46) is exploited to derive an alternative dual problem. We consider a recently-proposed CG-like method to perform the quadratic minimization arising from the alternative dual approach. This method produces, in exact arithmetic, the same iterates as those produced by a standard CG applied to the linear system (2.16). At first sight, even if the dimension may not explain everything, this algorithm might provide some advantage when the dual problem is defined on a space (named the dual space) of smaller dimension than that of the primal problem (named the primal space) as it can yield gains in terms of both memory usage and computational cost. The relation between this dual-space solver and existing dual-space techniques used in data assimilation problems is explained.

We also introduce practical warm-start preconditioners to accelerate the convergence of the dual algorithm when solving a convergent sequence of linear systems with multiple right hand sides. In particular, a dual-space counterpart to the warm-start LMPs explained in Chapter 3 is derived and its properties analyzed. We later adapt the preconditioners to the case where a sequence of slowly varying linear systems has to be solved as in the Gauss-Newton process.

We conclude the chapter by providing the convergence theory for the dual algorithm and inexpensive formulae to compute the quadratic cost function (2.13) and the norm of its gradient from by-products of the dual algorithm, that are useful quantities to monitor convergence.

Exploiting further the structure with the dual problem

In this section, we take into consideration the structure of the regularized subproblem (2.46) and derive its dual problem which has its Lagrange multiplier vector in IR m , the space spanned by the m-dimensional vectors associated by the data vector.

We reformulate the linear least-squares problem (2.46) as a convex quadratic problem with linear equality constraints given by min s,a

1 2 s + x k -x c 2 B -1 + 1 2 a 2 R -1 (4.1) subject to a = H k s + d k ,
which can, in turn, be solved using duality theory. We define the dual objective function q : IR m → IR for this problem as (Nocedal and Wright, 2006, p.343-349)

q(λ) = inf s,a L(s, a, λ) def = inf s,a 1 2 s + x k -x c 2 B -1 + 1 2 a 2 R -1 -λ T (H k s + d k -a), (4.2)
where λ ∈ IR m is a Lagrange multiplier vector. Note that contrary to Chapter 2, the Lagrange multiplier is not associated with the full residual of (4.1), it is associated only with the part related to the observations. The infimum is achieved when

∇ s L(s, a, λ) = B -1 (s + x k -x c ) -H T k λ = 0 ∇ a L(s, a, λ) = R -1 a + λ = 0 ∇ λ L(s, a, λ) = H k s + d k -a = 0
From the first two equations we obtain that

s = x c -x k + BH T k λ, (4.3) a = -Rλ.
We may therefore substitute s and a in the expression (4.2) and obtain the quadratic dual objective explicitly as follows:

q(λ) = - 1 2 λ T (H k BH T k + R)λ + λ T (H k (x k -x c ) -d k ),
Then the dual problem for the constrained primal problem (4.1) is defined as

max λ q(λ) (4.4)
The maximum of the dual objective is given by

λ * = (H k BH T k + R) -1 (H k (x k -x c ) -d k ), (4.5)
which is found as a stationary point. Therefore, from (4.5) and (4.3), the solution of the subproblem (2.46) can be written as

s k = x c -x k + BH T k (H k BH T k + R) -1 (H k (x k -x c ) -d k ). (4.6)
Note that this solution may also be obtained directly from

s k = x c -x k + (H T k R -1 H k + B -1 ) -1 H T k R -1 (H k (x k -x c ) -d k )
(written from (2.50)) by using the Sherman-Morrison-Woodbury formula (Nocedal and Wright, 2006, p. 612-613). This suggests that the step s k can be obtained by solving the dual problem (4.4) in the dual space of Lagrange multiplier and recovering the step from (4.3). We name this approach the dual approach. This approach solves the dual problem by applying the Krylov subspace method to the linear system in (4.5) given by

(H k BH T k + R)λ = H k (x k -x c ) -d k . (4.7)
In this chapter, we focus on solving this linear system with preconditioned conjugate gradients such that the iterates in IR n generated from (4.3) ensures the monotonic decrease on the quadratic cost function (2.46) along the inner-iterations.

Before explaining how to apply PCG in the context of dual approach, we next provide the link between applying CG method on the linear system (4.7) and the variants of the CRAIG method.

Relation with Extended CRAIG and Generalized CRAIG

The extended CRAIG proposed by [START_REF] Saunders | Solution of sparse rectangular systems using LSQR and CRAIG[END_REF] suggests to extend the CRAIG method [START_REF] Craig | The N-step iteration procedures[END_REF] for incompatible under determined systems where m n by including regularization. It is observed that the extended CRAIG solves the equivalent problem to that of (2.47) which can be stated as [START_REF] Saunders | Solution of sparse rectangular systems using LSQR and CRAIG[END_REF]

min δ x,z 1 2 δ x z 2 2 , (4.8)
subject to

H I δ x z = c
where z = c -Hδ x. It is pointed out (Saunders, 1995, Result 9) that since H T H + I n is symmetric and positive definite, the extended CRAIG iterates δ x i , i = 0, . . . , l -1 are related to the CG iterates for the problem

( H H T + I m ) λ = c (4.9) by δ x z = H T I λ according to δ x i = H T λ i and z = λ. Substituting H = R -1/2 HB 1/2 and c = R -1/2 H(x -x c ) -R -1/2 d into (4.9) yields that (R -1/2 HBH T R -1/2 + I m ) λ = R -1/2 (H(x -x c ) -d).
Note that this linear system is the same linear system as (4.7) preconditioned by R -1/2 . Therefore the iterates are related according to R -1/2 λ i = λ i .

The extended CRAIG is generalized by the Generalized CRAIG (G-CRAIG) proposed by [START_REF] Arioli | Iterative methods for symmetric quasi-definite linear systems[END_REF]; this method uses a regularization term different from (τ I) and a non-canonical inner product. The G-CRAIG solves the equivalent problem to (2.47) which can be given as min r,δx

1 2 δx r 2 D , subject to H R δx r = c where δx = B 1/2 δ x, r = R -1/2 z , c = R 1/2 c
and the D-norm is defined by the matrix

D := B -1 R .
The G-CRAIG iterates δx i , i = 0, . . . , l -1 are related to the PCG iterates for the problem [START_REF] Arioli | Iterative methods for symmetric quasi-definite linear systems[END_REF]. Note that the linear system (4.10) is the same linear system as (4.7).

(HBH T + R)λ = H(x -x c ) -d (4.10) δx r = BH T I λ with preconditioner R -1 according to δx i = BH T λ i (Arioli
The extended CRAIG and G-CRAIG minimize the direct error

δx r - δx * r * D ,
where δx * is the exact solution and r * = R -1 (c -Hδx * ).

Since the extended CRAIG and G-CRAIG are more expensive than LSQR (Saunders, 1995), [START_REF] Arioli | Iterative methods for symmetric quasi-definite linear systems[END_REF] we will not consider to use these methods within this study.

Solving the subproblem with Restricted PCG (RPCG)

In this section, we explain variants of dual approaches which solve the linear system (4.7) by using conjugate-gradients like methods with a preconditioner. In particular, we focus on the dual approach which produces, in exact arithmetic, the same iterates as those produced by a standard CG applied to the linear system (2.16). This method therefore preserves the monotonic decrease on the quadratic cost function (2.46). Note that within this section we have dropped the outer loop index k of a Gauss Newton method for simplicity.

A straightforward alternative for solving the system (4.7) with a preconditioner is to apply PCG Algorithm 2.5 with a preconditioner matrix R -1 and a canonical inner product. This gives the preconditioned system

(R -1/2 HBH T R -1/2 + I m ) λ = R -1/2 (H(x -x c ) -d), (4.11)
where λ = R -1/2 λ and I m is the identity matrix of order m. The approximate solution in IR n is then computed from the relation (4.3) given by

s = x c -x + BH T R -1/2 λ. (4.12)
We mentioned in Section 4.2 that CG applied to the linear system (4.11) generates the iterates λ i which are related to the iterates of the extended CRAIG applied to the problem (4.8) with δ x i = H T λ i . Using this relation CG applied to the linear system (4.11) minimizes the direct error

q( λ) =< ( H H T + I m )( λ -λ * ), ( λ -λ * ) > = ( λ -λ * ) T H H T ( λ -λ * ) + ( λ -λ * ) T ( λ -λ * ) = (δ x -δ x * ) T (δ x -δ x * ) + ( λ -λ * ) T ( λ -λ * ) = δx -δx * 2 B -1 + λ -λ * 2 R
which can be written as

q(δx, λ) = δx λ - δx * λ * 2 D , (4.13)
where λ * denotes the exact solution of the linear system (4.11), δ x * = H T λ * denotes the exact solution of the problem (4.8),

λ * = R -1/2 λ * , δx * = B 1/2 δ x * and δx = B 1/2 δ x.
The norm D is defined in Section 4.2. This dual approach that applies CG on the linear system (4.11) in IR m and recovers the solution in IR n from the relation (4.12) is of interest in atmospheric and ocean data assimilation systems since IR m represents the physical space and when the number of available observations m is much smaller than the number control variables n, this algorithm is more efficient in terms of computational cost. In ocean data assimilation this approach is known as the indirect representer method (Egbert, Bennett and Foreman, 1994) whereas in meteorological data assimilation it is called PSAS (Physical Space Assimilation System) (Courtier, 1997). Convergence properties of this dual approach is investigated by several authors (Akkraoui, Gauthier, Pellerin and Buis, 2008), (Gratton and Tshimanga, 2009) and it is shown that this dual approach produces iterates in the dual space of Lagrange multipliers (see Section 4.1), and that their corresponding primal-space counterparts (given by (4.12)) do not ensure monotonic decrease of the quadratic function (2.46) along the inner-iterations. Indeed, it turns out that this quadratic cost has an erratic behaviour, even on simple examples. From (4.13) it can also be seen that this dual approach is not minimizing the quadratic function (2.46), it minimizes the direct error for the vector [δx T , λ T ] T in the D-norm. Thus, if the iterations are stopped before exact optimality is attained, there is no guarantee that the value of the quadratic cost has decreased, which may then negatively affect the convergence of the Gauss Newton method.

A better alternative is to ensure the decrease of the quadratic cost function (2.46) by using a non-canonical inner-product. It can be shown that the iterates obtained by using the H H T -inner product within CG on the linear system (4.11):

( H H T + I m ) λ = c are related to standard CG on ( H T H + I n ) δ x = H T c
according to δ x i = H T λ i . From this relation using CG on the linear system (4.11) with the H H T (= R -1/2 HBH T R -1/2 )-inner product minimizes the cost function (2.47) which is equivalent to the quadratic cost function (2.46) under the assumption that the initial value satisfies s 0 = x cx. This assumption seems to be restrictive; however it can be overcome as explained in Section 4.4. Therefore, using this dual approach guarantees to have the same convergence properties as those of the primal approach.

This dual approach that uses a non-standard inner product in CG is the main idea behind the Restricted PCG (RPCG) method proposed by Gratton and Tshimanga (2009). RPCG solves the linear system

(R -1 HBH T + I m ) λ = R -1 (H(x -x c ) -d) (4.14)
with the (possibly semi-)definite HBH T -(semi) inner product, in which the unsymmetric system matrix

A = R -1 HBH T + I m (4.15)
becomes symmetric. Note that, RPCG generates the same iterates when applying standard CG Algorithm 2.2 on the linear system (4.7) with R -1 preconditioner (applying CG Algorithm on the linear system (4.11)) using the R -1/2 HBH T R -1/2 -inner product.

RPCG generates mathematically equivalent iterates to those of PCG Algorithm 2.5 applied to the linear system (2.50) with the canonical inner product and a precondi-tioner matrix B. Under the assumption that s 0 = x c -x, it also generates the same iterates as those obtained by applying PCG on the linear system (2.16) (primal approach) with a preconditioner matrix B. Note also that RPCG iterates are related with the iterates generated from CG on the linear system (4.11) with the R -1/2 HBH T R -1/2 -inner product according to λ i = R -1/2 λ i .

A suitable algorithm for RPCG with a preconditioner matrix D can be deduced from the primal approach with a preconditioner matrix P as follows.

It is shown in (Gratton and Tshimanga, 2009) that there exist m-dimensional vectors r i , z i , p i , q i , λ i that can be related to the corresponding n-dimensional vectors r i , z i , p i , q i , s i of the primal approach according to

r i = H T r i , z i = BH T z i p i = BH T p i q i = H T q i s i = s 0 + BH T λ i                (4.16)
where i ≥ 0. Equations (4.16) allow all the recurrence relations in the primal approach involving vectors of dimension n to be transformed directly into corresponding recurrence relations involving vectors of dimension m as done in Algorithm 4.1. This derivation is detailed in (Gratton and Tshimanga, 2009) and uses two assumptions. The first assumption is related with the initial residual vector which is Assumption 4.1 r 0 = H T r 0 , and holds if s 0 = x cx. Note that, the initial vector can also be chosen arbitrarily by using a generalized version of the RPCG algorithm in which augmented matrices and vectors are introduced (Gratton and Tshimanga, 2009). We postpone this discussion to Section 4.4 for the sake of simplicity. This assumption basically points out that accommodating the initial residual in the proper space is crucial when using the dual approach.

The second assumption is related to the preconditioner used in RPCG,

Assumption 4.2 P H T = BH T D,
where D is the preconditioner for RPCG and P is the preconditioner for the primal approach. At first sight, this assumption may appear restrictive because such a preconditioner D may not exist, in particular if, for some P , P H T is not included in the range of BH T . However, it is shown in Gratton, Gürol and Toint (2013) that the widespread warm-start preconditioning techniques based on limited memory methods (Tshimanga, Gratton, Weaver and Sartenaer, 2008), [START_REF] Morales | Automatic preconditioning by limited memory quasinewton updating[END_REF] do satisfy this condition.

The mathematical equivalence of the primal approach and RPCG (under Assumptions 4.1 and 4.2) ensures the monotonic decrease of the quadratic cost (2.46) along the RPCG inner iterations. This feature makes RPCG preferable to the first alternative which reduces the direct error.

1 λ 0 = 0 2 r 0 = R -1 (H(x -x c ) -d) 3 z 0 = D r 0 4 ρ 0 =< r 0 , z 0 > HBH T 5 p 0 = z 0 6 for i = 0, 1, ..., l -1 do 7 q i = (I m + R -1 HBH T ) p i 8 α i = ρ i / < q i , p i > HBH T 9 λ i+1 = λ i + α i p i 10 r i+1 = r i -α i q i 11 z i+1 = D r i+1 12
Check convergence and stop if desired accuracy is reached

13 ρ i+1 =< r i+1 , z i+1 > HBH T 14 β i = ρ i+1 /ρ i 15 p i+1 = z i+1 + β i p i 16 end 17 The solution is recovered from s l = x c -x + BH T λ l
However, the first version of the RPCG algorithm (stated as Algorithm 4.1) is expensive since it requires three matrix vector products involving HBH T for each inner loop. Fortunately, it can be rewritten in a much cheaper form by introducing additional dual-space vectors, reducing its cost per loop to a single matrix-vector product with HBH T . More precisely, consider w and t defined by

w i = HBH T z i and t i = HBH T p i , (4.17)
where z i and p i are defined in Algorithm 4.1. If we multiply lines 5 and 15 of Algorithm 4.1 by HBH T , we obtain that

t i = w 0 if i = 0 w i + β i-1 t i-1 if i > 0,
which yields the final version of RPCG (Algorithm 4.2).

As a summary, we outlined dual approaches which perform minimization in the data space IR m leading to algorithms that are computationally cheaper and require less memory comparing to that of primal approach when m n. Once it is decided to solve the dual problem instead of the primal problem due to the computational and memory advantages, a special attention has to be paid on

• accommodating the initial residual in the right space, i.e. Im(H T ), to ensure constructing the basis for the same solution subspace for dual and primal approaches,

• choosing the proper inner product, i.e. the HBH T inner product, in dual space to preserve monotonic decrease on the cost function (2.46), Algorithm 4.2: RPCG Algorithm

1 λ 0 = 0 2 r 0 = R -1 (H(x c -x) -d) 3 z 0 = D r 0 4 p 0 = z 0 5 w 0 = HBH T z 0 6 ρ 0 = r T 0 w 0 7 t 0 = w 0 8 for i = 0, 1, ..., l -1 do 9 q i = R -1 t i + p i 10 α i = ρ i / q T i t i 11 λ i+1 = λ i + α i p i 12 r i+1 = r i -α i q i 13 z i+1 = D r i+1 14 w i+1 = HBH T z i+1 15
Check convergence and stop if desired accuracy is reached

16 ρ i+1 = r T i+1 w i+1 17 β i = ρ i+1 /ρ i 18 p i+1 = z i+1 + β i p i 19 t i+1 = w i+1 + β i t i 20 end 21 The solution is recovered from s l = x c -x + BH T λ l
• choosing the proper preconditioner in dual space, i.e. a preconditioner that satisfies Assumption 4.2, to preserve one to one correspondance that keeps the desired monotonic convergence behaviour.

Preconditioning RPCG with LMPs

In Section 3.1 we present three approaches (Approaches (A), (B) and (C)) in primal space for preconditioning a sequence of linear systems given by (3.1). We assume that this sequence produces a convergent sequence of solutions within the corresponding algorithms (Algorithms 3.1, 3.2 and 3.5).

In this section, we explain in detail preconditioning RPCG with LMPs following the strategy of Approach (B) that does not require the factorization of the preconditioner. We later explain briefly how to use a split preconditioning in dual space. Note that, these approaches in dual space are developed such that the desired convergence properties in primal space are preserved. Since RPCG does not require matrix-vector products with B -1 , the third approach (Approach (C)) is not necessary with RPCG.

We consider two algorithms in primal space within the section, Algorithms 3.1 and 3.2. In these algorithms using the fact that the linear systems in sequence are not isolated and they produce a convergent sequence of solutions; the initial guess is zero, the right hand side is updated using the solution of the previous system, so that the minimization actually continues from the point obtained at the last iteration. By this way along the sequence, some information is carried over from one CG run to the next, which yields faster convergence in the case where the right hand sides slowly vary along the sequence of linear systems.

We start with the dual approach that does not require the factorization of the preconditioner and generates mathematically equivalent iterates to those of Approach (B) with the zero initial guess.

Approach (B) in dual space

In order to keep mathematically equivalent iterates between the dual and primal algorithm (Approach (B) with zero initial guess), we first need to ensure that Assumption 4.1 holds. We show that this assumption holds by transforming the linear system (4.14) into another linear system by using change of variables.

Let us now define the dual solution s k such that

s k = BH T s k . (4.18)
Using this vector the solution at the k-th outer loop can be written as

x k+1 = x k + s k = x 1 + k j=1 s j = x 1 + BH T k j=1 s j .
Taking x 1 = x c gives that

x k+1 -x c = BH T k j=1 s j . (4.19)
Using this expression with the relation

s k = x c -x k + BH T λ k (see line 21 of Algo- rithm 4.
2), we obtain that

s k = -BH T k-1 j=1 s j + BH T λ k = BH T   λ k - k-1 j=1 s j   . (4.20)
From (4.18) and (4.20) we can find the relation between the vectors s k and λ k as

s k = λ k - k-1 j=1 s j . (4.21)
Using the change of variable λ = s + k-1 j=1 s j , and the equation (4.19) in the linear system (4.14), we have

(R -1 HBH T + I m )( s + k-1 j=1 s j ) = R -1 (HBH T k-1 j=1 s j -d k ),
which can be written as

(R -1 HBH T + I m ) s = -R -1 d k - k-1 j=1 s j . (4.22)
Then taking s 0 = 0, the initial residual of the system (4.22) can be written as

r 0 = -R -1 d k - k-1 j=1 s j . (4.23)
Multiplying this expression from the left with H T and using the relations (4.18) and (4.19) gives that

H T r 0 = -H T R -1 d k -H T k-1 j=1 s j = -H T R -1 d k -B -1 (x k -x c ) = B -1 (x c -x k ) -H T R -1 d k ,
which is the inital residual of the linear system (3.1) with s 0 = 0. This ensures that Assumption 4.1 holds, i.e. H T r 0 = r 0 . (4.24)

Therefore, under the assumption that P H T = BH T D, applying RPCG on the linear system (4.22) with the HBH T -inner product and recovering the solution in IR n from the relation (4.18) yields mathematically equivalent iterates {s k } to those of applying PCG Algorithm 2.5 to the linear system (3.1) (Approach (B)) with zero initial guess. We name this variant of RPCG as Approach (DualB). This approach is given by Algorithm 4.3. Like Approach (B) this dual approach does not require a factorization of the preconditioner.

Algorithm 4.3: RPCG Algorithm (version for s 0 = 0)

1 s 0 = 0 2 r 0 = -R -1 d k - k j=1 s j 3 z 0 = D r 0 4 p 0 = z 0 5 w 0 = HBH T z 0 6 ρ 0 = r T 0 w 0 7 t 0 = w 0 8 for i = 0, 1, ..., l -1 do 9
Same pseudo-code as with Algorithm 4.2 for the variable s i 10 end 11 The solution is recovered from s l = BH T s l equation given in the lemma yields that

P H T = [I n -BH T S( S T HBH T A S) -1 S T HBA]M H T [I m -A S( S T HBH T A S) -1 S T HBH T ] +BH T S( S T HBH T A S) -1 S T HBH T = BH T [I m -S( S T HBH T A S) -1 S T HBH T A] M [I m -A S( S T HBH T A S) -1 S T HBH T ] +BH T S( S T HBH T A S) -1 S T HBH T = BH T [I m -S( S T HBH T A S) -1 S T HBH T A] M [I m -A S( S T HBH T A S) -1 S T HBH T ] + S( S T HBH T A S) -1 S T HBH T = BH T D
which completes the proof.

2

In this lemma since P is a symmetric positive definite preconditioner and HP H T = HBH T D, we have that when H has full row rank, the matrix HBH T D is not only symmetric but also positive definite. Note that the LMP D given in Lemma 4.1 is a preconditioner for A = I n + R -1 HBH T and HBH T D is an approximation of the inverse matrix HBH T A = HBH T + HBH T R -1 HBH T .

The formula (4.26) given in Lemma 4.1 can be rewritten inductively if S ∈ IR m×l is A-conjugate with respect to the HBH T -inner product. The next lemma provides such form of the preconditioner.

Lemma 4.2 Let HBH T A and HBH T M be m × m symmetric positive definite matrices. Assume also that S ∈ IR m×l is A-conjugate with respect to the HBH T -inner product. Then the matrix D defined by (4.26) can be written as

D l = I m - l i=1 s i s T i s T i HBH T A s i HBH T A M I m - l i=1 A s i s T i s T i HBH T A s i HBH T + l i=1 s i s T i s T i HBH T A s i HBH T , (4.28)
which can also be written inductively as

D l = I m - s l s T l HBH T A s T l HBH T A s l D l-1 I m - A s l s T l HBH T s T l HBH T A s l + s l s T l HBH T s T l HBH T A s l . (4.29) Proof. From S = [ s 1 , • • • , s l ]
and the A-conjugacy of the matrix S in the HBH T inner-product, i.e. s T i HBH T A s j = 0 for i = j, S T HBH T A S becomes a diagonal matrix. Then, S( S T HBH T A S) -1 S T can be written as

S( S T HBH T A S) -1 S T = [ s 1 , • • • , s l ]     s T 1 HBH T A s 1 . . . s T l HBH T A s l     -1     s T 1 . . . s T l     = [ s 1 , • • • , s l ]      1 s T 1 HBH T A s1
. . .

1 s T l HBH T A s l          s T 1 . . . s T l     = [ s 1 , • • • , s l ]      s T 1 s T 1 HBH T A s1 . . . s T l s T l HBH T A s l      = l i=1 s i s T i s T i HBH T A s i .
Substituting this expression in (4.26) gives the equation (4.28).

We next prove the second part of the lemma. Let us define D 0 = M and Q = HBH T . From the equation (4.28) for = 1 we obtain that

D 1 = I m - s 1 s T 1 s T 1 Q A s 1 Q A D 0 I m -A s 1 s T 1 s T 1 Q A s 1 Q + s 1 s T 1 s T i Q A s 1 Q.
We now assume that l > 1 and define

V l = I m - Q A s l s T l s T l Q A s l , (4.30) W l = I m - A s l s T l Q s T l Q A s l . (4.31)
Using these definitions, we have

V T l l-1 i=1 s i s T i s T i Q A s i = l-1 i=1 s i s T i s T i Q A s i - s l s T l A T Q s T l Q A s l l-1 i=1 s i s T i s T i Q A s i , (4.32) l-1 i=1 s i s T i s T i Q A s i QW l = l-1 i=1 s i s T i Q s T i Q A s i - l-1 i=1 s i s T i Q s T i Q A s i A s l s T l Q s T l Q A s l . (4.33)
Using the symmetry of A with respect to the HBH T -inner product and the assumption that the vectors s i , i = 1, . . . , are A-conjugate with respect to the HBH T -inner product, the second part of equations (4.32) and (4.33) vanishes yielding

V T l l-1 i=1 s i s T i s T i Q A s i = l-1 i=1 s i s T i s T i Q A s i (4.34) l-1 i=1 s i s T i s T i Q A s i QW l = l-1 i=1 s i s T i Q s T i Q A s i (4.35)
Using the definitions (4.30) and (4.31) we also have

V T l I m - l-1 i=1 s i s T i Q A s T i Q A s i = I m - l-1 i=1 s i s T i Q A s T i Q A s i - s l s T l A T Q s T l Q A s l + s l s T l A T Q s T l Q A s l l-1 i=1 s i s T i Q A s T i Q A s i = I m - l-1 i=1 s i s T i Q A s T i Q A s i - s l s T l A T Q s T l Q A s l = I m - l i=1 s i s T i Q A s T i Q A s i . (4.36) I m - l-1 i=1 A s i s T i s T i Q A s i Q W l = I m - A s l s T l Q s T l Q A s l - l-1 i=1 A s i s T i Q s T i Q A s i + l-1 i=1 A s i s T i Q s T i Q A s i A s l s T l Q s T l Q A s l = I m - A s l s T l Q s T l Q A s l - l-1 i=1 A s i s T i Q s T i Q A s i = I m - l i=1 A s i s T i Q s T i Q A s i . (4.37)
Let us now write the formula (4.28) for the matrix D l-1 , l > 1 which is given by

D l-1 = I m - l-1 i=1 s i s T i s T i Q A s i Q A M I m - l-1 i=1 A s i s T i s T i Q A s i Q + l-1 i=1 s i s T i s T i Q A s i Q.
Using this formula we have

V T l D l-1 W l = V T l I m - l-1 i=1 s i s T i s T i Q A s i Q A M I m - l-1 i=1 A s i s T i s T i Q A s i Q W l +V T l l-1 i=0 s i s T i s T i Q A s i QW l . (4.38)
Substituting the equations (4.36), (4.37), (4.34), and (4.34) into (4.38) we obtain that

V T l D l-1 W l + s l s T l Q s T l Q A s l = I m - l i=1 s i s T i Q A s T i Q A s i M I m - l i=1 A s i s T i Q s T i Q A s i + l-1 i=1 s i s T i Q s T i Q A s i + s l s T l Q s T l Q A s l = D l , (4.39)
which gives the desired result. 2

We are left with the task of integrating Approach (DualB) when solving a convergent sequence of linear systems (4.22) of the form of A s = b k , with multiple right-hand sides where preconditioning is achieved by the warm-start LMP technique.

Assume that we choose the initial guess x 1 = x c . Assume also that the initial preconditioner D 0 is chosen such that

P 0 H T = BH T D 0 , (4.40)
for instance P 0 = B and D 0 = I. Then for k = 1, RPCG Algorithm 4.3 solves the linear system A s = b 1 (4.41) for the variable s by using the preconditioner D 0 and recover the approximate solution from the relation s 1 = BH T s 1 . This approximate solution is then used to update the current iterate from x 2 = x 1 + s 1 .

While performing the RPCG algorithm on the linear system (4.41), the information can be stored to obtain the LMP D 1 for the system matrix A by using the formula (4.26). This preconditioner can then be used to precondition the second linear system. Note that while constructing D 1 from (4.26), M 0 is taken as the initial preconditioner matrix, i.e. M 0 = D 0 . From Lemma 4.1 we ensure that P 1 H T = BH T D 1 where P 1 is the preconditioner for the second linear system in Algorithm 3.2 assuming that S 0 = BH T S 0 .

Using the preconditioner D 1 for k = 2, RPCG solves the linear system

A s = b 2 (4.42)
for the variable s and recover the approximate solution from s 2 = BH T s 2 . This approximate solution is then used to update the current iterate from

x 3 = x 2 + s 2 .
As it is done for k = 1, while performing the RPCG algorithm on the linear system (4.42) the information can be stored to obtain the LMP D 2 for the system matrix A by using the formula (4.26). Note that while constructing D 2 from (4.26), M 1 is taken as the preconditioner used for k = 2, M 1 = D 1 . From Lemma 4.1 we ensure that P 2 H T = BH T D 2 where P 2 is the preconditioner for the third linear system in Algorithm 3.2 assuming that S 1 = BH T S 1 .

The same strategy can be carried over for later steps. Using the preconditioner D k , for the (k + 1)-th linear system RPCG Algorithm solves the preconditioned system

A s = b k+1 (4.43)
for the variable s and recover the approximate solution from s k+1 = BH T s k+1 . This approximate solution is then used to update the current iterate from x k+2 = x k+1 + s k+1 . Algorithm 4.4 outlines this implementation. Like Algorithm 3.2 this algorithm does not require the factorization of the preconditioner.

Algorithm 4.4: RPCG for solving a sequence of linear systems with multiple right-hand sides 1 Initialization: Choose an initial preconditioner D 0 such that Assumption 4.2 holds. Set

x 1 = x c .
2 Perform inner loop: Using preconditioner D k-1 solve the linear system

(R -1 HBH T + I m ) s = -R -1 d k - k j=1 s j ,
with RPCG Algorithm 4.3. During RPCG extract and save relevant information to precondition the next linear system with D k formulated by (4.26). The solution in IR n is recovered from

s k+1 = BH T s k+1
3 Update the iterate:

x k+1 = x k + s k+1
4 Increment k by 1 and go to step 2.

When solving the k-th linear system in Algorithm 4.4, RPCG Algorithm 4.3 requires a single matrix-vector product with each of the operators H, H T , R -1 , B and D k-1 .

Algorithm 4.4 generates a mathematically equivalent iterates of {s k } to those of Algorithm 3.2 under the assumptions that

• x 1 = x c ,
• s 0 = 0 for Approach (B),

• P 0 H T = BH T D 0 ,
while possibly yielding gains in terms of both memory usage and computational cost when m n. We summarize the characteristics of the equivalent primal and dual algorithms by Table 4.1.

We next explain briefly how to perform split preconditioning with RPCG.

Algorithm

Inner min. LMP formula Assumption 3.2 App. (B) P k+1 given by (2.38)

x 1 = x c with M k = P k s 0 = 0 4.4 App. (DualB) D k+1
given by (4.26)

P 0 H T = BH T D 0 with M k = D k and S k = BH T S k x 1 = x c
Table 4.1: A summary of the characteristics of Algorithms 3.2 and 4.4.

Approach (A) in the dual space

The linear system (4.22) can be equivalently solved by applying CG Algorithm 2.2 to the linear system

(R -1/2 HBH T R -1/2 + I m ) v = -R -1/2 d k -R 1/2 k-1 j=1 s j , (4.44)
with the (R -1/2 HBH T R -1/2 )-inner product where s = R -1/2 v. We can see this equivalence by observing that the linear system (4.44) is obtained after applying R 1/2left preconditioning and R -1/2 -right preconditioning to the linear system (4.22). In order to preserve mathematically equivalent iterates to RPCG Algorithm 4.3 (with D = I), the inner product should be changed accordingly (Chan, Chow, Saad and Yeung, 1999) which becomes in this case the (R -1/2 HBH T R -1/2 )-inner product.

We can rewrite the linear system (4.44) in the form of

R 1/2 AR -1/2 v = R 1/2 b k , (4.45)
where

A = R -1 HBH T + I m and b k = R -1 d k - k-1
j=1 s j . How can we construct and apply a LMP for the linear system (4.45) provided that we obtain mathematically equivalent iterates to those of Approach (DualB)? The answer is explained as follows.

Assume that Y = F F T is a preconditioner for the linear system matrix in (4.45), i.e. R 1/2 AR -1/2 . Using this preconditioner, CG Algorithm can be applied to the linear system

F T (R 1/2 AR -1/2 ) F v = F T R 1/2 b k , (4.46)
for the variable v using the ( F -1 R -1/2 HBH T R -1/2 F )-inner product. The solution in dual space is recovered from s = R -1/2 F v. We name this algorithm Approach (DualA).

Let us assume that

D = R -1/2 F F T R 1/2 , (4.47)
where D is the LMP used during Approach (DualB). Following the same strategy in (Gratton and Tshimanga, 2009), it can be easily shown that Approach (DualB) and Approach (DualA) generates mathematically equivalent iterates under the assumption (4.47).

When solving a sequence of linear systems for the (k + 1)-th linear system in sequence Approach (DualA) solves the linear system

F T k ( F T k-1 . . . F T 1 R 1/2 AR -1/2 F 1 . . . F k-1 ) F k v = F T k F T k-1 . . . F T 1 R 1/2 b k , (4.48) with the ( F -1 k . . . F -1 1 R -1/2 HBH T R -1/2 F 1 . . . F k )-inner product. The solution in dual space is recovered from s k+1 = R -1/2 F 1 . . . F k v k+1 .
In this case Assumption (4.47) can also be generalized as

D k = R -1/2 F 1 . . . F k F T k . . . F T 1 R 1/2 . ( 4
P k H T = BH T D k , P 1/2 k-1 Y k P 1/2 k-1 H T = BH T R -1/2 F 1 . . . F k F T k . . . F T 1 R 1/2 .
From this equation, using the relations P

1/2 k = F 0 . . . F k , Y k = F k F T k and taking F 0 = B 1/2 , we have B 1/2 F 1 . . . F k-1 F k F T k F T k-1 . . . F 1 B 1/2 H T = BH T R -1/2 F 1 . . . F k F T k . . . F T 1 R 1/2 .
Multiplying this equation from left by R -1/2 and from right by B -1/2 yields that

F 1 . . . F k-1 F k F T k F T k-1 . . . F 1 B 1/2 H T R -1/2 = B 1/2 H T R -1/2 F 1 . . . F k F T k . . . F T 1 . (4.50)
Therefore, under this assumption on the preconditioners Approach (A) and Approach (DualA) generates a mathematically equivalent sequence of iterates {s k }.

The formula for the LMP F k can be constructed by using the relation (4.47) and the LMP formula (4.26). Within this thesis preconditioning with LMPs is not explained in detail for Approach (DualA), however the necessary relations for the derivation of the LMPs are provided.

In the next section, we explain in detail preconditioning with particular LMPs: the quasi-Newton LMP, the Ritz LMP and the spectral LMP, for Approach (DualB). This approach is more efficient than Approach (DualA) in terms of computational cost for large-scale systems since it does not require the square-root factorization of the matrices B and R -1 . Like Chapter 3, this part will include for each preconditioner a definition of the preconditioner, its properties, an extraction methodology of the required information from Approach (DualB), and memory and flops requirements for the application of the preconditioner.

Preconditioning with the quasi-Newton LMP

In this section we want to derive the quasi-Newton LMP to precondition the linear system (4.22). We may follow the by now familiar pattern of deriving an equivalent preconditioner to that of the quasi-Newton LMP (3.37). It is shown in Gratton, Gürol and Toint (2013) that this is indeed possible and that the resulting formula satisfies a variational property similar to that described by (3.38)-(3.39). We first focus on deriving a quasi-Newton limited memory preconditioner for dual approach satisfying Assumption 4.2.

The search directions p i , i = 1, . . . , j generated by Algorithm 4.3 are A-conjugate with respect to the HBH T -inner product, i.e.

p T i HBH T A p j = 0 for i = j.
Therefore, choosing these directions in the formula (4.29), i.e. s i ≡ p i , i = 1, . . . , j gives that

D j+1 = I m - p j p T j HBH T A p T j HBH T A p j D j I m - A p j p T j HBH T p T j HBH T A p j + p j p T j HBH T p T j HBH T A p j ,
where A = I m + R -1 HBH T . Using this formula the quasi-Newton LMP for the matrix A in the k-th linear system (in Algorithm 4.4) can be constructed from D j+1 = I mτ j p j q T j HBH T D j I mτ j q j t T j + τ j p j t T j , (4.51)

where q i = A p i , t i = HBH T p i , and τ i = 1/( q T i t i ) are obtained during Algorithm 4.3 applied to the previous linear system. The initial preconditioning matrix for the formula (4.51) is chosen as the preconditioner that is used for the (k -1)-th linear system.

Using the relations (4.16), it can be easily shown that the preconditioner P j+1 formulated by (3.37) and the preconditioner D j+1 formulated by (4.51) satisfies the relation P j+1 H T = BH T D j+1 (Gratton, Gürol and Toint, 2013).

We now show that the preconditioner D obtained from (4.51) also satisfies variational properties in the dual space.

Lemma 4.3 Let Q = HBH T and suppose that Q is non-singular. Then the matrix ∆D j defined by ∆D j = D j+1 -D j where D j+1 defined in (4.51) is the solution of

min ∆Dj W 1/2 Q 1/2 ∆D j Q -1/2 W 1/2 F (4.52) subject to Q∆D j = ∆D T j Q, D j+1 q j = p j ,
where W is any symmetric positive definite matrix satisfying W Q 1/2 p j = Q 1/2 q j .

Proof. Using the change of variables

∆X j = Q 1/2 ∆D j Q -1/2 , p j = Q -1/2 s j and q j = Q -1/2 y j (4.53)
we can rewrite problem (4.52) as

min ∆Xj W 1/2 ∆X j W 1/2 F subject to ∆X j = ∆X T j , X j+1 y j = s j ,
which is structurally identical to problem (3.38). Using now (3.40) in this context yields that

∆X j = W -1 y j (s j -X j y j ) T + (s j -X j y j )y T j W -1 y T j W -1 y j - y T j (s j -X j y j )W -1 y j y T j W -1 (y T j W -1 y j ) 2 . (4.54)
Substituting (4.53) into this solution and multiplying by Q 1/2 on the right and Q -1/2 on the left gives that

∆D j = Q -1/2 W -1 Q 1/2 q j (Q 1/2 p j -Q 1/2 D j q j ) T Q 1/2 + ( p j -D j q j ) q T j Q 1/2 W -1 Q 1/2 q T j Q 1/2 W -1 Q 1/2 q j - Q -1/2 q T j Q 1/2 (Q 1/2 p j -Q 1/2 D j q j )W -1 Q 1/2 q j q T j Q 1/2 W -1 Q 1/2 ( q T j Q 1/2 W -1 Q 1/2 q j ) 2
.

From the relation W Q 1/2 p j = Q 1/2 q j , we deduce that Q 1/2 p j = W -1 Q 1/2 q j . Substituting this expression in the solution gives

∆D j = p j ( p j -D j q j ) T Q + ( p j -D j q j ) p T j Q q T j Q p j - q T j Q( p j -D j q j )p j p T j Q ( q T j Q p j ) 2 . (4.55)
On the other hand, we can reformulate (4.51) as:

D j+1 = D j - D j q j p T j Q q T j Q p j - p j q T j QD j q T j Q p j + p j q T j QD j q j p T j Q ( q T j Q p j ) 2 + p j p T j Q q T j Q p j = D j + ( p j -D j q j ) p T j Q q T j Q p j - p j q T j QD j q T j Q p j + p j q T j QD j q j p T j Q ( q T j Q p j ) 2
.

Adding and substracting the term

pj p T j Q q T j Q pj gives that, D j+1 = D j + ( p j -D j q j ) p T j Q q T j Q p j + p j p T j Q q T j Q p j - p j p T j Q q T j Q p j - p j q T j QD j q T j Q p j + p j q T j QD j q j p T j Q ( q T j Q p j ) 2
, which can be reorganized as

D j+1 = D j + ( p j -D j q j ) p T j Q + p j (Q p j -D T j Q q j ) T q T j Q p j - q T j Q( p j -D j q j ) p j p T j Q ( q T j Q p j ) 2
.

Using the property that QD j = D T j Q (see Lemma 4.1), we write that

D j+1 = D j + ( p j -D j q j ) p T j Q + p j ( p j -D j q j ) T Q q T j Q p j - q T j Q( p j -D j q j ) p j p T j Q ( q T j Q p j ) 2
, which is equivalent to D j + ∆D j where ∆D j is given by the formula (4.55). 2

Having found a suitable preconditioner (in the sense that it satisfies Assumption 4.2) and having verified that it shares desirable variational properties with its primal equivalent, we are left with the task of integrating it into the RPCG algorithm. From formula (4.51), we need to store the sequences of q, p, t, and HBH T q to obtain the successive preconditioner updates. Storing q, p and t does not require additional cost since they are already available from a run of RPCG Algorithm 4.3. On the other hand, the quantity HBH T q is not a by-product of the algorithm, and seems, at first sight, to require an additional matrix vector product, which may appear computationally costly. Fortunately, we can rewrite RPCG Algorithm 4.3 in a more computationally effective way by introducing a vector l defined by l i = HBH T r i .

Since z i = D r i and HBH T D is symmetric from Lemma 4.1, we may therefore write that

w i = HBH T D r i = D T HBH T r i = D T l i . (4.56)
Moreover, multiplying the line that updates the residual of Algorithm 4.3 by HBH T gives that HBH T q i = (l il i+1 )/α i which is the matrix vector product HBH T q that we need to store. Using all these relations, we can transform RPCG Algorithm 4.3 into Algorithm 4.5. As a result it is possible to find the corresponding quasi-Newton LMP (LMP (4.51)) for Algorithm 4.5 such that when it is used within Algorithm 4.4, it generates a mathematically equivalent sequence of iterates {s k } to those of Algorithm 3.2 with the quasi-Newton LMP (3.37), while possibly yielding gains in terms of both memory usage and computational cost when m n. We showed that this preconditioner can be constructed from vectors which are by-products of Algorithm 4.5.

We summarize the characteristics of the equivalent primal and dual algorithms when used with the quasi-Newton LMP by Table 4.2. In this table we define another approach for RPCG Algorithm when used with the quasi-Newton LMP : Algorithm 4.5: RPCG Algorithm (version for quasi-Newton LMP)

1 s 0 = 0 2 r 0 = -R -1 d k - k j=1 s j 3 l 0 = HBH T r 0 4 z 0 = D r 0 5 p 0 = z 0 6 w 0 = D T l 0 7 ρ 0 = r T 0 w 0 8 t 0 = w 0 9 for i = 0, 1, ..., l -1 do 10 q i = R -1 t i + p i 11 α i = ρ i / q T i t i 12 s i+1 = s i + α i p i 13 r i+1 = r i -α i q i 14 l i+1 = HBH T r i+1 15 i = (l i -l i+1 )/α i 16 z i+1 = D r i+1 17 w i+1 = D T l i+1 18
Check convergence and stop if desired accuracy is reached We now want to discuss the implementation issues for the quasi-Newton LMP (4.51). In Algorithm 4.5, matrix-vector products with D j and D T j can be calculated by adapting the L-BFGS two-loop recursion which results in Algorithm 4.6 and 4.7. Each algorithm requires 8jm flops and a matrix vector product with D 0 .

19 ρ i+1 = r T i+1 w i+1 20 β i = ρ i+1 /ρ i 21 p i+1 = z i+1 + β i p i 22 t i+1 = w i+1 + β i t i
The memory requirement for the quasi-Newton preconditioner D is (4m + 1)j. Therefore when comparing the cost and memory requirements for the primal approach Algorithm 4.6: Compute z = D r

1 Given r set v ← r 2 for i = j : -1 : 1 do 3 η i ← τ i t T i v 4 v ← v -η i q i 5 end 6 z ← D 0 v 7 for i = 1:j do 8 ζ ← τ i T i z 9 z ← z + p i (η i -ζ) 10 end Algorithm 4.7: Compute w = D T l 1 Given l set v ← l 2 for i = j : -1 : 1 do 3 η i ← τ i p T i v 4 v ← v -η i i 5 end 6 w ← D T 0 v 7 for i = 1:j do 8 ζ ← τ i q T i w 9 w ← z + t i (η i -ζ)
10 end with a quasi-Newton LMP given in Table 3.4, the cost for appying the preconditioner for the dual approach is much less than that of the primal approach when m n.

Preconditioning with the Ritz LMP

We now derive a Ritz LMP for Algorithm 4.3 with a similar idea as in the previous subsection that this preconditioner satisfies Assumption 4.2 so that one to one correspondance between the primal approach and RPCG remains. We first explain in Lemma 4.4 and 4.5 how to extract the spectral information and the Lanczos vectors generated in IR m from RPCG Algorithm 4.3. Using later the relations given in these lemmas we derive the Ritz-LMP expressions in dual space.

Lemma 4.4 Let v i , i = 1, . . . , l be l Lanczos vectors generated from (3.51) during PCG Algorithm 2.5 applied to the linear system (3.1) in the form of A s = b with a symmetric positive definite preconditioner P and zero initial guess. Assume also that D is a preconditioner for RPCG Algorithm 4.3 satisfying the relation P H T = BH T D.

Then the Lanczos vectors v i , i = 1, . . . , l generated from the formula

v i+1 = (-1) i r i r T i w i (4.57)
during RPCG Algorithm 4.3 applied to the linear system (4.22) in the form of A s = b satisfy the relation

v i = H T v i . (4.58)
Let us define g i = P v i to be the preconditioned Lanczos vectors in IR n . Then, the preconditioned Lanczos vectors in IR m defined by

g i = D v i (4.59)
satisfy the relation

g i = BH T g i . (4.60)
The column vectors g i construct an orthonormal basis with respect to the HBH T -inner product for the Krylov subspace K l (D A, D r 0 ). The projected matrix given by (3.55) can be written in terms of the matrix G l = [ g 1 , . . . , g l ] as

T l = G T l HBH T A G l (4.61)
which defines an orthogonal projection of A with respect to the HBH T -inner product onto the Krylov subspace K l (D A, D r 0 ).

Proof. From (4.24) and the assumption on the preconditioners, the residuals r i = b -As i and r i = b i -A s i generated by Algorithm 2.5 and Algorithm 4.3 respectively satisfy the following relation (Gratton and Tshimanga, 2009)

r i = H T r i .
with i = 0, . . . , l -1. Substituting this relation into (3.51) and using the relation

P H T = BH T D gives that v i+1 = (-1) i r i r i P = (-1) i H T r i H T r i P = H T (-1) i r i r i HBH T D = H T v i+1
which proves the first part of the Lemma. Using this relation with (4.60) and the assumption on preconditioners in the definition of g i gives that

g i = P H T v i = BH T D v i = BH T g i .
We now show that g i vectors form an orthornormal basis for K l (D A, D r 0 ). Using the relations (4.24), P H T = BH T D and the fact that ABH T = H T A, the subspace K l (P A, P r 0 ) spanned by the vectors g i can be written as K l (P A, P r 0 ) = span{P H T r 0 , P AP H T r 0 , . . . , (P A) l-1 P H T r 0 } = span{BH T D r 0 , P ABH T D r 0 , . . . ,

(P A) l-1 BH T D r 0 } = span{BH T D r 0 , BH T D AD r 0 , . . . , BH T (D A) l-1 D r 0 } = BH T K l (D A, D r 0 ). (4.62)
From this relation and (4.60), it can be seen that the columns of the matrix G l form an orthonormal basis for the Krylov subspace K l (D A, D r 0 ). It remains to show that T l matrix (3.55) can be written as

T l = G T l HBABH T G l = G T l HBH T A G l
where we used the relations (4.60) and ABH T = H T A. 2

Lemma 4.5 Suppose that (θ i , u i ) = (θ i , V l ūi ), i = 1, . . . , l are the Ritz pairs given in Lemma 3.4. Let us define the vectors u i such that

u i = V ūi (4.63)
where V l = [ v 1 , . . . , v l ] having the column vectors v i defined in Lemma 4.4. Then the vectors u i satisfies the relation

u i = H T u i . (4.64)
Suppose also that x i are the vectors given in Lemma 3.4. Then the vectors defined by

x i = G l ūi (4.65)
where the matrix G l = [ g 1 , . . . , g l ] is defined in Lemma 4.4 satisfy the relation

x i = BH T x i . (4.66)
Proof. Using the equations (4.58) and (4.63), from the definition of u i given by (3.59) we deduce that

u i = V l ūi = H T V l ūi = H T u i .
(4.67)

The vector x i can be written from its definition (3.61) as

x i = H l ūi = BH T G l ūi
where we used the relation (4.60). Therefore, it can be seen that the vectors defined by (4.65) satisfies the relation (4.66). 2

We mentioned that the PCG Algorithm 2.5 searches for an approximate solution over the set s 0 + K l (P A, P r 0 ), therefore from Lemma 4.4 it can be seen that RPCG searches for the solution s over the set s 0 + BH T K l (D A, D r 0 ). Our task is now to use the relations given in Lemma 4.4 and Lemma 4.5 to derive the Ritz preconditioner D satisfying Assumption 4.2. Let us consider applying RPCG Algorithm 4.3 to the linear system (4.22) of the form of A s = b k within Algorithm 4.4. We suppose that P 0 H T = BH T D 0 where P 0 is the initial preconditioner for Algorithm 3.2 and D 0 is the initial preconditioner for Algorithm 4.4. Our aim is to find the preconditioner D k-1 for the k-th linear system in Algorithm 4.4 such that P k-1 H T = BH T D k-1 where P k-1 is the Ritz-LMP constructed from (3.64) and used for the k-th linear system in Algorithm 3.2.

Assume that when solving the (k -1)-th linear systems within Algorithm 3.2 and Algorithm 4.4 the relation P k-2 H T = BH T D k-2 holds. Let us define a matrix U = [ u 1 , . . . , u l ] whose columns are defined in Lemma 4.5. Multiplying the equation (3.64) on the left by H T and substituting the relation (4.64) for i = 1, . . . , l, the relation (4.58) for i = l + 1 and P k-2 H T = BH T D k-2 into (3.64) gives that

P k-1 H T = P k-2 H T + P k-2 U (Θ -1 -I )U T P k-2 H T -P k-2 U ωv T l+1 P k-2 H T -P k-2 v l+1 ω T U T P k-2 H T + P k-2 U ωω T U T P k-2 H T , = BH T D k-2 + P k-2 H T U (Θ -1 -I ) U T HP k-2 H T -P k-2 H T U ω v T l+1 HP k-2 H T -P k-2 H T v l+1 ω T U T HP k-2 H T +P k-2 H T U ωω T U T HP k-2 H T , = BH T D k-2 + BH T D k-2 U (Θ -1 -I ) U T D T k-2 HBH T -BH T D k-2 U ω v T l+1 D T k-2 HBH T -BH T D k-2 v l+1 ω T U T D T k-2 HBH T +BH T D k-2 U ωω T U T D T k-2 HBH T , = BH T D k-2 + D k-2 U (Θ -1 -I ) U T D T k-2 HBH T -D k-2 U ω v T l+1 D T k-2 HBH T -D k-2 v l+1 ω T U T D T k-2 HBH T +D k-2 U ωω T U T D T k-2 HBH T ,
from which we can write the preliminary formula for D k-1 as

D k-1 = D k-2 + D k-2 U (Θ -1 -I ) U T D T k-2 HBH T -D k-2 U ω v T l+1 D T k-2 HBH T -D k-2 v l+1 ω T U T D T k-2 HBH T + D k-2 U ωω T U T D T k-2 HBH T . (4.68)
This formula can be written in a more compact form as explained below. Let us define a matrix X = [ x 1 , . . . , x l ] whose column vectors are defined in Lemma 4.5. From the relations (4.65), (4.59) and (4.63) for i = 1, . . . , , it can be easily seen that

D k-2 U = X.
Substituting this relation with the relation (4.59) for i = + 1 into (4.68) gives that

D k-1 = D k-2 + X(Θ -1 -I ) X T HBH T -Xω g T l+1 HBH T -g l+1 ω T X T HBH T + Xωω T X T HBH T .
(4.69)

In order to obtain the successive preconditioner updates for Ritz LMP (4.69) we first need to obtain the triadiagonal matrix T l (see Sect. 2.6.4). Then the diagonal matrix Θ (having eigenvalues of the matrix T l on its diagonal) and the vector ω (defined by (3.48)) can be calculated from the eigenpairs of the matrix T l and the scalar β +1 . This scalar can be obtained from formula (2.36) where η +1 = β +1 . The vectors x i are calculated from (4.65) by using the matrix G l whose column vectors are generated from

g i+1 = (-1) i z i r T i w i , (4.70)
where we used the relations (4.57) and (4.59). The vector g l+1 can also be calculated from this expression for i = l.

In the LMP formula (4.69), the vectors HBH T x i and HBH T g l+1 require matrixvector product with HBH T which may be computationally expensive. This can be handled by storing an additional matrix and obtaining HBH T matrix-vector products from a by-product of Algorithm 4.3. This is explained as follows:

Let us define a vector υ i such that

υ i+1 = HBH T g i+1 = (-1) i w i r T i w i . (4.71)
Then, using the relations (4.65) and (4.71) the vectors y i = HBH T x i can be calculated from

y i = HBH T x i = Υ l ūi (4.72)
where Υ l = [υ 1 , . . . , υ l ]. The vector υ l+1 can also be obtained from (4.71) for i = l. Defining a matrix Y = [y 1 , . . . , y ] and substituting the relations (4.72) for i = 1, . . . , , and (4.71) into (4.69) yields that

D k-1 = D k-2 + X(Θ -1 -I l )Y T -X ω υ T l+1 -g l+1 ω T Y T + Xω ω T Y T . (4.73)
This formula requires to generate and store matrices G l , Υ l and T l . Once we have all the required information from RPCG Algorithm 4.3, Algorithm 4.8 summarizes how to built the the Ritz-LMP (4.73). As a result it is possible to find the corresponding Ritz LMP (LMP (4.73)) for Algorithm 4.3 such that when it is used within Algorithm 4.4, it generates a mathematically equivalent sequence of iterates {s k } to those of Algorithm 3.2 with the Ritz LMP (3.65), while possibly yielding gains in terms of both memory usage and computational cost when m n. We showed that this preconditioner can be constructed from vectors which are by-products of Algorithm 4.3.

We summarize the characteristics of the equivalent primal and dual algorithms independent eigenvectors such that y T i u i = δ ij . Then the LMP matrix given by (4.26) with S = X, X = [ M u 1 , . . . , M u l ] can be written as

D = M + X(Λ -1 -I l ) X T HBH T , (4.74)
where Λ = diag(λ i ).

Proof. Since y i are the eigenvectors of M T A T , we can write

M T A T HBH T M U = HBH T M U Λ, (4.75)
where U = [ u 1 , . . . , u l ]. From y T i u i = δ ij , we can also write that

U T HBH T M U = I l .
These relations with the symmetry property of HBH T M and HBH T A imply that

( X T HBH T A X) -1 = Λ -1 X( X T HBH T A X) -1 X T HBH T A M = X X T HBH T A X = U Λ (4.76)
Substituting these relations into (4.26) yields that

D = ( M -X X T HBH T )(I m -A XΛ -1 X T HBH T ) + XΛ -1 X T HBH T = M -M A XΛ -1 X T HBH T -X X T HBH T + X X T HBH T A XΛ -1 X T HBH T + XΛ -1 X T HBH T = M -X X T HBH T -X X T HBH T + X X T HBH T + XΛ -1 X T HBH T = M + X(Λ -1 -I l ) X T HBH T
which gives the desired result. 2

We now show that the preconditioner D k-1 constructed from the formula (4.74) for the k-th linear system in Algorithm 4.4 satisfy the relation P k-1 H T = BH T D k-1 where P k-1 is the spectral LMP constructed from (3.78) for the k-th linear system in Algorithm 3.2.

We suppose that P 0 H T = BH T D 0 where P 0 is the initial preconditioner for Algorithm 3.2 and D 0 is the initial preconditioner for Algorithm 4.4. Assume that for the (k -1)-th linear system in Algorithm 3.2, the preconditioner P k-2 is generated from the formula (3.78) where (λ i , x i ) = (λ i , P k-2 u i ), i = 1, . . . , l are eigenpairs of the matrix P k-2 A, i.e.

P k-2 Ax i = x i λ i . ( 4
H T = BH T D k-1 is satisfied.
As a result it is possible to find the corresponding spectral LMP (LMP (4.74)) for Algorithm 4.3 such that when it is used within Algorithm 4.4, it generates a mathematically equivalent sequence of iterates {s k } to those of Algorithm 3.2 with the spectral LMP (3.78).

We summarize the characteristics of the equivalent primal and dual algorithms when used with the spectral LMP by Table 4

P 0 H T = BH T D 0 x 1 = x c
Table 4.4: A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the spectral LMPs. Approach (A) and (DualD) are the approaches defined in Section 3.1 and 4.3.1.1 respectively.

The cost for applying the spectral preconditioner given by (4.74) is (2m + 1) flops and one matrix-vector product with M .

Solving the subproblem with Augmented RPCG

In this section, we present the final version of the RPCG algorithm that allows for an arbitrary initial guess. This generalized algorithm is proposed by (Gratton and Tshimanga, 2009) and relies on augmentation. We name this algorithm Augmented RPCG.

By using augmentation the linear system (3.1) is transformed into an equivalent linear system, i.e.

(B -1

+ H T k R -1 H k )s = B -1 s 0 + H T k d k , (4.80)
having an initial residual in the range of an augmented matrix H T k ∈ IR (m+1)×n , i.e.

r 0 = H T k (d k -R -1 k H k s 0 ), (4.81)
(satisfying the assumption on the initial residual). For instance, H k , R -1 ∈ IR (m+1)×n and the vector d k ∈ IR m+1 are defined as

H k = H k (x c -x k -s 0 ) T B -1 , R -1 k = R -1 0 , d k = -R -1 d k 1 .
As a result applying PCG Algorithm 2.5 on the augmented linear system generates the same iterates as those of PCG applied to the linear system (3.1). Therefore, an equivalent algorithm (Algorithm 4.9) to PCG on the augmented system consisting the recurrence relations in IR m+1 can be obtained (Gratton and Tshimanga, 2009) under the following assumption on the preconditioner:

Assumption 4.3 There exists a preconditioner D ∈ IR (m+1)×(m+1) for a given preconditioner P used during PCG satisfying

P H T k = BH T k D.
Note that this assumption is an adaptation of Assumption 4.2 when augmentation is introduced.

Algorithm 4.9: Augmented RPCG Algorithm

1 Choose an initial estimate s 0 2 λ 0 = 0 3 r 0 = d -R -1 Hs 0 4 z 0 = D r 0 5 p 0 = z 0 6 w 0 = HBH T z 0 7 ρ 0 = r T 0 w 0 8 t 0 = w 0 9 for i = 0, 1, ..., l -1 do 10 q i = R -1 t i + p i 11 α i = ρ i /q T i t i 12 λ i+1 = λ i + α i p i 13 r i+1 = r i -α i q i 14 z i+1 = D r i+1 15 w i+1 = HBH T z i+1 16
Check convergence and stop if desired accuracy is reached

17 ρ i+1 = r T i+1 w i+1 18 β i = ρ i+1 /ρ i 19 p i+1 = z i+1 + β i p i 20 t i+1 = w i+1 + β i t i 21 end
Defining the vectors a and a scalar σ (Gratton and Tshimanga, 2009)

a = H k (x c -x k -s 0 ) σ = (x c -x k -s 0 ) T B -1 (x c -x k -s 0 )
the matrix H k BH T k can be written as

H k BH T k = H k (x c -x k -s 0 ) T B -1 B H T k , B -1 (x c -x k -s 0 ) = H k BH T k a a T σ
Substituting these definitions into Algorithm 4.9, the algorithm can be decomposed in Algorithm 4.10 (Gratton and Tshimanga, 2009, Algorithm 8) which avoids explicit reference to the augmented matrices.

Algorithm 4.10: Augmented RPCG Algorithm

1 Choose an initial estimate s 0 2 λ 0 = 0 3 Compute the vector a 4 Compute the scalar σ 5 r 0 (1 : m) = d -R -1 Hs 0 6 r 0 (m + 1) = 1 7 z 0 = D r 0 8 p 0 = z 0 9 w 0 (1 : m) = HBH T z 0 (1 : m) + z 0 (m + 1)a 10 w 0 (m + 1) = a T z 0 (1 : m) + σz 0 (m + 1) 11 ρ 0 = r T 0 w 0 12 t 0 = w 0 13 for i = 0, 1, ..., l -1 do

14 q i (1 : m) = R -1 t i (1 : m) + p i (1 : m) 15 q i (m + 1) = p i (m + 1) 16 α i = ρ i /q T i t i 17 λ i+1 = λ i + α i p i 18 r i+1 = r i -α i q i 19 z i+1 = D r i+1 20 w i+1 (1 : m) = HBH T z i+1 (1 : m) + z i+1 (m + 1)a 21 w i+1 (m + 1) = a T z i+1 (1 : m) + σz i+1 (m + 1) 22
Check convergence and stop if desired accuracy is reached

23 ρ i+1 = r T i+1 w i+1 24 β i = ρ i+1 /ρ i 25 p i+1 = z i+1 + β i p i 26 t i+1 = w i+1 + β i t i 27 end 28 The solution is recovered from s l = s 0 + BH T λ l (1 : m) + λ l (m + 1)(x c -x k -s 0 )
In Algorithm 4.10, the notation is adapted similar to that of MATLAB, for instance r 0 (1 : m) refers to the first m entries of the vector r 0 ∈ IR m+1 and r 0 (m + 1) refers to the last (augmented) entry of the vector.

The comparison on required operations during the inner loops of RPCG Algorithm 4.3 and Augmented RPCG Algorithm 4.10 is given by Table 4

Preconditioning Augmented RPCG with LMPs

In this section we are interested in preconditioning the linear system at the k-th outer loop of Algorithm 4.11 with a preconditioner inherited from the previous outer loop. So the question is whether we may derive a preconditioner D ∈ IR (m+1)×(m+1) from by-products of Algorithm 4.10 satisfying Assumption 4.3. We want to find a formula for the LMP so that Augmented RPCG would behave the same way as non augmented RPCG whenever augmentation is not needed, i.e. there exists a vector c k ∈ IR m such that B -1 (x cx ks 0 ) = H T c k . The following lemma provides such a preconditioner. Lemma 4.7 Suppose that H k is defined as

H k = H (x c -x k -s 0 ) T B -1
where H ∈ IR m×n has full row rank. Suppose also that there exists a vector c k ∈ IR m such that B -1 (x cx ks 0 ) = H T c k . Then the matrix D defined by

D = D Dc k -c k 0 1 (4.82)
where the matrix HBH T D is symmetric and positive definite satisfies HBH T D = D T HBH T .

If P H T = BH T D holds for a given matrix P ∈ IR n×n , then the matrix D defined by (4.82) satisfies P H T = BH T D.

Proof. Replacing the relation B -1 (x cx ks 0 ) = H T c k in H and multiplying D on the left gives that

HBH T D = HBH T D HBH T Dc k c T k HBH T D c T k HBH T Dc k .
Using the symmetry of D in HBH T , we deduce that

HBH T D = D T HBH T D T HBH T c k c T k D T HBH T c T k D T HBH T c k = D T HBH T .
The second part of the proof is as follows. Using the assumption P H T = BH T D given in the lemma, we obtain that

P H T = P H T P H T c k = BH T D BH T Dc k = BH T D,
which completes the proof.

2

After defining a preconditioner in IR m+1 by Lemma 4.7, we are left with finding such a vector c k and a matrix D given in Lemma 4.7 from by-products of Algorithm 4.10. Let us start with explaining how to obtain the vector c k .

We showed in Subsection 3.1.1 that (x kx c ) can be written as

x k -x c = BH T k-1 j=1 s j (4.83)
under the assumption that x 1 = x c where s j is the solution of Algorithm 4.4 at the j-th outer loop. Therefore taking s 0 = 0, and choosing the vector c k as

c k = - k-1 j=1 s j (4.84)
satisfies the assumption in Lemma 4.7. We showed in Section 4.3 that it is possible to derive a preconditioner D from RPCG Algorithm 4.3 such that HBH T D is symmetric and positive definite. It is also shown that this preconditioner also satisfies the relation P H T = BH T D for a given P constructed from PCG Algorithm. Therefore, it is possible to derive a preconditioner D from the pairs obtained during RPCG Algorithm 4.3 that satisfies the relation P H T = BH T D. The question is now how to obtain these m-dimensional pairs from Augmented RPCG. The answer is as follows.

When solving the k-th linear systems within Algorithms 3.2 and 4.4 the relations between the m-dimensional vectors of RPCG Algorithm 4.3 and n-dimensional vectors of PCG Algorithm 2.5 applied to the linear system (3.1) with zero initial guess are given as

r i = H T r i , z i = BH T z i p i = BH T p i q i = H T q i s i = BH T s i                (4.85)
with i ≥ 0 under the assumption that P H T = BH T D.

The relation between the (m + 1)-dimensional vectors of Augmented RPCG Algorithm 4.9 and those of PCG Algorithm can be similarly given as

r i = H T k r i z i = BH T k z i p i = BH T k p i q i = H T k q i s i = BH T k λ i                (4.86)
with i ≥ 0 under the assumption that P H T k = BH T k D. From the definition of c k and the relation (4.83), H T k can be written as

H T k = H T H T c k .
Substituting this definition into the relations given in (4.86) and using the equality of right hand sides of (4.85) and (4.86), m-dimensional vectors obtained during RPCG Algorithm can be generated from the (m + 1)-dimensional vectors obtained during Augmented RPCG by the following relations:

r i = r i (1 : m) + c k r i (m + 1) z i = z i (1 : m) + c k z i (m + 1) p i = p i (1 : m) + c k p i (m + 1) q i = q i (1 : m) + c k q i (m + 1) s i = λ i (1 : m) + c k λ i (m + 1)                (4.87)
The vector t i = H k BH T k p i defined in Augmented RPCG Algorithm 4.10 can be written also as

t i = H k p i = Hp i c T k Hp i = HBH T p i c T k Hp i = t i c T k Hp i
where t i is defined in RPCG Algorithm 4.3. Therefore the vector t i ∈ IR m can be obtained from the vector t i ∈ IR m+1 by using the relation

t i = t i (1 : m).
Similarly, the vector w i = H k BH T k z i defined in Augmented RPCG Algorithm 4.10 can be written as

w i = H k z i = Hz i c T k Hz i = HBH T z i c T k Hz i = w i c T k Hz i
where w i is defined in RPCG Algorithm 4.3. The vector w i ∈ IR m can be obtained from the vector w i ∈ IR m+1 by using the relation

w i = w i (1 : m).
Therefore, these relations allow to construct the limited memory preconditioner D ∈ IR m×m at the (k + 1)-th outer loop of Algorithm 4.11 (with the same H k along the outer loops) from the recurrence relations of Augmented RPCG that satisfies the relation P H T = BH T D. Then using the preconditioner D, a preconditioner D ∈ IR (m+1)×(m+1) for the (k + 1)-th outer loop of Algorithm 4.11 can be generated from (4.82) which satisfies the relation P H T k+1 = BH T k+1 D. In case of using quasi-Newton LMP Augmented RPCG, Algorithm 4.9 can be rewritten in a more computationally effective way like RPCG that does not require to perform additional matrix vector products to calculate HBH T q. Introducing a vector

l i = HBH T r i (4.88)
and using the symmetry of D in HBH T from Lemma (4.7), Algorithm 4.9 can be transformed into Algorithm 4.12 by using the similar relations used to transform Algorithm 4.3 to Algorithm 4.5. While constructing the quasi-Newton LMP, we need to save the vectors i = HBH T q i defined in Algorithm 4.5. These vectors can be

q i = R -1 t i + p i 12 α i = ρ i /q T i t i 13 λ i+1 = λ i + α i p i 14 r i+1 = r i -α i q i 15 l i+1 = HBH T r i+1 16 i = (l i -l i+1 )/α i 17 z i+1 = D r i+1 18 w i+1 = D T l i+1 19
Check convergence and stop if desired accuracy is reached

20 ρ i+1 = r T i+1 w i+1 21 β i = ρ i+1 /ρ i 22 p i+1 = z i+1 + β i p i 23 t i+1 = w i+1 + β i t i 24 end 25
The solution is recovered from s l = s 0 + BH T λ l

Preconditioning varying systems

In this section we are interested in solving a sequence of slowly varying linear systems (4.14) in the form of

A k λ = b k (4.89)
with conjugate gradients equipped with an LMP. Our aim is not only solving this linear system, but also to preserve the desired convergence properties of the primal approach. For this reason, for varying linear systems Algorithm 4.10 has to be used since this algorithm accomodates the initial residual in the proper space which is a requirement to ensure monotonic decrease on the cost function (2.13). In case of preconditioning another requirement is to find an LMP D that satisfies Assumption 4.3 (the augmented version of Assumption 4.2). When A k is constant (the case of H k is constant) along the outer loops, an LMP D that satisfies P H T k = BH T k D is given by Lemma 4.7. Now, we are interested in using this preconditioner D when H k slowly varies along the outer loops. In this case depending on the change along the outer loops the preconditioner D may loose its symmetry according the inner product H k BH T k since D may loose its symmetry in H k BH T k . This may happen since D is inherited from the preceding outer loop (k -1) and used in the current outer loop k. This symmetry problem can be handled by regenerating the required pairs for the LMP by using the current inner product. For instance, for a quasi-Newton preconditioner, the preconditioner D can be derived from the formula (4.51) as

D j = I m -τ j p j q T j H k BH T k D j-1 I m -τ j q j t T j + τ j p j t T j , (4.90)
where p i ∈ IR m , i = 0, 1, ..., j are the search directions generated from the relation given in (4.87) at the (k -1)-th outer loop of Algorithm 4.11,

q i = (I m + R -1 H k BH T k ) p i , t i = H k BH T k p i and τ i = 1/( q T i t i ).
Therefore replacing H k-1 BH T k-1 with H k BH T k in the formula makes the preconditioner D(= D j ) be symmetric with respect to the current inner product H k BH T k . Generating this preconditioner requires 2j matrix-vector products with H k BH T k which may be computationally expensive if the acceleration attained with the preconditioner is not strong enough.

In this situation a measure on the symmetry of the preconditioner D can be defined and used as a criterion to precondition the next linear system with D or not. A possible necessary condition for symmetry could be η k < , where

η k = H k BH T k D x -D T H k BH T k x H k BH T k D x , (4.91)
x ∈ IR m+1 is a random vector, and is a user defined threshold. This measure requires two more H k BH T k matrix inner product.

Convergence issues

In this section we present the convergence properties of the variants of RPCG. First the re-orthogonalization will be introduced which is an important issue in the convergence of the conjugate-gradient-like methods in finite arithmetic. Afterwards the convergence theory for RPCG will be provided and the section will be concluded by providing inexpensive formulas for computing the necessary information on each iteration from the recurrence relations of RPCG algorithms for monitoring the convergence of the minimization.

Re-orthogonalization

It is known that round-off errors typically cause loss of orthogonality between successive residuals [START_REF] Meurant | The Lanczos and conjugate-gradient algorithms[END_REF], a central property ensuring fast convergence of the conjugate-gradient methods in exact arithmetic. As a result the rate of convergence might be substantially deteriorated. A possible cure for this problem is to consider reorthogonalization of the residuals, either explicitly [START_REF] Roux | Acceleration of the outer conjugate gradient by reorthogonalization for a domain decomposition method for structural analysis problems[END_REF] or in the form of the mathematically equivalent Full-Orthogonalization-Method (FOM) [START_REF] Van Der Vorst | Iterative krylov methods for large linear systems[END_REF]. We present the reorthogonalization of the residuals using a modified Gram-Schmidt (MGS) procedure (Saad, 1996, p. 11-12) in this subsection. With reference to Algorithm 2.5 for PCG, the re-orthogonalization procedure acts on the vectors r i . Making use of the orthogonality relationship r T i P r j = 0, for i = j, MGS re-orthogonalization can be described in compact notation by

r i ←   1 j=i-1 I n - r j r T j P r T j P r j   r i , (4.92) =   1 j=i-1 I n - r j z T j r T j z j   r i . (4.93)
Equations (4.92) and (4.93) require the storage of the residuals r j from all previous iterations j = 1, . . . , i -1. Equation (4.92) can be less practical than Equation (4.93) when matrix vector products with P is expensive. In this case, the vectors z j can also be stored during PCG and used in Equation (4.93).

The need to store and manipulate the sequence of n-dimensional vectors r j and z j can lead to significant computational overhead. In this respect, the dual algorithms, which involve sequences of m (or (m+ 1))-dimensional vectors, are clearly an attractive alternative when m n. From the relationship r i = H T r i and P H T = BH T D the orthogonality condition for the residuals can be written as (Gratton and Tshimanga, 2009) r

T i P r j = r T i HP H T r j = r T i HBH T D r j = r T i w i (4.94)
for i = j. Combining Eq. (4.94) with Eqs (4.92) and (4.93) leads to the MGS reorthogonalization scheme for RPCG as

r i ←   1 j=i-1 I m - r j r T j D T HBH T r T j HBH T D r j   r i , =   1 j=i-1 I m - r j w T j r T j w j   r i .
Note that, the corresponding reorthogonalization scheme for Augmented RPCG follows the similar derivation.

It is remarkable that re-orthogonalization of residuals may be considered as costly in terms of memory and computational cost in the primal setting, turns out to be much cheaper in the dual frameworks since the dimension of the residual vectors in dual space is (typically much) smaller than the dimension of the residual vectors in primal space.

Our observation is that reorthogonalization in the dual space has a good effect on the convergence in the presence of round-off errors. Comparing this reorthogonalization with what is done in the primal space remains an open question.

Convergence theory

In this subsection we consider bounds on the efficiency of the dual-space preconditioners and compare them to those that can be derived for its primal-space equivalents. For this purpose, we start by recalling known properties of the preconditioned conjugategradient method. From the construction of the PCG method, the iteration error e k can be related to the initial error e 0 = s *s 0 by (Axelsson, 1996, p. 560) e k = (P l (P A)P A + I n )e 0 where s * is the solution of the linear system (3.1) in IR n , A = B -1 + H T R -1 H is a symmetric positive definite matrix, P is a symmetric positive definite preconditioner, and P l is a polynomial defined by P l (P A) = a 0 I + a 1 P A + ... + a l (P A) l .

Taking s 0 = 0, this method implicitly computes the coefficients of the polynomial P * l (P A) that solves the minimization problem (Axelsson, 1996, p. 560) min

P l (P l (P A)P A + I n )s * 2 A .
(4.95)

If P A has eigenvalues µ 1 ≤ µ 2 ≤ ... ≤ µ n , the PCG algorithm (Golub and Van Loan, 1989, p. 534) with zero initial starting vector ensures the inequality

s l+1 -s * A ≤ 2 √ µ n - √ µ 1 √ µ n + √ µ 1 l s * A (4.96)
(see (Conn, Gould and Toint, 2000, p. 89), for instance). Note that, when a dual approach is used, the iterates all belong to the affine subspace IM(BH T ) for RPCG or IM(BH T ) for Augmented RPCG. This information can be taken into account to obtain a better bound on the convergence rate. We first find this bound for RPCG Algorithm 4.3 shown in the next lemma.

Lemma 4.8 Suppose that HBH T is non-singular, and that D is a preconditioner satisfying Assumption 4.2. If D(I m + R -1 HBH T ) has eigenvalues ν 1 ≤ ν 2 ≤ ... ≤ ν m , then the RPCG Algorithm 4.3 and its primal equivalent with zero initial starting vector ensure the inequality

s l+1 -s * A ≤ 2 √ ν m - √ ν 1 √ ν m + √ ν 1 l s * A ≤ 2 √ µ n - √ µ 1 √ µ n + √ µ 1 l s * A .
(4.97)

Proof. From (4.18) the solution of the linear system (3.1) can be written as s * = BH T s * , where s * is the solution of the linear system (4.22). Substituting this form for the solution in the objective function of (4.95) then yields the new form

l i=0 a i (P A) i+1 BH T + BH T s * 2 A
and this objective is minimized over all choices of the coefficients {a i } l i=0 . Using the fact that (B -1 + H T R -1 H)BH T = H T (I m + R -1 HBH T ), which we can simply write as ABH T = H T A, we obtain that our objective may now be written as

l i=0 a i (P A) i P H T A + BH T s * 2 A
Using Assumption 4.2, we obtain the further form

l i=0 a i (P A) i BH T D A + BH T s * 2 A .
Substituting the term ABH T with H T A and using Assumption 4.2 then yields an objective of the form

BH T l i=0 a i (D A) i+1 + I m s * 2 A = BH T (P l (D A)D A + I m ) s * 2 A = (P l (D A)D A + I m ) s * 2 HBABH T = (P l (D A)D A + I m ) s * 2 HBH T A (4.98)
Performing the change of variables A = HBH T A and P = D(HBH T ) -1 in (4.98), we may write the minimization problem in dual space as:

min P l (P l ( P A) P A + I m ) s * 2 A (4.99)
Using Assumption 4.2, we then write (HBH T ) -1 HP H T (HBH T ) -1 = D(HBH T ) -1 which shows that the matrix P is symmetric positive definite. On the other hand, A = HBH T A = HBH T + HBH T R -1 HBH T is also a symmetric positive definite matrix. Therefore, from (4.95) and (4.96), if P A has eigenvalues ν 1 ≤ ν 2 ≤ ... ≤ ν m , the RPCG Algorithm 4.3 ensures the inequality

s l+1 -s * A ≤ 2 √ ν m - √ ν 1 √ ν m + √ ν 1 l s * A . (4.100)
One also has that

s * HBH T A = s * HBABH T = BH T s * A = s * A . (4.101) s l+1 -s * HBH T A = s l+1 -s * HBABH T = BH T ( s l+1 -s * ) A = s l+1 -s * A . (4.102)
Finally, substituting A with HBH T A and P with D(HBH T ) -1 in P A, and using the relations (4.101) and (4.102) in (4.100) proves the first part of the inequality (4.97).

For the second part of the inequality, we start from the equality ABH T = H T A. If we multiply both sides of this equality from the left by P , we obtain P ABH T = P H T A and using Assumption 4.2, we deduce that (P A)BH T = BH T (D A). This equality tells us that BH T spans an invariant subspace of P A, from which we may deduce that every eigenvalue of D A is an eigenvalue of P A. So, µ 1 ≤ ν 1 and µ n ν m . From these two inequalities we obtain that

√ µ 1 √ µ n ≤ √ ν 1 √ ν m .
Using the fact that the function 1-x 1+x is decreasing for x 0, we then deduce the desired result and complete the proof.

2

This lemma shows that the condition number of P A is generally worse than that of D A and we now show that it can be arbitrarily worse. For example, taking m = n -2, B as the identity matrix, R is a diagonal matrix, H = [I 0], P = [D 0; 0 diag(ξ, 1)] where diag(ξ, 1) is the 2 × 2 diagonal matrix with the diagonal entries ξ and 1 and D = (I m +R -1 HH T ) -1 we easily verify that Assumption 4.2 holds. Then the diagonal preconditioned system matrices are

P A = diag(d i ) where d i = 1 for 1 ≤ i ≤ m and i = n, d i = ξ for i = n -1. D A = diag(d i ) where d i = 1 for 1 ≤ i ≤ m.
We conclude that P A is ill-conditioned with a condition number of 1/ξ whereas D A has a condition number of 1. Therefore, for a given preconditioner D in dual space, we can find a preconditioner P in primal space satisfying Assumption 4.2 that is arbitrarily ill-conditioned, showing the relevance of the bound (4.97) in terms of the ν's. Note that, in this case the iterates of the dual and primal algorithms are the same and convergence takes place in one iteration both in dual and primal spaces.

Note that we assumed HBH T to be non-singular for Lemma 4.8, an assumption which has been used to solve the minimization problem (4.99) on the space of polyno-mials. Lemma 4.8 can nevertheless be generalized to the case where HBH T is singular, which results in the following improvement on Lemma 4.8. Lemma 4.9 Suppose that HBH T is singular, and that H is a preconditioner satisfying Assumption 4.2. Then, it is possible to find a r × n matrix of rank r Ȟ and a r × r matrix of rank r Ř-1 satisfying

B -1 + H T R -1 H = B -1 + ȞT Ř-1 Ȟ (4.103)
where r < m, and a r × r matrix Ď satisfying

P ȞT = B ȞT Ď. (4.104)
Moreover, the RPCG Algorithm 4.3 and its primal equivalent ensure the inequality

s l+1 -s * A ≤ 2 √ ν r - √ ν 1 √ ν r + √ ν 1 l s * A ≤ 2 √ µ n - √ µ 1 √ µ n + √ µ 1 l s * A , (4.105)
where ν 1 ≤ ν 2 ≤ ... ≤ ν r are the eigenvalues of the full rank matrix Ď(I r + Ř-1 ȞB ȞT ).

Proof. If the singular value decomposition for H is given by

H = [U 1 U 2 ] Σ r 0 0 0 V T 1 V T 2 , (4.106)
a possible theoretical choice for Ȟ could be

Ȟ = Σ r V T 1 . Denoting Ř = U T 1 R -1 U 1 -1 , direct computations show that B -1 + H T R -1 H = B -1 + ȞT Ř-1 Ȟ.
Using Assumption 4.2 and denoting Ď = U T 1 DU 1 , we also obtain that P ȞT = B ȞT Ď. The matrix Ȟ is now a r × n matrix of rank r and Lemma 4.8 can then be applied using r, Ř, Ȟ and Ď instead of m, R, H and D, yielding the desired result where ν 1 , ..., ν r the eigenvalues of Ď(I r + Ř-1 ȞB ȞT ), replace those of D(I m + R -1 HBH T ) in (4.97). We next investigate the relations between these two sets of eigenvalues.

Using the relation on Ȟ, Ř and Ď, it can be shown that

[Z A]U 1 = U 1 [ Ď Ǎ]
where Z = U 1 U T 1 DU 1 U T 1 and A = I m + R -1 HBH T . This says that U 1 is an invariant subspace of Z A and every eigenvalue of Ď Ǎ is an eigenvalue of Z A. Therefore, the nonzero eigenvalues of Z A are equal to the eigenvalues of Ď Ǎ using the fact that Z A has (mr) null eigenvalues. We now consider the relations between the eigenvalues of Z A and D A, and start by rewriting these matrices blockwise.

Using the relation (4.106), it can be shown that

HBH T = U 1 Σ r V T 1 BV 1 Σ r U T 1 .
Defining

U 1 = U r 0 , (4.107)
HBH T can thus be rewritten in a block matrix form as

HBH T = U r Σ r V T 1 0 B V 1 Σ r U T r 0 = M r 0 0 0 , (4.108)
where M r = U r Σ r V T 1 BV 1 Σ r U T r has full rank r. Using the equality P H T = BH T D, we can write that HP H T = HBH T D.

Hence, HBH T D is symmetric due to the symmetry of HP H T . Using the relation (4.108) and defining

D = D r D 2 D 3 D m-r ,
where D r is r ×r matrix and D m-r is a (m-r)×(m-r) matrix, we can write HBH T D as

HBH T D = M r D r M r D 2 0 0 . (4.109)
From the symmetry of HBH T D given by (4.109), M r D 2 = 0 which implies that D 2 = 0 since M r is a full rank matrix. Thus, D has the form

D = D r 0 D 3 D m-r . (4.110)
We next derive a block matrix form of A. Defining R -1 as

R -1 = R -1 r R -1 2 R -1 3 R -1 m-r
, and using (4.108) and the definition of A, we can write A as

A = I + R -1 r R -1 2 R -1 3 R -1 m-r M r 0 0 0 = A r 0 R -1 3 M r I , ( 4

.111)

where A r = I + R -1 r M r . From (4.107), (4.110) and (4.111), we deduce that

D A = D r A r 0 D 3 A r + D m-r R -1 3 M r D m-r and Z A = D r A r 0 0 0 . (4.112)
From (4.112), the eigenvalues of D A are the eigenvalues of D r A r and the eigenvalues of D m-r . Also, the eigenvalues of Z A are the eigenvalues of D r A r and (mr) null eigenvalues. Therefore, the nonzero eigenvalues of Z A which are equal to the eigenvalues of Ď Ǎ form a subset of the eigenvalues of D A. As a result, the eigenvalues of Ď(I r + Ř-1 ȞB ȞT ) can be used in (4.97) instead of those D(I m + R -1 HBH T ), which completes the proof. 2

Monitoring convergence

Inexpensive formulae for computing the quadratic cost function and the norm of the cost function gradient on each iteration from the recurrence relations of RPCG and Augmented RPCG (Algorithms 4.3 and 4.10) are derived in this section. Note that for the sake of simplicity, the initial guess is chosen as s 0 = 0 for all derivations.

Quadratic cost function

For RPCG, the quadratic cost function is calculated from Eq. (3.87) using the relations s i = BH T s i and r 0 = H T r 0 :

Q[s i ] = Q[0] - 1 2 s T i HBH T r 0
Using the symmetry of D in HBH T in RPCG Algorithm 4.3 the vector w 0 can be calculated also from w 0 = D T HBH T r 0 . Therefore defining l 0 = HBH T r 0 the inexpensive formula for the quadratic cost function can be given as

Q[s i ] = Q[0] - 1 2 s T i l 0 .
Using the realtion s i = BH T s i , the Q b term can be written as

Q b [s i ] = 1 2 s T i f i + 1 2 (x k -x c ) T B -1 (x k -x c )
where f i = HBH T s i can be computed without the need to apply HBH T by including an additional recurrence relation in Algorithm 4.3 as follows:

f i = 0 if i = 0 f i-1 + α i-1 t i-1 if i > 0 Taking x 1 = x c the term B -1 (x k -x c
) can also be calculated without the need to apply B -1 from Eq. (4.19). The Q o term can then be computed from Eq. (3.89).

For Augmented RPCG, the formula can be obtained similarly following the same derivation by using augmented matrices and vectors. While replacing the term B -1 (x kx c ), since H k varies along the outer loop, the following equation has to be used instead of the Eq. ( 4.19)

B -1 (x k -x c ) = k-1 j=0 H T j λ j .

Norm of the quadratic cost function gradient

For RPCG, the norm is obtained from

∇Q[s i ] P = H T r i P = r i HBH T D = r T i w i
where the vectors r i and w i are defined in Algorithm 4.3. Similarly for Augmented RPCG the norm is obtained from

∇Q[s i ] P = r T i w i
where the vectors r i and w i are defined in Algorithm 4.9.

CHAPTER 5

Solving the subproblem with a Lanczos method

This chapter considers applying a preconditioned Lanczos method to a sequence of symmetric and positive-definite linear systems of the form of A k s = b k , where A k and b k are defined in (2.16), and where preconditioning is achieved by using the Limited Memory Preconditioners (LMPs) described in Chapter 3.

Like Chapter 3, we first focus on warm-start preconditioning of a sequence of linear systems with multiple right-hand sides, As = b k by using directions generated by the Lanczos algorithm applied on the previous linear system. Note that, each linear system is part of a sequence, which is supposed to produce a convergent sequence of solutions. We discuss three different ways of preconditioning the linear system As = b k with a warm-start preconditioner. These approaches can be considered as the Lanczos version of the variants of the CG algorithm described in Section 3.1.

We later explain how to extract the information from Lanczos-type algorithms to construct particular LMPs such as the quasi-Newton LMP, Ritz LMP and spectral LMP. The adaptation of these preconditioners to the case of solving a sequence of slowly varying linear systems in a Gauss-Newton process follows the same results as explained in Section 3.1.4.

This chapter then introduces a Lanczos-type method in dual space. This dual method produces, in exact arithmetic, the same iterates as those produced by a standard Lanczos method applied to the linear system (2.16) and RPCG. Like RPCG, this algorithm might yield gains in terms of both memory usage and computational cost compared to primal algorithms when m n. We then explain how to use practical warm-start LMPs described in Chapter 4 to accelerate the convergence of the dual Lanczos algorithm when solving a sequence of linear systems with multiple right hand sides. The adaptation of these preconditioners when used for a sequence of slowly varying linear systems is similar to what is explained in Section 4.4.2.

We conclude the chapter by providing inexpensive formulae to compute the quadratic cost function (2.13) and the norm of its gradient, which are useful quantities to monitor convergence.

Preconditioning the Lanczos method with LMPs

In this section we are dealing with applying the Lanczos method to the systems of equations (3.1) with multiple right hand sides of the form of

As = b k ,
with a warm-start LMP technique. Each linear system is part of a sequence and is not isolated. It is assumed that this sequence produces a convergent sequence of solutions.

We explain different approaches on preconditioning the linear system (3.1) with a symmetric and positive definite LMP defined by (2.38). This chapter follows the same structure as Section 3.1 and the algorithms introduced in this section are the Lanczos version of the CG algorithms presented in Section 3.1.

We start with the approach that uses a split preconditioning with the Lanczos method (Algorithm 2.3) as explained below.

Preconditioning with Lanczos Algorithm 2.3

This approach can be considered as the Lanczos version of Approach (A) presented in Section 3.1. It considers solving the linear system (3.1) with a LMP P formulated by (2.38) by using Lanczos Algorithm 2.3 on the preconditioned linear system

F T AF s = F T b k , (5.1)
where F is the factored form the LMP P , i.e. P = F F T , and formulated by equation (3.2). The approximate solution s is obtained from the relation s = F s. We name this approach Approach (E).

The same strategy as with solving a convergent sequence of linear systems (3.1) by Approach (A) can be used to solve these linear systems with Approach (E). Therefore, an algorithm similar to Algorithm 3.1 can be used to solve a sequence of linear systems (3.1) with Lanczos Algorithm 2.3, by performing the inner minimization of Algorithm 3.1 with the Lanczos algorithm instead of the CG algorithm.

The characteristics of the Lanczos version of Algorithm 3.1 summarized in Section 3.1 (for instance, the need for the factored form of the LMP, matrix-vector products in particular case P 0 = B) remain the same as those of Algorithm 3.5. For further details we refer to Section 3.1.

We now present an alternative algorithm for preconditioning a sequence of linear systems (3.1) with Lanczos Algorithm 2.6. approach, is the same as with Approaches (A), (B) and (C) given in Table 3.2.

As a result, Algorithms 3.1, 3.2 and 3.5 when used with Lanczos type methods whose characteristics are listed in Table 5.1 generate a mathematically equivalent sequence of iterates {s k }. The choice of the algorithm depends on the availability of the required form of the preconditioner, operators and their matrix-vector product cost.

We next explain how to construct particular LMPs: the quasi-Newton LMP, the Ritz LMP and the spectral LMP described in Chapter 3 in the context of the given Lanczos-type algorithms. This part will include for each preconditioner an extraction methodology of the required information from the Lanczos algorithm.

Preconditioning with the quasi-Newton LMP

We consider in this section the use of the quasi-Newton LMPs described in Section 3.1.1 when solving a convergent sequence of linear systems 3.1 with a Lanczos-type method. We start with explaining how to construct a quasi-Newton LMP within Algorithm 3.2 in which the linear system in sequence is solved by Approach (E).

Approach (E) with the quasi-Newton LMP

Let us first consider to apply Lanczos Algorithm 2.3 on the linear system (3.7) of the form of A k s = b k within Algorithm 3.1. Remember that while applying the quasi-Newton LMP Y j+1 formulated by (3.41) during Approach (A) the square-root factorization of Y j+1 is required. This factored form Y j+1 = F j+1 F T j+1 is calculated from the formula (3.42) given as

F j+1 = I n -p j τ j q T j + τ j r T j r j 2 F j
where F 0 = I n , p i , i = 0, . . . , j are the search directions, τ j = 1/( q T j p j ) and q j = A p j obtained during the CG run when solving the (k -1)-th linear system.

The pairs required in the formula (3.42) are not by-products of Lanczos Algorithm 2.3. Can we construct the pairs required in this formula from the recurrence relations of Lanczos Algorithm 2.3? The answer is provided by Algorithm 2.4 in which the iterates of CG Algorithm 2.2 are obtained from those of Lanczos Algorithm 2.3. Therefore, when using Lanczos Algorithm 2.3 with the quasi-Newton LMP (3.42) during Algorithm 3.1, before starting the next inner loop, Algorithm 2.4 has to be run to obtain the required pairs for constructing the quasi-Newton LMP.

We now explain how to construct a quasi-Newton LMP within Algorithm 3.2 in which the linear system in sequence is solved by Approach (F).

Approach (F) with the quasi-Newton LMP

Let us now consider solving the system of linear equations (3.1) of the form of As = b k with PLanczos Algorithm 2.6 within Algorithm 3.2. In this case the iterates of PCG Algorithm 2.5 can be obtained from the iterates of PLanczos Algorithm 2.6 by using Algorithm 2.7. This allows to construct the quasi-Newton LMP for the matrix A in the k-th linear system from formula (3.37) given as P j+1 = (I nτ j p j q T j )P j (I nτ j q j p T j ) + τ j p j p T j ,

where p i , i = 1, . . . , j, q j = Ap j and τ j = 1/(q T j p j ). These pairs are extracted from the iterates of PLanczos Algorithm obtained when solving the (k -1)-th linear system by using Algorithm 2.7. Note that, the initial preconditioning matrix (which can be interpreted as the first-level preconditioner) for the formula (3.37) is chosen as the preconditioner used for the previous linear system.

We are left with the task of deriving the quasi-Newton LMP for Approach (G) that is used within Algorithm 3.5.

Approach (G) with the quasi-Newton LMP

Let us consider solving the system of linear equations (3.14) of the form of As = b k with PLanczosIF Algorithm 5.2 within Algorithm 3.5. Remember that the quasi-Newton LMP for the system matrix A is constructed from the formula (3.43) given as C j+1 = I nτ j p j q T j B C j I nτ j q j p T j + τ j p j p T j where p i , i = 1, . . . , j, p i = Bp i , q i = Ap i and τ i = 1/(q T i p i ) are obtained when solving the (k -1)-th linear system with PCGIF Algorithm 3.6. We remind that the initial matrix in (3.43) is chosen as the preconditioner used for the previous linear system.

We are left with the task of obtaining the iterates of PCGIF algorithm from those of PLanczosIF. Using the relations given in Section 2.6.3 and adapting them to Algorithms PCGIF and PLanczosIF results in Algorithm 5.3. This algorithm extracts the iterates of PCGIF algorithm from the iterates of PLanczosIF Algorithm.

As a result it is possible to find the corresponding quasi-Newton LMPs for variants of Lanczos algorithms (Approaches (E), (F) and (G)) such that when they are used within the corresponding algorithms (Algorithms 3.1, 3.2 and 3.5), all these algorithms generate mathematically equivalent iterates {s k }. We showed that such preconditioners can be constructed from vectors which can be extracted from by-products of the corresponding algorithms (Algorithms 2.3, 2.6 and 5.2).

We summarize the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with the Lanczos method in which the preconditioner is achieved by the quasi-Newton LMP in Table 5.2. We assume that, for Algorithm 3.1 F 0 is available and that, for Algorithm 3.5 B -1 P 0 is available.

The cost for applying the preconditioner and the memory requirements of the algorithms of the present section are summarized in Table 3.4. To obtain the quasi-Newton preconditioner, Lanczos type methods require additional O(n) operations when compared with conjugate-gradients type methods. Note that, the algorithms listed in Table 5 Table 5.2: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs. Approaches (E), (F) and (G) are defined in Section 5.1.

Table 3.3.

Preconditioning with the Ritz LMP

In this section we consider preconditioning the linear system (3.1) by using Ritz pairs computed during Lanczos type methods (Approaches (E), (F) and (G)). Ritz LMP formulas for each conjugate gradient type method (Approaches (A), (B) and (C)) are already provided in Section 3.1.2. The pairs needed to construct these Ritz LMPs are directly available from a Lanczos type method therefore there is no need to generate these pairs as done in conjugate gradients. We next explain shortly how to construct the Ritz LMPs for the Lanczos type methods.

Approach (E) with the Ritz LMP While using Approach (E) for the k-th linear system in sequence within Algorithm 3.1, the Ritz LMP formulated by (3.49) can be constructed from Algorithm 3.9. Note that, the required inputs to Algorithm 3.9 (the matrices V and T , the vector v +1 , and the scalar β +1 ) are directly available from Approach (E).

Approach (F) with the Ritz LMP

When applying Approach (F) on the k-th linear system in sequence within Algorithm 3.2, the Ritz LMP (3.65) can be constructed from Algorithm 3.11. The required inputs (the matrices P k-2 , T , G , the vector g +1 and the scalar β +1 ) to Algorithm 3.11 are directly available from Approach (F). The column vectors g i , i = 1, . . . , of matrix G and the vector g +1 are the normalized vectors z i , i = 1, . . . , + 1 that is generated during PLanczos Algorithm 2.6. These vectors then need to be stored to construct the Ritz-LMP by using Algorithm 3.11.

Approach (G) with the Ritz LMP

When applying Approach (G) on the k-th linear system in sequence within Algorithm 3.5, the Ritz LMP (3.73) can be constructed from Algorithm 3.12. The required inputs (the matrices C k-2 , T , G , V , the vectors g +1 and v +1 , and the scalar β +1 ) to Algorithm 3.11 are directly available from Approach (G). The column vectors g i , i = 1, . . . , of matrix G and the vector g +1 are the normalized vectors z i , i = 1, . . . , + 1 that is generated during PLanczosIF Algorithm 5.2, i.e. G = Z and g +1 = z +1 . Note that, the column vectors v i , i = 1, . . . , of matrix V and the vector v +1 in Algorithm 3.11 are the normalized vectors z i , i = 1, . . . , + 1 that is generated during PLanczosIF Algorithm 5.2, i.e. V = Z and v +1 = z +1 . As a result it is possible to find the corresponding Ritz LMPs for variants of Lanczos algorithms (Approaches (E), (F) and (G)) such that when they are used within the corresponding algorithms (Algorithms 3.1, 3.2 and 3.5), all these algorithms generate mathematically equivalent iterates {s k }. We showed that such preconditioners can be constructed from vectors which are by-products of the corresponding algorithms (Algorithms 2.3, 2.6 and 5.2).

We summarize the characteristics of Algorithms 3.1, 3.2 and 3.5 when used with the Lanczos method in which the preconditioner is achieved by the Ritz LMP in Table 5.3. We assume that, for Algorithm 3.1 F 0 is available and that, for Algorithm 3.5 B -1 P 0 is available.

Algorithm inner min.

Ritz-LMP calculation Ritz-LMP formula

3.1

Approach (E) Algorithm 3.9 F k given by (3.49)

3.2 Approach (F) Algorithm 3.11 P k given by (3.65)

3.5

Approach (G) Algorithm 3.12 C k given by (3.73) Table 5.3: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the Ritz LMPs. Approaches (E), (F) and (G) are defined in Section 5.1.

The cost for applying the preconditioner and memory requirements is summarized in Table 3.6. The algorithms listed in Table 5.3 are mathematically equivalent to the corresponding algorithms listed in Table 3.5.

Preconditioning with the spectral LMP

The spectral LMPs for each conjugate gradient algorithm are given in Section 3.1.3. The same formulas can be applied to the Lanczos type methods. We refer to Section 3.1.3 for details in deriving these formulas. In this section we listed the corresponding spectral LMP formula for each Lanczos type method in Table 5.4.

Preconditioning varying systems

The same results explained in Section 3.1.4 apply for Lanczos-type methods when solving a sequence of varying linear systems (3.1) preconditioned by an LMP.

Algorithm inner min. spectral LMP formula Assumption

3.1 Approach (E) F k given by (3.76) F 0 is available 3.2 Approach (F) P k given by (3.78)

Approach (G)

C k given by (3.80) B -1 P 0 is available Table 5.4: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the spectral LMPs. Approaches (E), (F) and (G) are defined in Section 5.1.

Solving the subproblem with Restricted Preconditioned Lanczos (RPLanczos)

We mentioned in Section 4.1 that the solution of the linearized subproblem (2.46) can be alternatively found by using the dual approach. We focused on the dual approach called RPCG (Gratton and Tshimanga, 2009) which solves the linear system (4.14),

(R -1 HBH T + I m ) λ = R -1 (H(x -x c ) -d)
with the (possibly semi-)definite HBH T -(semi) inner product in which the unsymmetric system matrix A = R -1 HBH T + I m becomes symmetric. Note that we have dropped the outer loop index k for the sake of simplicity. As discussed by Gratton and Tshimanga (2009), RPCG generates, in exact arithmetic, the same iterates as those generated by the primal approach (PCG on the linear system (2.16)).

In the same way that RPCG is the dual equivalent of PCG, there exists a Lanczos algorithm that is the dual equivalent of PLanczos. We call this algorithm Restricted PLanczos (RPLanczos). It produces identical iterates, in exact arithmetic, to RPCG, PCG and PLanczos. RPLanczos simply solves the linear system (4.14) using a Lanczos algorithm equipped with the (possibly semi-definite) HBH T -inner product. The solution is then recovered from Eq. (4.3). RPLanczos is a variant of the Restricted FOM (RSFOM) algorithm (Gratton, Toint and Tshimanga, 2009) adapted to symmetric and positive-definite matrices. The derivation of RPLanczos is as follows.

We refer the primal approach within this section as applying PLanczos Algorithm 2.6 on the linear system (2.16). Assume for now that the initial guess for primal approach is taken as s 0 = x cx. Taking this initial guess to satisfy Assumption 4.1,

r 0 = H T r 0 (5.3)
where r 0 ∈ IR n is the initial residual for primal approach and r 0 ∈ IR m is the initial residual for RPLanczos. We assume also that RPLanczos uses the preconditioner D ∈ IR m×m such that Assumption 4.2,

P H T = BH T D (5.4)
we obtain from (5.16) and (5.17) that

v i+1 = H T v i+1 , (5.18)
z i+1 = BH T z i+1 . (5.19)
Similarly, from lines 10 and 12 of Algorithm 2.6 and using the relations (5.14), (5.18) and (5.19), we have

w i+1 = (B -1 z i+1 + H T R -1 Hz i+1 ) -β i v i -α i+1 v i+1 , = (H T z i+1 + H T R -1 HBH T z i+1 ) -β i+1 H T v i -α i+1 H T v i+1 , = H T (I m + R -1 HBH T ) z i+1 -β i+1 v i -α i+1 v i+1 , = H T w i+1 , (5.20)
where

w i+1 = (I m + R -1 HBH T ) z i+1 -β i+1 v i -α i+1 v i+1 , (5.21)
which completes the proof. Equations (5.13)-(5.15) allow all the recurrence relations in PLanczos Algorithm 2.6 involving vectors of dimension n to be transformed directly into corresponding recurrence relations involving vectors of dimension m. This yields the preliminary version of RPLanczos algorithm given by Algorithm 5.4.

The cost of Algorithm 5.4 per loop can be reduced by introducing a new vector t defined by t = HBH T z, (5.22) which yields the final version of the RPLanczos (Algorithm 5.5). As with PLanczos, one loop of RPLanczos requires a single matrix-vector product with each of the operators H, H T , R -1 and B, but with the important difference that all vectors are defined in IR m instead of IR n . The two algorithms are mathematically equivalent. As discussed in Section 5.1, PLanczos searches for the solution s in the subspace s 0 + K (P A, P r 0 ) where the Krylov subspace K (P A, P r 0 ) is given by (5.2). Replacing r 0 in (5.2) with Eq. (5.3) yields K (P A, P H T r 0 ) = span P H T r 0 , . . . , (P A) l-1 P H T r 0 .

(5.23)

Using the relations ABH T = H T A, (where

A = I m + R -1 HBH T , A = B -1 + H T R -1 H)
and P H T = BH T D in (5.23) we obtain that

K (P A, P H T r 0 ) = span BH T D r 0 , . . . , BH T D A l-1 D r 0 = BH T K (D A, D r 0 ) α i = w T i HBH T z i 12 w i = q i -α i v i 13 z i+1 = D w i 14 β i+1 = z T i+1 HBH T w i 15 v i+1 = w i /β i+1 16 z i+1 = z i+1 /β i+1 17 V = [ V , v i+1 ] 18 (T i ) i,i = α i 19 if i > 1 then 20 (T i ) i-1,i = (T i ) i,i-1 = β i 21 end 22
Check convergence and stop if desired accuracy is reached

23 end 24 y = T -1 β 0 e 1 25 s = s 0 + BH T D V y
where

K (D A, D r 0 ) = span D r 0 , (D A)D r 0 , . . . , (D A) l-1 D r 0 .
From these relations it can be easily seen that RPLanczos searches for the solution s l from the subspace s 0 + K (D A, D r 0 ).

Preconditioning RPLanczos with LMPs

In this section we are interested in preconditioning RPLanczos with LMPs such that the one-to-one correspondance between primal and dual iterates are preserved. The term "primal approach" denotes here solving a sequence of linear systems (3.1) with multiple right-hand sides with PLanczos Algorithm 2.6, where the preconditioning is achieved by the LMP formulated by (2.38) (Approach (F)). Note that, the dual version of Approach (E) follows similar strategy as explained in Section 4.3.1 and Approach (G) is not necessary since RPLanczos does not require matrix-vector products with B -1 .

q i = ( z i + R -1 t i ) -β i v i-1 13 α i = w T i t i 14 w i = q i -α i v i 15 z i+1 = D w i 16 t i+1 = HBH T z i+1 17 β i+1 = t T i+1 w i 18 v i+1 = w i /β i+1 19 z i+1 = z i+1 /β i+1 20 t i+1 = t i+1 /β i+1 21 V = [ V , v i+1 ] 22 (T i ) i,i = α i 23 if i > 1 then 24 (T i ) i-1,i = (T i ) i,i-1 = β i 25 end 26
Check convergence and stop if desired accuracy is reached

27 end 28 y = T -1 β 0 e 1 29 s = s 0 + BH T D V y
In this section we focus on finding an algorithm that generates a mathematically equivalent sequence of iterates {s k } to those of Algorithm 3.2 where the inner minimization is performed by PLanczos, while working in the dual space.

In Algorithm 3.2, the initial guess is chosen as a zero vector because of the same reason explained in Section 4.3.1. In this case, RPLanczos is applied to the linear system (4.22) given as

(R -1 HBH T + I m ) s = -R -1 d k - k-1 j=1 s j .
whose initial residual r 0 with s = 0 satisfies Assumption 4.2 for the initial residual of the linear system (3.1) with s 0 = 0 (see subsection 4.3.1). This algorithm is provided by Algorithm 5.6. We name this approach Approach (DualF).

Preconditioning with the quasi-Newton LMP

In this section, we are interested in preconditioning the linear system (4.22) with the quasi-Newton LMP given by (4.51). The pairs to construct the quasi-Newton LMP are not available from a Lanczos-type algorithm, however, they can be extracted from the iterates of Lanczos algorithm as explained in subsection 5.1.1. Following the similar strategy for deriving the iterates of CG Algorithm from those of Lanczos Algorithm as explained in Section 2.6.3, we can derive an algorithm to obtain the iterates of RPCG Algorithm 4.3 from those of RPLanczos Algorithm 5.6. This algorithm is provided by Algorithm 5.7.

Algorithm 5.7: RPLanczos2RPCG

1 Given , r 0 , s 0 , T , β +1 , V from Algorithm 4.3 2 (T ) +1, = β +1 3 z 0 = D r 0 4 l 0 = HBH T r 0 5 w 0 = D T l 0 6 β 0 = r T 0 w 0 7 p 0 = z 0 8 t 0 = w 0 9 ρ = β 2 0 10 for i = 1, . . . , do 11 y i = T -1 i (β 0 e 1 ) 12 r i = -(T l ) i+1,i v i+1 e T i y i 13 l i = -(T l ) i+1,i HBH T v i+1 e T i y i 14 z i = D r i 15 w i = D T l i 16 ρ i = r T i w i 17 β i = ρ i /ρ i-1 18 p i = z i + β i p i-1 19 t i = w i + β i t i-1 20 s i = s 0 + DV i y i 21 α i = ( s i -s i-1 ) 2 / p i-1 2 22 q i = -( r i -r i-1 )/α i 23 i = -(l i -l i-1 )/α i 24 end
Algorithm 5.7 requires matrix-vector products with HBH T at each loop. By introducing a new vector l i = HBH T v i , Algorithm 5.6 can be transformed into Algorithm 5.8 in which the matrix-vector products, HBH T v i , can be obtained as byproducts of RPLanczos Algorithm. formula (4.51) given as D j+1 = I mτ j p j q T j HBH T D j I mτ j q j t T j + τ j p j t T j .

(5.24)

Note that, p i , i = 1, . . . , j, q i = A p i , t i = HBH T p i , and τ i = 1/( q T i t i ) are extracted from the iterates of RPLanczos Algorithm 5.8 obtained at the (k -1)-th outer loop by using Algorithm 5.7. We remind that the initial matrix to construct the preconditioner (4.51) at each outer loop k is chosen as the preconditioner that is generated from the previous outer loop (k -1). To obtain this preconditioner with RPLanczos requires additional O(m) operations when compared with RPCG.

As a result it is possible to find the corresponding quasi-Newton LMP (LMP (4.51)) for Algorithm 5.8 such that when it is used within the Lanczos version of Algorithm 4.4, it generates a mathematically equivalent sequence of iterates {s k } to those of the Lanczos version of Algorithm 3.2 with the quasi-Newton LMP (3.37), while possibly yielding gains in terms of both memory usage and computational cost when m n. We showed that the quasi-Newton LMP can be constructed from vectors which can be extracted from by-products of Algorithm 5.8.

We summarize the characteristics of the equivalent primal and dual Lanczos algorithms when used with the quasi-Newton LMP by Table 5.6. We assume that for Algorithm 3.2 x 1 = x c and s 0 = 0, and for Algorithm 4.4, P 0 H T = BH T D 0 and x 1 = x c . In this table we define another approach for RPLanczos Algorithm when used with the quasi-Newton LMP: Approach (H): Apply RPLanczos Algorithm 5.8 on the linear system (4.22) with the HBH T -inner product.

Algorithm

inner min. pairs computation quasi-Newton LMP 3.2 Approach (F) Algorithm 2.7 P j+1 given by (3.37)

4.4

Approach (H) Algorithm 5.7 D j+1 given by (4.51)

Table 5.6: A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs. Approach (F) is defined in Section 5.1.

Preconditioning with the Ritz LMP

In this section we explain how to construct the Ritz-LMP (4.73), from the pairs obtained during RPLanczos Algorithm 5.8. As mentioned before, the information to construct a Ritz-LMP is available as by-products of a Lanczos algorithm. Therefore, when applying RPLanczos on the linear system (4.22) at the k-th outer loop of Algorithm 4.4 (in which the linear system matrix is constant), the Ritz-LMP (4.73) can be constructed from Algorithm 4.8. The inputs, the matrices D k-2 and T and the scalar β +1 , to Algorithm 4.8 are directly available from RPLanczos Algorithm 5.8. The inputs, the vectors υ i and g i , i = 1, . . . , + 1, to Algorithm 4.8 can be obtained as explained below.

From the definition of the vector g i = D v i (see Lemma 4.4), it can be easily seen that g i = z i where z i are defined in RPLanczos Algorithm. The vectors υ i = HBH T z i can be obtained from

HBH T z i = D T k-2 HBH T v i = D T k-2 l i ,
where we used the symmetry of D k-2 in the HBH T -inner product (see Section 4.3.1). As a result it is possible to find the corresponding Ritz LMP (LMP (4.73)) for Algorithm 5.8 such that when it is used within the Lanczos version of Algorithm 4.4, it generates a mathematically equivalent sequence of iterates {s k } to those of the Lanczos version of Algorithm 3.2 with the Ritz LMP (3.65), while possibly yielding gains in terms of both memory usage and computational cost when m n. We showed that the Ritz LMP can be constructed from vectors available from Algorithm 5.8.

We summarize the characteristics of the equivalent primal and dual Lanczos algorithms when used with the Ritz LMP by Table 5.7. We assume that for Algorithm 3. Table 5.7: A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the quasi-Newton LMPs. Approaches (F) and (H) are defined in Section 5.1 and 5.2.1.1 respectively.

The cost and memory requirements for the Ritz LMP (4.73) are given in subsection 4.3.1.2.

Preconditioning with the spectral LMP

The spectral LMP for RPCG is given by Lemma 4.6. The same formula can be applied to RPLanczos Algorithm. We refer to subsection 4.3.1.3 for details in deriving these formulas. In this section we summarize the characteristics of the equivalent primal and dual Lanczos algorithms when used with the spectral LMP by Table 5.8.

Algorithm

inner min. spectral LMP formula Assumption 3.2 Approach (F) P k given by (3.78)

x 1 = x c , s 0 = 0 4.4 Approach (DualF) D k given by (4.74)

P 0 H T = BH T D 0 x 1 = x c
Table 5.8: A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the spectral LMPs. Approaches (F) and (DualF) are the approaches defined in Section 3.1 and 4.3.1 respectively.

Solving the subproblem with Augmented RPLanczos

This section presents the final version of RPLanczos that allows for an arbitrary initial guess. The generalized algorithm relies on augmentation like augmented RPCG. We name this algorithm augmented RPLanczos, which is mathematically equivalent to augmented RPCG. Under Assumption 4.3 on the preconditioner, the augmented RPLanczos algorithm can be derived from PLanczos Algorithm 2.6 on the linear system (4.80) following the basic approach previously used (to derive RPLanczos Algorithm 5.5 from PLanczos Algorithm 2.6 on the linear system (3.1)). The augmented RPLanczos algorithm is given by Algorithm 5.9. This algorithm can be applied to a convergent sequence of linear systems (4.14) by using Algorithm 4.11.

The preconditioner D ∈ IR (m+1)×(m+1) derived in Lemma 4.7 can be used for the augmented RPLanczos algorithm as explained in subsections 4.4.1 and 4.4.2. The required pairs for this preconditioner can be found following the same approach for the augmented RPCG.

Preconditioning varying systems

The same results explained in subsection 4.4.2 apply for RPLanczos when solving a sequence of varying linear systems (4.14) preconditioned by an LMP.

Convergence issues

The convergence behaviour of the Lanczos algorithm is well known and for details we refer to [START_REF] Meurant | The Lanczos and conjugate-gradient algorithms[END_REF]Golub and Van Loan, 1996;Saad, 1996). When Lanczos method is used with one of the particular LMPs, the convergence properties follow similar properties to PCG with an LMP. The convergence properties of RPLanczos with an LMP also follow similar properties to RPCG since these algorithms are mathematically equivalent.

In this section we explain how to re-orthogonalize the Lanczos vectors generated by the RPLanczos algorithm in presence of round-off errors. The section will be concluded by providing inexpensive formulas for computing the values of quadratic cost

q i = (z i + R -1 t i ) -β i v i-1 14 α i = w T i t i 15 w i = q i -α i v i 16 z i+1 = D w i 17 t i+1 = HBH T z i+1 18 β i+1 = t T i+1 w i 19 v i+1 = w i /β i+1 20 z i+1 = z i+1 /β i+1 21 t i+1 = t i+1 /β i+1 22 V = [V , v i+1 ] 23 (T i ) i,i = α i 24 if i > 1 then 25 (T i ) i-1,i = (T i ) i,i-1 = β i 26 end 27
Check convergence and continue if necessary

28 end 29 y = T -1 β 0 e 1 30 s = s 0 + BH T D V y function (2.13
) and its gradient norm on each iteration from the recurrence relations of PLanczos and RPLanczos algorithms.

Re-orthogonalization

The Lanczos vectors in PLanczos (like the residuals in PCG) are, in exact arithmetic, mutually orthogonal. Round-off errors can result in a loss of the orthogonality that can significantly hinder the convergence of these methods. It is possible to alleviate this problem by re-orthogonalizing the Lanczos vectors on each iteration using a modified Gram-Schmidt (MGS) procedure (Saad, 1996, p. 11-12). Considering Algorithm 2.6, the re-orthogonalization procedure acts on the vectors w i . Making use of the orthogonality relationship v T i P v j = 0, for i = j, the MGS re-orthogonalization can be written in compact notation by

w i ←   1 j=i-1 (I n -v j v T j P )   w i , (5.25)
=   1 j=i-1 (I n -v j z T j )   w i .
(5.26) Equations ( 5.25) and (5.26) require the storage of the Lanczos vectors v j from all previous iterations j = 1, . . . , i -1. Equation (5.25) can be less practical than Eq. (5.26) when matrix-vector product with P is expensive. In this case, the vectors z j can also be stored during PLanczos and used in Eq. (5.26).

The need to store and manipulate the sequence of n-dimensional vectors v j and z j can lead to significant computational overhead. In this respect, the dual algorithms, which involve sequences of m-dimensional vectors, are clearly an attractive alternative when m n. From the relations v i = H T v i and P H T = BH T D, the orthogonality condition for the Lanczos vectors can be written as

v T i P v j = v T i HP H T v j = v T i HBH T D v j = 0 (5.27)
for i = j. Combining Eq. (5.27) with Eqs (5.25) and (5.26) leads to the MGS reorthogonalization scheme for RPLanczos as

w i ←   1 j=i-1 (I m -v j v T j D T HBH T )   w i , =   1 j=i-1 (I m -v j t T j )   w i .
Note that the corresponding re-orthogonalization scheme for the augmented RPLanczos follows the similar derivation.

It is remarkable that re-orthogonalization of residuals may be considered as costly in terms of memory and computational cost in the primal setting, turns out to be much cheaper in the dual frameworks since the dimension of the residual vectors in dual space is (typically much) smaller than the dimension of the residual vectors in primal space.

Monitoring convergence

The values of the quadratic cost function and the norm of the cost function gradient are important for monitoring the convergence of the minimization. Cheap formulae for computing all these quantities on each iteration from the recurrence relations of the PLanczos and RPLanczos algorithms (Algorithms 2.6, 5.2, and 5.6, 5.9) are derived in this section. Note that for the sake of simplicity, the initial guess is chosen as s 0 = 0 for all derivations.

Quadratic cost function

The quadratic cost function can be expressed in terms of quantities from PLanczos by substituting the relation s i = Z i y i into Equation (3.87) as

Q[s i ] = Q[0] - 1 2 y T i Z T i r 0 , (5.28)
where Z i = P V i , i = 0, 1, ..., -1, is the number of iterations performed during PLanczos and

Q[0] = f (x k ) = 1 2 d T R -1 d + 1 2 (x c -x k ) T B -1 (x c -x k ).
(5.29)

The background term Q b can be calculated from (3.88) as

Q b [s i ] = 1 2 y T i Z T i B -1 Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c ).
(5.30)

The observation term, Q o , can then be calculated from Equation (3.89).

The expressions for Q and Q b depend on the vectors y i . These vectors can be computed on each iteration by solving the (inexpensive for small i) tridiagonal system

T i y i = β 0 e 1 .
For PLanczosIF the same formula can be used for the quadratic cost function value

Q[s i ] and the term Q o [s i ]. The term Q b [s i ] can alternatively calculated from Q b [s i ] = 1 2 y T i Z T i B -1 Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c ) = 1 2 y T i Z T i B -1 BZ i y i + 1 2 (x k -x c ) T B -1 (x k -x c ) = 1 2 y T i Z T i Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c )
where Z i = [z 1 , . . . , z i ] can be stored during PLanczosIF. Taking x 1 = x c and using the relation s i = BZ i y i , the term B -1 (x kx c ) can also be calculated from

B -1 (x k -x c ) = k-1 j=0 B -1 s j = k-1 j=0 Z (j) l y (j) l
where Z (j) l denotes the last iteration (i = l) of PLanczosIF at the j-th outer loop of Gauss-Newton method.

The values of Q and Q b can be evaluated in terms of quantities from RPLanczos Algorithm 5.6 by using the relations r 0 = H T r 0 , and the relation for z i given by Equation (5.13) in the expressions (5.28) and (5.30). This yields

Q[s i ] = Q[0] - 1 2 y T i Z T i H T r 0 = Q[0] - 1 2 y T i Z T i HBH T r 0 and Q b [s i ] = 1 2 y T i Z T i B -1 Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c ) = 1 2 y T i Z T i HBB -1 BH T Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c ) = 1 2 y T i Z T i HBH T Z i y i + 1 2 (x k -x c ) T B -1 (x k -x c )
where Q[0] is given by Eq. (5.29). Using the symmetry of D with respect to the HBH T -(semi) inner product in RPLanczos Algorithm 5.6 , the vector t 0 can be calculated also from t 0 = D T HBH T r 0 . Similarly the vector t i , i > 1 can be calculated also from t i = D T HBH T w i . Therefore, defining l 0 = HBH T r 0 , and l i = HBH T w i the inexpensive formulas for Q[s i ] and Q b [s i ] can be written as

Q[s i ] = Q[0] - 1 2 y T i Z T i l 0 and Q b [s i ] = 1 2 y T i Z T i D T L i y i + 1 2 (x k -x c ) T B -1 (x k -x c ) Taking x 0 = x c the term B -1 (x k -x c
) can also be calculated without the need to apply B -1 from Eq. (4.19). The Q o term can then be computed from Eq. (3.89). The matrices Z i and L i contain the j = 1, . . . , i column vectors z j and l j . For Augmented RPLanczos, the formula can be obtained similarly following the same derivation by using augmented matrices and vectors. While replacing the term B -1 (x kx c ), since H k varies along the outer loop, the following equation has to be used instead of the Eq. (4.19)

B -1 (x k -x c ) = k-1 j=0 H T j Z (j) l y (j)
l .

Norm of the quadratic cost function gradient

The residual vector of the approximate solution s i computed by PLanczos satisfies (Saad, 1996, p.153)

r i = b -As i = -β i+1 e T i y i v i+1
, where e i is the i-th column of the i × i identity matrix, and β i+1 , y i and v i are as defined in Algorithm 2.6. The P -norm of the cost function gradient can be readily computed as

∇Q[s i ] P = r i P = β i+1 |e T i y i |.
The gradient norm for PCGIF, RPLanczos and the augmented RPLanczos can be calculated equivalently from the same formula.

CHAPTER 6

Towards globally convergent algorithms

In this chapter we consider solving the regularized nonlinear least-squares problem (2.3) by using a method based on a Gauss-Newton technique, made globally convergent with a trust-region strategy. The sequence of linear least-squares problems involved in the method is iteratively solved by a conjugate-gradients method, appropriately truncated by the Steihaug-Toint strategy, and which is accelerated by limited memory preconditioners.

We present numerical experiments on a test problem based on the heat equation which can be considered as an idealized data assimilation system.

The Steihaug-Toint truncated conjugate gradient method

When using the simple Gauss-Newton approach described at Section 2.5 for more than mildly non-linear cost functions, the iterations can unfortunately diverge, and the function value can increase with the Gauss-Newton step computed from (2.46), see for instance (Kelley, 1999, p.39). This problem is not purely theoretical, and is also discussed in a real life problem in (Tshimanga, Gratton, Weaver and Sartenaer, 2008), where the necessity for global minimization is emphasized. As indicated above, global convergence can be ensured by inserting the Gauss-Newton strategy in a trustregion framework. For the nonlinear least-squares problem (2.3), trust-region methods amount to solving approximately a sequence of quadratic problems min

s k 1 2 s k + x k -x c 2 B -1 + 1 2 H k s k + d k 2 R -1 (6.1) subject to s k P -1 k ≤ ∆ k , (6.2)
where ∆ k is the radius of the "trust region", which is the region where we believe that the objective function (2.3) of our nonlinear problem is adequately approximated by that of (6.1). It is important to note that preconditioning appears in this problem as the norm • P -1 k used in (6.2). After solving this subproblem, as explained in Section 2.5.3 the step s k is accepted or rejected and the trust region radius is updated accordingly. The acceptance of the trial point and trust region radius update are decided by considering the ratio

ρ k = f (x k ) -f (x k + s k ) m k (x k ) -m k (x k + s k )
where f is the objective function (2.4) and m k is its quadratic approximation (6.1). This ratio of achieved to predicted reductions gives an indication of the model's quality.

If it is larger than some constant, the step is accepted and the trust-region radius possibly enlarged, while, if it is too small or negative, the step is rejected and the trust-region radius decreased. We refer the reader to (Conn, Gould and Toint, 2000, p. 116) for a more complete description. For large scale instances, the subproblem (6.1)-( 6.2) can be solved approximately using iterative techniques, for instance the Large-Scale Trust-Region Subproblem (LSTRS) algorithm [START_REF] Lampe | Accelerating the LSTRS algorithm[END_REF][START_REF] Rojas | Algorithm 873: LSTRS: MATLAB software for large-scale trust-region subproblems and regularization[END_REF], the Generalized Lanczos Trust-Region method (GLTR) [START_REF] Gould | Solving the trust-region subproblem using the Lanczos method[END_REF][START_REF] Conn | Trust-region methods[END_REF], or the Steihaug-Toint truncated conjugate-gradient method (Conn, Gould and Toint, 2000, p. 205). We consider below an efficient implementation of the Steihaug-Toint truncated conjugate gradient method. In Section 6.2, we briefly explain the GLTR method. A similar adaptation of LSTRS is out of the scope of this thesis.

In the Steihaug-Toint truncated conjugate-gradient technique, the model (6.1) is approximately minimized using PCG until the boundary of the trust region (6.2) is encountered. More specifically, (dropping again the outer-iterations index k for simplicity) three different cases may occur when applying PCG to (6.1) (Conn, Gould and Toint, 2000, pp. 202-204):

1. the curvature p i , Ap i remains positive at each inner iteration, and the PCG iterates remain inside the trust region (the standard PCG stopping rule depending on the relative gradient norm then applies); 2. the curvature p i , Ap i remains positive at each inner iteration, and the PCG iterates leave the trust region, in which case the iterates are stopped when the trust region boundary is met;

3. the curvature p i , Ap i is negative at some PCG step, in which case, the associated descent direction is followed until the trust region boundary is met.

This strategy can be shown to yield a sufficient decrease condition (Nocedal and Wright, 2006, p. 33), (Conn, Gould and Toint, 2000, p. 205) which guarantees global convergence of the iterates. Again the same question arises: may we derive an equivalent dual-space version of this method? In particular, how easy is it to compute a final iterate on the boundary of the trust region following a descent direction form a given inner iterate? For answering these questions, we start by rewriting the Steihaug-Toint algorithm described in (Conn, Gould and Toint, 2000, p. 205) in terms of the vectors defined in Algo-rithm 4.9, using the relations (Gratton and Tshimanga, 2009)

r i = H T r i , p i = BH T p i , s i = s 0 + BH T λ i , z i = BH T z i , q i =
H T q i and the equality P H T = BH T D where D ∈ IR (m+1)×(m+1) is the preconditioner. This gives a first version of the Steihaug-Toint truncated conjugate gradient algorithm (Algorithm 6.1) for the dual approach.

Algorithm 6.1: The Steihaug-Toint truncated CG method in dual space (version 1)

1 λ 0 = 0 2 r 0 = d -R -1 Hs 0 3 z 0 = D r 0 4 p 0 = z 0 5 for i = 0, 1, ..., l -1 do 6 q i = (R -1 HBH T + I m+1 )p i 7 K i = p T i HBH T q i 8 if K i ≤ 0 then 9 compute α i as the positive root of λ i + α i p i HBH T D -1 = ∆ 10 λ i+1 = λ i + α i p i 11 return 12 end 13 α i = r i , z i HBH T /K i 14 if λ i + α i p i HBH T D -1 ∆ then 15 compute α i as the positive root of λ i + α i p i HBH T D -1 = ∆ 16 λ i+1 = λ i + α i p i 17 return 18 end 19 λ i+1 = λ i + α i p i 20 r i+1 = r i -α i q i 21 z i+1 = D r i+1 22
Check convergence and continue if necessary

23 β i = r i+1 , z i+1 HBH T / r i , z i HBH T 24 p i+1 = z i+1 + β i p i 25 end 26 The solution is recovered from s l = s 0 + BH T λ l
As before, this version of the algorithm turns out to be very expensive in terms of HBH T matrix-vector products, and we introduce new vectors to transform it into a computationally efficient method. From

λ i + α i p i 2 HBH T D -1 = λ i 2 HBH T D -1 + 2α i λ i , HBH T D -1 p i + α 2 i p i 2 HBH T D -1 , the positive root of λ i + α i p i 2 HBH T D -1 = ∆ is given by α i = -λ i , HBH T D -1 p i + λ i , HBH T D -1 p i 2 + p i 2 HBH T D -1 ∆ 2 -λ i 2 HBH T D -1 p i 2 HBH T D -1
.

We now explain how to compute the terms given in this expression recursively. Let us define a i+1 = p i+1 2 HBH T D -1 . From line 24 of Algorithm 6.1 we can write

a i+1 = z T i+1 HBH T r i+1 + 2 β i p T i HBH T r i+1 + β 2 i p i 2 HBH T D -1
Since the vectors r i+1 and p i are orthogonal with respect to HBH T , we obtain that

a i+1 = r i+1 , z i+1 HBH T + β 2 i a i .
The term b i+1 = λ i+1 , HBH T D -1 p i+1 can be calculated by using line 24 of Algorithm 6.1 and the relation λ i+1 = i j=0 α j p j (see line 19 of Algorithm 6.1) as follows

b i+1 =   i j=0 α j p j   T (HBH T r i+1 + β i HBH T D -1 p i ) =   i j=0 α j p j   T HBH T r i+1 +   i j=0 α j p j   T β i HBH T D -1 p i =   i-1 j=0 α j p j   T β i HBH T D -1 p i + α i p T i β i HBH T D -1 p i = λ T i β i HBH T D -1 p i + β i α i p T i HBH T D -1 p i = β i λ i , HBH T D -1 p i + α i p i 2 HBH T D -1 = β i (b i + α i a i ) .
Finally, defining c i+1 = λ i+1 2 HBH T D -1 , this term can be calculated from

c i+1 = λ i + α i p i 2 HBH T D -1 = λ i 2 HBH T D -1 + 2 α i λ i , HBH T D -1 p i + α 2 i p i 2 HBH T D -1 = c i + 2 α i b i + α 2 i a i .
We may now use these scalars to calculate α i , which yields

α i = -b i + b 2 i + a i (∆ 2 -c i ) a i . (6.3)
Introducing this change, we now obtain Algorithm 6.2 on the next page. This last version requires a single HBH T matrix-vector product and products with D and D T in each inner loop.

Algorithm 6.2: The Steihaug-Toint truncated CG method in dual space

1 λ 0 = 0; 2 r 0 = d -R -1 Hs 0 3 l 0 = HBH T r 0 4 z 0 = Dr 0 5 p 0 = z 0 6 w 0 = D T l 0 7 t 0 = w 0 8 ρ 0 = w T 0 r 0 9 a 0 = ρ 0 b 0 = 0 c 0 = 0 for i = 0, 1, ... do q i = R -1 t i + p i K i = t T i q i if K i ≤ 0 then
Calculate α i from the formula (6.3) return end

α i = ρ i /K i γ = c i + 2 α i b i + α 2 i a i if γ ∆ then
Calculate α i from the formula (6.3)

λ i+1 = λ i + α i p i return end λ i+1 = λ i + α i p i r i+1 = r i -α i q i l i+1 = HBH T r i+1 i = (l i -l i+1 )/α i z i+1 = Dr i+1 w i+1 = D T l i+1
Check convergence and stop if desired accuracy is reached

ρ i+1 = w T i+1 r i+1 β i = ρ i+1 /ρ i p i+1 = z i+1 + β i p i t i+1 = w i+1 + β i t i c i+1 = c i + 2α i b i + α 2 i a i b i+1 = β i (b i + α i a i ) a i+1 = ρ i+1 + β 2 i a i end
The solution is recovered from s l = s 0 + BH T λ l Therefore we need adapt the strategy in the trust region algorithm.

We first ensure that the preconditioner is positive definite along the steepest descent direction -g k (= -∇ x f (x k )). Then we search for the Cauchy step s k [START_REF] Conn | Trust-region methods[END_REF], i.e

x C k = x k -t C k g k = arg min t ≥ 0 x k -tg k ∈ B k m k (x k -tg k ).
(6.4)

From the Cauchy point, we continue with Augmented RPCG to get x CG k = x k + s k . The iteration is stopped either when the residual norm reduces by a factor of a given tolerance or the maximum number of iterations is reached. Note that, we do not impose the trust-region bound for searching the step after the Cauchy step. Once we obtain the trial step, the step beyond the Cauchy step is accepted if

f (x CG k ) < f (x C k ),
otherwise we choose the trial point as the Cauchy point. This strategy leads to the magical step strategy explained in Conn, Gould and Toint (2000, p. 387) where the step determined within the trust region is taken as the Cauchy step and the magical step is calculated from Algorithm 4.9. We next check the change from x k to x (CG) k

including the contribution of the objective function since the improvement from s C k to s CG k does not depend on any prediction using the model approximation (Conn, Gould and Toint, 2000, p. 388). If this change is larger than some small constant, the step is accepted and the trust-region radius is possibly enlarged, while, if it is too small or negative, the step is rejected and the trust-region radius is decreased. We outline this strategy by Algorithm 6.5 in which the subproblem is solved by Algorithm 6.4.

The Generalized Lanczos Trust-Region method

As indicated before, global convergence of a Gauss Newton method can be ensured by using a trust-region method. A globally convergent algorithm stated as the Steihaug-Toint truncated conjugate gradient method is presented in Section 6.1. This technique minimizes the model (6.1) approximately by using PCG and stops when the boundary of the trust region (6.2) is encountered. Till the trust region boundary is met, this method searches the solution in a subspace s 0 + K (P A, P r 0 ) where the basis for the Krylov subspace K (P A, P r 0 ) is spanned by search directions. [START_REF] Gould | Solving the trust-region subproblem using the Lanczos method[END_REF] investigates the way of continuing the process that approximates the solution by using the generated Krylov subspace also when the trust region boundary is met. They proposed the Generalized Lanczos Trust-Region method (GLTR) which searches for the approximate solution from the same subspace s 0 +K (P A, P r 0 ) where the basis for the Krylov subspace K (P A, P r 0 ) is formed by the preconditioned Lanczos vectors {P v 1 , . . . , P v }. Using the matrix Z l = [P v 1 , . . . , P v ] the approximate solution is given by s = Z l y l where s l solves the problem min s ∈ s0+K (P A,P r0) s, r 0 + 1 2 s, As (6.5) subject to s P -1 ≤ ∆ (6.6) which can be written as (Conn, Gould and Toint, 2000, p. 222) min y∈IR +1 y, β 0 e 1 + 1 2 y, T y (6.7) subject to y 2 ≤ ∆, (6.8)

where T l = Z T l AZ l is the tridiagonal projected matrix. The solution is then found from the relation s = Z l y l . The GLTR Algorithm (Conn, Gould and Toint, 2000, p. 228) is similar to the Steihaug-Toint truncated conjugate gradient algorithm inside the trust region and when the boundary is met or A is indefinite, it solves the problem (6.7) with the constraint (6.8).

Hence, when using the GLTR Algorithm, inside the trust region the Steihaug-Toint truncated conjugate gradient algorithms described in Section 6.1 can be used and when the boundary is met or A is indefinite the Lanczos algorithms (Algorithm 2.6 and Algorithm 5.9) can used to solve problem (6.7) with the constraint (6.8).

When there is a trouble with a preconditioner that is, when it fails to be symmetric and positive definite in dual space, Algorithm 6.5 has to be used. Note that within this algorithm one can use also the augmented RPLanczos algorithm to compute the step beyond the Cauchy step.

Numerical experiments

In this section we provide a brief description of the academic test problem introduced in (Gratton and Tshimanga, 2009) based on heat equation and considered as an idealized data assimilation system. The numerical experiments are performed using this test problem, and are designed to analyze the effect of second-level preconditioners in dual space when used for slowly varying linear systems in a Gauss-Newton method, made globally convergent with a Steihaug-Toint truncated CG method.

The test problem

This test problem consists in finding the best initial temperature distribution using a priori initial temperature distribution, x c , and a given set of approximate temperature distributions, y(t j ), at different times t j . The evolution model (dynamical model) is considered to be the nonlinear heat equation defined by (Gratton and Tshimanga, 2009)

∂x ∂t - ∂ 2 x ∂u 2 - ∂ 2 x ∂v 2 + f [x] = 0 in Ω × (0, ∞) (6.9) x[u, v, t] = 0 on ∂Ω × (0, ∞) (6.10)
where the temperature variable x[u, v, t] depends both on time t and position given by spatial coordinates u and v. The function f [x] is defined by (6.11) An uniform 5-point finite difference scheme with n = 14 × 14 = 196 nodes, and consequently, a space step length h = 1/33 is used. The implicit Euler time scheme is used for the integration over time with a time step length τ = 2 • 10 -4 . Experimental data (x c ∈ IR n and y(t j ) ∈ IR mj ) are produced using twin experiments, where the data are perturbed according the given covariance diagonal matrices B ∈ IR n×n and R ∈ IR m×m where m = N j=1 m j (for details we refer to (Gratton and Tshimanga, 2009)).

f [x] = exp[ηx].
The prediction model H j that maps the model fields to the observed data space is chosen as a restriction operator which selects 1 point every 16 points on the discretization grid and applies the operator C which is the diagonal matrix of the eigenvalues of the 5-point discrete Laplacian of order m j .

We assume that observations are available at times t j = j • τ with j = 1, . . . , 4. Then, we have m = 4 × m j = 64 and n = 196, i.e m < n.

Results

We run experiments for different values of η given in the source term (6.11). The values of η affects the degree of nonlinearity and accordingly the changes along a sequence of linear systems (2.16). As discussed in Chapter 4 depending on the change along the sequence, a CG method in dual space may be in trouble since the inherited preconditioner may not be symmetric with respect to the HBH T -inner product.

We first show the numerical results for the primal and dual approach when a Gauss-Newton method is used without a trust-region strategy to solve a sequence of varying systems (2.16) for different values of η. For the primal approach, PCG Algorithm 2.5 is applied on the linear system (2.16), where the preconditioning is achieved by the quasi-Newton LMP (3.37). In order to see the effect of second-level preconditioning we also run experiments with P k = B. For the dual approach, Algorithm 4.11 is used where the preconditioning is achieved by the formula (4.82) in which the LMP D is constructed from the quasi-Newton LMP (4.51).

For all experiments, the maximum number of outer iterations is set to 4 and the maximum number of inner iterations is set to 10, which somehow reflects a common practice in data assimilation (Tshimanga, Gratton, Weaver and Sartenaer, 2008). To construct the quasi-Newton LMP, all the pairs inherited from the previous iteration are used.

We present these experiments by Figure 3.1, 3.2 and 3.3 for η = 1, η = 2, and η = 3 respectively. First of all from the experiments perfomed with the primal approach, we can see the effect of second-level preconditioning. When η = 1 and η = 2 applying second-level preconditioner in dual space gives very similar results to that of primal space. When η = 3, augmented RPCG has trouble because of the loss of symmetry of the preconditioner with respect to the HBH T -inner product. In Figure 3.4, P-GNTR-ST refers the primal approach which solves a sequence of slowly varying systems (2.16) with the Steihaug-Toint truncated CG algorithm (Conn, Gould and Toint, 2000, p. 205). D-GNTR-F refers the dual approach which solves a sequence of slowly varying systems (2.16) with the flexible trust-region Algorithm 6.5 in which the subproblem is solved by the flexible truncated CG algorithm 6.4 in dual space.

As it can be seen from Figure 3.4 that when η = 3, using Algorithm 6.5 the global convergence can be ensured for the dual approach. Algorithm 6.5 accepts the step that is calculated by Algorithm 6.4 for the first two outer loops, and the third outer loop the algorithm only accepts the Cauchy point. Note that this algorithm requires an additional function evaluation at the Cauchy point for each outer loop.

With the numerical experiments presented in this section, we see that the dual solver performs well on mildly nonlinear systems and can also be safely used in severe cases using the proposed flexible algorithm. This algorithm enables a global convergence disregarding the starting point of the iterations, at the cost of one function evaluation per outer loop. However the robustness that is gained with this algorithm is especially important for operational systems.

CHAPTER 7

Application to variational ocean data assimilation Data assimilation is a key technique to improve the estimation of physical parameters arising for instance in oceanography, atmospheric sciences and also land studies. In this thesis; one of the efficient solution approximation for these systems, four dimensional variational (4D-Var) data assimilation method, is considered with an application to ocean data assimilation systems.

Variational assimilation solves a regularized nonlinear least-squares problem for determining a model state that optimally fits both observational information and a priori information in the form of a model background state. The fit is quantified by a cost function that measures the sum of the weighted squared differences between the available information (observations and background state) and the corresponding model-predicted fields. The weights are defined by matrix operators that define the error statistics (inverse error covariance) of the information.

The main characteristic of the variational data assimilation problems arising in oceanography and meteorology is that these problems are nonlinear large-scale problems. Hence, variational assimilation is implemented using an iterative technique based on the incremental approach (Courtier et al., 1994), which in optimization theory is known as a Truncated Gauss-Newton (TGN) method (Gratton et al., 2007). This approach is also widely used in oceanographic applications (Weaver et al., 2003;Moore et al., 2011). In some applications one can also observe that m n, where m is the number of observations and n is the dimension of the control variable.

There is a lot of interest in data assimilation community to develop efficient algorithms that accelerate the convergence of TGN while possibly reducing the computational and memory cost. In this chapter, we show that the algorithms proposed in this thesis contribute in developing such efficient algorithms for large-scale problems where m n. This chapter includes a recent paper that presents the numerical results obtained from ocean data assimilation systems where the inner minimization in a Gauss-Newton method is performed by RPCG and RPLanczos with first-level preconditioners.

Introduction

Variational assimilation seeks to solve a regularized nonlinear least-squares problem to determine a model state that optimally fits both observational information and a priori information in the form of a model background state. The fit is quantified by a cost function that measures the sum of the weighted squared differences between the available information (observations and background state) and the corresponding model-predicted fields. The weights are defined by matrix operators that define the error statistics (inverse error covariance) of the information.

The basic problem involves the optimization of a set of n control variables given m physical observations and a background estimate for each control variable. Iterative techniques must be used to identify an approximate minimum of the cost function when n is large. In meteorological applications, variational assimilation is implemented using an iterative technique based on the incremental approach (Courtier et al., 1994), which in optimization theory is known as a Truncated Gauss-Newton (TGN) method [START_REF] Lawless | Approximate iterative methods for variational data assimilation[END_REF]Gratton et al., 2007). This approach is also widely used in oceanographic applications (Weaver et al., 2003;Moore et al., 2011a).

The incremental approach solves a sequence of linear least-squares (quadratic minimization) problems where each member of the sequence is a local quadratic approximation of the original nonlinear least-squares problem. Conjugate Gradient (CG) or Lanczos methods, which belong to the general class of Krylov subspace methods, are effective for solving quadratic minimization problems when n is large and when the system (Hessian) matrix is symmetric and positive definite, and only available in operator form (i.e. as a matrix-vector product). When the quadratic minimization is performed directly in R n , the method is referred to as the primal approach. Alternatively, the solution can be found using the so-called dual approach (Egbert et al., 1994;Da Silva et al., 1995;Courtier, 1997;Cohn et al., 1998;[START_REF] Daley | NAVDAS: Formulation and diagnostics[END_REF][START_REF] Bennett | Inverse Modelling of the Ocean and Atmosphere[END_REF] which performs the quadratic minimization in R m . The solution in R m is then mapped to R n through the application of an n × m matrix operator that defines the background-error covariances of the model-equivalent of the observations with the control variables.

The dual approach can provide a significant reduction in the computational cost and storage when m n since all the recurrence formulae in the minimization algorithm involve m-dimensional vectors instead of n-dimensional vectors as in the primal approach. For weak-constraint variational assimilation problems (Courtier, 1997;[START_REF] Bennett | Inverse Modelling of the Ocean and Atmosphere[END_REF][START_REF] Trémolet | Accounting for an imperfect model in 4D-Var[END_REF], n can be very large since the control vector includes a time-sequence of corrective terms or model states in order to account for model error. For those problems, the primal approach may become intractable.

The dual approach of Courtier (1997) consists of solving the quadratic minimization problem in R m with a first-level preconditioner given by the inverse of the observation-error covariance matrix (R -1 ). The preconditioner was applied using a factorized form involving R -1/2 . The method is commonly referred to as PSAS (Physical-space Statistical Analysis System), where the dual approach was first used for operational meteorological applications (Da Silva et al., 1995;Cohn et al., 1998). However, it has been shown by El [START_REF] Akkraoui | Intercomparison of the primal and dual formulations of variational data assimilation[END_REF], Gratton and Tshimanga (2009) (hereafter referred to as GT09) and El [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF] that this version of the dual approach does not generate corresponding n-dimensional iterates that reduce monotonically the quadratic cost function in R n . A prohibitively large number of iterations may then be required to obtain an acceptable solution, which should be measured by the reduction of the cost function in primal space [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF]. If the minimization is terminated after a limited number of iterations, it can even yield a result that is inferior to the initial guess. El [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF] demonstrate experimentally that this problem can be alleviated by minimizing the (R -1preconditioned) cost function in R m using the MINimum RESidual (MINRES) method, although no mathematical proof was provided to support this result.

Another dual algorithm known as the Restricted Preconditioned Conjugate Gradient (RPCG) method has been proposed by GT09. This method again performs the minimization in R m , but contrary to PSAS, it generates mathematically equivalent iterates to those of the primal approach in which the cost function is minimized using a CG method. This allows the dual approach to benefit from the computational savings when m < n, while preserving the desired convergence properties of the primal approach.

In this article, we derive a new dual algorithm called the Restricted B-preconditioned Lanczos method (RBLanczos) where B is the background-error covariance matrix. RBLanczos generates mathematically identical iterates to the B-preconditioned Lanczos algorithm often used in primal approaches. This algorithm can also be interpreted as the Lanczos version of a special case of RPCG in which the corresponding primal first-level preconditioner is B. We call this specific algorithm Restricted B-preconditioned Conjugate Gradient (RBCG). The Lanczos vectors, which are directly computed by RBLanczos, are important for preconditioning (Tshimanga et al., 2008;[START_REF] Fisher | Data assimilation in weather forecasting: A case study in PDE-constrained optimization[END_REF][START_REF] Desroziers | Accelerating and parallelizing minimizations in ensemble and deterministic variational assimilations[END_REF] and for quantifying the performance of the data assimilation system [START_REF] Cardinali | Influence-matrix diagnostic of a data assimilation system[END_REF][START_REF] Gelaro | Examination of observation impacts derived from observing system experiments (OSEs) and adjoint models[END_REF]Moore et al., 2011c).

A practical demonstration of the benefits of RBCG and RBLanczos is provided using two operational variational data assimilation systems for the ocean. RBCG is implemented in a three-dimensional variational assimilation (3D-Var) system for a global configuration of the NEMO (Nucleus for European Modelling of the Ocean) model. This system, called NEMOVAR, is used for operational monthly/seasonal forecasting and ocean reanalysis at the European Centre for Medium Range Weather Forecasts (ECMWF) [START_REF] Mogensen | The NEMOVAR ocean data assimilation system as implemented in the ECMWF ocean analysis for System 4[END_REF][START_REF] Balmaseda | Evaluation of the ECMWF ocean reanalysis ORAS4[END_REF]. RBLanczos is implemented in a four-dimensional variational assimilation (4D-Var) system for a California Current configuration of the ROMS (Regional Ocean Modeling Systems) model. This system, which includes a weakconstraint formulation of 4D-Var, is used for regional ocean forecasting and reanalysis applications (Moore et al., 2011a,b,c).

The outline of the article is as follows. In section 2 the variational assimilation problem is formulated and the Lanczos algorithm for the primal approach is introduced. The RBLanczos algorithm is given in the same section. Practical formulae for diagnosing quantities needed for monitoring the convergence of the algorithm are also derived. Numerical experiments with NEMOVAR and ROMS are presented in section 3. Conclusions and B-preconditioned minimization algorithms future directions are given in section 4. Appendix A provides simplified versions of the BLanczos and RBLanczos algorithms that do not include re-orthogonalization. Appendix B provides a derivation of the relations between the vectors defined in the primal and dual formulations of the Lanczos algorithm.

Lanczos method with the primal and dual approaches

Problem formulation

In its standard formulation, 4D-Var is designed to estimate the initial state of a dynamical system by combining observations over a given time window with an a priori estimate of the initial conditions called the background state. This approach has its origin in maximum likelihood estimation under a Gaussian assumption (Tarantola, 2005, pp 24-32). It leads to the minimization of the cost function

J[u] = 1 2 (u -u b ) T B -1 (u -u b ) + 1 2 N t j=0 H j (x j ) -y j T R -1 j H j (x j ) -y j (1)
with respect to a control vector u = x(t 0 ), chosen to be the initial state of the dynamical model at time t 0 . In this formulation, observations are given by an m j -dimensional vector y j at time t j , and the background state is given by an n-dimensional vector u b = x b (t 0 ). The n × n matrix B is an estimate of the background-error covariance matrix, and the m j × m j matrix R j is an estimate of the observationerror covariance matrix at time t j , with the observation errors assumed here to be uncorrelated in time. The inverse of these matrices defines the weighting matrix of the quadratic terms in (1). In order to calculate the model counterpart of the observation vector at t j , first the state x j = x(t j ) = M 0,j {x(t 0 )} is estimated by propagating the initial state to time t j using the dynamical model operator M 0,j ≡ M(t 0 , t j ), and then the state is mapped to observation space using the observation operator H j . Note that this formulation of 4D-Var assumes that the model operator M 0,j is perfect. Adopting the terminology of [START_REF] Sasaki | Some basic formalisms in numerical variational analysis[END_REF], it is referred to as strong-constraint 4D-Var. The incremental approach (Courtier et al., 1994) is a practical algorithm for solving the 4D-Var problem when the system is weakly nonlinear and of large dimension n. Incremental 4D-Var consists of solving a sequence of linear least-squares approximations of the nonlinear least-squares problem (1). On each iteration (k), a quadratic cost function

J[δu (k) ] = 1 2 u (k-1) -u b +δu (k) T B -1 u (k-1) -u b +δu (k) + 1 2 N t j=0 H (k-1) j M (k-1) 0,j δu (k) -d (k-1) j T ×R -1 j H (k-1) j M (k-1) 0,j δu (k) -d (k-1) j (2)
is minimized to determine a correction (increment) δu (k) = δx (k) (t 0 ) to the initial state estimate u (k-1) = x (k-1) (t 0 ), such that the updated estimate is u (k) = u (k-1) + δu (k) .

The initial estimate is usually taken to be the background state, u (0) = u b . In the quadratic formulation (2), M (k-1) 0,j = M (k-1) (t 0 , t j ) is the tangent-linear (TL) of the nonlinear model M 0,j defined with respect to the time sequence of reference states x (k-1) i , i = 1, . . . , j, and H (k-1) j is the TL of the observation operator H (k) j defined with respect to x (k-1) j . In practice, the TL operators are often approximated. The vector d (k-1) j = y j -H j (x (k-1) j ) is the difference between the observation vector and the corresponding model-predicted values at time t j . The main loop of the incremental 4D-Var algorithm that generates the time sequence of states x (k-1) j and difference vectors d (k-1) j , for j = 1, . . . , N t , is called the outer loop. For large problems, an iterative method is used to solve the quadratic minimization problem (2). The iterative loop of the quadratic minimization problem is called the inner loop since it is nested within the outer loop.

This article focuses on minimization algorithms for the inner loop. For clarity and without loss of generality, we consider a single outer iteration (k = 1) starting from the background state u b . For this special case, the quadratic cost function of the inner-loop problem can be written as

J[δu] = 1 2 δu T B -1 δu J b + 1 2 (G δu -d) T R -1 (G δu -d) J o , (3)
which is a compact form of the linearized problem (2) with k = 1 and u (0) = u b . The underbraces highlight the background term J b and observation term J o . The outer iteration counter (k) has been dropped for clarity of notation. In (3), the generalized observation operator G is a m × n matrix of concatenated operators H j M 0,j over time where m = N t j=0 m j , R is a m × m block-diagonal matrix whose jth block is R j , and d is the vector of concatenated difference vectors d j .

The exact solution, δu * = argmin J[δu], is obtained by setting the gradient of the cost function to zero which yields

δu * = (B -1 + G T R -1 G) -1 G T R -1 d. ( 4
)
Since the matrices appearing in (4) are large and typically only available in operator form (i.e. as a matrix-vector product), an approximate solution is usually found by solving the n × n linear system

(B -1 + G T R -1 G) δu = G T R -1 d , ( 5 )
using a Krylov subspace iterative method (Saad, 1996, Chapter 6). Equation ( 5), which involves directly optimizing in control space to determine δu, is referred to as the primal problem.

Alternatively, the solution (4) can be expressed as (Courtier, 1997)

δu * = BG T (GBG T + R) -1 d , ( 6 )
by using the Sherman-Morrison-Woodbury formula (Nocedal and Wright, 2006, pp 612-613) or from duality theory (Gratton et al., 2013). An approximate solution of ( 6) can be obtained by applying a Krylov subspace method to the m × m linear system

(GBG T + R)λ = d, ( 7 )
and then transforming the solution as

δu = BG T λ. ( 8
)
In ocean data assimilation, this approach has been referred to as the indirect representer method (Egbert et al., 1994) whereas in meteorological data assimilation it has been called PSAS (Da Silva et al., 1995;Courtier, 1997;Cohn et al., 1998). Equations ( 7) and ( 8), which involve optimizing the m-dimensional vector λ, constitute the dual problem.

The dual problem can be preferable to the primal problem for computational reasons when the dimension (m) of observation space is much smaller than the dimension (n) of control space.

For some problems it may be desirable to include additional variables in the control vector u. For example, extra control variables can be included to account for errors in the dynamical model M 0,j , leading to the so-called weakconstraint formulation of 4D-Var [START_REF] Sasaki | Some basic formalisms in numerical variational analysis[END_REF], or to account for errors in boundary conditions. The dimension of the primal problem is determined by the size of the control vector. In contrast, the dimension of the dual problem is determined by the size of the observation vector and thus does not change by including additional control variables. For weak-constraint 4D-Var, the number of extra variables can be so large that the primal problem becomes impractical. In such cases, the dual problem is particularly appealing.

Solving the linearized problem: primal approaches

Equation ( 5) is a problem of the standard form

A δu = b, where A = ∇ 2 J = B -1 + G T R -1 G (9)
is the (symmetric and positive-definite) Hessian or system matrix, and

b = -∇J[0] = G T R -1 d (10)
is the negative of the cost function gradient evaluated at δu = 0. Krylov subspace methods search for an approximate solution δu l in a subspace δu 0 + K l (A, r 0 ) where δu 0 is the initial guess,

r 0 = b -A δu 0 = -∇J[δu 0 ] (11)
is the initial residual (the negative of the gradient evaluated at δu = δu 0 ), and K l (A, r 0 ) = span r 0 , Ar 0 , A 2 r 0 , . . . , A l-1 r 0 is the Krylov subspace of dimension l.

To define the lth iterate uniquely, several Krylov subspace methods impose the Petrov-Galerkin condition (Saad, 1996, p 144)

r l ⊥ L l (A, r 0 ), (12)
where r l is the residual of the lth iterate, and L l (A, r 0 ) is an l-dimensional subspace. The choice of the subspace L l (A, r 0 ) and properties of the matrix A yield different Krylov subspace methods. For instance, when L l (A, r 0 ) = K l (A, r 0 ), the condition ( 12) is called a Galerkin condition. It leads to the Lanczos method when A is symmetric, which finds an approximate solution δu l by minimizing the A -1norm of the residual; r l A -1 = r T l A -1 r l . On the other hand, choosing L l (A, r 0 ) = A K l (A, r 0 ) with A symmetric gives the Minimum RESidual method (MINRES) (Paige and Saunders, 1975). It finds the approximate solution δu l by minimizing the Euclidean norm of the residual; r l 2 = r T l r l . Furthermore, if A is symmetric and positive definite then the Lanczos method is mathematically equivalent to the CG method (Saad, 1996, p 176).

This article focuses on the Lanczos method which is widely used in meteorological and ocean variational data assimilation (Tshimanga et al., 2008;[START_REF] Fisher | Data assimilation in weather forecasting: A case study in PDE-constrained optimization[END_REF]Moore et al., 2011a). As mentioned above, the Lanczos method imposes the Galerkin condition which implies

V T l (b -Aδu l ) = 0 ,
where

V l = [v 1 , v 2 , . . . , v l
] is an n × l matrix whose column vectors v i , i = 1, . . . , l, form an orthonormal basis for K l (A, r 0 ). The basis vectors are known as Lanczos vectors and are constructed by using a simplified version of Arnoldi's algorithm when A is symmetric (Saad, 1996, pp 174-175).

They can be shown to be related to the normalized residual (gradient) vectors generated by CG on the same iterations (Paige and Saunders, 1975). Using this orthonormal basis, the solution on the lth iteration becomes

δu l = δu 0 + V l s l , (13)
where s l is the solution of the linear system

T l s l = V T l r 0 (14)
involving the l × l tridiagonal matrix

T l = V T l AV l . ( 15
)
The tridiagonal system (14) which is of size l, l being typically small, can be easily solved using standard factorization methods (Golub and Van Loan, 1996, pp 138-139). The eigenpairs of the matrix A can be approximated by the Ritz pairs, (θ i , V l ξ i ), i = 1, . . . , l, defined with respect to the subspace K l (A, r 0 ), where (θ i , ξ i ) are the eigenpairs of T l (Parlett, 1980;Golub and Van Loan, 1996;Saad, 1996). The right-hand side of ( 14) can be simplified by recalling that the vectors v i , i = 1, . . . , l, form an orthonormal basis for K l (A, r 0 ) and that v 1 = r 0 / r 0 2 , so that

V T l r 0 = r 0 2 e 1 , (16)
where r 0 2 = r T 0 r 0 and e 1 = [1, 0, . . . , 0] T .

When applying the Lanczos or CG method, a preconditioner is desirable to accelerate the convergence. A preconditioner typically transforms the linear system into one with 'better' spectral properties; e.g. more clustered eigenvalues. At the very least, a first-level preconditioner is needed to non-dimensionalize the control vector when it involves more than one physical variable. In this way, results obtained using the standard implementations of Lanczos or CG (with the canonical inner product) will be independent of the choice of physical units. This can be easily done by separating out the units using the factorization B = D 1/2 CD 1/2 where D 1/2 is a diagonal matrix of (dimensional) standard deviations, and C is a (dimensionless) correlation matrix, with 1s on the diagonal. Equation ( 5) can then be transformed as

C -1 + D 1/2 G T R -1 G D 1/2 δ u = D 1/2 G T R -1 d (17)
and solved for δ u = D -1/2 δu using Lanczos or CG equipped with the canonical inner product. The solution in control space is then obtained from δu = D 1/2 δ u. Equation ( 17) is poorly conditioned in general since typical correlation matrices C used in variational data assimilation contain a wide range of eigenvalues [START_REF] Lorenc | Development of an operational variational assimilation scheme[END_REF]. It also requires specification of the inverse correlation operator C -1 which is difficult in practice. To alleviate these problems, the full background-error covariance matrix B is used as a first-level preconditioner [START_REF] Lorenc | Optimal nonlinear objective analysis[END_REF]. Assuming B can be factored as B = UU T , where U is a n × n left square-root matrix of B, with n possibly different from n, then the preconditioned system becomes

(I n + U T G T R -1 GU) δ u = U T G T R -1 d , (18)
where I n is the n × n identity matrix. Equation ( 18) is solved for δ u using Lanczos or CG formulated with the canonical inner product, and the solution in control space is obtained from δu = U δ u. The eigenvalue spectrum of the system matrix in ( 18) is bounded below by 1, and has a cluster at 1 of size at least max (0, nm). A second-level preconditioner that approximates the inverse of the linear system matrix can also be used to further accelerate the convergence. Tshimanga et al. (2008) and Gratton et al. (2011b) discuss second-level preconditioning techniques within the context of multi-outer-loop iterations of incremental 4D-Var. However, in this article, only first-level preconditioning with B will be considered. Furthermore, the initial guess δu 0 will be assumed to be zero, as is typically the case in applications of 3D-Var and 4D-Var that involve a single outer-loop iteration.

Notice that the dimension of minimization space for solving (18) is determined by n , the dimension of δ u. The case n < n arises with reduced-rank formulations of B where, for example, U is defined by a limited number of ensemble perturbations or some other appropriately chosen basis vectors [START_REF] Evensen | Data Assimilation: The Ensemble Kalman Filter[END_REF]Gratton et al., 2011a). Squareroot preconditioning is clearly advantageous when n n. The case n > n arises when B is formulated as a weighted sum of two or more matrices. This situation can arise with general covariance models constructed from spectral or grid-point filters [START_REF] Fisher | Background-error covariance modelling[END_REF][START_REF] Purser | Numerical aspects of the application of recursive filters to variational statistical analysis. Part I: Spatially homogeneous and isotropic Gaussian covariances[END_REF][START_REF] Weaver | On the diffusion equation and its application to isotropic and anisotropic correlation modelling in variational assimilation[END_REF] and with hybrid ensemble-variational B formulations [START_REF] Buehner | Ensemble-derived stationary and flow-dependent background-error covariances: evaluation in a quasi-operational NWP setting[END_REF][START_REF] Wang | On the theoretical equivalence of differently proposed ensemble-3DVAR hybrid analysis schemes[END_REF][START_REF] Wang | A hybrid ETKF-3DVAR data assimilation scheme for the WRF model. Part I: Observing system simulation experiment[END_REF]. Square-root preconditioning can be less convenient in these latter cases. The preconditioned linear system (18) can be written in the standard form

A δ u = b , (19)
where A = U T AU and b = U T b , with A and b given by ( 9) and ( 10). The Lanczos algorithm applied to (19) searches for a solution in the Krylov subspace K l ( A, r 0 ), where r 0 = U T r 0 . Applying the standard version of the Lanczos algorithm described by ( 13)-(15) to the system (19), with the assumption δu 0 = 0, gives an approximate solution on the lth iteration as

δ u l = V l s l , (20)
where

T l s l = V T l r 0 , and T l = V T l A V l . ( 21
)
The Lanczos algorithm applied to the preconditioned linear system ( 19) is based on the availability of a factored form for B [START_REF] Fisher | Data assimilation in weather forecasting: A case study in PDE-constrained optimization[END_REF]. Hereafter we use the term Lanczos to refer to this particular form of the Lanczos algorithm. Alternatively, B-preconditioning of the linear system (5) can be achieved by employing B as a right symmetric preconditioner (Nour-Omid et al., 1988;Axelsson, 1996;Chan et al., 1999). This leads to

(I n + G T R -1 G B) δu = G T R -1 d. (22)
The system matrix in ( 22) is symmetric (self-adjoint) with respect to the B-inner product. The solution in control space itself is obtained from δu = B δu. This approach generates mathematically equivalent iterates to those of the Lanczos approach using U and U T , but is more general since it does not require B to be factored, this factorization being particularly inconvenient when n n as mentioned above. The Lanczos algorithm that solves the linear system ( 22) with the B-inner product is given in Algorithm 1. For future reference, we refer to it as BLanczos.

BLanczos searches for a solution in the Krylov subspace (Chan et al., 1999) K l (BA, Br 0 ) = span Br 0 , (BA)Br 0 , . . . , (BA) l-1 Br 0 .

(

) 23
It can be easily shown that the matrix V l constructed by BLanczos is related to the matrix V l via the relation V l = U T V l . From this relation, ( 20) and ( 21), and the definitions of A and r 0 , it is straightforward to show that the solution on the lth iteration of BLanczos can be written as

δu l = Z l s l , (24)
where

Z l = BV l T l s l = Z T l r 0 , and T l = Z T l AZ l .    ( 25
)
q i = (v i + G T R -1 G z i ) -β i v i-1 12 α i = q T i z i 13 w i = q i -α i v i 14
Re-orthogonalize w i using V i and Z i 15

t i = B w i 16 β i+1 = t T i w i 17 v i+1 = w i /β i+1 18 z i+1 = t i /β i+1 19 Z i := [ Z i , z i+1 ] 20 V i := [ V i , v i+1 ] 21 (T i ) i,i = α i 22 if i > 1 then 23 (T i ) i-1,i = (T i ) i,i-1 = β i 24 end 25 end 26 Solve T l s l = β 0 e 1 27 δu l = Z l s l
From the orthonormality of the column vectors of V l , i.e. V T l V l = I l , and the relation V l = U T V l , we obtain that V T l BV l = I l , i.e. that the column vectors of V l are Borthonormal. From the B-orthonormality of the vectors v i , i = 1, . . . , l, and the fact that v 1 = r 0 /β 0 , where β 0 = r 0 B = r T 0 Br 0 , the right-hand side of the second expression in (25) can be simplified as

Z T l r 0 = β 0 e 1 .
Each iteration of BLanczos requires only one matrixvector multiplication with each of the matrices G, G T , R -1 and B. The n × l matrix Z l and the two vectors consisting of the diagonal and off-diagonal entries of the l × l symmetric matrix T l need to be stored for computing the solution with (24). As mentioned earlier, these matrices contain approximate information on the eigenspectrum of the Hessian matrix which is valuable for diagnostic studies and for building second-level preconditioners. The matrix Z l is also required for re-orthogonalization with the matrix V l , as will be explained in section 2.5.

When re-orthogonalization is not necessary, the solution can also be found using recurrence relationships that do not require the matrices Z l and T l [START_REF] Papadrakakis | A new implementation of the Lanczos method in linear problems[END_REF]Saad, 1996;[START_REF] Chien | Application of the Lanczos algorithm for solving the linear systems that occur in continuation problems[END_REF]. Therefore, the memory requirements can be reduced by using this version of the Lanczos algorithm (known also as the direct version of the Lanczos algorithm (Saad, 1996, p 177)) which is derived from the LU factorization of T l . This direct version of BLanczos is given in Appendix A.

As mentioned earlier, another well-known method for solving the linear system (5) is CG. CG is mathematically equivalent to the Lanczos method in exact arithmetic when the system matrix A is symmetric and positive definite.

A B-preconditioned CG algorithm for solving the linear system (5), which we call BCG, is given in Algorithm 2. Like BLanczos, BCG only requires B for preconditioning (not its factorization or inverse) and only one application per iteration of the matrix operators G, G T , R -1 and B (Derber and Rosati, 1989).

Algorithm 2: BCG 1 δu 0 = 0 2 f 0 = 0 3 r 0 = G T R -1 d 4 z 0 = Br 0 5 p 0 = z 0 6 h 0 = r 0 7 for i = 0, 1, ..., l -1 do 8 q i = h i + G T R -1 Gp i 9 α i = r T i z i /q T i p i 10 δu i+1 = δu i + α i p i 11 f i+1 = f i + α i h i } for diagnosing J b 12 r i+1 = r i -α i q i 13 Re-orthogonalize r i+1 14 z i+1 = Br i+1 15 β i = r T i+1 z i+1 /r T i z i 16 p i+1 = z i+1 + β i p i 17 h i+1 = r i+1 + β i h i 18 end
From the theoretical equivalence of BLanczos and BCG, it is possible to construct the tridiagonal matrix T l from the coefficients α i and β i generated by BCG (Saad, 1996, pp 181-182), and hence to obtain approximate eigenvalues of the Hessian matrix from the eigenvalues T l . In particular, the diagonal and off-diagonal entries of T l can be obtained from the relations

(T l ) i,i =      1 α i-1 if i = 1, 1 α i-1 + β i-2 α i-2 if i > 1, (26) and (
T l ) i+1,i = (T l ) i,i+1 = √ β i-1 α i-1 . ( 27
)
Another way to avoid specifying a factorization of B is to treat (22) as a non-symmetric system (with respect to the canonical inner product) and to solve it with the Full Orthogonalization Method (FOM), which is a generalization of the Lanczos method that does not require A to be symmetric but is more computationally expensive. El [START_REF] El Akkraoui | Preconditioning of variational data assimilation and the use of a bi-conjugate gradient method[END_REF] used a BiConjugate Gradient (BiCG) algorithm * (Saad, 1996, pp 211-212) to solve the nonsymmetric system ( 22) with second-level preconditioning. BiCG is less robust numerically than PCG (Golub and Van Loan, 1996, p 551), although this problem can be handled by using the BiCG Stabilized method (Saad, 1996, p 217). When using only first-level (B)-preconditioning, El [START_REF] El Akkraoui | Preconditioning of variational data assimilation and the use of a bi-conjugate gradient method[END_REF] show that BiCG is mathematically equivalent to BCG or 'double CG' as referred to in their article.

B-preconditioned minimization algorithms

Solving the linearized problem: dual approaches

Alternatively, the solution of the linearized problem (3) can be found by solving the dual problem using a CG or Lanczos method. By analogy with the use of B as a preconditioner for the primal problem, one possibility is to use R -1 as a preconditioner for the dual problem. This is the basic approach taken in PSAS. For example, employing R -1 left symmetric preconditioning on (7) leads to solving the system (R -1 GBG T + I m ) λ = R -1 d (28) using a CG or Lanczos algorithm formulated with the Rinner product. In many data assimilation systems, R is assumed to be diagonal in which case applying R and R -1 is trivial. Conventional implementations of PSAS (Courtier, 1997;[START_REF] El Akkraoui | Intercomparison of the primal and dual formulations of variational data assimilation[END_REF][START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF] employ R -1 -preconditioning via a square-root matrix

R -1/2 , where R -1 = R -1/2 R -1/2 (assuming R is diagonal), and solve (R -1/2 GBG T R -1/2 + I m ) λ = R -1/2 d , (29)
where λ = R -1/2 λ, using CG or Lanczos furnished with the canonical inner product.

As illustrated by GT09 and El [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF], PSAS has a non-monotonic convergence behaviour when viewed in terms of the reduction of the quadratic cost function (3). This can yield a quadratic cost value that is larger than the initial value when the minimization is terminated after a few iterations. As a remedy, El [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF] suggested the use of MINRES to solve the linear system (29). Their experimental results showed that MINRES could produce a monotonically decreasing quadratic cost function (3), although no mathematical theory was provided to guarantee this behaviour.

A better alternative is the Restricted Preconditioned Conjugate Gradient (RPCG) method (GT09) which also solves (28) using a CG method but equipped with the (possibly semi-definite) GBG T -inner product instead of the R-inner product. As discussed by GT09, RPCG generates, in exact arithmetic, the same iterates as those generated by PCG. This can also be achieved by solving (29) with the R -1/2 GBG T R -1/2 -inner product instead of the canonical inner product as in PSAS.

Since only first-level preconditioning with B is considered in this study, we restrict our attention to RBCG which is the dual equivalent of BCG and a special case of RPCG. For reference, the RBCG algorithm, which is based on Algorithm 5 of GT09, is provided in Algorithm 3. (GT09 and Gratton et al. (2013) provide a more general presentation of the algorithm that allows for second-level preconditioning.)

As with BCG, each loop of the RBCG algorithm requires one matrix-vector multiplication with G, G T , R -1 and B. Note that, the tridiagonal matrix T l can be generated using ( 26) and (27) since the coefficients α i and β i are equivalent in BCG and RBCG.

In the same way that RBCG is the dual equivalent of BCG, there exists a Lanczos algorithm that is the dual equivalent of BLanczos. We call this algorithm Restricted BLanczos (RBLanczos). It produces identical iterates, in exact arithmetic, to RBCG, BCG and BLanczos. RBLanczos simply solves the linear system (28) using a Lanczos algorithm equipped with the (possibly semi-definite) GBG Tinner product. The solution in control space is then 8). RBLanczos is a variant of the Restricted FOM (RSFOM) algorithm (Gratton et al., 2009) adapted to symmetric and positive-definite matrices. The derivation of RBLanczos is sketched in the next section, with more technical details provided in Appendix B.

Algorithm 3: RBCG 1 λ 0 = 0 2 c 0 = 0 3 r 0 = R -1 d 4 p 0 = r 0 5 w 0 = GBG T r 0 6 t 0 = w 0 7 for i = 0, 1, . . . l -1 do 8 q i = R -1 t i + p i 9 α i = w T i r i / q T i t i 10 λ i+1 = λ i + α i p i 11 c i+1 = c i + α i t i } for diagnosing J b 12 r i+1 = r i -α i q i 13 Re-orthogonalize r i+1 14 w i+1 = GBG T r i+1 15 β i = w T i+1 r i+1 /w T i r i 16 p i+1 = r i+1 + β i p i 17 t i+1 = w i+1 + β i t i 18 end 19 δu l = BG T λ l recovered from (

RBLanczos algorithm

Following GT09 and Gratton et al. (2009), we assume that the initial residual r 0 ∈ R n is in the range of the operator G T . This assumption is valid when the initial guess δu 0 is taken to be zero, which is the case here. When the initial guess is different from zero, the assumption still holds but a generalized algorithm with augmented vectors and matrices is required (GT09).

The residual can be written as

r 0 = G T r 0 , (30)
where r 0 = R -1 d ∈ R m . From the basic assumption (30), it is shown in Appendix B that there exist m-dimensional vectors v i , z i , q i , w i and t i that can be related to the corresponding n-dimensional vectors v i , z i , q i , w i and t i of BLanczos Algorithm 1 according to

Gt i = t i , v i = G T v i ( 31
)
for i ≥ 0, and

Gz i = z i , q i = G T q i , w i = G T w i    ( 32
)
where i ≥ 1. Equations ( 30)-( 32) allow all the recurrence relations in BLanczos Algorithm 1 involving vectors of dimension n to be transformed directly into corresponding recurrence relations involving vectors of dimension m. This yields the RBLanczos algorithm given by Algorithm 4.

As with BLanczos, one loop of RBLanczos requires a single matrix-vector product with each of the operators G, Algorithm 4: RBLanczos with re-orthogonalization

1 r 0 = R -1 d 2 t 0 = GBG T r 0 3 β 0 = t T 0 r 0 4 v 1 = r 0 /β 0 5 z 1 = t 0 /β 0 6 β 1 = 0 7 v 0 = 0 8 Z 1 := [ z 1 ] 9 V 1 := [ v 1 ] 10 for i = 1, 2, . . . , l do 11 q i = ( v i + R -1 z i ) -β i v i-1 12 α i = q T i z i 13 w i = q i -α i v i 14
Re-orthogonalize w i using V i and Z i 15

t i = GBG T w i 16 β i+1 = t T i w i 17 v i+1 = w i /β i+1 18 z i+1 := t i /β i+1 19 Z i := [ Z i , z i+1 ] 20 V i := [ V i , v i+1 ] 21 (T i ) i,i = α i 22 if i > 1 then 23 (T i ) i-1,i = (T i ) i,i-1 = β i 24 end 25 end 26 Solve T l s l = β 0 e 1 27 δu l = BG T V l s l G T ,
R -1 and B, but with the important difference that all vectors are defined in R m instead of R n . The two algorithms are mathematically equivalent.

As discussed in section 2.2, BLanczos searches for the solution δu l in the Krylov subspace (23). It is instructive to determine the corresponding Krylov subspace for RBLanczos. Replacing r 0 in ( 23) with (30) yields K l (BA, BG T r 0 ) = span BG T r 0 , . . . , (BA) l-1 BG T r 0 . (33) It can be easily shown that

ABG T = G T A,
where A = I m + R -1 GBG T . Equation (33) can then be written as

K l (BA, BG T r 0 ) = span BG T r 0 , . . . , BG T A l-1 r 0 = BG T K l ( A, r 0 ),
where K l ( A, r 0 ) = span r 0 , A r 0 , . . . , A l-1 r 0 is the Krylov subspace generated by RBLanczos.

When re-orthogonalization is not required, as with BLanczos, RBLanczos can be transformed into an equivalent algorithm that does not require storing the matrices T l and V l to find the solution. This algorithm is provided in Appendix A.

Re-orthogonalization

The Lanczos vectors in BLanczos (or the residuals in BCG) are, in exact arithmetic, mutually orthogonal with respect to the B-inner product. Round-off errors can result in a loss of B-orthogonality and can significantly hinder the convergence of these methods as a consequence. It is possible to alleviate this problem by re-orthogonalizing the Lanczos vectors (or the CG residuals) on each iteration using a modified Gram-Schmidt (MGS) procedure (Saad, 1996, pp 11-12).

With reference to Algorithm 1 for BLanczos, the reorthogonalization procedure acts on the vectors w i . Making use of the orthogonality relationship v T i Bv j = 0, for i = j, MGS re-orthogonalization can be described in compact notation by

w i ←   1 j=i-1 (I n -v j v T j B)   w i , ( 34
) =   1 j=i-1 (I n -v j z T j )   w i . ( 35
)
Equations ( 34) and ( 35) require the storage of the Lanczos vectors v j from all previous iterations j = 1, . . . , i -1. Equation ( 34) is clearly less practical than (35) since it requires i -1 additional, and generally expensive, applications of B to compute the B-inner product of v j and w i . This expensive matrix-vector product can be avoided by storing the vectors z j and using them in (35).

The need to store and manipulate the sequence of n-dimensional vectors v j and z j can lead to significant computational overhead, as will be illustrated in the experiments in section 3. In this respect, the dual algorithms, which involve sequences of m-dimensional vectors, are clearly an attractive alternative when m n. From the relationship v i = G T v i , the orthogonality condition for the Lanczos vectors can be written as

v T i Bv j = v T i GBG T v j = 0 (36)
for i = j. Combining (36) with ( 34) and ( 35) leads to the MGS re-orthogonalization scheme for RBLanczos in Algorithm 4:

w i ←   1 j=i-1 (I m -v j v T j GBG T )   w i , =   1 j=i-1 (I m -v j z T j )   w i .

Monitoring convergence

The values of the quadratic cost function and the norm of the cost function gradient are important for monitoring the convergence of the minimization. In addition to the total value of the cost function, J, the relative contributions to J from the background term J b and observation term J o provide additional useful diagnostic information.

Inexpensive formulae for computing all these quantities on each iteration from the recurrence relations in BLanczos, RBLanczos, BCG and RBCG (Algorithms 1, 4, 2 and 3) are derived in this section. Similar formulae can also be derived for the corresponding direct Lanczos algorithms (Algorithms 5 and 6) given in Appendix A.

Quadratic cost function

BLanczos

Using a Taylor series expansion about the initial guess δu 0 = 0, the quadratic cost function J[δu i ] in (3) can be expressed as

J[δu i ] = J[0] + δu T i ∇J[0] + 1 2 δu T i (∇ 2 J) δu i .
Replacing ∇ 2 J by A (Eq. ( 9)) and ∇J[0] by -r 0 (Eq. ( 11)), and using the relations ( 24) and ( 25), the quadratic cost function can be expressed in terms of quantities from BLanczos as

J[δu i ] = J[0] -s T i Z T i r 0 + 1 2 s T i Z T i AZ i s i = J[0] -s T i Z T i r 0 + 1 2 s T i T i s i = J[0] -s T i Z T i r 0 + 1 2 s T i Z T i r 0 = J[0] - 1 2 s T i Z T i r 0 , (37)
where

J[0] = 1 2 d T R -1 d. (38)
Using again ( 24) and ( 25), the J b term can be evaluated as

J b [δu i ] = 1 2 δu T i B -1 δu i = 1 2 s T i Z T i B -1 Z i s i = 1 2 s T i V T i Z i s i . ( 39
)
The expressions for J and J b depend on the vectors s i . These vectors can be computed on each iteration by solving the (inexpensive for small i) tridiagonal system T i s i = β 0 e 1 .

The J o term is expensive to compute directly since it requires evaluating the term Gδu i which is not directly available on each iteration of BLanczos. It also requires application of R -1 which may be costly when correlated observation error is accounted for in R. Fortunately, given J and J b , J o can be deduced simply from their difference:

J o [δu i ] = J[δu i ] -J b [δu i ]. (40)
However, in some applications, additional constraints are included in the cost function via a penalty term J c , in which case (40) would allow only the sum of J o and J c to be recovered. A typical example is when a digital filter is employed as a weak constraint in 4D-Var in order to penalize high-frequency noise in the solution trajectory [START_REF] Gauthier | Impact of the digital filter as a weak constraint in the pre-operational 4D-VAR assimilation system of Météo-France[END_REF]. In order to isolate J o , the J c term would then need to be evaluated explicitly on each iteration.

RBLanczos

The values of J and J b can be evaluated in terms of quantities from RBLanczos by using the relation (30) and the relations for v i and z i in ( 31) and (32) in the expressions (37) and (39). This yields

J[δu i ] = J[0] - 1 2 s T i Z T i G T r 0 = J[0] - 1 2 s T i Z T i r 0 , and
J b [δu i ] = 1 2 s T i V T i GZ i s i = 1 2 s T i V T i Z i s i ,
where J[0] is again given by ( 38). The J o term can then be evaluated from (40). The matrices V i and Z i contain the j = 1, . . . , i column vectors v j and z j . These matrices are also needed for re-orthogonalization.

BCG

For BCG, the quadratic cost function can be calculated from (37) using the relation (24):

J[δu i ] = J[0] - 1 2 δu T i r 0 , (41)
where J[0] is given by Eq. ( 38).

The J b term can be calculated as

J b [δu i ] = 1 2 δu T i B -1 δu i = 1 2 δu T i f i ,
where f i = B -1 δu i can be computed without the need to apply B -1 by including an additional recurrence relation in Algorithm 2. The J o term can then be computed from (40).

RBCG

For RBCG, the quadratic cost function is calculated from (41) using the relations ( 8) and (30):

J[δu i ] = J[0] - 1 2 λ T i GBG T r 0 = J[0] - 1 2 λ T i w 0 ,
where w 0 = GBG T r 0 is available from Algorithm 3. Using (8), the J b term can be written as

J b [δu i ] = 1 2 λ T i GBG T λ i = 1 2 λ T i c i ,
where c i = GBG T λ i can be diagnosed at little extra cost by including an additional recurrence relation in Algorithm 3. The J o term can then be computed from (40).

S. Gürol et al.

Norm of the quadratic cost function gradient

BLanczos

The residual vector of the approximate solution δu i computed by BLanczos satisfies (Saad, 1996, p 153)

r i = b -Aδu i = -β i+1 e T i s i v i+1 ,
where e i is the ith column of the i × i identity matrix, and β i+1 , s i and v i are as defined in Algorithm 1. Since r i = -∇J[δu i ], the B-norm of the cost function gradient can be readily computed as

∇J[δu i ] B = r i B = β i+1 |e T i s i |. (42)

RBLanczos

Formula (42) can also be used to compute the gradient norm with RBLanczos.

BCG

For BCG, the gradient norm is given by

∇J[δu i ] B = r i B = r T i z i ,
where the vectors z i and r i are defined in Algorithm 2.

RBCG

For RBCG, the norm is obtained from

∇J[δu i ] B = G T r i B = r i GBG T = r T i w i
where the vectors r i and w i are defined in Algorithm 3.

Numerical experiments

Experimental framework

This section provides a brief description of the NEMOVAR and ROMS data assimilation systems and the framework in which the experiments are performed.

NEMOVAR

NEMOVAR is a variational data assimilation system [START_REF] Mogensen | NEMOVAR: A variational data assimilation system for the NEMO model[END_REF] for the NEMO ocean model [START_REF] Madec | NEMO ocean engine[END_REF]. It is designed as an incremental 4D-Var algorithm. 3D-Var is also supported, using the First-Guess at Appropriate Time (FGAT) approach. 3D-Var FGAT follows the basic formulation of incremental 4D-Var but replaces the linearized model propagator with the identity operator (M(t j , t 0 ) ≡ I n ). Dynamical information is still incorporated into the 3D-Var analysis through a balance operator, which relates the control vector to the initial state vector (see later).

A close variant of the operational version of NEMOVAR used at ECMWF for seasonal forecasting and ocean reanalysis [START_REF] Mogensen | The NEMOVAR ocean data assimilation system as implemented in the ECMWF ocean analysis for System 4[END_REF] is used for the experiments presented here. The system is based on 3D-Var FGAT. The configuration is global, with 42 vertical levels and approximately 1 • resolution in the Extratropics. In the Tropics, the meridional resolution is refined, reaching a minimum value of 0.3 • directly at the Equator.

Both BCG and RBCG have been implemented in NEMOVAR, following very similar technical specifications. NEMOVAR also includes a version of Lanczos (the Lanczos method on the linear system (19) with the canonical inner product). Here the experiments focus on comparing BCG and RBCG, and will be presented in section 3.2. These experiments are conducted on a single 3D-Var analysis cycle where the time period covered by the cycle is 10 days. The background initial conditions are taken from the ECMWF ORAS4 ocean reanalysis [START_REF] Balmaseda | Evaluation of the ECMWF ocean reanalysis ORAS4[END_REF] on 1 January 2006. The model state variables comprise potential temperature (T), salinity (S), sea-surface height (η) and the horizontal components (u, v) of velocity. The background-state trajectory on the 10-day window is generated by integrating forward the NEMO model using the same daily surface forcing fluxes and relaxation strategies to sea-surface temperature and climatology as used in ORAS4 for the same period. The observations that are assimilated on this cycle are the same as those used in ORAS4 on the same cycle. They consist of temperature and salinity (T/S) profiles from the quality-controlled EN3 v2a dataset † and along-track sea-level anomaly (SLA) data from altimeter database at AVISO (Archiving, Validation and Interpretation of Satellite Oceanographic data) ‡ . The SLA data are referenced to a mean dynamic topography defined as the time-mean sea level from an ocean reanalysis similar to ORAS4 but assimilating only T/S profiles. The total number of individual observations assimilated on the 10-day 3D-Var cycle from 1 to 11 January 2006 is approximately 5.0 × 10 5 , with 2.1 × 10 5 for T, 1.6 × 10 5 for S, and 1.3 × 10 5 for SLA.

The variables in the control vector δu consist of increments δT, δS U and δη U where the subscript U denotes an 'unbalanced' component of that variable [START_REF] Weaver | A multivariate balance operator for variational ocean data assimilation[END_REF]. The control variables are assumed to be mutually uncorrelated. A linearized balance operator K relates δu to the increments δT, δS, δη, δu and δv comprising the initial state δx(t 0 ). The balance operator acts as a multivariate constraint in the linearized generalized observation operator G in the observation term of the cost function (3). In this experiment, the horizontal velocity increments δu and δv are determined entirely by geostrophic constraints in K [START_REF] Mogensen | The NEMOVAR ocean data assimilation system as implemented in the ECMWF ocean analysis for System 4[END_REF]. The total number of active control variables is approximately 4.6 × 10 6 .

For computational convenience, a land-ocean mask with values of 0 (land) and 1 (ocean) is used in NEMO, as in many ocean models, to account for the complex geometry of land-ocean boundaries. As a consequence, (passive) land points are included along with (active) ocean points in the model-variable arrays in the computer code. By including the land points, the size of the control vector is roughly 9.2 × 10 6 . Half of the memory allocation in a control vector is thus associated with land points. A control vector in this experiment requires 19 times more memory than an observation vector.

B-preconditioned minimization algorithms

ROMS 4D-Var

ROMS is a hydrostatic, primitive equation, Boussinesq ocean general circulation model designed primarily for coastal applications. Terrain-following vertical coordinates are employed allowing for greater vertical resolution in shallow water and regions with complex bathymetry [START_REF] Marchesiello | Open boundary conditions for long-term integration of regional oceanic models[END_REF]Shchepetkin andMcWilliams, 2003, 2005;[START_REF] Haidvogel | Ocean forecasting in terrain-following coordinates: Formulation and skill assessment of the Regional Ocean Modeling System[END_REF]. ROMS supports both a primal and dual formulation of incremental 4D-Var. The primal form follows very closely that of Weaver et al. (2003) and solves (19) using the standard Lanczos algorithm which is equivalent to solving (5) using the BLanczos algorithm.

Two forms of dual 4D-Var are available in ROMS, which differ in the choice of linearization strategy on the outer loop (Moore et al., 2011a,b). One follows the incremental approach (Courtier et al., 1994), while the other is based on the indirect representer approach of Egbert et al. (1994).

For the current study, there is no distinction between these approaches since the experiments are conducted with a single outer loop where the linearization trajectory is the background.

As noted earlier, one of the primary advantages of the dual approach to 4D-Var is in the weak-constraint formulation. Since ( 29) is solved in a space spanned by vectors whose size is determined by the number of observations, the dimension of the problem is not changed by the addition of control variables to account for model error. Conversely, in the primal form embodied by ( 5), the dimension of the control vector swells very quickly as additional control variables are added.

Four options are available in ROMS for solving the quadratic problem (3), namely Lanczos (the Lanczos method on the linear system (19) with the canonical inner product), dual-Lanczos (the Lanczos method on the linear system (29) with the canonical inner product), dual-MINRES (MINRES on the linear system (29) with the canonical inner product), and RBLanczos. A comparison of the four approaches under the assumption of both strong and weak constraint will be presented in section 3.2.

The configuration of ROMS used for the experiments presented here is for the California Current System (CCS), an eastern boundary current characterized by a pronounced seasonal cycle of upwelling and by energetic mesoscale circulations [START_REF] Hickey | Coastal oceanography of western North America from the tip of Baja California to Vancouver Island[END_REF][START_REF] Checkley | Patterns and process in the California Current system[END_REF], providing a challenge for linear-based data assimilation methods such as incremental 4D-Var. The ROMS CCS domain and circulation is described by Veneziani et al. (2009a,b) and Broquet et al. (2009a,b). It spans the region 134-116 • W, 31-48 • N, with 10 km resolution in the horizontal and 42 σ -levels. The minimization experiments described here are conducted for a single 4D-Var analysis cycle spanning the 7-day period 29 March-4 April 2003. The background initial conditions are taken from a 4D-Var sequence described in Moore et al. (2011b), and the model control variables are the same as those of NEMOVAR, with the addition of unbalanced variables δu U and δv U for horizontal velocity. The background trajectory is generated using forcing derived from daily averaged output of atmospheric boundary-layer fields from the Naval Research Laboratory's Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS) [START_REF] Doyle | High-resolution atmospheric modeling over the Monterey Bay during AOSN II[END_REF]. The ocean surface fluxes were derived using the bulk formulations of [START_REF] Liu | Bulk parameterization of the air-sea exchange of heat and water vapor including the molecular constraints at the interface[END_REF] and Fairall et al. (1996a,b), and represent the background surface forcing for 4D-Var. The model domain has open boundaries at the northern, southern, and western edges, and at these boundaries the tracer and velocity fields were prescribed, while the free surface and vertically integrated flow are subject to [START_REF] Chapman | Numerical treatment of cross-shelf open boundaries in a barotropic coastal ocean model[END_REF] and [START_REF] Flather | A tidal model of the northwest European continental shelf[END_REF] boundary conditions respectively. The prescribed open-boundary solution was taken from the Estimating the Circulation and Climate of the Ocean (ECCO) global data assimilation product [START_REF] Wunsch | Practical global ocean state estimation[END_REF], and represents the background openboundary conditions for 4D-Var.

In the strong-constraint 4D-Var experiments, the control vector comprises increments to the model initial conditions, the ocean surface forcing (namely the surface wind stress, and the surface fluxes of heat and freshwater), and open boundary conditions. The number of control variables is 7.3 × 10 6 . In the weak-constraint 4D-Var experiments, the control vector is augmented to include corrections for model error as well, which take the form of space-time corrections to the model equations of motion. The size of the control vector in this case is 5.2 × 10 7 . The implementation of the ROMS weak-constraint 4D-Var used here is the same as that described in Moore et al. (2011b).

The implementation of ROMS weak-constraint 4D-Var follows the dual approach of [START_REF] Chua | An inverse ocean modeling system[END_REF] where a correction term for model error enters as a forcing on the right-hand side of the ROMS TL equations in the inner loop, and in the ROMS nonlinear equations in the outer loop. The ROMS formulation is described in detail by Moore et al. (2011b). While the structure of the cost function is unchanged in this approach, the background term J b now contains additional terms that penalize departures of the model-error correction terms from a zero modelerror background. The background-error covariance matrix B is expanded and contains block-diagonal entries Q that describe the model-error covariance. Similarly, the structure of the Hessian matrix does not change and would have the same structure implied by the system matrix in (5) but with B now expanded to include the Q matrix blocks. Therefore, the problem can be preconditioned by B in the usual way, although the addition of Q to the expanded B will undoubtedly influence the conditioning of the Hessian.

Since we do not have a primal weak-constraint algorithm for ROMS, we cannot explore directly the change in properties of the Hessian due to the addition of Q. However, the influence of Q on the condition number of the preconditioned stabilized representer matrix R -1/2 GBG T R -1/2 + I m in (29) has been examined. The condition number of the preconditioned Hessian matrix I n + U T G T R -1 GU in (18) can then be estimated since this matrix has the same eigenvalues as the preconditioned stabilized representer matrix, together with an additional nm eigenvalues equal to 1 (Courtier, 1997;Horn and Johnson, 1999, pp 53-54). A one-year sequence of strongand weak-constraint 4D-Var simulations using a lowerresolution version of the California Current ROMS reveals that the addition of Q to B generally raises the condition number of the Hessian matrix, although the difference in condition number between the strong-and weak-constaint cases is typically less than an order of magnitude (P. J. Smith, personal communication).

The observations assimilated are the same as those described in Moore et al. (2011b) and include: gridded seasurface height analyses in the form of dynamic topography from AVISO at approximately 1/3 • resolution every 7 days; a blended sea-surface temperature (SST) product with 10 km resolution, available daily from National Oceanic and Atmospheric Administration (NOAA) CoastWatch and consisting of 5-day means derived from the Goddard Earth Observing System (GEOS), Advanced Very High Resolution Radiometer (AVHRR) and Moderate Resolution Imaging Spectroradiometer (MODIS) satellite instruments; and in situ hydrographic observations extracted from the quality-controlled EN3 v1d dataset. The total number of observations is of the order 10 5 (1.0 × 10 5 for T, 5.0 × 10 2 for S, and 2.4 × 10 3 for SSH).

Results

This section describes the results from numerical experiments with NEMOVAR and ROMS that are designed to compare the various primal and dual algorithms discussed in this article. The distinguishing characteristics of these algorithms are summarized in Table 1. The different experiments are summarized in Table 2.

NEMOVAR

The experiments with NEMOVAR are based on 3D-Var FGAT with a single outer-loop iteration and 40 innerloop iterations. The properties of the minimization are summarized in Figure 1. Figure 1(a) shows the iterative evolution of the quadratic cost function (3) using the primal formulation (BCG) and dual formulation (RBCG) of the CG algorithm. The curves are indistinguishable as expected from the theory. Numerical results with Lanczos (not shown) also produce indistinguishable results from those of BCG and RBCG.

The results also illustrate that, for this 3D-Var experiment, there is no visible improvement in the cost function minimization when using re-orthogonalization. However, a positive impact from re-orthogonalization is seen after iteration 22 in the evolution of the gradient norm (Figure 1(b)). With re-orthogonalization, the gradient norms in the BCG O and RBCG O experiments are indistinguishable (when the norm is defined with respect to the appropriate metric: B in BCG, and GBG T in RBCG).

While Figure 1 shows that the algorithms for the primal and dual formulations have near-identical convergence properties, Figure 2 illustrates that the memory requirements of the algorithms can be very different, particularly with reorthogonalization. This figure shows the memory usage as a function of execution time, where the initial point has been chosen just before entering the minimization loop. The jump in the curves near the start corresponds to static memory allocation within the CG algorithms (∼2.5 Gb for BCG; ∼2 Gb for RBCG). Thereafter, the total memory usage remains constant if re-orthogonalization is not activated, resulting in a saving of ∼0.5 Gb with RBCG. Re-orthogonalization requires storing an extra pair of vectors on each iteration as discussed in section 2.5. The required memory is allocated dynamically in NEMOVAR, which explains the steady increase of the curves BCG O and RBCG O. Re-orthogonalization produces only small memory overhead with RBCG, with the total memory being comparable to that required by BCG without reorthogonalization. On the other hand, re-orthogonalization with BCG requires a significantly larger amount of extra memory (∼6 Gb). With higher-resolution model NEMO, Global, 3D-Var, RBCG with re-orthogonalization RBCG NEMO, Global, 3D-Var, RBCG without re-orthogonalization BLanc OS ROMS, Regional, Strong-constraint 4D-Var, Lanczos with re-orthogonalization DLanc OS ROMS, Regional, Strong-constraint 4D-Var, dual-Lanczos with re-orthogonalization DMin OS ROMS, Regional, Strong-constraint 4D-Var, dual-MINRES with re-orthogonalization RBLanc OS ROMS, Regional, Strong-constraint 4D-Var, RBLanczos with re-orthogonalization DLanc OW ROMS, Regional, Weak-constraint 4D-Var, dual-Lanczos with re-orthogonalization DMin OW ROMS, Regional, Weak-constraint 4D-Var, dual-MINRES with re-orthogonalization RBLanc OW ROMS, Regional, Weak-constraint 4D-Var, RBLanczos with re-orthogonalization DMin W ROMS, Regional, Weak-constraint 4D-Var, dual-MINRES without re-orthogonalization RBLanc W ROMS, Regional, Weak-constraint 4D-Var, RBLanczos without re-orthogonalization Naming convention: The first few letters are shorthand notation for the minimization algorithm used (see Table 1 2 gives a description of the experiments, which have been performed on a Dell T5500 using a single-CPU processor.

configurations or with applications, such as 4D-Var, that involve additional control variables, the memory requirements could easily become prohibitive. This figure also illustrates that the cost of algebraic computations, such as scalar products, is noticeably cheaper with the dual approach than the primal approach. The total execution time with RBCG is roughly equivalent with or without reorthogonalization, and is noticeably shorter than with BCG. This is especially noticeable when re-orthogonalization is activated where the computational saving with RBCG is about 22%.

ROMS 4D-Var

In all of the experiments presented here, ROMS 4D-Var was run, as was NEMO 3D-Var, with a single outer-loop iteration and 40 inner-loop iterations. Figures 3 and4 summarize the results of various experiments employing either strong-or weak-constraint 4D-Var. These figures show the behaviour of the quadratic cost function (3) as a function of the number of inner-loop iterations.

The results for the strong-constraint 4D-Var are shown in Figure 3(a). The curves correspond to experiments with the primal solver BLanczos and the dual solvers RBLanczos, dual-MINRES and dual-Lanczos with re-orthogonalization. As anticipated, RBLanczos and BLanczos yield the same sequence of cost function values. Clearly the performance of RBLanczos is superior to both dual-MINRES and dual-Lanczos. Figure 3(a) illustrates the poor convergence properties of dual-Lanczos, in agreement with results from GT09 and El [START_REF] El Akkraoui | Convergence properties of the primal and dual forms of variational data assimilation[END_REF] who employed CG to the linear system (29) with a canonical inner product (which is mathematically equivalent to applying dual-Lanczos to system (29)). In fact, approximately 100 inner-loop iterations are generally required to achieve an acceptable level of convergence with dual-Lanczos, as shown by Moore et al. (2011b). The results of three corresponding weak-constraint dual 4D-Var calculations are shown in Figure 3(b), revealing again the superior convergence properties of RBLanczos. The convergence of weak-constraint 4D-Var with the dual-Lanczos algorithm required more than 200 inner-loop iterations to yield an acceptable level of convergence (Moore et al., 2011b). Note that this plot does not include results for the primal approach since it was not feasible to perform weak-constraint 4D-Var experiments with BLanczos.

The importance of re-orthogonalization with the dual approach is illustrated in Figure 4 which shows, for the case of weak-constraint 4D-Var, the results from applying dual-MINRES and RBLanczos without re-orthogonalization (cf. Figure 3). After an initial period of convergence, the cost function diverges. This is in stark contrast to the NEMO 3D-Var experiment where re-orthogonalization had a relatively minor impact on convergence. The need for re-orthogonalization in weak-constraint 4D-Var places an enormous memory burden on the primal approach. Indeed, in the ROMS weak-constraint 4D-Var experiments, only the dual approach was feasible. Unlike NEMO 3D-Var, the computational cost of ROMS 4D-Var, with strong-or weakconstraints, is comparable for both the primal and dual formulations for the same number of inner-loop iterations since the majority of the computational burden comes from the integration of the TL and adjoint models.

Summary and conclusions

Variational data assimilation involves the solution of a nonlinear least-squares problem to estimate an ndimensional control vector given an m-dimensional observation vector and an n-dimensional background estimate of the control vector. A nonlinear generalized observation operator relates the model control vector to the observation vector. Variational data assimilation problems in meteorology and oceanography are commonly solved using an incremental or truncated Gauss-Newton approach. This approach requires minimizing a sequence of quadratic cost functions, where each member of the sequence is derived by linearizing the generalized observation operator about a recent estimate.

The quadratic minimization problems can be solved using a primal or dual approach. Primal approaches search for a solution directly in an n-dimensional space associated with the model control variables. Dual approaches exploit the fact that the solution of the quadratic minimization problem is in the range of the adjoint of the generalized observation operator to search for the solution in an m-dimensional space associated with the observations. The dual approach becomes especially attractive from a computational viewpoint when m n which is a typical situation in ocean data assimilation and with weakconstraint formulations of 4D-Var in general. CG and Lanczos methods, which belong to the general class of Krylov subspace iterative methods, are well suited for solving quadratic minimization problems when n and m are large. Conventional implementations of incremental 4D-Var employ a primal approach in conjunction with a CG or Lanczos algorithm. The system is preconditioned by the background-error covariance matrix, B, which in practice is achieved through a variable transformation that requires a square-root factorization of B. B-preconditioning of the CG and Lanczos algorithms can also be achieved without the need to factor B. This property, which is convenient with general B formulations, is shared by the BCG and BLanczos primal algorithms which were presented in this article. Both algorithms are equivalent in exact arithmetic.

The dual equivalent of BCG is the RPCG algorithm of GT09 with no second-level preconditioning. We called this specific algorithm RBCG to distinguish it from RPCG which allows for general preconditioners. Numerical experiments comparing the performance of RBCG and BCG in a globalocean 3D-Var assimilation system for the NEMO model were presented. In the experiments, the control vector was roughly 19 times larger than the observation vector. As expected from theory, and as demonstrated in an idealized framework by GT09, RBCG and BCG produce identical iterates to within machine precision. However, the memory requirements and computational costs of the algorithms were very different, particularly when a re-orthogonalization scheme was used to compensate for the effects of roundoff error. While RBCG was relatively insensitive to reorthogonalization in terms of memory overhead, BCG with re-orthogonalization required nearly twice as much total memory as BCG without re-orthogonalization. RBCG was also nearly 22% faster than BCG.

The dual equivalent of BLanczos is RBLanczos which is a new algorithm which was derived in this article. The derivation of RBLanczos follows directly from BLanczos by exploiting the fundamental property that, under the assumption of a zero initial guess, the initial value of the cost function gradient is in the range of the adjoint of the generalized observation operator. As a result, all recurrence relations involving n-dimensional vectors in BLanczos can be transformed into recurrence relations involving mdimensional vectors. The resulting algorithm requires the same number of matrix-vector products as BLanczos.

Numerical experiments comparing the performance of RBLanczos and Lanczos (the equivalent form of BLanczos which employs the factorized form of B as a preconditioner) in a regional-ocean 4D-Var assimilation system for the ROMS model were presented. As expected from theory, RBLanczos and Lanczos produce identical iterates to within machine precision. In experiments which employed both strong-and weak-constraint formulations of 4D-Var, the convergence properties of RBLanczos were also clearly superior to those of two other dual algorithms (dual-Lanczos and dual-MINRES) which have been proposed in the literature. In all algorithms, re-orthogonalization was found to be crucial for convergence and produced much smaller memory overhead with the dual algorithms than with the primal algorithms. It is also important to remark that only the dual approach was feasible in the weak-constraint 4D-Var experiment due to the very large size of the control vector.

The two ocean models and data assimilation systems used here differ significantly. On the one hand, NEMO B-preconditioned minimization algorithms is configured as a coarse-resolution model of the global ocean and captures the dominant large-scale features of the global circulation. ROMS, on the other hand, is configured as an eddy-resolving model of the west coast of North America, and captures the energetic mesoscale circulation environment associated with the CCS. In addition, in this study, NEMO employs 3D-Var while ROMS employs 4D-Var with the model applied as either a strong or weak constraint. It is therefore noteworthy that RBCG and RBLanczos are robust in two very different circulation environments, and across a range of data assimilation approaches within the variational framework.

This work has illustrated the benefits of RBCG and RBLanczos from different perspectives. In this study, we considered only a single outer loop of the incremental algorithm and only first-level preconditioning using B. Extensions of this work to account for multiple outerloop iterations and second-level preconditioners based on limited-memory preconditioners such as quasi-Newton and Ritz (Gratton et al., 2011b) will be described in a future publication. In view of the importance of reorthogonalization in the 4D-Var experiments, it would also be of interest to investigate the use of FOM, which is more stable with respect to loss of orthogonality since it includes naturally an explicit full re-orthogonalization of the Krylov subspace basis.

CHAPTER 8 Conclusions and Future Directions

The thesis concentrates on the solution of the nonlinear least squares problem, in which there are many fewer observations than variables to be estimated, and where a quadratic regularization term has therefore to be introduced in order to guarantee uniqueness of the solution. Within this study, a solution method based on a Gauss-Newton technique, made globally convergent with a trust-region strategy is considered. Each member of the sequence of linear least-squares problems involved in the method is solved iteratively by the conjugate gradient method (or the Lanczos method), appropriately truncated by the Steihaug-Toint strategy, and which is accelerated by limited memory preconditioners (Gratton, Sartenaer and Tshimanga, 2011).

It has been recently shown that it is possible to use duality theory and rewrite the linear least-squares solver in an equivalent algorithm (in exact arithmetic) in which all vectors of the short-term recurrences are represented by vectors of dimension m, m being the number of physical observations. Two proposed dual approaches, called PSAS and RPCG, are shown to differ in the way they define the scalar product in dual space (Gratton, Gürol and Toint, 2013). It is argued that RPCG is preferable to PSAS because it maintains the convergence properties of the initial Gauss-Newton process (Gratton and Tshimanga, 2009).

In this thesis, a further step is taken in making the RPCG dual solver relevant to practice for large scale, nonlinear problems. A general LMP is first introduced for the case when the system matrix of a sequence of linear systems (arising from solving the nonlinear least squares problem with a Gauss-Newton method) is assumed to be constant. This leads to solving a convergent sequence of linear systems with multiple right hand sides. Later particular LMPs in dual space such as the Ritz-LMP, the spectral LMP and the quasi-Newton LMP are introduced. All these techniques are implemented in such a way that RPCG and the primal approach generate the same sequence of iterates. A further advantage of the proposed dual approach in the common situation where m n, is that storing vectors for the preconditioner or performing re-orthogonalization is computationally much cheaper than with standard primal algorithms, making both of these techniques applicable in realistic cases.

These preconditioners are later adapted to the case where a sequence of slowly varying linear systems has to be solved as in the Gauss-Newton process by introducing an efficient implementation of the Steihaug-Toint truncation of CG in dual space. Numerical experiments are presented on a test problem based on the heat equation which can be considered as an idealized data assimilation system. With the numerical experiments, it can be seen that the dual solver performs well on mildly nonlinear systems and can also be safely used in severe cases using the proposed flexible algorithm. This algorithm enables a global convergence disregarding the starting point of the iterations, at the cost of one function evaluation per outer loop. However the robustness that is gained with this algorithm is especially important for operational systems.

In addition to designing memory efficient preconditioners in dual space, an alternative memory efficient preconditioning is proposed in primal space by using the PCGIF algorithm. This algorithm avoids expensive matrix vector products (especially for large-scale problems) with the inverse of the a priori error covariance matrix B and does not require the factorization of the preconditioner. All the particular LMPs are adapted to be used in the PCGIF algorithm.

Knowing that Lanczos type algorithms are widely used in data assimilation, Lanczos versions of all CG variant algorithms are also derived both in primal and dual space. The implementation of the limited memory preconditioners are explained in detail for these Lanczos type algorithms.

Important applications of the problem stated in this thesis arise in environmental sciences like meteorology or oceanography, where the estimated vector is the initial state of a dynamical system which is then integrated forward in time to produce a forecast. In this thesis, RPCG and its Lanczos version RPLanczos were implemented in two different variational ocean data assimilation systems: NEMOVAR and ROMS, where the a priori error covariance matrix B is used as a preconditioner and a single outer loop is considered. NEMOVAR is based on a global ocean model and captures the dominant large-scale features of the global circulation whereas ROMS is based on a regional model and is configured as an eddy-resolving model of the west coast of North America, and captures the energetic meso-scale circulation environment. ROMS accounts for model errors also. Therefore, RPCG and RPLanczos are validated in two ocean systems which represent the main approaches for such systems.

There are some issues worth further exploration. It would be interesting to further explore globalization strategies by developing algorithms that are similar in spirit to the Moré-Sorensen (Conn, Gould and Toint, 2000, Section 7.3) approach, or by considering techniques based on cubic regularization [START_REF] Cartis | Adaptive cubic overestimation methods for unconstrained optimization. Part I: motivation, convergence and numerical results[END_REF]. In both cases, as for the Steihaug-Toint truncated CG algorithm we expect that keeping the symmetry with respect to the inner product in dual space may be a key issue to obtain a robust algorithm. A further improvement of the algorithms presented in ths thesis is to design computationally efficient criteria to detect the loss of symmetry with respect to the inner product in dual space and thereby avoid the computation of a magical step that would not result in a decrease of the objective function.

It would be also interesting to further explore the adaptation of the proposed algorithms when the errors of assimilated data (observations and a priori background estimates) have non-Gaussian probability density functions (pdfs) [START_REF] Bocquet | Modélisation inverse et assimilation de données nongaussiennes pour les traceurs atmosphériques. application a etex, algésiras et tchernobyl[END_REF].

In view of the importance of re-orthogonalization in ocean data assimilation systems the use of the Full Orthogonalization Method (FOM), which is more stable with respect to loss of orthogonality, can also be investigated.

The implementation of the particular LMPs such as the quasi-Newton LMP and the Ritz LMP in dual space with multiple outer loops in ocean data assimilation systems: NEMOVAR and ROMS, and atmospheric data assimilation system: OOPS (Object Oriented Prediction System) developed by ECMWF (European Center for Medium-Range Weather Forecast) are of interest for future studies.

  3.4 Memory and cost requirements for applying the quasi-Newton LMPs during Approaches (A), (B) and (D) . . . . . . . . . . . . . . . . . . . . 3.5 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where preconditioning is achieved by using the Ritz LMPs. . . . . . . . . . . . 3.6 Memory and cost requirements for applying the Ritz LMPs during Approaches (A), (B) and (C) . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where preconditioning is achieved by using the spectral LMPs. . . . . . . . . . 3.8 Memory and cost requirements for applying the spectral LMPs during Approaches (A), (B) and (C) . . . . . . . . . . . . . . . . . . . . . . . . 4.1 A summary of the characteristics of Algorithms 3.2 and 4.4. . . . . . . . 4.2 A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the quasi-Newton LMPs. Approach (B) is defined in Section 3.1. . . . . . . . . . . . . . . . . . . . . . . . . 4.3 A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the Ritz LMPs. Approach (A) and (DualD) are the approaches defined in Section 3.1 and 4.3.1.1 respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the spectral LMPs. Approach (A) and (DualD) are the approaches defined in Section 3.1 and 4.3.1.1 respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Summary of operations for each inner iteration i of RPCG and Augmented RPCG (A-RPCG): flops requirements, matrix-vector product, required operations to apply preconditioner. . . . . . . . . . . . . . . . .

  v

  2.1.1 respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 A summary of the characteristics of Algorithms 3.2 and 4.4 where the inner minimization is performed with corresponding Lanczos algorithms and preconditioning is achieved by using the spectral LMPs. Approaches (F) and (DualF) are the approaches defined in Section 3.1 and 4.3.1 respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. 1

 1 CG for solving a convergent sequence of linear systems with multiple right-hand sides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 PCG for solving a convergent sequence of linear systems with multiple right-hand sides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 PCGIF Algorithm (version 1) . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 PCGIF Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 PCGIF for solving a convergent sequence of linear systems with multiple right-hand sides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 PCGIF Algorithm (version for quasi-Newton LMP) . . . . . . . . . . . . 3.7 Compute z= Cr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Compute z = C T l . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Construct the factored form of the Ritz-LMP for the matrix A k . . . . . 3.10 Construct the Ritz-LMP for the matrix A (version 1) . . . . . . . . . . . 3.11 Construct the Ritz-LMP for the matrix A (version 2) . . . . . . . . . . . 3.12 Construct the Ritz-LMP for the matrix A . . . . . . . . . . . . . . . . . . 4.1 PCG Algorithm in IR m (RPCG, version 1) . . . . . . . . . . . . . . . . . 4.2 RPCG Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 RPCG Algorithm (version for s 0 = 0) . . . . . . . . . . . . . . . . . . . . 4.4 RPCG for solving a sequence of linear systems with multiple right-hand sides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 RPCG Algorithm (version for quasi-Newton LMP) . . . . . . . . . . . . . 4.6 Compute z = D r . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Compute w = D T l . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Construct the Ritz-LMP for the matrix A . . . . . . . . . . . . . . . . . . vii CHAPTER 1

d

  : IR n → IR m , by defining d(x) = [d 1 (x), d 2 (x), . . . , d m (x)] T .

  Memory and cost requirements for applying the quasi-Newton LMPs during Approaches (A), (B) and (D)

  .71) Proof. Multiplying the equation (3.59) from right by C and using the relation (3.68) gives that Cu i = CV ūi = V ūi from which it can be seen that the vectors defined by (3.70) satisfy the relation (3.71). 2

  Algorithm 4.1: PCG Algorithm in IR m (RPCG, version 1)

23 end 24

 24 The solution is recovered from s l = BH T s l Approach (DualD): Apply RPCG Algorithm 4.5 on the linear system (4.22) with a preconditioner constructed from (4) D j+1 given by (4.51)P H T = BH T D 0 x = x cTable4.2: A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the quasi-Newton LMPs. Approach (B) is defined in Section 3.1.

  2,x 1 = x c and s 0 = 0, and for Algorithm 4.4 P 0 H T = BH T D 0 and x 1 = x c .

  truncated CG, that reduces the model along the steepest descent direction while satisfying the bound s (k) c ≤ ∆ k . The resulting point is known as the Cauchy point x C

34

  Check the positive-definiteness of the preconditioner along r (k) 0 Compute the Cauchy step s C k by using Algorithm 6.2 5 Compute the step beyond the Cauchy step s CG k with the augmented RPCG algorithm (Algorithm 4.9)

Figure 3

 3 Figure 3.1: The quadratic cost function (2.13) values versus the inner iterations for η = 1.

Figure 3 . 2 :

 32 Figure 3.2: The quadratic cost function (2.13) values versus the inner iterations for η = 2.

Figure 3 . 3 :

 33 Figure 3.3: The quadratic cost function (2.13) values versus the inner iterations for η = 3.

Figure 3 . 4 :

 34 Figure 3.4: The quadratic cost function (2.13) values versus the inner iterations for η = 3 when the Gauss-Newton method is used with a trust-region strategy.

Figure 1 .Figure 2 .

 12 Figure 1. (a) The value of the cost function as a function of the CG iteration counter, and (b) the value of the norm of the cost function gradient relative to the norm of the initial cost function gradient, both as a function of the CG iteration counter. The gradient norm is measured with respect to the B metric. The experiments are as described in Table 2. The curves in (a), and the curves for BCG O and RBCG O in (b), are indistinguishable. A logarithmic vertical scale is used in both panels.

Figure 3 .

 3 Figure 3. The value of the cost function as a function of the Lanczos iteration counter for (a) strong-constraint 4D-Var and (b) weak-constraint 4D-Var, using the Lanczos algorithms based on the primal and dual formulations with re-orthogonalization. Table2gives a description of the experiments. A logarithmic vertical scale is used in both panels.

Figure 4 .

 4 Figure 4. As Figure 3, but for the case of weak-constraint 4D-Var without re-orthogonalization of the Lanczos vectors.

  3.1 The quadratic cost function (2.13) values versus the inner iterations for η = 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 The quadratic cost function (2.13) values versus the inner iterations for η = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 The quadratic cost function (2.13) values versus the inner iterations for

η = 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 The quadratic cost function (2.13) values versus the inner iterations for η = 3 when the Gauss-Newton method is used with a trust-region strategy.149 vi List of Algorithms

2.1 Basic Gauss-Newton trust region algorithm . . . . . . . . . . . . . . . . . 2.2 CG Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Lanczos Algorithm for Linear Systems . . . . . . . . . . . . . . . . . . . . 2.4 Lanczos2CG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 PCG Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 PLanczos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 PLanczos2PCG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  .8.

	Algorithm inner min. spectral LMP formula	Assumption
	3.1	App. (A)	F k given by (3.76)	F 0 is available
	3.2	App. (B)	P k given by (3.78)
	3.5 C Approach App. (C) LMP Ref.	Memory req. Flops req.	mat-vec prod.
	(A)	F and F T (3.76)		(n + 1)l	≈ 8ln
	(B)	P	(3.78)		(n + 1)l	≈ 4ln	P k-2
	(C)	C and C T (3.80)		(2n + 1)l	≈ 8ln	C k-2 and C T k-2

k given by (3.80) B -1 P 0 is available Table

3

.7: A summary of the characteristics of Algorithms 3.1, 3.2 and 3.5 where preconditioning is achieved by using the spectral LMPs. Table

3

.8: Memory and cost requirements for applying the spectral LMPs during Approaches (A), (B) and (C)

  Assume also thatP k-2 H T = BH T D k-2 where D k-2 = M is defined in Lemma (4.6). H T = BH T D k-2 and BH T A = H T A.From (4.78) and (4.79) it can be easily seen that U = H T U where U = [u 1 , . . . , u l ]. Using this relation in (4.75), we get X = BH T X, X = [x 1 , . . . , x l ]. Therefore, from Lemma 4.1 the relation P k-1

	Multiplying equation (4.76) from left by H T gives that
	H T AD k-2 U = H T U Λ
	from which we obtain that
	AP k-2 H T U = H T U Λ,	(4.79)
	by using the relations P k-2

.77) and (λ i , u i ) are eigenpairs of the matrix AP k-2 , i.e.

AP k-2 u i = u i λ i .

(4.78)

  .4.

	Algorithm	inner min.	spectral LMP formula Assumption
	3.2	Approach (B)	P k given by (3.78)	x 1 = x c , s 0 = 0
	4.4	Approach (DualD) D k given by (4.74)

  .5. From this table it can be seen that the computational cost of Augmented RPCG Algorithm 4.10 is very close to that of RPCG Algorithm 4.3.

	Algorithm flops req.	matvec pr.	precond op.
	RPCG	12m -2	B, H T , R -1 and H matvec with D
	A-RPCG 16m + 13 B, H T , R -1 and H matvec with D
	Table 4.5: Summary of operations for each inner iteration i of RPCG and Augmented
	RPCG (A-RPCG): flops requirements, matrix-vector product, required operations to
	apply preconditioner.
	Algorithm 4.10 can be integrated into a Gauss-Newton algorithm given by Al-
	gorithm 4.11. This algorithm finds the solution of a sequence of varying linear sys-
	tems (3.1) by performing minimization in IR m+1 .
	Algorithm 4.11: Gauss-Newton method with Augmented RPCG
	1 Initialization: Choose an initial preconditioner D 0 such that Assumption 4.3
	holds. Choose an initial estimate s 0
	2 Perform inner loop: Solve the linear system (3.1) with Augmented RPCG
	Algorithm 4.10. During Augmented RPCG extract and save relevant
	information to precondition the next linear system using D k such that P k H T k+1 = BH T k+1 D k .
	3 Update the iterate:
			x

k+1 = x k + s k 4 Increment k by 1 and go to step 2.

  .2 are mathematically equivalent to the corresponding algorithms listed in

	Algorithm	inner min.	pairs computation quasi-Newton LMP
	3.1	Approach (E)	Algorithm 2.4	F j+1 given by (3.42)
	3.2	Approach (F)	Algorithm 2.7	P j+1 given by (3.37)
	3.5	Approach (G)	Algorithm 5.3	C j+1 given by (3.43)

Table 1 .

 1 A summary of the distinguishing characteristics of the different minimization algorithms used in the data assimilation experiments for solving either the primal problem (5) or the dual problem (7).

	Algorithm	Formulation	Description

Table 2 .

 2 A list of the experiments performed using data assimilation systems with the NEMO and ROMS models.

	Experiment	Description
	BCG O	NEMO, Global, 3D-Var, BCG with re-orthogonalization
	BCG	NEMO, Global, 3D-Var, BCG without re-orthogonalization
	RBCG O

Copyright c 2013 Royal Meteorological Society Q. J. R.Meteorol. Soc. (2013)

Acknowledgements

Acknowledgements

This work is a contribution to the ADTAO and FILAOS projects which are financed by the RTRA STAE foundation. Additional support from the French National Research Agency (ANR) COSINUS programme (VODA project, no. ANR-08-COSI-016) and the French LEFE programme is also acknowledged. The authors also gratefully acknowledge the support of the US Office of Naval Research (N00014-10-1-0476, N00014-10-1-0322) and the National Science Foundation (OCE-1061434). Any opinions, findings, and conclusions or recommendations expressed here are those of the authors and do not necessarily reflect the views of the National Science Foundation. Kristian Mogensen is thanked for his help in implementing the minimization algorithms in NEMOVAR. Two anonymous reviewers provided helpful remarks for improving the article.

Note that the iterates of Algorithm 4.3 and the iterates of Algorithm 4.2 are related to each other with equation (4.21).

We now want to find a LMP formula for the linear system matrix (4.25) satisfying Assumption 4.2 for a given general LMP (2.38). It is shown in Gratton, Gürol and Toint (2013) that it is possible to find a quasi-Newton LMP D ∈ IR m satisfying P H T = BH T D (where P is a preconditioner for the primal approach) from the pairs that are available as by-products of RPCG. The following lemma generalizes this quasi-Newton LMP for the system matrix A. is symmetric with respect to the HBH T -inner product.

Suppose also that M H T = BH T M . If we denote S = BH T S and ABH T = H T A then the LMP matrix P given by (2.38) and the matrix D defined by (4.26) satisfies P H T = BH T D.

Proof. Let us define Q = HBH T . Multiplying D on the left by Q gives that

Using the assumption that Q A and Q M are symmetric, we have

which proves the symmetry of D in HBH T .

For the second part of the proof, multiplying (2.38) from right by H T gives that

Substituting the relations M H T = BH T M , S = BH T S and ABH T = H T A into this Algorithm 4.8: Construct the Ritz-LMP for the matrix A

1 Given D k-2 , G , Υ l , T , υ +1 , g +1 and β +1 from RPCG Algorithm 4.3

2 Calculate the eigenpairs (θ i , ūi ), i = 1, . . . , of the tridiagonal matrix T :

where Θ = diag(θ i ) and Ū = [ū 1 , . . . , ū ].

3 Generate the vector ω = (ω 1 , ..., ω ) T with ω i calculated from the formula (3.48)

4 Generate the matrix Y = [y 1 , . . . , y ] whose column vectors are calculated from . . . , x ] whose column vectors are calculated from

6 Construct the Ritz-LMP from the formula (4.73)

when used with the Ritz LMP by

Table 4.3: A summary of the characteristics of Algorithms 3.2 and 4.4 where preconditioning is achieved by using the Ritz LMPs. Approach (A) and (DualD) are the approaches defined in Section 3.1 and 4.3.1.1 respectively.

The cost for applying preconditioner D k-1 given by (4.73) is (8lm + 4m) flops and one matrix-vector product with D k-2 . The memory requirement is (m + 2)l + m scalars. Therefore when comparing the cost and memory requirements for the primal approach with a Ritz LMP given in Table 3.6, the cost for appying the preconditioner in the dual approach is much less than that of the primal approach when m n.

Preconditioning with the spectral LMP

In this section, we derive the spectral LMP to precondition the linear system (4.22) during RPCG Algorithm 4.3. This spectral LMP satisfying Assumption 4.2 is given by the following lemma.

Lemma 4.6 Suppose that HBH T M = M T HBH T where HBH T is a nonsingular matrix. Let (λ i , y i ) be eigenpairs of ( A M ) T , with y i = HBH T M u i , i = 1, . . . , l be l obtained from vectors i as explained below.

The vector i = HBH T q i defined in Augmented RPCG Algorithm 4.10 can be written as

Then, the vector i ∈ IR m can be obtained from the vector i ∈ IR m+1 by using the relation

Algorithm 4.12: Augmented RPCG (version for quasi-Newton LMP)

1 Choose an initial estimate s 0

Preconditioning with PLanczos Algorithm 2.6

This approach can be considered as the Lanczos version of Approach (B) presented in Section 3.1. Similar to the case of the CG solvers, the main difference in solving the linear system (3.1) with PLanczos Algorithm 2.6 rather than Lanczos Algorithm 2.3 with a preconditioner is that Lanczos Algorithm 2.3 does not require the factored form of the preconditioner. This can be algorithmically significant when the preconditioner factorization is not available or expensive to compute.

Therefore, another preconditioning technique results from applying Lanczos Algorithm 2.6 to the linear system As = b k preconditioned with the LMP (2.38). We name this approach Approach (F).

The same strategy for solving a convergent sequence of linear systems (3.1) with Approach (B) can be used to solve these linear systems by Approach (F). This yields Algorithm 3.2 in which the inner minimization is performed by Approach (F).

The characteristics of the Lanczos version of Algorithm 3.2 summarized in Section 3.1 (for instance, matrix-vector products in particular case P 0 = B) remain the same as those of Algorithm 3.5. For further details we refer to Section 3.1.

We are left with explaining the third preconditioning approach which avoids matrixvector products with B -1 as well as the factorization of the preconditioner by using Algorithm PCGIF 3.4. We next introduce the Lanczos version of Approach (C) presented in Section 3.1.

Preconditioning with PLanczos Inverse Free Algorithm

Remember that the idea in this approach is to apply the symmetric and positive definite matrix P formulated by (2.38) as a right symmetric preconditioner while solving the linear system (3.1). Under the assumption that the matrix B -1 P is available, this strategy yields solving the linear system (3.14), which is given by

equipped with the B-inner product for the variable s by using the LMP matrix C formulated by (3.19). The approximate solution is recovered from s = Bs. This results in the PLanczos Inverse Free (PLanczosIF) algorithm (Algorithm 5.1) that avoids matrix-vector multiplication with B -1 . We name this approach Approach (G).

Algorithm 5.1 can be transformed into a cheaper form by introducing an additional vector as z i = Bz i , reducing its cost per loop to a single matrix-vector product with B. Substituting this definition into Algorithm 5.1, yields the final version of PLanczosIF Algorithm (Algorithm 5.2).

Therefore, as a third alternative to preconditioning, the linear system (3.14) can be solved by Approach (G). This approach can be applied on a convergent sequence of linear systems (3.1) by using Algorithm 3.5.

The characteristics of the Lanczos version of Algorithm 3.5 summarized in Section 3.1 remain the same as those of Algorithm 3.5. So, this algorithm requires neither Algorithm 5.1: PLanczosIF (version 1)

Check convergence and stop if desired accuracy is reached

Check convergence and stop if desired accuracy is reached

Gürol -Solving regularized nonlinear least-squares problem in dual space 116 the factorization of the preconditioner nor matrix-vector products with B -1 under the assumption that C 0 = B -1 P 0 is available.

Summary

We have presented different Lanczos algorithms for the solution of the linear system (3.1) with the LMP given by (2.38) in the context of solving a convergent sequence of linear systems with multiple right-hand sides. These algorithms differ in the assumption they made on the preconditioner form and matrix-vector products. They are listed below:

• Approach (E): Apply Lanczos Algorithm 2.3 to the linear system (3.7) of the form of A s = b k with a preconditioner constructed from (3.2),

• Approach (F): Apply Lanczos Algorithm 2.6 to the linear system (3.1) of the form of As = b k with a preconditioner constructed from (2.38),

• Approach (G): Apply PLanczosIF Algorithm 5.2 to the linear system (3.14) of the form of A s = b k with a preconditioner constructed from (3.19).

As in the variants of the conjugate gradients method, the iterates of Lanczos Algorithm 2.3 are related to the iterates of PLanczos Algorithm 2.6 with (3.33). Under the assumption that P = BC, the iterates of PLanczosIF Algorithm 3.4 are related to the iterates of PLanczos Algorithm 2.5 with (3.36). These three variants of the Lanczos algorithms search for an approximate solution s in a subspace s 0 + K(P r 0 , P A) with K(P r 0 , P A) = span P r 0 , (P A)P r 0 , . . . , (P A) l-1 P r 0

(5.2)

where r 0 = As 0b is the initial residual and is the number of inner iterations. Note that these approaches are mathematically equivalent to the approaches (A), (B) and (C) described in Section 3.1. The main properties of these approaches in terms of preconditioning are listed in Section 3.1.

These three approaches can be used within Algorithms 3.1, 3.2 and 3.5 for solving a sequence of linear systems with multiple right hand sides. In this case, the characteristics of these algorithms are listed in Considering the particular case that the initial preconditioner is chosen as P 0 = B when solving a sequence of linear systems, the required matrix-vector products for each Algorithm 5.3: PLanczosIF2PCGIF

holds where P is the preconditioner for the primal approach.

Let us define the vectors in R m such that z 0 = D r 0 , (5.5)

)

(5.7)

From the definitions (5.3) -(5.7), it is possible to derive the following relations from PLanczos:

)

(5.9)

(5.10)

(5.11) Equation ( 5.11) can be written as (5.12) where

We now prove by induction that the relations given by (5.9), (5.10) and (5.12), also hold for i > 1. First, assume that the relations are satisfied for a given i:

(5.13)

(5.15)

We show that the relations hold for i + 1. From lines 15 and 16 of Algorithm 2.6 for PLanczos, we have, using Eqs (5.15) and (5.4), (5.16)

(5.17) Defining

Algorithm 5.5: RPLanczos Algorithm

Algorithm 5.6: RPLanczos Algorithm (version for s 0 = 0)

Therefore, this algorithm can be applied to a convergent sequence of linear systems (4.22) by using Algorithm 4.4 whose inner loop is performed by RPLanczos Algorithm 5.6, and where the preconditioning is achieved by the LMP D formulated by (4.26).

As a result, the Lanczos version of Algorithm 4.4 generates a mathematically equivalent iterates of {s k } to those of the Lanczos version of Algorithm 3.2 under the assumptions that P 0 H T = BH T D 0 and x 1 = x c , while possibly yielding gains in terms of both memory usage and computational cost when m n. We summarize the characteristics of the equivalent primal and dual Lanczos algorithms by Table 5

Table 5.5: A summary of the characteristics of Algorithms 3.2 and 4.4. Approach (E) is defined in Section 3.1.

We next explain how to construct particular LMPs: the quasi-Newton LMP, the Ritz LMP and the spectral LMP described in Chapter 4.3 in the context of Algorithm 4.4 when used with RPLanczos. This part will include for each preconditioner an extraction methodology of the required information from the RPLanczos algorithm.

Algorithm 5.8: RPLanczos Algorithm (version for quasi-Newton LMP)

Check convergence and stop if desired accuracy is reached

Therefore, when solving the system of linear equations (4.22) of the form of A s = b k with RPLanczos Algorithm 5.8 (at the k-th outer loop of Algorithm 4.4 with a constant A), the quasi-Newton LMP for the system matrix A can be constructed from the Algorithm 5.9: Augmented RPLanczos Algorithm

Inserting this algorithm in a Gauss Newton method in the framework of trust region yields Algorithm 6.3. This is the algorithm which we recommend for solving truly nonlinear instances of our original problem (2.3) when m n and under Assumption 4.3.

Algorithm 6.3: Gauss-Newton trust region algorithm for dual approach 1 Initialization: Choose an initial point x 0 and an initial trust region radius ∆ 0 . Choose a first level preconditioner D 0 such that Assumption 4.3 holds. Choose an initial estimate s 0 . Choose the constants η 1 , η 2 , γ 1 , γ 2 that satisfy

Set k = 0 and calculate the function value at the initial, f (x 0 ).

2 Calculation of s k approximately: Solve the subproblem (2.46) by using Steihaug-Toint CG Algorithm 6.2.

3 Acceptance of the trial point: Compute the ratio

4 Trust-region radius update:

Increment k by 1 and go to step 2.

When an LMP defined in Lemma 4.7 used in Algorithm 6.3 with a symmetry measure as explained in Section 4.4.2, it is clear that efficient preconditioning depends on the choice of threshold value. When this value is not chosen properly, the conjugate gradient method may be in trouble with a preconditioner that is not symmetric and positive definite. Moreover, to be used in the trust region framework, the trust region may not be defined since the norm is defined by the preconditioner. Therefore, in this case we could not use Algorithm 6.3 directly. We need to adapt this algorithm as detailed in the next section.

Flexible truncated conjugate gradients

In this section we adapt the Steihaug-Toint truncated conjugate gradients for the dual approach given by Algorithm 6.2 when the trust region could not be defined with the standard approach, since the norm is not defined. As we mentioned this may happen when the preconditioner that is inherited from previous iteration may not be symmetric in the current inner product and may not be positive-defnite in the full dual space. Algorithm 6.5: Flexible trust region algorithm for dual approach 1 Initialization: Choose an initial point x 0 and an initial trust region radius ∆ 0 .

Choose a first level preconditioner D 0 such that Assumption 4.3 holds. Choose an initial estimate s 0 . Choose the constants η 1 , η 2 , γ 1 , γ 2 that satisfy

Set k = 0 and calculate the function value at the initial, f (x 0 ).

2 Norm definition: Define • k .

3 Preconditioner construction: Decide which pairs to use according to the success of the previous inner loop 4 Calculation of s k approximately: Solve the subproblem (2.46) by using flexible conjugate gradients Algorithm 6.4. During Algorithm 6.4 extract and save relevant information to precondition the next linear system 5 Acceptance of the step beyond the Cauchy step: If

6 Acceptance of the trial point: Compute the ratio

7 Trust-region radius update:

Increment k by 1 and go to step 2.

Note that Algorithm 6.5 requires the function evaluation both at x C k and x CG k . Note also that in Algorithm 6.5, finite number of iterations are allowed. Convergence theory of trust region with magical steps is given in (Conn, Gould and Toint, 2000, p. 388-391).

Algorithm 1: BLanczos with re-orthogonalization

BLanczos and RBLanczos without re-orthogonalization

This appendix provides the direct versions of BLanczos and RBLanczos which can be used instead of Algorithms 1 and 4 to reduce memory requirements when re-orthogonalization is not desired.

The direct BLanczos algorithm is given in Algorithm 5. The direct RBLanczos algorithm can be derived following the basic approach used to derive RBLanczos Algorithm 4 from BLanczos Algorithm 1. This is done by using the relations between the vectors defined in R n and R m given by ( 30)-( 32) and the additional equations

)

that relate m-dimensional vectors p i and λ i with the vectors p i and δu i in Algorithm 5, for i ≥ 0. It can be shown that relations (A1) and (A2) are satisfied by choosing

where p 0 = λ 0 = 0 and the scalars β i , η i and ζ i are defined as

for i > 1. Direct substitution of these recurrence relations in BLanczos Algorithm 5 yields Algorithm 6.

Algorithm 5: BLanczos without re-orthogonalization

Relationships between vectors in RBLanczos and BLanczos

Following GT09 and Gratton et al. (2009) in their derivation of RPCG, we suppose that r 0 ∈ range(G T ) which is satisfied under the assumption that the initial guess δu 0 = 0. In particular,

where

Algorithm 6: RBLanczos without re-orthogonalization

where β 0 = r T 0 t 0 . From the definitions (B1)-(B4), it is possible to derive the following relations from BLanczos:

)

)

)

)

Equations ( B7)-(B9) can be written as q 1 = G T q 1 , (B10)

)

where q 1 = v 1 + R -1 z 1 , w 1 = q 1α 1 v 1 , α 1 = q T 1 z 1 and t 1 = GBG T w 1 .

We now prove by induction that the relations given by (B5)-( B6) and (B10)-(B12) also hold for i > 1. First, assume that the relations are satisfied for a given i:

We now show that the relations hold for i + 1. From lines 17 and 18 of Algorithm 1 for BLanczos, we have, using (B14) and (B15),

Defining

we obtain from (B16) and (B17) that

)

From line 13 of Algorithm 1 and using the relations (B13), (B18) and (B19), we have

where

Similarly, from line 13 of Algorithm 1 and using the relations (B18) and (B20), we have

where

which completes the proof.