Kimon Antonakopoulos

Lipschitz Continuous Operators * | F t ] ≤ σ 2

A B S T R A C T

S everal important problems in learning theory and data science involve high- dimensional optimization objectives that transcend the Lipschitz regularity conditions that are standard in the field. This absence of Lipschitz regularity -smoothness or continuity -poses significant challenges to the convergence analysis of most optimization algorithms and, in many cases, it requires the introduction of novel analytical and algorithmic tools. In this thesis, we aim to partially fill this gap via the design and analysis of universal first-order methods in two general optimization frameworks: (a) online convex optimization (which contains as special cases deterministic and stochastic convex optimization problems); and (b) abstract variational inequalities (which contain as special cases min-max problems and games) both without global Lipschitz continuity/smoothness conditions.

In this "NoLips" setting, we take a geometric approach -Riemannian, Finslerian, or Bregman-based -that allows us to handle vector fields and functions whose norm or variation becomes infinite at the boundary of the problem's domain. Using these non-Euclidean surrogates for Lipschitz continuity and smoothness, we propose a range of adaptive first-order methods that concurrently achieve order-optimal convergence rates in different problem classes, without any prior knowledge of the class or the problem's (relative) smoothness parameters. These methods are based on a suitable mirror descent or mirror-prox template (for convex minimization and monotone variational inequalities respectively), and they revolve around adaptive step-size policies that exploit the geometry of the gradient data observed at earlier iterations to perform more informative (extra-)gradient steps in later ones. Our results do not always coincide with what one would expect in standard Lipschitz problems, and serve to further highlight the differences between the "Lispschitz" and "NoLips" frameworks.

vii R E S U M E P lusieurs problèmes importants issus de l'apprentissage statistique et de la sci- ence des données concernen des objectifs d'optimisation à très haute dimension qui vont au delà des hypothèses de régularité de Lipschitziennes. L'absence de régularité de Lipschitz -continuité ou lissitude -pose des défis importants à l'analyse de convergence des algorithmes existants d'optimisation et nécessite souvent de nouveaux outils analytiques et algorithmiques pour être traitée de manière efficace. Dans cette thèse, nous visons à combler partiellement cette lacune en proposant de nouvelles méthodes universelles du premier ordre dans deux cadres généraux : (a) l'optimisation convexe en ligne (qui contient comme cas particuliers les problèmes d'optimisation convexe déterministes et stochastiques) ; et (b) les inégalités variationnelles (qui contiennent comme cas particulier les problèmes de point-selle et les jeux).

Nous étudions ces deux problèmes génériques dans un "NoLips" et nous adoptons une approche géométrique -Riemannienne ou Bregmanienne -qui nous permet de traiter des champs vectoriels et/ou des fonctions dont la norme ou la variation explose vers le bord du domaine du problème. En utilisant ces substituts non-euclidiens, nous proposons des nouvelles méthodes adaptatives du premier ordre qui atteignent simultanément des taux de convergence optimaux dans différentes classes de problèmes, sans aucune connaissance préalable des paramètres de régularité du problème. Nos méthodes sont basées sur un template "mirror descent" ou "mirror-prox" (pour les problèmes de minimisation convexe et les inégalités variationnelles monotones respectivement), et elles se basent sur des politiques adaptatives de pas qui exploitent l'historique des gradients observés afin d'effectuer des pas de gradient mieux adaptés à la régularité du problème. Nos résultats ne coïncident pas toujours avec ce que l'on attendrait dans le cadre Lipschitz, et ainsi apportent une intuition très utile pour comprendre les différences fondamentales entre les problèmes "Lispschitz" et "NoLips". F irst of all, I would like to warmly thank my advisor Panayotis Mertikopoulos, without him this thesis (literally) wouldn't have never been written. During the many ups and downs of this journey, he was always the voice of reason in this madness, helping me out to focus on the important at any given moment. Whenever I despaired with a proof or a calculation (and happened very often), Panayotis was always there to lend a helping hand and a word of support; he would always identify the essence and the correct path to proceed. But, above all else, Panayotis has been a friend, a wise friend who gave advice (very) carefully but freely, and on whose experience and kindness I could always draw when needed. His door was always open to discuss and to empathize my desperation at the (endlessly) many dead ends. For these reasons (and many more) I am truly indebted to Panayotis and I can only hope that it does not end with this thesis.

Moreover, I would like to warmly thank my co-advisor Veronica Belmega who did also her fair part to make the write-up of this document possible. She literally was the enthusiastic voice which, even at the most difficult times, made always the glass to seem half full; and for this I truly grateful.

Of course, this document would never have existed without the great effort and diligence of all the members of my committee. Therefore, I would first like to thank professors Marc Teboulle and Walid Hachem for their extremely detailed reviews (especially under these tight time constraints) which helped to improve the rigour of this document significantly. Moreover, I thank each member of my committee, Roberto Cominetti, Niao He and Jerome Malick for the fruitful discussion during my defence and of course for their time and effort to read this thesis.

Last but not least, I would like to thank all the people who put up with me during this long journey: my parents without their sacrifices nothing would have been possible, my friends (Eleni, Giannis) for their support and my girlfriend Archontia for tolerating me all these years. financial support. The author of this thesis gratefully acknowledges financial support by the French National Research Agency (ANR) in the framework of the "Investissements d'avenir" program (ANR-15-IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19-P3IA-0003), and the grants ORACLESS (ANR-16-CE33-0004) and ALIAS (ANR-19-CE48-0018-01). This research was also supported by the COST Action CA16228 "European Network for Game Theory" (GAMENET). part i the role of regularity in optimization T he rise of machine learning protocols has reaffirmed the interest in the theory of optimization problems. To that end, two important settings stand out, that of online convex optimization and (monotone) variational inequality problems.

The first framework refers to a scenario where the optimizer faces a (possibly adversarial) sequence of time-varying loss functions f t , t = 1, 2, . . . , one at a time -for instance, when drawing different sample points from a large training set [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multi-armed bandit problems[END_REF][START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF]. Specifically, if the optimizer faces a sequence of G-Lipschitz convex losses, the incurred min-max regret, a standard performance criterion that will be discussed in detail later, is Ω(GT 1/2 ) after T rounds and this bound can be achieved by inexpensive first-order methods -such as online mirror descent and its variants [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multi-armed bandit problems[END_REF][START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF][START_REF] Shalev | Convex repeated games and Fenchel duality[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF].

This setting properly includes (static) convex minimization problems, but the situation in this case changes dramatically. The analysis of static minimization problems typically revolves around two main regularity conditions for the problem at hand: (a) Lipschitz continuity of the problem's objective function and/or (b) Lipschitz continuity of its gradient (also referred to as Lipschitz smoothness). Depending on which of these conditions holds, the lower bounds for first-order methods with perfect gradient input are Θ(1/ √ T) and Θ(1/T 2 ) after T gradient queries, and they are achieved by gradient descent and Nesterov's fast gradient algorithm respectively [START_REF] Nesterov | A method for unconstrained convex minimization problem with the rate of convergence O(1/k 2 )[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. By contrast, if the optimizer only has access to stochastic gradients (as is often the case in machine learning and distributed control), the corresponding lower bound is Θ(1/ √ T) for both problem classes [START_REF] Bubeck | Convex optimization: Algorithms and complexity[END_REF][START_REF] Semen | Problem Complexity and Method Efficiency in Optimization[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

On the other hand, the surge of recent breakthroughs in generative adversarial networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF], robust reinforcement learning [START_REF] Pinto | Robust adversarial reinforcement learning[END_REF], and other adversarial learning models [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] has sparked renewed interest in the theory of min-max optimization problems and games. In this broad setting, it has become empirically clear that, ceteris paribus, the simultaneous training of two (or more) antagonistic models faces drastically new challenges relative to the training of a single one. Perhaps the most prominent of these challenges is the appearance of cycles and recurrent (or even chaotic) behavior in min-max games. This has been studied extensively in the context of learning in bilinear games, in both continuous [START_REF] Flokas | Poincaré recurrence, cycles and spurious equilibria in gradient-descent-ascent for non-convex non-concave zero-sum games[END_REF][START_REF] Mertikopoulos | Cycles in adversarial regularized learning[END_REF][START_REF] Piliouras | Optimization despite chaos: Convex relaxations to complex limit sets via Poincaré recurrence[END_REF] and discrete time [START_REF] Daskalakis | Training GANs with optimism[END_REF][START_REF] Gauthier Gidel | A variational inequality perspective on generative adversarial networks[END_REF][START_REF] Gauthier Gidel | Negative momentum for improved game dynamics[END_REF][START_REF] Mertikopoulos | Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile[END_REF], and the methods proposed to overcome recurrence typically focus on mitigating the rotational component of min-max games.

The method with the richest history in this context is the extra-gradient (EG) algorithm of [START_REF] Korpelevich | The extragradient method for finding saddle points and other problems[END_REF] and its variants. The EG algorithm exploits the Lipschitz smoothness of the problem and, if coupled with a Polyak-Ruppert averaging scheme, it achieves an O(1/T) rate of convergence in smooth, convexconcave min-max problems [START_REF] Semen | Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convexconcave saddle point problems[END_REF]. This rate is known to be tight [START_REF] Semen | Information-based complexity of linear operator equations[END_REF][START_REF] Ouyang | Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems[END_REF] but, in order to achieve it, the original method requires the problem's Lipschitz constant to be known in advance. If the problem is not Lipschitz smooth (or the algorithm is run with a vanishing step-size schedule), the method's rate of convergence drops to O(1/ √ T).

From the above, one may directly observe that from a practical perspective the challenging part in order to apply the respective optimal solution method to the problem at hand is to able to identify which regularity condition and/or oracle feedback she has at hand. Therefore, a question that naturally arises in this context is the following:

Is it possible to design methods that simultaneously achieve optimal convergence rates without any prior knowledge of the problem's regularity features ?

The positive answer to the above question gives rise to the so-called adaptive methods.

In its general context, adaptivity of a method may refer to (at least) two different things:

1. Automatic adjustment to the function's regularity parameters within a fixed problem class (Lipschitz continuous, Lipschitz smooth, etc.). In what follows we treat both questions in tandem.

related work minimization problems.

There is an extensive corpus of literature concerning the convex minimization framework. To name out the methods of [START_REF] Hu | Accelerated gradient methods for stochastic optimization and online learning[END_REF] and [START_REF] Lan | An optimal method for stochastic composite optimization[END_REF] successfully interpolate between the stochastic and smooth deterministic regimes achieving a O(1/ √ T) convergence rate for the former and an O(1/T 2 ) rate for the latter; however, their interpolation guarantees require prior knowledge of the function's smoothness parameter. More recently, [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] proposed a method that adjusts automatically to the Lipschitz (or Hölder) modulus of the function based on line-search queries of the objective1 ; in the Lipschitz smooth case, the method of Nesterov [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF] attains an accelerated rate of convergence of the. order O(1/T 2 ). However, in order to establish an implementable stopping criterion, said method requires as an input parameter an estimate of the distance between the algorithm's initial state to the problem's solution set (i.e., this upper bound should be known to the optimizer a priori).

Such an estimate is difficult to come by in problems with unbounded domains, so the performance of the method is unclear in this case.

By contrast, the AcceleGrad method of [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF] and the more recent UnixGrad algorithm of [START_REF] Kavis | UnixGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization[END_REF] successfully interpolate between the O(1/ √ T) and O(1/T 2 )

rates for the Lipschitz continuous and/or stochastic settings and smooth regimes respectively without requiring a line search -but the boundedness caveat is still present. Finally, beyond the minimization framework, [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] proposed a universal mirror-prox method for solving (stochastic) variational inequalities, with or without smoothness requirements. When applied to function minimization, the algorithm of [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] interpolates between the O(1/ √ T) and an unaccelerated O(1/T) rate. What makes this particularly interesting for our purposes is that this scheme does not require vanishing gradients near a minimizer. variational inequalities. There have been several works focusing on the convergence guarantees of the original EG / mirror-prox (MP) template. We review the most relevant of these works below. In unconstrained problems with an operator that is locally Lipschitz continuous (but not necessarily globally so), the golden ratio algorithm (Graal) of [START_REF] Malitsky | Golden ratio algorithms for variational inequalities[END_REF] achieves convergence without requiring prior knowledge of the problem's Lipschitz parameter. However, Graal provides no rate guarantees for non-smooth problems -and hence, a fortiori, no interpolation guarantees either. By contrast, such guarantees are provided in problems with a bounded domain by the generalized mirror-prox (GMP) algorithm of [START_REF] Stonyakin | Generalized mirror prox for monotone variational inequalities: Universality and inexact oracle[END_REF] under the umbrella of Hölder continuity.

Another method that simultaneously achieves an O(1/ √ T) rate in non-smooth problems and an O(1/T) rate in smooth ones is the recent algorithm of [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF]. The BL algorithm employs an adaptive, AdaGrad-like step-size policy which allows the method to interpolate between the two regimes -and this, even with noisy gradient feedback. On the negative side, the BL algorithm requires a bounded domain with a (Bregman) diameter that is known in advance; as a result, its theoretical guarantees do not apply to unbounded problems. beyond lipschitz regularity. Despite the fact that the (Euclidean-based) Lipschitz regularity conditions appear quite generic there exists a whole set of real life situations where both of these conditions fail, either because the loss profile of the problem grows too rapidly (e.g., as in support vector machines or GAN models with Kullback-Leibler losses), or because the problem exhibits singularities near the boundary of the feasible region (e.g., as in resource allocation and inverse problems). A prominent example that will serve as motivation for the NoLips setting is that of Poisson Inverse Problems. We examine this in detail below.

Example 1.1 (Poisson Inverse Problems). Poisson inverse problem (PIP) arise in various practical problems stemming from image sciences and machine learning problems. Informally, this consists of two components: a matrix A ∈ R m×n which models the experimental protocol and a vector b ∈ R m + represents the measurements made by the optimizer. With all this in hand, the objective would be to recover the signal or image x ∈ R n + from the noisy measurements b such that:

Ax b (1.1)

A natural measure that evaluates the proximity of these two vectors is that of the Kullback-Leibler (KL) divergence. Namely, we are facing the following convex minimization problem:

minimize d(b, Ax) = m ∑ i=1 b i log b i (Ax) i + (Ax) i -b i subject to x ∈ R n + (1.2)
As one may recognize the above minimization objective is neither Lipschitz contiuous nor smooth due to the singular behaviour of the logarithm near the origin.

The above schemes all rely intrinsically on Lipschitz/Hölder continuity and/or smoothness. Achieving convergence beyond the Lipschitz framework has been the focal point of a recent strand in the literature, starting with the work of [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] and the concurrent paper of [START_REF] Lu | Relatively-smooth convex optimization by first-order methods and applications[END_REF]. More recent works have provided different extensions to non-convex [START_REF] Bolte | First order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems[END_REF] and stochastic optimization [START_REF] Hanzely | Fastest rates for stochastic mirror descent methods[END_REF], including a tentative path towards acceleration [START_REF] Hanzely | Accelerated Bregman proximal gradient methods for relatively smooth convex optimization[END_REF]; however, these methods are neither universal nor adaptive.

In more detail Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] introduced a "Lipschitz-like" smoothness condition for convex minimization problems and used it to establish a O(1/T) value convergence rate for mirror descent methods (as opposed to mirror-prox). Always in the context of loss minimization problems, Bolte et al. [START_REF] Bolte | First order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems[END_REF] subsequently extended the results of Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] to non-convex problems that satisfy the Kurdyka-Łojasiewicz (KL) inequality, while Lu et al. [START_REF] Lu | Relatively-smooth convex optimization by first-order methods and applications[END_REF] considered functions that are also relatively strongly convex and showed that mirror descent achieves a geometric convergence rate in this context.

The condition of Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] is remarkably simple as it only posits that the problem's loss function f is such that :

βh -f is convex (RS)
for some reference Bregman function h and some β > 0. A straightforward extension of this condition to an operator setting would be to require the monotonicity of β∇h -A, where A is the operator defining the variational inequality under study. However, the cornerstone of this "Lipschitz-like" condition is a descent lemma which does not carry over to variational inequalities, so it does not seem possible to extend the analysis of Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] to an operator setting at least not directly.

Insofar as Lipschitz continuity of the objective is concerned, Lu [START_REF] Lu | Relative-continuity" for non-Lipschitz non-smooth convex optimization using stochastic (or deterministic) mirror descent[END_REF] also considered a "relative continuity" condition for loss minimization problems positing that

∇ f (x) ≤ G inf x 2D(x , x)/ x -x (1.3)
(where f is the problem's objective and D is the Bregman divergence of h). Written this way, the condition of Lu [START_REF] Lu | Relative-continuity" for non-Lipschitz non-smooth convex optimization using stochastic (or deterministic) mirror descent[END_REF] can also be extended to an operator setting, but this would provide a surrogate for operator boundedness, not Lipschitz continuity (since A = ∇ f in minimization problems). Extending the above definition Zhou et al. [START_REF] Zhou | Regret bounds without Lipschitz continuity: Online learning with relative Lipschitz losses[END_REF] proposed a similar notion, i.e.,

∇ f (x), x -x ≤ G 2D(x , x) (RC)
and applied it for the context of online convex optimization problems. Finally, in Teboulle [START_REF] Teboulle | A simplified view of first order methods for optimization[END_REF] the notion of W[h]-continuity is proposed by singling out particular properties of Bregman divergences; formally, given an appropriate regularizer h an operator A is called to be W[h]continuous:

t A(x), x -x -D(x , x) ≤ t 2 2 G 2 for all x ∈ dom h, x ∈ dom ∂h. (W)
In the sequel, we shall introduce an alternative way that will allow us to extend the Lipschitz continuity conditions in a unified manner for both minimization and (VI) problems.

main objective and contributions of this thesis

In view of the above, the objective of this thesis is twofold:

1. Introduce novel regularity conditions, which are able to include variational inequality problems whose associated operator exhibits a "singular" behaviour.

2.

Bridge the gap between the development of general Lipschitz continuity conditions on the one hand and the lack of respective adaptive methods on the other.

Tackling each objective separately, we begin by introducing two novel classes of operators. In particular inspired by the idea that Lipschitz continuity is first and foremost a metric space property we use the notion of local norms extensively as a primal geometrical tool in order to capture finer geometrical aspects of the problem. More precisely, in contrast to the traditional setting, local norms dependent on the point where it is evaluated, i.e., we have a continuous assignment • x for all x ∈ X . This in turn defines the associated dual norm in the standard way, i.e., for all w ∈ V * , w x, * = max{ w, x :

x x = 1} (1.4)
Armed with this geometry-aware local norm machinery we revisit the Euclidean based regularity conditions. In particular, we define two new operator classes that of metrical boundedness and metrical smoothness (see [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF][START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Formally, an operator A is called metrically bounded when:

A(x) x, * ≤ G (MB)
and metrically smooth whenever the following inequality holds:

A(x) -A(x ) x, * ≤ β x -x x (MS)
In this context, the adaptivity results evolve throughout this thesis gradually. More precisely, our contributions can be summarized as follows:

• We begin gently by investigating online convex optimization (OCO) problems by recovering optimal regret minimization upper bounds under (MB).

• We proceed by taking a closer look at static/ stochastic convex minimization problems. More precisely we establish optimal interpolation guarantees for both stochastic and/or deterministic oracle feedback under the blanket assumptions of (RS) and (RC).

• In the last part of this thesis, we focus on the generic framework of variational inequalities. To that end, for this setting we provide convergence rates starting from non-adaptive to adaptive to the "Lipschitz"-like modulus and finally regime-agnostic order optimal interpolation guarantees for both deterministic and stochastic (VI)'s under (MB) and/or (MS).

In what follows we present the content of each chapter in a more detailed manner.

diagrammatic outline

This thesis consists of two parts. In Part i the general theoretical setup is presented, while Part ii examines the particular algorithmic guarantees achieved in each setting. We now provide a quick overview of the content of each chapter individually.

• Chapter 2 contains the main ingredients of this thesis; the particular problem Preliminaries set-ups along with the state of the art first order methods and the respective convergence rate guarantees. An important part of this chapter is devoted to the pivotal role that Lipschitz continuity plays in all these optimization scenarios.

• Chapter 3 introduces and examines in detail the NoLips conditions discussed NoLips above. In doing, we distinguish our presentation for the different optmization frameworks.

• Chapter 4 provides concrete definitions of the main algorithmic schemes

Bregman Methods which will be of interest throughout the sequel. More precisely, we start with the basic mathematical toolkit of Bregman divergences which serves as the key ingredient for generalizing the standard Euclidean based projection operators. Based on this machinery, we describe a set of Bregman driven iterative methods for both optimization scenarios.

• Chapter 5 Motivated by applications to machine learning and imaging sci-

Online Convex Optimization

ence, we study a class of online and stochastic optimization problems with loss functions that are not Lipschitz continuous; in particular, the loss functions encountered by the optimizer could exhibit gradient singularities or be singular themselves. Drawing on tools and techniques from Finsler geometry, we examine the (MB) continuity condition which is tailored to the singularity landscape of the problem's loss functions. In this way, we are able to tackle cases beyond the Lipschitz framework provided by a global norm, and we derive optimal regret bounds and last iterate convergence results through the use of regularized learning methods (such as online mirror descent).

• Chapter 6 We propose a new family of adaptive first-order methods for a class

Convex Optimization

of convex minimization problems that may fail to be Lipschitz continuous or smooth in the standard sense. Specifically, we consider problems that are continuous or smooth relative to a reference Bregman function -as opposed to a global, ambient norm (Euclidean or otherwise). In this setting, the application of existing order-optimal adaptive methods -like UniXGrad or AcceleGrad-is not possible, especially in the presence of randomness and uncertainty. The proposed method, adaptive mirror descent (AdaMir), aims to close this gap by concurrently achieving min-max optimal rates in problems that are relatively continuous or smooth, including stochastic ones.

• Chapter 7 We present a new family of min-max optimization algorithms that Variational Inequalities automatically exploit the geometry of the gradient data observed at earlier iterations to perform more informative extra-gradient steps in later ones.

Thanks to this adaptation mechanism, our proposed method, adaptive mirrorprox (AdaProx) automatically detects whether the problem is smooth or not, without requiring any prior tuning by the optimizer. As a result, AdaProx simultaneously achieves order-optimal convergence rates, i.e., it converges with a rate of O(1/T) iterations in smooth problems, and O(1/ √ T) in nonsmooth ones. Importantly, these guarantees do not require any of the standard boundedness or Lipschitz continuity conditions that are typically assumed in the literature; in particular, they apply even to problems with singularities (such as resource allocation problems and the like). This adaptation is achieved through the use of a geometric apparatus based on Finsler metrics and a suitably chosen mirror-prox template that allows us to derive sharp convergence rates for the methods at hand.

Moving forward, we finally illustrate the full potential of our results. Namely, by employing the dual extrapolation (DualX) template run with a similar adaptive learning as is AdaProx, we are able to show optimal convergence rates for both deterministic and stochastic oracles and smooth and nonsmooth settings.

notational conventions

Throughout the sequel, V ∼ = R n will denote an n-dimensional space with norm • and V * will denote its (algebraic) dual. We will also write w, x for the canonical pairing between w ∈ V * and x ∈ V, and w * ≡ max{ w, x : x ≤ 1} for the associated dual norm on V * . We also use the notation O(•) to dismiss logarithmic factors.

T he main objective of this introductory chapter is to present the basic concepts of two general optimization scenarios: a) the time-varying setting of online convex optimization (OCO); and b) the operator-based setting of variational inequalities (VIs). In both frameworks, we seek to briefly review the main definitions, applications, and state-of-the-art solution methods.

To begin with, the online convex optimization setting -presented in detail in Section 2.1 -concerns decision-making processes that unfold in an otherwise unknown and time-varying environment. More precisely, the optimizer is assumed to be facing a sequence of convex losses f t which evolves from round to round, possibly in an adversarial manner. This framework properly includes as special cases the class of convex minimization problems, deterministic and/or stochastic; these problems will be of individual interest throughout as well.

Moving forward, Section 2.2 provides a detailed description of an optimization framework that goes beyond ordinary minimization problems -the general setting of variational inequalities. This setup serves as a unifying framework for various "convex-structured" optimization problems so, in addition to standard minimization problems, it allows us to put under the same umbrella cases such as saddle-point, fixed-point and Nash equilibrium problems.

Having described these two settings of interest, in Section 2.3 we discuss two generic regularity conditions -boundedness and Lipschitz continuity of the defining operators of each problem class. Subsequently, in Section 2.4 we present the general framework of first-order methods which will be our main candidate solution methods. Moreover, we illustrate how the performance of these methods is influenced under each specific regularity condition, by providing "worst-case" optimal lower bounds. Finally, in Section 2.5 and Section 2.6 we present the stateof-the-art first-order methods that match the optimal lower bounds along with their adaptive counterparts.

online convex optimization

Problem setup and examples

We begin by presenting the core protocol of online convex optimization (OCO), i.e., when the optimizer faces a sequence of time-varying loss functions f t , t = 1, 2, . . . , one at a time. Formally, this can be described by the following sequence of events:

1. At each round t = 1, 2, . . . , the optimizer chooses an action X t from a convex -but not necessarily closed or compact -subset X of an ambient normed space V ∼ = R n .

2. The optimizer incurs a loss f t (X t ) based on some (a priori unknown) loss function f t : X → (-∞, +∞] which is assumed to be proper, lower semicontinuous (l.s.c.) and convex.

3. The optimizer updates their action and the process repeats.

Online Convex Optimization Protocol

This broad setting captures a wide range of convex problems, for instance, when drawing different sample points from a large training set [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multi-armed bandit problems[END_REF][START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF]. To that end, we distinguish below two iconic examples of OCO problems which are going to be of individual interest in the sequel.

Example 2.1 (Static convex minimization). Consider a convex minimization problem of the general form:

Convex Minimization minimize f (x) subject to x ∈ X (Opt)
where f : X → R is a convex function. The notion of "stationarity" refers here to the fact that (Opt) is obtained by the online protocol by assuming that the optimizer faces at each round the same convex loss function, i.e., f t = f .

Example 2.2 (Stochastic convex minimization)

. A variant of (Opt) with important applications to machine learning, distributed control and data science is the socalled stochastic optimization problem:

Stochastic Convex Minimization minimize f (x) = E [F(x; ω)] subject to x ∈ X . (StochOpt)
where F : X × Ω → R is a stochastic objective defined over a (complete) probability space (Ω, F , P) and F(•; ω) is assumed convex for all ω ∈ Ω. Clearly, (StochOpt) can be seen as a special case of an OCO protocol where the optimizer faces at each round the convex loss function f t = F(•; ω t ) with ω t drawn i.i.d. from Ω at each round.

Performance evaluation and merit functions

The most widely used figure of merit in OCO problems is the optimizer's regret.

Intuitively, this notion compares the average loss incurred by the agent to the minimum loss they could have incurred in hindsight by playing a fixed x ∈ X . Formally, the regret of a policy X t ∈ X , t = 1, 2, . . . , against a "benchmark action"

x ∈ X is defined as

Regret Reg x (T) = T ∑ t=1 [ f t (X t ) -f t (x)] (2.1)
and we define the optimizer's static (or external) regret (without any benchmark quantifiers) as

Reg(T) = sup x∈X Reg x (T) = sup x∈X T ∑ t=1 [ f t (X t ) -f t (x)]. (2.2)
With all this in hand, a natural property that the optimizer would like to attain is for their regret to remain "small" over time; this amounts to the requirement:

Reg x (T) = o(T) for all x ∈ X . (2.3)
This, in turn, yields that on average the cumulative loss compared to the best

No-Regret action in hindsight becomes asymptotically non-positive.

For concreteness, we discuss below the implications of attaining no regret in the special cases of static and stochastic minimization problems discussed above. To begin with, if the optimizer is facing (Opt) while deploying an iterative method generating the sequence of actions X t , t = 1, 2, . . . , the regret given by (2.2) becomes

Regret Conversion

Reg(T) = Reg x * (T) = T ∑ t=1 f (X t ) -T f (x * ) (2.4)
with x * ∈ arg min x∈X f (assumed here to be nonempty). Now, since f is assumed to be convex, Jensen's inequality shows that the performance (in terms of function values) of the time-averaged sequence

X T = 1 T T ∑ t=1 X t (2.5)
is bounded by the optimizer's regret as

Optimality Gap f (X T ) -f (x * ) ≤ Reg(T) T (2.6)
In a similar fashion, if the optimizer is facing (StochOpt), we get

Expected Optimality Gap E f (X T ) -f (x * ) ≤ E [Reg(T)] T . (2.7)
As a result, in view of (2.6) and (2.7), no-regret policies clearly guarantee an "optimality gap" f (•)min x∈X f (x) that vanishes asymptotically for the associated time-average sequence X t .

variational inequalities

Problem setup and examples

Despite the generality of OCO protocols, there are relevant instances that arise in practice and which necessitate a framework for "optimization beyond minimization". A large class of such problems can be captured by the variational inequality (VI) framework:

Variational Inequality Problem Find x * ∈ X such that A(x * ), x -x * ≥ 0 for all x ∈ X (VI)
where A : X → V * is a single-valued operator, which we call the problem's defining vector field. Moreover, for the time being we shall assume that the feasible region X is a convex and closed subset of R n . Following [START_REF] Facchinei | Finite-Dimensional Variational Inequalities and Complementarity Problems[END_REF], we will refer to this problem as VI(X , A) and we will write X * ≡ Sol(X , A) for its set of solutions. 1 Moreover, to avoid trivialities, we will also assume that the solution set X * of (VI) is nonempty and we will reserve the notation x * for solutions thereof.

X TC(x * ) PC(x * ) . x * x A(x * )
In terms of blanket requirements, we will assume throughout that A is continuous and monotone, i.e.,

Monotone Operators A(x) -A(x ), x -x ≥ 0 for all x, x ∈ X . (Mon)
This condition translates the notion of convexity to the language of operators: indeed, if A = ∇ f for some smooth function f , then A satisfies (Mon). For a panoramic overview of monotone operators we refer the reader to Bauschke and Combettes [START_REF] Heinz | Convex Analysis and Monotone Operator Theory in Hilbert Spaces[END_REF] For illustration purposes, we present some archetypal examples of such problems below:

Example 2.3 (Function minimization). If A = ∇ f for some smooth convex function f on X = R n , solutions of (VI) coincide with the global minimizers of f , i.e., the Minimization Problems solutions of (Opt).

Example 2.4 (Min-max optimization). Suppose that

A = (∇ x 1 f , -∇ x 2 f ) for some real-valued function f (x 1 , x 2 ) with x 1 ∈ X 1 , x 2 ∈ X 2 , and X 1 , X 2 convex. If f is Min-Max Problems convex-concave (i.e., convex in x 1 and concave in x 2 ), any solution x * = (x * 1 , x * 2 ) of (VI) is a global saddle-point of f , i.e., f (x * 1 , x * 2 ) ≤ f (x 1 , x * 2 ) and f (x * 1 , x * 2 ) ≥ f (x * 1 , x 2 ) (2.8) for all x 1 ∈ X 1 , x 2 ∈ X 2 .
Problems of this type have attracted considerable interest in the fields of machine learning and artificial intelligence because they constitute the basic optimization framework for GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF]. For a series of recent papers focusing on this interplay, see [START_REF] Daskalakis | Training GANs with optimism[END_REF][START_REF] Gauthier Gidel | A variational inequality perspective on generative adversarial networks[END_REF][START_REF] Liang | Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks[END_REF][START_REF] Mertikopoulos | Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile[END_REF][START_REF] Yadav | Stabilizing adversarial nets with prediction methods[END_REF] and references therein.

Convex Games

Example 2.5 (Convex Games). A continuous game in normal form is defined as follows: Consider a finite set of players N = {1, . . . , N}, each with their own action space X i ⊂ R n i (convex but possibly not closed). During play, each player selects an action x i from X i with the aim of minimizing a loss determined by the ensemble x ≡ (x i ; x -i ) = (x 1 , . . . , x N ) of all players' actions. In more detail, writing X = ∏ i X i for the game's total action space, we assume that the loss incurred by the i-th player is i (x i ; x -i ), where i : X → R is the player's loss function.

In this context, a Nash equilibrium is any action profile x * ∈ X that is unilaterally stable, i.e.,

i (x * i ; x * -i ) ≤ i (x i ; x * -i ) for all x i ∈ X i and all i ∈ N . (NE)
In most cases of interest, the players' loss functions are individually subdifferentiable on a subset X of X with ri X ⊆ X ⊆ X [START_REF] Hiriart-Urruty | Fundamentals of Convex Analysis[END_REF][START_REF] Tyrrell | Convex Analysis[END_REF]. This means that there exists a (possibly discontinuous) vector field

A i : X → R n i such that i (x i ; x -i ) ≥ i (x i ; x -i ) + A i (x), x i -x i (2.9)
for all x ∈ X , x ∈ X and all i ∈ N [START_REF] Hiriart-Urruty | Fundamentals of Convex Analysis[END_REF]. In the simplest case, if i is differentiable at x, then A i (x) can be interpreted as the gradient of i with respect to x i . In turn, this means that Nash equilibria of the game are solutions of VI(X , A).

Merit functions

Due to the lack of a single objective function the quality of a candidate solution of (VI) becomes much trickier to assess compared to the minimization case. To that end, we start with the unconstrained case, i.e., when X = R n . Then (VI) is reduced to the zero-finding problem:

Stationarity Problem Find x * ∈ R n such that A(x * ) = 0 (Zer)
Therefore, a natural performance criterion of a given policy X t for this case would be to examine how fast A(X t ) * converges to 0.2 However, if X is a strict subset of R n , i.e., when we are facing a genuine constrained problem, then the operator may not necessarily vanish at a solution of (VI). Therefore, we shall need a more general measure in order to be able to capture cases where the solution lies on the border of the domain X .

A popular performance criterion in this context is that of the restricted merit function, first introduced in [START_REF] Auslender | Optimisation: Méthodes numériques[END_REF][START_REF] Auslender | Asymptotic Cones and Functions in Optimization and Variational Inequalities[END_REF]:

Gap Function Gap C ( x) = sup x∈C A(x), x -x , (2.10)
where the "test domain" C is a nonempty convex subset of X [START_REF] Facchinei | Finite-Dimensional Variational Inequalities and Complementarity Problems[END_REF][START_REF] Juditsky | Solving variational inequalities with stochastic mirror-prox algorithm[END_REF][START_REF] Nesterov | Dual extrapolation and its applications to solving variational inequalities and related problems[END_REF]. The following proposition generalizes earlier characterizations by [START_REF] Auslender | Optimisation: Méthodes numériques[END_REF][START_REF] Nesterov | Dual extrapolation and its applications to solving variational inequalities and related problems[END_REF] and justifies the use of Gap C (x) as a merit function for (VI); since every solution of (VI) is a zero of (2.10) and vice versa.

Proposition 2.1 [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF]. Let C be a nonempty convex subset of X . Then: a) Gap C ( x) ≥ 0 whenever x ∈ C; and b) if Gap C ( x) = 0 and C contains a neighborhood of x, then x is a solution of (VI).

Proof. Let x * ∈ X be a solution of (VI) so A(x * ), xx * ≥ 0 for all x ∈ X . Then, by monotonicity, we get:

A(x), x * -x ≤ A(x) -A(x * ), x * -x + A(x * ), x * -x = -A(x * ) -A(x), x * -x -A(x * ), x -x * ≤ 0, (2.11)
so Gap C (x * ) ≤ 0. On the other hand, if x * ∈ C, we also get Gap(x * ) ≥ A(x * ), x *x * = 0, so we conclude that Gap C (x * ) = 0.

For the converse statement, assume that Gap C ( x) = 0 for some x ∈ C and suppose that C contains a neighborhood of x in X . First, we claim that the following inequality holds:

A(x), x -x ≥ 0 for all x ∈ C.
(2.12) Indeed, assume to the contrary that there exists some x 1 ∈ C such that

A(x 1 ), x 1 -x < 0. (2.13)
This would then give

0 = Gap C ( x) ≥ A(x 1 ), x -x 1 > 0, (2.14)
which is a contradiction. Now, we further claim that x is a solution of (VI),i.e.,:

A( x), x -x ≥ 0 for all x ∈ X . (2.15)
If we suppose that there exists some z 1 ∈ X such that A( x), z 1x < 0, then, by the continuity of A, there exists a neighborhood U of x in X such that A(x), z 1x < 0 for all x ∈ U .

(2.16)

Hence, assuming without loss of generality that U ⊂ U ⊂ C (the latter assumption due to the assumption that C contains a neighborhood of x), and taking λ > 0 sufficiently small so that x = x + λ(z 1x) ∈ U , we get that A(x), xx = λ A(x), z 1x < 0, in contradiction to (2.12). We conclude that x is a solution of (VI), as claimed.

lipschitz regularity

Having described the problems of interest, besides the structural assumption of convexity (or monotonicity for the (VI) context) there are two additional regularity conditions which heavily determine the performance of the respective merit functions of each framework. In what follows, we shall present them in a nutshell. More precisely, given an operator A : X → R n we have the following definitions:

1. A is bounded, i.e., there exists some positive constant G > 0 such that :

Bounded Operators A(x) * ≤ G for all x ∈ X (Bd)
2. A is Lipschitz continous, i.e., there exists some positive constant β > 0 such that :

A(x) -A(x ) * ≤ β x -x for all x, x ∈ X (LC)
As said these conditions play a crucial role in determining the performance of the various algorithmic methods at play; this fact will become apparent in Section 2.4. Now, when A = ∇ f for some convex objective f , (Bd), (LC) give rise to a series of explicit properties for f . Starting with (Bd) one can straightforwardly derive that said property essentially boils to Lipschitz continuity of f ,i.e., :

Lipschitz Objectives | f (x) -f (x )| ≤ G x -x for all x, x ∈ X (2.17)
On the other hand under (LC), f satisfies the descent inequality

Descent Inequality f (x ) ≤ f (x) + ∇ f (x), x -x + β 2 x -x 2 for all x, x ∈ X , (2.18)
which lies at the core of the success of first order "descent" methods. In particular, we have the following proposition:

Smoothness Properties Proposition 2.2. Assume that X is a convex and closed subset of R n and f : X → R is a continuously differentiable convex function on int X . Then, the following statements are equivalent:

1. ∇ f satisfies (LC) 2. f satisfies (2.18) 3. β 2 • 2 -f is a convex function 4. ∇ f (x) -∇ f (x ), x -x ≤ β x -x 2 for all x, x ∈ X
Finally, a quite interesting equivalence holds whenever X = R n which is known as the Baillon-Haddad theorem [START_REF] Baillon | Quelques propriétés des opérateurs anglebornés et n-cycliquement monotones[END_REF]. In particular, we have:

Baillon-Haddad Theorem Theorem 2.3. Assume that f : R n → R is a continuously differentiable convex function. Then, the following statements are equivalent:

1. ∇ f satisfies (LC) 2. ∇ f is 1/β-cocoercive 3 : 1 β ∇ f (x) -∇ f (x ) 2 * ≤ ∇ f (x) -∇ f (x ), x -x for all x, x ∈ R n (2.19)

first-order methods

Now we turn our attention towards the respective iterative solution methods. In particular, our focal point would be the so-called first-order methods, i.e., methods that require at each iteration access on a first order/ gradient feedback. The surge of recent breakthroughs in machine learning and artificial intelligence has reaffirmed the prominence of these methods in solving large-scale optimization problems. One of the main reasons for this is that the computation of higher-order derivatives of functions with thousands -if not millions -of variables quickly becomes prohibitive; another is that gradient calculations are typically easier to distribute and parallelize, especially in large-scale problems. In view of this, firstorder methods have met with prolific success in many diverse fields, from machine learning and signal processing to wireless communications, nuclear medicine, and many others [START_REF] Bubeck | Convex optimization: Algorithms and complexity[END_REF][START_REF] Scutari | Convex optimization, game theory, and variational inequality theory in multiuser communication systems[END_REF][START_REF] Sra | Optimization for Machine Learning[END_REF]. In what follows, we present the main structure of these methods combined with the respective optimal lower bounds.

Oracle mechanism and feedback

From an algorithmic point of view, we aim to solve (Opt) and/or (VI) by using iterative methods that require access to a stochastic first-order oracle (SFO) [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. This means that, at each stage of the process, the optimizer can query a black-box mechanism that returns an estimate of the objective's gradient (or subgradient) at the queried point. Formally, when called at x ∈ X , an SFO is assumed to return a random (dual) vector V(x; ω) ∈ V * where ω belongs to some (complete) probability space (Ω, F , P). In practice, the oracle will be called repeatedly at a (possibly) random sequence of points X t ∈ X generated by the algorithm under study. Thus, once X t has been generated at stage t, the oracle draws an i.i.d. sample sample ω t ∈ Ω and returns the dual vector:

First Order Oracle V t ≡ V(X t ; ω t ) = A(X t ) + U t (SFO)
with U t ≡ U(X t ; ω t ) ∈ V * denoting the "measurement error" of the oracle. In terms of measurability, we will write F t for the history (natural filtration) of X t ; in particular, X t is F t -adapted, but ω t , V t and U t are not.

Finally, we will also make the following statistical assumptions. First, we shall

First Order Oracle's Statistics assume that (SFO) is an unbiased estimator:

E[U t | F t ] = 0 (2.20)
Moreover, we shall assume that for some (known) q ∈ (2, +∞] we have:

U t L q , * = E[ U t q * ] 1/q ≤ σ 2 for all t = 1, 2, . . . (2.21)
For concreteness, we will refer to the oracles with σ = 0 as "perfect" -since, in that case, U t = 0 for all t almost surely. Otherwise, if U t L q , * > 0 the noise will be called persistent and the model will be called stochastic.

Lower bounds

With all this in hand, the first question that arises is what is the worst performance that the optimizer may expect and how is this influenced by the different feedback and regularity conditions at play. The answer to the above question is formally stated by the notion of worst-case lower bounds and differs depending on the respective setting. So, we shall investigate each setting individually.

• Online Convex Optimization/ Stochastic Minimization: We begin with the online convex optimization framework. More precisely, under (Bd) the regret optimal lower bound is

Reg(T) = Θ(1/ √ T) (2.22)
Moreover, (LC) does not help the optimizer to improve upon this lower

Regret Lower Bound bound [START_REF] Abernethy | Optimal strategies and minimax lower bounds for online convex games[END_REF].

• Static Convex Minimization: Now, we turn our attention towards the particular case of (Opt). We distinguish the deterministic (σ = 0) and the (purely)

Convex Minimization Lower Bounds stochastic (σ > 0) instances of (SFO). Starting with the deterministic one, the sub-optimality gap for first-order methods with perfect gradient input possesses a "worst-case" guarantee

f (X T ) -f (x * ) = Ω(1/ √ T) (2.23)
under (Bd). This guarantee is improved significantly, i.e.,

f (X T ) -f (x * ) = Ω(1/T 2 ) (2.24)
whenever (LC) kicks in [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF]. On the other hand, if the optimizer has only access to stochastic gradients (as is often the case in machine learning and distributed control), the corresponding lower bound for the expected suboptimality gap is

E[ f (X T ) -f (x * )] = Ω(1/ √ T) (2.25)
For details we refer the reader to [START_REF] Bubeck | Convex optimization: Algorithms and complexity[END_REF][START_REF] Semen | Problem Complexity and Method Efficiency in Optimization[END_REF][START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF].

• Variational inequalities: Finally, we describe the worst case guarantees for the generic framework of (VI). In doing so, if the optimizer has access to a perfect

Variational Inequality Lower Bound

(SFO) oracle then the respective optimal lower bound for the restricted merit function (2.10) under (Bd) is:

Gap C (X T ) = Ω(1/ √ T), (2.26)
while under (LC) a lower bound of Θ(1/T) is achievable [START_REF] Semen | On optimality of Krylov's information when solving linear operator equations[END_REF][START_REF] Semen | Information-based complexity of linear operator equations[END_REF]; the latter illustrates also a significant gap between VI's and the static smooth minimization setting. Finally, for a purely stochastic (SFO) the respective lower bound relative to the restricted merit function (2.10) would be that of Θ(1/ √ T) under (Bd).

2.5 first-order methods for online convex optimization 2.5.1 Gradient descent and its primal-dual variant For OCO, the most popular first order methods are the so called greedy/lazy (projected) gradient descent algorithms. In what follows, we describe these methods in detail.

To start with, the greedy version is defined formally as: In the above pr X (x) = arg min x ∈X xx denotes the euclidean projection onto the convex and closed feasible domain X , γ t > 0 is the method's step-size and V t is the (SFO) feedback at X t . We refer to (GD) also as greedy gradient descent in order to distinguish it from the so-called lazy variant [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF] which is defined by the following recursion:

Gradient Descent X t+1 = pr X (X t -γ t V t ) (GD) X X 1 X 2 X 3 -γ 1 V 1 -γ 2 V 2 Figure 2.2: Schematic representation of (projected) gradient descent. X X 1 Y 2 X eager 2 = X lazy 2 Y 3 X lazy 3 X eager 3 +γ 1 V 1 +γ 2 V 2 +γ 2 V 2
Lazy Gradient Descent Y t+1 = Y t -γ t V t X t+1 = pr X (Y t+1 ) (LGD)
A different perspective of the above method is given by the so-called dual averaging scheme, originally introduced by Nesterov in [START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF] and further developed in [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]. This is formally given by the following recursion:

Dual Averaging Y t+1 = Y t -V t X t+1 = pr X (η t+1 Y t+1 ) (DA)
More precisely, the critical difference between (LGD) and (DA) is that in the latter the learning rate η t changes its role. In particular, in (DA) η t acts as a post-multiplier over the ensemble aggregation of V t instead of allocating a specific weight on each individual V t . As we discuss in the sequel this key feature of (DA) would enable us to deal with unbounded feasible domains X .

Performance guarantees

We now proceed to describe the regret minimization guarantees of the family of algorithms presented in Section 2.5.1. More precisely, if (GD)/(LGD) are run with a "horizon"-dependent step-size policy γ t ≡ 1/ √ T4 we have the following the proposition [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF]:

Regret of Descent

Variants (Horizon-dependent step-size)

Proposition 2.4. Assume that X t are the iterates of (GD) or (LGD) run with a stepsize γ t = 1/ √ T and a "perfect" oracle feedback. Then, if f t satisfies (2.17). for all t = 1, 2, . . . , T with sup t ∇ f t (x) 2 * ≤ G, we have:

1 T T ∑ t=1 f t (X t ) - T ∑ t=1 f t (x) = O X 1 -x 2 + G 2 √ T for all x ∈ X (2.27)
Some comments concerning the particular step-size are in order. More precisely, this step-size policy is based on the idea on dividing the infinite play into epochs (or time-windows) of length T. Hence, the optimizer practically applies a constant, within the time window [1, T] and then repeats the same idea for the next window [T, 2T] and the procedure repeats to infinity. Moreover, if the feasible domain X is a compact set, one may apply a "dynamic" step-size γ t ∝ 1/ √ t and derive an "any-time" regret bound in contrast to that of Proposition 2.4. This is described by the following proposition.

Regret of Descent

Variants (Dynamic step-size) Proposition 2.5. Assume that X is a compact set and let X t be the iterates of (GD) or (LGD) run with γ t ∝ 1/ √ t and a "perfect" oracle feedback. Then, if f t satisfy (2.17) with sup t ∇ f t (x) 2 * ≤ G 2 , we have:

1 T T ∑ t=1 f t (X t ) - T ∑ t=1 f t (x) = O diam X + G 2 √ T for all x ∈ X (2.28)
where diam X = sup x,x ∈X xx .

Moving forward our next step is to illustrate the respective regret guarantees for (DA). Similarly with the above guarantees we have the following result for the (DA) [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF]:

Regret of Dual Averaging

Proposition 2.6. Assume that X t are the iterates of (DA) run with a learning rate η t ∝ 1/ √ t and a "perfect" oracle feedback. Then, if f t satisfy (2.17) with sup t ∇ f t (x) 2 * ≤ G 2 , we have:

1 T T ∑ t=1 f t (X t ) - T ∑ t=1 f t (x) = O x 2 + G 2 √ T for all x ∈ X (2.29)
An important difference between Proposition 2.5 and Proposition 2.6 is that in the latter no compactness-or rather boundedness-assumption for the domain X is required.

Sub-optimality for smooth minimization and accelerated methods

In this section we shall investigate the particular case of (Opt) in a more detailed manner in accordance to the optimal worst case lower bounds (cf. Section 2.4.2).

In doing so, the first candidate would be the (GD) methods presented in Section 2.5.1. More precisely, we first describe their performance under the different regularity conditions (Bd) and (LC). A preliminary result under (Bd) can be obtained via a straightforward adaptation of Propositions 2.5 and 2.6; more precisely this yields that under (Bd):

f (X t ) -f (x * ) = O(1/ √ T) (2.30
) with X T denoting the time average of the (GD)/(LGD) and (DA) iterates run with a step-size policy γ t ∝ 1/ √ t. Hence, the generic (GD) algorithms exhibit an optimal convergence rate within this class of objectives.

That said, the situation changes drastically under (LC). For that particular case, (GD) and (LGD) run with a constant step-size γ t ≡ γ ≤ 1/β guarantees a performance rate of order O(1/T). This result confirms the sub-optimality of (GD) family of methods for smooth deterministic minimization problems, since its performance does not match the iconic 1/T 2 lower bound. This 1/T 2 rate was first achieved by Nesterov in his seminal paper [START_REF] Nesterov | A method for unconstrained convex minimization problem with the rate of convergence O(1/k 2 )[END_REF]. This algorithm has since generated an immense literature with several hallmark contributions like the fast iterative shrinkage-thresholding algorithm (FISTA) method, [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], for composite minimization problems and many others. More precisely, following [START_REF] Auslender | Interior gradient and proximal methods for convex and conic optimization[END_REF] we consider the improved interior gradient algorithm (IGA) algorithm:

Acceleration Schemes Y t = (1 -λ t )X t + λ t Z t Z t+1 = pr X (Z t - λ t β V t ) X t+1 = (1 -λ t )X t + λ t Z t+1 (IGA)
with the weight sequence λ t being defined recursively as follows:

1 -λ t+1 λ 2 t+1 = 1 λ 2 t (2.31)
The crucial difference of (IGA) is the particular averaging part that serves as an acceleration mechanism of the (GD) template. This is described formally by the following proposition.

Acceleration Rate Proposition 2.7. Assume that X t are the iterates of (IGA) run with a step-size given by (2.31). Then, if f satisfies (LC) we have:

f (X t ) -f (x * ) ≤ 2β X 1 -x * 2 T 2 (2.32)
Variants of this method can be also found in [START_REF] Hanzely | Accelerated Bregman proximal gradient methods for relatively smooth convex optimization[END_REF].

2.6 optimal methods: the variational inequality case 2.6.1 Extra-Gradient method and its primal-dual variant Now we turn our attention towards defining optimal iterative methods for (VI).

Perhaps the most widely used solution method for VIs is the EG algorithm of [START_REF] Korpelevich | The extragradient method for finding saddle points and other problems[END_REF] and its variants [START_REF] Malitsky | Projected reflected gradient methods for monotone variational inequalities[END_REF][START_REF] Denisovich | A modification of the Arrow-Hurwicz method for search of saddle points[END_REF][START_REF] Rakhlin | Optimization, learning, and games with predictable sequences[END_REF]. This algorithm has a rich history in optimization, and it has recently attracted considerable interest in the fields of machine learning and AI, see e.g., [START_REF] Chavdarova | Reducing noise in GAN training with variance reduced extragradient[END_REF][START_REF] Daskalakis | Training GANs with optimism[END_REF][START_REF] Gauthier Gidel | A variational inequality perspective on generative adversarial networks[END_REF][START_REF] Hsieh | On the convergence of single-call stochastic extra-gradient methods[END_REF][START_REF] Hsieh | Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling[END_REF][START_REF] Mertikopoulos | Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile[END_REF][START_REF] Mokhtari | Convergence rate of O(1/k) for optimistic gradient and extra-gradient methods in smooth convex-concave saddle point problems[END_REF] and references therein.

In its simplest form, for problems with closed and convex domains, the algorithm proceeds recursively as

X t+1/2 = pr x (X t -γ t V t ) X t+1 = pr X (X t -γ t V t+1/2 ) (EG)
In a nutshell (EG) suggests first to generate a leading state X t+1/2 by taking a Extra-Gradient "gradient" step as usual. Then, instead of continuing from X t+1/2 , (EG) samples V t+1/2 and goes back to the original state X t in order to generate a new state X t+1 via a "gradient" step along the direction of V t+1/2 .

Let us now present its primal-dual counterpart, firstly introduced by Nesterov in [START_REF] Nesterov | Dual extrapolation and its applications to solving variational inequalities and related problems[END_REF]. In particular, the (euclidean based) dual extrapolation (DualX) method is given via the following recursive formula:

Dual Extrapolation X t+1/2 = pr x (X t -γ t V t ) Y t+1 = Y t -V t+1/2 X t+1 = pr X (γ t+1 Y t+1 ) (DualX)
In turn, the (DualX) template hinges on a combination of the (GD) and (DA) methods. In particular, it suggests the following updating rule: first generate a leading state X t+1/2 by taking a "gradient" step as in (EG) and again samples V t+1/2 . Then, the method aggregates these feedbacks and finally the method's update is obtained by applying a dual averaging step.

Performance guarantees

Building on the templates of (EG) and (DualX) in this section we present the performance guarantees (in terms of the restricted merit function (2.10)) under the light of the different regularity conditions (Bd) and/or (LC). Starting with the (EG) template, we have the proposition for the case where A is not necessarily Lipschitz continuous:

Proposition 2.8 (Juditsky et al. [START_REF] Juditsky | Solving variational inequalities with stochastic mirror-prox algorithm[END_REF]). Assume that X t , t = 1, 1/2, . . . are the iterates of (EG) with an oracle satisfying (SFO) with σ 2 > 0 and A satisfies (Bd). Moreover, let Extra-Gradient Guarantees (Stochastic)

X T = ∑ T t=1 γ t -1
∑ T t=1 γ t X t+1/2 , let C be a compact neighbourhood of a solution of the (VI) and set D 2 = sup x∈C x -X 1 2 . Then, if (EG) run with a decreasing (deterministic) step-size γ t satisfies the following estimate:

E Gap C (X T ) ≤ D 2 + G 2 + σ 2 ∑ T t=1 γ 2 t ∑ T t=1 γ t (2.33)
In particular, if (EG) is run with

γ t ∝ 1/ √ t, then E Gap C (X T ) = O(1/ √ T) (2.34)
We distinguish the "perfect" (SFO) case, i.e., σ 2 = 0, where the particular influence of the respective regularity conditions becomes more apparent. In particular, this is illustrated by the following result.

Extra-Gradient

Guarantees (Deterministic) Proposition 2.9 (Nemirovski [START_REF] Semen | Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convexconcave saddle point problems[END_REF]). Assume that X t , X t+1/2 are the iterates of (EG) with a "perfect" (SFO) and A satisfies (Bd). Let us denote X T = ∑ T t=1 γ t -1

∑ T t=1 γ t X t+1/2 , C is a compact neighbourhood of a solution of the (VI) and D = sup x∈C x -X 1 2 . Then, the following hold:

1. Under (Bd) then, Gap C (X T ) ≤ D + G 2 ∑ T t=1 γ 2 t ∑ T t=1 γ t (2.35)
In particular, if

γ t ∝ 1/ √ t, then Gap C (X T ) = O(1/ √ T)
2. Under (LC) and 0 < inf t γ t ≤ γ t ≤ 1/β then,

Gap C (X T ) ≤ D 2T inf t γ t (2.36)
On the other the hand, for the (DualX) template we may obtain similar convergence rate guarantees by consider the time average, 1/T ∑ T t=1 X t+1/2 , as the method's output. Formally, we have the following proposition. 5Dual-Extrapolation Guarantees (Stochastic) Proposition 2.10. Assume that X t , X t+1/2 are the iterates of (DualX) with an oracle satisfying (SFO) with σ 2 > 0. Moreover, X T = 1/T ∑ T t=1 X t+1/2 and C is a compact neighbourhood of a solution of the (VI) and D 2 = sup x∈C x -X 1 2 . Then, if (DualX) is run with a (deterministic) decreasing step-size γ t the following holds:

E Gap C (X T ) ≤ D + G 2 + σ 2 ∑ T t=1 γ t T (2.37)
In particular, if (DualX) is run with

γ t ∝ 1/ √ t then E Gap C (X T ) = O(1/ √ T). (2.38)
Now in the same spirit as for the (EG) for the deterministic case we obtain the respective range of rates as in Proposition 2.11:

Dual-Extrapolation Guarantees (Deterministic)
Proposition 2.11 (Nesterov [91]). Assume that X t , X t+1/2 are the iterates of (DualX) with a "perfect" (SFO). Moreover, let us denote X T = 1/T ∑ T t=1 X t+1/2 , C is a compact neighbourhood of a solution of the (VI) and D 2 = sup x∈C x -X 1 2 . Then, the following hold:

1. Under (Bd) then, Gap C (X T ) ≤ D + G 2 ∑ T t=1 γ t T (2.39)
In particular, if and

γ t ∝ 1/ √ t then Gap C (X T ) = O(1/ √ T).
2. Under (LC) and γ t ≤ 1/β then,

Gap C (X T ) ≤ D T (2.40)
As one may observe the set of results presented above rely their success on prior knowledge of the Lipschitz modulus of the associated operator. In what follows, we describe the state-of-the-art methods which transcend this restriction.

adaptive methods

Having described the performance guarantees under different regularity conditions and step-sizes, we move forward by introducing a range of "adaptive" methods that automatically detect the level of regularity in the problem and the quality of the oracle. In particular, adaptivity of a method refers (at least) to two different scenarios:

• The method automatically adjusts its performance to parameters within a fixed operator class (Lipschitz/Hölder smoothness and the like).

• The method automatically detects the respective Lipschitz modulus at hand and exhibits (optimal) rate interpolation guarantees between different classesfor example between non-smooth and smooth objectives etc. 6 Of course, one may straightforwardly recognize the fact that the second type properly includes the other two. In Section 2.7.1 and Section 2.7.2, we squarely focus on the latter. Moreover, in what will follow we address each framework individually.

The minimization case

We assume first that the optimizer is facing an unconstrained (Opt) version and/or (StochOpt). For this particular framework one may show that the (GD) template run with the adaptive step-size of the form:

AdaGrad Step-Size γ t = 1 ∑ t j=1 V j 2 * (2.41) 6
The methods that satisfy this property are also denoted in the literature as universal.

achieve the following convergence rate:

• For the deterministic case, (GD) run with the step-size policy (2.41) interpolates between O(1/ √ T) for non-smooth and stochastic regimes and O(1/T) whenever smoothness kicks in.

• For the stochastic case, guarantees an O(1/ √ T) under (Bd) and O( β

T + σ √ T ) under (LC).
Remark 2.1. This iterative scheme is often refered as adaptive inverse-norm-squared (AdaNorm) and is a simplified variant of the general AdaGrad firstly introduced in [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF][START_REF] Mcmahan | Adaptive bound optimization for online convex optimization[END_REF].

More precisely, the following result describes formally the above properties for (GD). For detailed proof we defer the reader to [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF].

AdaGrad Guarantees Proposition 2.12 (Levy et al. [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF]). Assume X t are the iterates of (GD) run with adaptive step-size policy (2.41) and an oracle feedback of the form (SFO) and X T = 1/T ∑ T t=1 X t . Moreover we assume that sup t∈N X tx * ≤ D. Then the following hold:

1. Under (Bd) we have:

E f (X T ) -f (x * ) ≤ D 2 (G + σ) 2 √ T (2.42)
2. Under (LC) we have:

E f (X T ) -f (x * ) ≤ βD 2 T + σD √ T (2.43)
The above result indicates that even if (GD) is run with an adaptive step-size this does not seem to match the worst-case 1/T 2 lower bound remaining suboptimal for smooth objectives. In order to overcome this and achieve an adaptive optimal rate interpolation from O(1/ √ T) to O(1/T 2 ), more elaborate schemes are required.

The first result of this kind is done by Nesterov in [START_REF] Nesterov | Universal gradient methods for convex optimization problems[END_REF]; in particular it shown that under "perfect" oracle feedback an optimal rate interpolation for objectives with gradient whose variance satisfy: For some β > 0 and q ∈ [0, 1]7 :

∇ f (x) -∇ f (x ) * ≤ β x -x q for all x, x ∈ R n (2.44)
That said, since this result requires a perfect oracle feedback in an essential manner er we shall not dive into more detail. A different approach which captures adaptivity even for noisy settings is to mimic the idea of (2.41); a step-size that is updated "on the fly". That idea is incarnated by AcceleGrad [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF] and Unixgrad [START_REF] Kavis | UnixGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization[END_REF] methods for the unconstrained and the constrained case. In particular, for X = R n the Accelegrad method suggests:

AcceleGrad Scheme X t+1 = λ t Z t + (1 -λ t )Y t Z t+1 = pr K (Z t -α t γ t V t ) Y t+1 = X t+1 -γ t V t (AcceleGrad)
Some notational comments are here in order to describe each part of the method.

In particular, K is a convex and compact subset of R n and denotes a "domain of interest",i.e., an initial speculation of a subset where the global minimizer lives. Moreover, D denotes the diameter of K. In terms of the weighting sequences we set α t = t, λ t = 1/α t . Having all this in hand, the method's step-size step-size is defined:

γ t = 2D   θ 2 + t ∑ j=1 α 2 j V j 2 *   -1
(2.45)

The following result describes the precise convergence rates of (AcceleGrad); for details we refer the reader to [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF].

AcceleGrad Guarantees Proposition 2.13 (Levy et al. [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF]). Assume that Y t are the iterates of (AcceleGrad)

and let Y T = ∑ T t=1 α t -1
∑ T t=1 α t Y t with oracle feedback satisfying (SFO). Then, the following hold:

1. If σ = 0, then • Under (Bd) we have: f (Y T ) -f (x * ) ≤ GD log T √ T (2.46)
• Under (LC) we have:

f (Y T ) -f (x * ) ≤ DG 2 + βD 2 log(βD/G) T 2 (2.47) 2. If σ > 0, then E f (Y T ) -f (x * ) ≤ GD log T √ T (2.48)
Now, we turn our attention towards the constrained setting and the (UniXGrad) method, which hinges on the (EG) template. More precisely, this is given by the UniXGrad Scheme following:

X t+1/2 = pr X (X t -α t γ t V t ) X t+1 = pr X (X t -α t γ t V t+1/2 ) (UniXGrad)
The crucial difference with generic (EG) is that V t and V t+1/2 are the oracle queries for the gradient evaluated at the averaged points:

X t = α t X t + ∑ t-1 j=1 α j X j+1/2 ∑ t j=1 α j and X t+1/2 = ∑ t j=1 α j X j+1/2 ∑ t j=1 α j (2.49)
with α t = t. Having induced this acceleration mechanism in the (EG) routine, one may obtain the first result concerning the universal properties of (UniXGrad) for the deterministic framework.

UniXGrad Guarantees

Proposition 2.14 (Kavis et al. [START_REF] Kavis | UnixGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization[END_REF]). Assume that X t t = 1, 1/2, . . . are the iterates of (UniXGrad) under an oracle of the form (SFO). Then, we have the following:

1. If f satisfies (Bd), then, E f (X T+1/2 ) -f (x * ) ≤ 6D T 2 + 14σD √ T (2.50) 2. If f satisfies (LC), then, E f (X T+1/2 ) -f (x * ) ≤ 224 √ 14D 2 L T 2 + 14 √ 2σD √ T (2.51)
An important remark concerning Proposition 2.14 is that the compactness assumption for the feasible region X is crucial for establishing the desired agnostic rate interpolation.

The variational inequality case

Now we move forward towards adaptive methods for (VI). To that end a reasonable candidate would be that of the (EG). Indeed, in [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] a novel adaptive step-size is proposed for constrained (VI) problems in the following manner: If X is convex and compact with diam X = D, then, Bach-Levy in [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] propose:

Universal Extra-Gradient γ t = 2D θ 2 + ∑ t-1 j=1 Z 2 j (2.52)
with θ > 0 being an arbitrarily chosen positive constant and Z 2 j :

Z 2 j = X j+1/2 -X j 2 + X j+1/2 -X j+1 2 γ 2 j (2.53)
As it becomes apparent the (2.53) is the crucial ingredient of the adaptive step-size (2.52). In terms of convergence rate guarantees for the stochastic case [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] provides us the following result:

Universal Extra-Gradient Guarantees Proposition 2.15 (Bach and Levy [14]). Assume that X t , X t+1/2 are the iterates of (EG) un with the adaptive step-size policy (2.52) and an oracle feedback of the form (SFO). Moreover, assume that X T = 1/T ∑ T t=1 X t+1/2 and C is a convex and compact neighbourhood of a solution x * of the (VI). Then, the following hold:

1. If A satisfies (Bd), then E Gap C (X T ) ≤ αD(G + σ) log T √ T (2.54) 2. If A satisfies (Bd) and (LC), then E Gap C (X T ) ≤ αGD + α 2 βD 2 + βD 2 log βD/θ 0 T + α √ T (2.55)
The analysis of Proposition 2.15 is that in order to achieve rate adaptivity, even for the case of perfect oracle feedback hinges on the following limitations:

• Compactness of the feasible domain X .

• The associated operator A should satisfy simultaneously both (Bd) and (LC).

As a prelude of our contributions our general beyond Lipschitz analysis will allow us to drop both these restrictions.

# This section incorporates material from the papers [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF][START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF][START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF] T hroughout this chapter we focus on extending the Lipschitz regularity condi- tions presented in Section 2.3. Even though boundedness and Lipschitz continuity conditions (Bd) and (LC) appear to be fairly mild, they may fail to hold in a wide array of practical applications. These unboundedness issues also arise for the case of bounded domains. Indeed, consider as a toy example the 1-dimensional minimization objective:

f (x) = -log x for x > 0. ( 3.1)
In that case, since ∇ f (x) = -1 x , one may straightforwardly verify that ∇ f remains unbounded for all positive intervals that include the origin; so it fails to satisfy both (Bd) and (LC). The main objective would be to design efficient definitions which are able to account for possible "blow ups" of the associated operators. In doing so, one should apply more "geometry aware" toolkits than the standard geometryblind Euclidean setup of the previous chapter. Therefore in Section 3.2 we present two frameworks of that kind. We first describe the notion of a regularization function (or regularizer for short) along with the associated Bregman divergence. The Bregman divergence will serve in the sequel as a generalized distance function surrogate; despite the fact that it is not a distance function per se (it does not satisfy neither symmetry nor the triangle inequality). Moreover, drawing arguments from differential geometry we provide an alternative approach based on the notion of a Finsler metric. This framework allows us to induce families of local 1 norms over the ambient space which are able to capture the geometry of the feasible region in a more efficient way. Armed with these mathematical tools, we introduce in Section 3.3 and Section 3.4 the main classes of objectives that transcend the traditional Lipschitz regularity conditions. To motivate all the above, we first present some prominent examples of widely studied problems with "gradient singularities" in Section 3.1.

motivating examples

Poisson Inverse Problems Poisson Inverse Problems

Many problems in machine learning and the imaging sciences focus on the reconstruction of an unknown object from a set of imperfect observations (e.g., noisy 2D cross-sections of a 3D object). This is especially true in the fields of emission tomography and optical/infrared astronomy, where images are obtained by counting particles (usually photons) reaching a detector. In this case, factors such as fluorescence emissions, radioactive decay and thermal noise can severely affect particle counts, typically by introducing Poisson-distributed errors in the measurement process [START_REF] Bertero | Image deblurring with Poisson data: from cells to galaxies[END_REF].

Mathematically, inverse problems of this kind boil down to solving linear systems of the form

y = Hx + z (3.2)
where:

• x ∈ R n + is the object under study (a signal, image, . . . ). • y ∈ R m + is the observed data (usually m n).
• The kernel matrix H ∈ R m×n + is a representation of the data-gathering protocol and is typically ill-conditioned (e.g., a Toeplitz matrix in the case of image deconvolution problems).

• z ∈ R m is the noise affecting the measurements.

When data points are obtained by means of a counting process, measurements can be modeled as Poisson random variables of the form y j ∼ Pois(Hx) j . 2 Then, up to an additive constant, the log-likelihood of x ∈ R n given an observation y ∈ R m will be

(x; y) = - m ∑ j=1 y j log y j (Hx) j + (Hx) j -y j . (3.3)
Hence, obtaining a maximum likelihood estimate for x leads to the archetypal Poisson inverse problem:

minimize f (x) ≡ D KL (y, Hx), subject to x ∈ R n + , (PIP)
where D KL (p, q) = ∑ m j=1 [p j log(p j /q j ) + q jp j ] denotes the generalized KL divergence on R m + .

In many cases of practical interest, measurements arrive in distinct batches over time -e.g., as sequential optical sections in microscopy and tomography. Moreover, due to the large numbers of pixels/voxels involved (a typical range of values for m is between 10 6 and 10 7 ), gradients of f are very costly to compute; as such, optimization methods that rely on accurate gradient data are difficult to apply in this setting. Accordingly, a natural workaround to this obstacle is to exploit the online nature of the measurement process, model (PIP) as an online optimization problem, and then to use an online-to-batch conversion to get a candidate solution [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF].

On the downside, this online optimization analysis crucially requires the loss functions faced by the optimizer to be Lipschitz continuous, and this assumption does not hold for (PIP): Indeed, if f j (x) = -y j log(y j /(Hx) j ) denotes the singular part of the KL divergence for the j-th sample, we readily get

∂ f j ∂x j = y j H ji (Hx) j . ( 3.4)
This shows that the gradient of f j exhibits an O(1/x) singularity at the boundary of R n + , so f cannot be Lipschitz under any global norm on R n . The same of course holds for (LC).

Resource sharing problems

Consider a set of resources r ∈ R = {1, . . . , R} serving a stream of demands that arrive at a rate of ρ per unit of time (for instance, a GPU cluster or a computing grid processing a stream of jobs). If the load on the r-th resource is x r , the expected

Resource Sharing Formulation

service time in the standard Kleinrock model [START_REF] Kleinrock | Queueing Systems[END_REF] is given by the M/M/1 loss function

r (x r ) = 1 c r -x r , (3.5)
where c r denotes the capacity of the resource. In this setting, the set of feasible resource allocations is 3 and we say that a resource allocation profile x * ∈ x * is at Nash/Wardrop equilibrium [START_REF] Nisan | Algorithmic Game Theory[END_REF][START_REF] John | Some theoretical aspects of road traffic research[END_REF] if

X ≡ {(x 1 , . . . , x r ) : 0 ≤ x r < c r , x 1 + • • • + x R = ρ},
r (x * r ) ≤ r (x r ) for all x ∈ X and all r ∈ R such that x * r > 0 (3.6)
i.e., when no job would be better served by transferring it to a different priority queue. In this case, if we let A(x) = ( 1 (x 1 ), . . . , R (x R )), a standard calculation shows that x * is an equilibrium allocation if and only if it solves the associated variational inequality problem for A.

Fisher market model

Following [START_REF] Nisan | Algorithmic Game Theory[END_REF], a Fisher market consists of a set N = {1, . . . , N} of N buyers -or players -that seek to share a set A = {1, . . . , n} of n perfectly divisible goods (ad space, CPU/GPU runtime, bandwidth, etc.). The allocation mechanism for these

Fisher Model Formulation

goods follows a proportionally fair price-setting rule that is sometimes referred to as a Kelly auction [START_REF] Kelly | Rate control for communication networks: shadow prices, proportional fairness and stability[END_REF]: each player i = 1, . . . , N bids x ia per unit of the a-th good, up the player's individual budget; for the sake of simplicity, we assume that this budget is equal to 1 for all players, so ∑ n a=1 x ia ≤ 1 for all i = 1, . . . , N. The price of the p-th good is then set to be the sum of the players' bids, i.e., p a = ∑ i∈N x ia ; then, each player gets a prorated fraction of each good, namely w ia = x ia /p a . Now, if the marginal utility of the i-th player per unit of the a-th good is θ ip , the agent's total utility will be

u i (x i ; x -i ) = ∑ a∈A θ ia w ia = ∑ a∈A θ ia x ia ∑ j∈N x ja , (3.7)
where x i = (x ia ) a∈A denotes the bid profile of the i-th player, and we use the shorthand (x i ; x -i ) = (x 1 , . . . , x i , . . . , x N ). A Fisher equilibrium is then reached when the players' prices bids follow a profile x * = (x * 1 , . . . , x * N ) such that

u i (x * i ; x * -i ) ≥ u i (x i ; x * -i ) (Eq)
for all i ∈ N and all x i = (x ia ) a∈A such that x ia ≥ 0 and ∑ a∈A x ia = 1. 4As was observed by Shmyrev [START_REF] Vadim | An algorithm for finding equilibrium in the linear exchange model with fixed budgets[END_REF], the equilibrium problem (Eq) can be rewritten equivalently as

minimize F(x; θ) ≡ ∑ a∈A p a log p a -∑ i∈N ∑ a∈A x ia log θ ia subject to p a = ∑ i∈N x ia , ∑ a∈A x ia = 1, and x ia ≥ 0 for all a ∈ A, i ∈ N , (Opt)
with the standard continuity convention 0 log 0 = 0. In the above, the agents' marginal utilities are implicitly assumed fixed throughout the duration of the game. On the other hand, if these utilities fluctuate stochastically over time, the corresponding reformulation instead involves the mean objective

f (x) = E[F(x; ω)]. (3.8)
Because of the logarithmic terms involved, F (and, a fortiori, f ) cannot be Lipschitz continuous or smooth in the standard sense.

tools for transcending the euclidean framework

In this section we present the necessary mathematical machinery that will allow us to generalize the notions of (Bd) and (LC). In doing so, we shall use two key notions. The first is that of the so-called Bregman divergence, whereas the second consists of the geometrical tool of a local norm, i.e., a norm that depends on the point upon which it is calculated.

Bregman functions and divergences

The notion of a Bregman divergence was first introduced by Bregman [START_REF] Bregman | The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming[END_REF]. The building block for this pseudo-distance function is that of a suitable "reference" Bregman function. This is defined as follows:

Definition 3.1. A convex l.s.c. function h : V → R ∪ {∞} is a Bregman function on Bregman Functions X , if
1. The subdifferential of h admits a continuous selection, i.e., there exists a continuous mapping

∇ : dom ∂h → ∇h(x) ∈ ∂h(x) (3.9)
for all x ∈ dom ∂h.

2. h is strongly convex, i.e., there exists some K > 0 such that

h(x ) ≥ h(x) + ∇h(x), x -x + K 2 x -x 2 (3.10)
for all x ∈ dom ∂h, x ∈ dom ∂h.

The induced Bregman divergence of h is then defined for all x ∈ dom ∂h, x ∈ dom h as

Bregman Divergence D(x , x) = h(x ) -h(x) -∇h(x), x -x . (3.11)
Remark. Our definition follows [START_REF] Juditsky | Solving variational inequalities with stochastic mirror-prox algorithm[END_REF][START_REF] Arkadi Semen Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF][START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF], but there are variant definitions where h is not necessarily assumed strongly convex, cf. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF][START_REF] Censor | An iterative row action method for internal convex programming[END_REF][START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF] and references therein.

Some standard examples of Bregman functions are as follows:

Example 3.1. Euclidean regularizer: Let X be a convex subset of R n endowed with the Euclidean norm • 2 . Then, the Euclidean regularizer on X is defined as h(x) = x 2 2 /2 and the induced Bregman divergence is the standard square

distance D(x , x) = x -x 2 2 for all x, x ∈ X Example 3.2. Entropic regularizer: Let X = {x ∈ R n + : ∑ n i=1 x i = 1} be the unit simplex of R n endowed with the L 1 -norm • 1 . Then, the entropic regularizer on X is h(x) = ∑ i x i log x i and the induced divergence is the relative entropy D(x , x) = ∑ i x i log(x i /x i ) for all x ∈ X x ∈ ri X . In particular, h is 1-strongly convex with respect to • 1 . Example 3.3. Log-barrier: Let X = R n ++ denote the (open) positive orthant of R n .
Then, the log-barrier regularizer on X is defined as h(x) = -∑ n i=1 log x i for all x ∈ R n ++ . The corresponding divergence is known as the Itakura-Saito divergence and is given by D(x, x ) = ∑ n i=1 (x i /x ilog(x i /x i ) -1) [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF].

We conclude this presentation by providing some elementary properties of a Bregman [START_REF] Juditsky | Solving variational inequalities with stochastic mirror-prox algorithm[END_REF]. Lemma 3.1. Let h be a Bregman function on X with associated divergence D. Then:

1. D(x , x) is convex with respect to x (but not necessarily with respect to x).

2. D(x, x ) ≥ K 2 x -x 2 for all x ∈ dom h, x ∈ dom ∂h.
Remark. In a nutshell, the first part of Lemma 3.1 is directly derived by the convexity of h, whereas the second is obtained by collecting the terms that constitute the Bregman divergence in (3.16). In the sequel we shall revisit Lemma 3.1 under the light of the notion of local norms.

Finsler geometry and local norms

Following [START_REF] Dai-Wai | An Introduction to Riemann-Finsler Geometry[END_REF][START_REF] Chern | Finsler geometry is just Riemannian geometry without the quadratic restriction[END_REF] a Finsler metric [START_REF] Dai-Wai | An Introduction to Riemann-Finsler Geometry[END_REF][START_REF] Chern | Finsler geometry is just Riemannian geometry without the quadratic restriction[END_REF] is described as follows:

Definition 3.2. A Finsler metric on a convex subset X of V is a continuous function

Finsler Metrics

Φ : X × V → R + which satisfies the following properties for all x ∈ X and all z, z ∈ V:

1. Subadditivity: Φ(x; z + z ) ≤ Φ(x; z) + Φ(x; z ). 2.
Absolute homogeneity: Φ(x; λz) = |λ|Φ(x; z) for all λ ∈ R.

3.

Positive-definiteness: Φ(x; z) ≥ 0 with equality if and only if z = 0.

Given a Finsler metric on X , the induced primal / dual local norms on X are respectively defined as

Finslerian Local Norms z x = Φ(x; z) and w x, * = max{ w, z : Φ(x; z) = 1} (3.12)
for all x ∈ X and all z, w ∈ V. We will also say that a Finsler metric on X is regular when w x , * / w x, * = 1 + O( xx x ) for all x, x ∈ X , w ∈ V * . Finally, for simplicity, we will also assume in the sequel that • x ≥ ν • for some ν > 0 and all x ∈ X (this last assumption is for convenience only, as the norm could be redefined to

• x ← • x + ν • without affecting our theoretical analysis).
When X is equipped with a regular Finsler metric as above, we will say that it is a Finsler space.

Example 3.4. Let Φ(x; z) = z where • denotes the reference norm of X = V. Then the properties of Definition 3.2 are satisfied trivially.

Example 3.5. For a more interesting example of a Finsler structure, consider the set X = (0, 1] n and the metric

z x = max i |z i |/x i , z ∈ R n , x ∈ X . In this case w x, * = ∑ n i=1 x i |w i | for all w ∈ R n ,
and the only property of Definition 3.2 that remains to be proved is that of regularity. To that end, we have

w x , * -w x, * ≤ ∑ n i=1 |w i | • |x i -x i | = ∑ n i=1 x i |w i | • |x i -x i |/x i ≤ w x, * • x -x x .
(3.13) Hence, by dividing by w x, * , we readily get w x , * / w x, * ≤ 1 + xx x i.e.,

• x is regular in the sense of Definition 3.2.

Example 3.6 (Riemannian metrics). In its simplest form, a Riemannian metric on C ⊆ R n is a field of positive-definite matrices g(x) 0, x ∈ C; for a panoramic view of the subject we refer the reader to [START_REF] Attouch | Singular Riemannian barrier methods and gradient-projection dynamical systems for constrained optimization[END_REF][START_REF] Lee | Riemannian Manifolds: an Introduction to Curvature[END_REF]. This defines a local norm as

z x = z g(x)
z, and a dual local norm as w x, * = w g(x) -1 w. In this way, Riemannian metrics can be seen as special cases of Finsler metrics; the converse however is not true (35; see also Example 3.7 below).

Examples of Finsler Spaces

Example 3.7 (Shahshahani p-norm). Consider the Finsler metric on

C = R n ++ given by Φ(x; z) = (∑ n i=1 |z i | p /x i ) 1/p (3.14)
By a straightforward application of Hölder's inequality, the associated dual norm is given by

w x, * = ∑ n i=1 x c-1 i |w| c i 1/q (3.15)
with the convention p -1 + q -1 = 1. This metric is known as the Shahshahani p-norm [START_REF] Mirshams | A New Mathematical Framework for the Study of Linkage and Selection[END_REF] and it plays an important role in game theory, optimal transport, evolutionary biology, and many other fields -see e.g., [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF][START_REF] Alvarez | Hessian Riemannian gradient flows in convex programming[END_REF][START_REF] Sham | Regularization techniques for learning with matrices[END_REF][START_REF] Sra | Optimization for Machine Learning[END_REF], and references therein. The Shahshahani p-norm comes from a Riemannian metric if p = 2 but not otherwise (since it does not satisfy the parallelogram law for p = 2).

We are now in the position to revisit Definition 3.1 under the light of the local norms. More precisely, we may assume that the respective Bregman function is compatible with the said norm, resulting to the new notion of a Bregman-Finsler regularizer. In particular, we propose the following definition firstly introduced in Antonakopoulos et

≥ h(x) + ∇h(x), p -x + 1 2 α p -x 2 x (3.16)
for some α > 0 and all p ∈ X , x ∈ dom ∂h.

As a consequence of the above, we have:

Lemma 3.2. A Bregman function h is α-strongly convex relative to • x if and only if D(p, x) ≥ 1 2 α p -x 2
x for all p ∈ X and all x ∈ dom ∂h.

(3.17)

Proof. The result follows by rearranging (3.16) along with the definition of the Bregman divergence described in (3.11).

The main difference between Definition 3.3 and Definition 3.1 or the standard assumptions in the literature [19, 27, 28, 56, 77-79, 87, 90, 91] is the strong convexity requirement relative to the local norm • x (whose choice, in turn, is aimed to capture the singularity landscape of the operator). We illustrate this with two examples of Bregman-Finsler functions below: Example 3.8. Suppose that X = R n is endowed with the Euclidean norm. Then, setting h(x) = (1/2) x 2 2 , we get the standard expression

D(p, x) = (1/2) p -x 2 2
for the associated Bregman divergence. Obviously, h is 1-strongly convex relative to • 2 .

Example 3.9. Let X = [0, 1) n (so X is neither open nor closed), and consider the local norm

z 2 x = ∑ n i=1 |z| 2 i /(1 -x i ) 2 for x ∈ X , z ∈ R n (cf. Example 3.7 above). If we set h(x) = ∑ n i=1 1/(1 -x i ) (3.18)
a straightforward calculation gives

D(p, x) = n ∑ i=1 (p i -x i ) 2 (1 -p i )(1 -x i ) 2 ≥ n ∑ i=1 (p i -x i ) 2 (1 -x i ) 2 = p -x 2 x , (3.19)
i.e., h is strongly convex relative to • x . Importantly, since • x ≥ • 2 , this Bregman function is also strongly convex relative to the standard Euclidean norm. However, even though the Euclidean regularizer of Example 3.8 is strongly convex relative to any global norm on X , it cannot be strongly convex relative to the local norm • x because of the singularity of the latter when x i → 1 -.

surrogates for operator boundedness

Having described this background material, we now proceed to discuss the particular generalizations of Section 2.3 in order to account for problems with singular objective functions. We divide our presentation into two parts. The first concerns these notions that are based on the Bregman divergence, whereas the second part considers the notions based on Finsler induced norms. The first extension of (LC) is due to [START_REF] Teboulle | A simplified view of first order methods for optimization[END_REF].

Definition 3.4 (Teboulle [START_REF] Teboulle | A simplified view of first order methods for optimization[END_REF]). An operator A is said to be W[h]-continuous relative to h on X if there exists some G > 0 such that, for all t > 0, we have

Weakly Continuous Objectives t A(x), x -x -D(x , x) ≤ t 2 2 G 2 for all x ∈ dom h, x ∈ dom ∂h. (W)
As a prelude we mention that (W) notion intends to single out sufficient conditions for the convergence of "proximal-like" methods like mirror descent. The standard Euclidean (Bd) condition satisfies Definition 3.4. Indeed, if one chooses h(x) = 1/2 x 2 and its respective Bregman divergence D(x, x ) = 1/2 xx 2 then by applying Fenchel-Young inequality we get:

t ∇ f (x), x -x - 1 2 x -x 2 ≤ t 2 ∇ f (x) 2 * 2 + 1 2 x -x 2 - 1 2 x -x 2 ≤ t 2 G 2 2
which yields that f is weakly continuous. Moreover, another related notion is that of (RC), as introduced by [START_REF] Lu | Relative-continuity" for non-Lipschitz non-smooth convex optimization using stochastic (or deterministic) mirror descent[END_REF] and extended further in a recent paper by [START_REF] Zhou | Regret bounds without Lipschitz continuity: Online learning with relative Lipschitz losses[END_REF]: Definition 3.5 (Zhou et al. [START_REF] Zhou | Regret bounds without Lipschitz continuity: Online learning with relative Lipschitz losses[END_REF]). An operator A : X → R n is said to be relatively continuous if there exists some G > 0 such that

Relative Continuous Objectives A(x), x -x ≤ G 2D(x , x) for all x ∈ dom h, x ∈ dom ∂h. (RC)
The two notions above are linked in the following manner: Consider an objective f which satisfies Definition 3.4. Then, we have:

t ∇ f (x), x -x -D(x , x) ≤ t 2 2 G 2 for all x ∈ dom h, x ∈ dom ∂h. (W)
By rearranging the above quadratic polynomial in t, we note that its discriminant is

∆ = [ ∇ f (x), x -x ] 2 -2G 2 D(x , x)
, so it is immediate to check that (RC) holds.

Let us now turn our attention towards the Finsler driven generalized notion of (Bd). In order to give some intuition, let us recall the toy-example presented in Section 3.1. In particular, the 1-dimensional logistic regression f (x) =log x for x > 0 one may straightforwardly detect that the optimizer is dealing with a gradient "singularity" of order O(1/x). Therefore, if one chooses a local norm of the form:

x x = |x |/x for all x ∈ R, x > 0 along with the corresponding dual (local) norm:

w x, * = x|w| for all w ∈ R, x > 0 (3.20)
which allows us to obtain:

∇ f (x) x, * = x(1/x) =
Riemann Lipschitz Continuity ∇ f (x) x, * ≤ G for all x ∈ X . (RLC)
That said, for the sake of generality we prefer the more general formulation of Definition 3.6.

We conclude this section by presenting the connection between (W) and (MB). More precisely, given a Bregman-Finsler fuction (cf. Definition 3.3) Fenchel-Young inequality ensures:

t ∇ f (x), x -x -D(x , x) ≤ t 2 ∇ f (x) 2 x, * 2K + K 2 x -x 2 x -D(x , x) (3.22)
which yields that f is weakly K-continuous.

surrogates for operator lipschitz continuity

Now we turn our attention towards the generalization of (LC). A popular notion, closely linked with the particular minimization framework, is that of Lipschitz-like (or relative smoothness) introduced by [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] (see also [23] [72]). Formally following [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] we have:

Lipschitz-like Objectives Definition 3.7 (Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF]). A convex l.s.c. function f : V → R ∪ {∞} is said to be Lipschitz-like if there exists some β > 0 such that

βh -f is convex on int dom h. (RS)
Here, we recall that the domain of the respective regularizer h is contained in the domain of f . The main motivation behind this elegant definition Definition 3.7 is to generalize the standard descent inequality satisfied by smooth objectives:

f (x) ≤ f (x ) + ∇ f (x ), x -x + β x -x 2 (3.23)
by substituting it with the more geometry-sensitive Bregman divergence, i.e.,

f (x) ≤ f (x ) + ∇ f (x ), x -x + βD(x, x ) (3.24)
For the sake of completeness we provide an overview of some properties of Lipschitz-like functions; proofs of the following appear in [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF] (see also [START_REF] Birnbaum | Distributed algorithms via gradient descent for Fisher markets[END_REF][START_REF] Lu | Relatively-smooth convex optimization by first-order methods and applications[END_REF]), so we omit it.

Lipschitz-like Properties Proposition 3.3 (Bauschke et al. [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF]). The following statements are equivalent:

1. f satisfies (RS) in int X . 2. f satisfies the inequality f (x) ≤ f (x ) + ∇ f (x ), x -x + βD(x, x ), for allx, x ∈ int X 3. f satisfies the inequality ∇ f (x) -∇ f (x ), x -x ≤ β [D(x, x ) + D(x , x)] .
That said, the success of (RS) condition is limited for minimization settings; the reason is that descent-type inequalities of the form (3.23) and/or (3.24), are not available for general (VI) problems. Therefore, in order to be able to include optimization beyond minimization settings we should follow a different approach. In particular, by applying similar reasoning with Definition 3.6, we have the following definition again for a generic operator A. Following Antonakopoulos et al. ( 2021), we propose a novel regularity condition based on the local norm framework.

Metric Smoothness and Variants

Definition 3.8 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Given a local norm • x , x ∈ X , we say that A is metrically smooth

(relative to • x ) if A(x) -A(x ) x, * ≤ β x -x x for all x, x ∈ dom A. (MS)
Following [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF], one may also get a similar notion to (MS); namely that of Bregman continuity of an operator. Formally, this is given by the following. Definition 3.9. [Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF]] Let h be a Bregman-Finsler regularizer relative to some local norm • x on X . The operator A : X → V * is said to be β-Bregman continuous if

A(x ) -A(x) x, * ≤ β 2D(x, x ) for all x, x ∈ dom A.
(BC) Remark 3.2. If A satisfies (MS) and we are given with some h Finsler-Bregman regularizer adapted to the associated norm then one may straightforwardly obtain that (MS) implies (BC).

Finally, we conclude by describing the connection between (RS) and (MS) conditions; of course under the assumption that A = ∇ f for some f convex function. Indeed, if h is a Bregman-Finsler function (cf. Definition 3.3), we have:

∇ f (x) -∇ f (x ), x -x ≤ ∇ f (x) -∇ f (x ) x, * x -x x ≤ β x -x x x -x x ≤ β 2 x -x 2 x + x -x 2 x . (3.25)
Thus by the compatibility of h and • x , we readily obtain

∇ f (x) -∇ f (x ), x -x ≤ β K D(x, x ) + D(x , x) . (3.26)
Therefore, the claim that (MS) implies (RS) follows from Proposition 3.3.

B R E G M A N F I R S T O R D E R M E T H O D S
W e now turn to the presentation of the necessary algorithmic machinery that will allow us transcend the limitations of Euclidean based algorithmic schemes. The main ingredient of these methods boils down to a generalized notion of "projections" which are based on a appropriately chosen reference function in the sense of Definition 3.1 and/or Definition 3.3. We describe this toolkit in Section 4.1, where we also present more precisely the so-called prox and mirror mappings.

As an additional feature we present a novel primal-dual variant of the Bregman divergences; the so-called Fenchel coupling. This will serve a "primal-dual" measure of distance and will come in handy for the particular analysis of primal-dual methods in the sequel.

Finally, in Section 4.2 and Section 4.2 we illustrate the explicit defining recursive formulas of the particular algorithmic schemes. In particular, these methods are obtained by revisiting the (GD) and (EG) methods (see in Section 2.5.1 and Section 2.6.1) under the light of these new projection operators, i.e., the prox and mirror mappings. These generic iterative methods will enable us in the sequel to capture in an efficient manner the finer geometrical aspects that arise from the non-Lipschitz framework.

prox-and mirror mappings

In order to describe the algorithmic methods we first present their key ingredients, that of Bregman prox-and mirror mappings. In addition we provide their crucial properties and template inequalities. Versions of these are known in the literature [see e.g., 20, 34, 91, 105, and references therein] and mostly rely on global -norms. However, in our case we revisit these results and provide here complete statements and proofs armed with the notion local -norm.

In particular, we shall assume that h is a Bregman-Finsler regularization function in the sense of Definition 3.3. To begin, we introduce two key notions that will be useful in the sequel. The first is the convex conjugate of a Bregman function h, i.e.,

h * (y) = max x∈dom h { y, x -h(x)} (4.1)
and the associated primal-dual mirror map Q : V * → dom ∂h:

Mirror Map Q(y) = arg max x∈dom h { y, x -h(x)} (4.2)
That the above is well-defined is a consequence of the fact that h is proper, l.s.c., convex and coercive;1 in addition, the fact that Q takes values in dom ∂h follows from the fact that any solution of (4.2) must necessarily have nonempty subdifferential (see below Lemma 4.1). We also recall here the definition of the Bregman proximal mapping:

Proximal Mapping P x (w) = arg min x ∈dom h { w, x -x + D(x , x)} (4.3)
valid for all x ∈ dom ∂h and all w ∈ V * .

We then have the following basic lemma connecting the above notions:

Mirror and Prox Mapping Links Lemma 4.1. Let h be a K-strongly convex Bregman-Finsler regularizer. Then, for all x ∈ dom ∂h and all w, y ∈ V * we have:

1. x = Q(y) ⇐⇒ y ∈ ∂h(x). 2. x + = P x (w) ⇐⇒ ∇h(x) + w ∈ ∂h(x) ⇐⇒ x + = Q(∇h(x) + w).
3. Finally, if x = Q(y) and p ∈ X , we get:

∇h(x), x -p ≤ y, x -p . ( 4
+ = Q(∇h(x) + w).
For our last claim, by a simple continuity argument, it is sufficient to show that the inequality holds for the relative interior ri X of X (which, in particular, is contained in dom ∂h). In order to show this, pick a base point p ∈ ri X , and let

φ(t) = h(x + t(p -x)) -[h(x) + y, t(p -x) ] for all t ∈ [0, 1]. (4.5)
Since, h is strongly convex and y ∈ ∂h(x) due to the first equivalence, it follows that φ(t) ≥ 0 with equality if and only if t = 0. Since, ψ(t) = ∇h(x + t(px))y, px is a continuous selection of subgradients of φ and both φ and ψ are continuous over [0, 1], it follows that φ is continuously differentiable with φ = ψ on [0, 1]. Hence, with φ convex and φ(t) ≥ 0 = φ(0) for all t ∈ [0, 1], we conclude that φ (0) = ∇h(x)y, px ≥ 0 and thus we obtain the result.

To proceed, the basic ingredient for establishing connections between Bregman proximal steps is a generalization of the rule of cosines which is known in the literature as the "three-point identity" [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF]. This will be our main tool for deriving the main estimates for our results. Being more precise, we have the following lemma:

Bregman Three-Point Identity Lemma 4.2 (Chen and Teboulle [START_REF] Chen | Convergence analysis of a proximal-like minimization algorithm using Bregman functions[END_REF]). Let h be a Bregman-Finsler regularizer. Then, for all p ∈ dom h and all x, x ∈ dom ∂h, we have:

D(p, x ) = D(p, x) + D(x, x ) + ∇h(x ) -∇h(x), x -p . (4.6)
Proof. By definition:

D(p, x ) = h(p) -h(x ) -∇h(x ), p -x D(p, x) = h(p) -h(x) -∇h(x), p -x D(x, x ) = h(x) -h(x ) -∇h(x ), x -x . (4.7)
The lemma then follows by adding the two last lines and subtracting the first.

Remark 4.1. As one may directly observe from the proof of the above, Lemma 4.2 holds for a general convex function h. However, for the need of our analysis we constrain our selves to the specific regularizer class of interest; namely that of Bregman-Finsler ones.

Thanks to the three-point identity, we obtain the following estimate for the Bregman divergence before and after a mirror descent step:

One Step Mirror Template Inequality Proposition 4.3. Let h be a Bregman-Finsler function with strong convexity modulus K > 0. Fix some p ∈ dom h and let x + = P x (w) for some x ∈ dom ∂h and w ∈ V * . We then have:

D(p, x + ) ≤ D(p, x) -D(x + , x) + w, x + -p (4.8) and D
(p, x + ) ≤ D(p, x) + D(x, x + ) -w, x -p . ( 4.9)
Proof. By the three-point identity established in Lemma 4.2, we have:

D(p, x) = D(p, x + ) + D(x + , x) + ∇h(x) -∇h(x + ), x + -p (4.10)
Rearranging terms then yields:

D(p, x + ) = D(p, x) -D(x + , x) + ∇h(x + ) -∇h(x), x + -p (4.11)
By (4.4) and the fact that x + = P x (w) so ∇h(x) + w ∈ ∂h(x + ), the first inequality follows; the second one is obtained similarly.

Thanks to the above estimations, we obtain the following inequalities relating the Bregman divergence between two prox-steps:

Two Steps Mirror Template Inequality Proposition 4.4. Let h be a Bregman function on X and fix some p ∈ X , x ∈ X • . Letting x + 1 = P x (w 1 ) and x + 2 = P x (w 2 ), we have:

D(p, x + 2 ) ≤ D(p, x) + w 2 , x + 1 -p + [ w 2 , x + 2 -x + 1 -D(x + 2 , x)] (4.12) and D(p, x + 2 ) ≤ D(p, x) + w 2 , x + 1 -p + w 2 -w 1 , x + 2 -x + 1 -D(x + 2 , x + 1 ) -D(x + 1 , x). (4.13)
Proof. For the first inequality, by applying (4.8) for x + 2 = P x (w 2 ), we get:

D(p, x + 2 ) ≤ D(p, x) -D(x + 2 , x) + w 2 , x + 2 -p = D(p, x) + w 2 , x + 1 -p + [ w 2 , x + 2 -x + 1 -D(x + 2 , x)] (4.14)
For the second inequality, we need to bound w 2 ,

x + 2 -x + 1 -D h (x + 2 , x).
In particular, applying again (4.8) for p = x + 2 , we get:

D(x + 2 , x + 1 ) ≤ D(x + 2 , x) + w 1 , x + 1 -x + 2 -D(x + 1 , x) (4.15)
and hence:

D(x + 2 , x) ≥ D(x + 2 , x + 1 ) + D(x + 1 , x) -w 1 , x + 1 -x + 2 . (4.16)
So, combining the above inequalities we get:

w 2 , x + 2 -x + 1 -D(x + 2 , x) ≤ w 2 , x + 2 -x + 1 -D(x + 2 , x + 1 ) -D(x + 1 , x) -w 1 , x + 2 -x + 1 (4.17)
and thus we get the second inequality as well.

On the other hand, much of our analysis of primal-dual methods revolves around a "primal-dual" divergence between a target point p ∈ X and a dual vector y ∈ Y. This gives rise to the primal-dual counterpart of the Bregman divergence, the so-called Fenchel coupling. Following [START_REF] Mertikopoulos | Learning in games with continuous action sets and unknown payoff functions[END_REF], this is defined as follows for all p ∈ X , y ∈ Y:

Fenchel Coupling F(p, y) = h(p) + h * (y) -y, p . (4.18)
The following lemma illustrates basic properties of the Fenchel coupling and generalizes similar properties derived in [START_REF] Mertikopoulos | Learning in games with continuous action sets and unknown payoff functions[END_REF]:

Fenchel Coupling Properties Lemma 4.5. Let h be a Bregman-Finsler regularizer on X with convexity modulus α.

Then, for all p ∈ X and all y ∈ Y, we have:

1. F(p, y) ≥ D(p, Q(y)).
2. Moreover,

F(p, y) = D(p, Q(y)) if Q(y) ∈ X • (but not necessarily otherwise). (4.19) 3. If x = Q(y), then F(p, y) ≥ α 2 x -p 2 x
Proof. For the first inequality we have,

F(p, y) = h(p) + h * (y) -y, y = h(p) -h(Q(y)) + y, Q(y) + y, -p = h(p) -h(Q(y)) -y, p -Q(y) Since y ∈ ∂h(Q(x)), by Lemma 4.1 we get ∇h(Q(y)), Q(y) -p ≤ y, Q(y) -p
With all the above we then have

F(p, y) = h(p) -h(Q(y)) -y, p -Q(y) ≥ h(p) -h(Q(y)) -∇h(Q(y)), p -Q(y) = D(p, Q(y))
For the equality, let x = Q(y). Then, by definition we have:

F(p, y) = h(p) -y, Q(y) -h(Q(y)) -y, p = h(p) -h(x) -y, p -x .
Since y ∈ ∂h(x), we have h (x; px) = y, px whenever x ∈ X • , thus proving our first claim. For our second claim, working in the previous spirit we get that:

F(p, y) = h(p) -h(x) -y, p -x (4.20)
Thus, we obtain the result by recalling the strong convexity assumption for h with respect to the local norm • x .

We continue with some basic relations connecting the Fenchel coupling relative to a target point before and after a gradient step. The basic ingredient for this is a primal-dual analogue of Lemma 4.2

Fenchel Three-Point Identity Lemma 4.6. Let h be a Bregman-Finsler regularizer on X . Fix some p ∈ X and let y, y + ∈ Y. Then, letting x = Q(y), we have

F(p, y + ) = F(p, y) + F(x, y + ) + y + -y, x -p . ( 4.21)
Proof. By definition, we get:

F(p, y + ) = h(p) + h * (y + ) -y + , p F(p, y) = h(p) + h * (y) -y, p . (4.22)
Then, by subtracting the above we get:

F(p, y + ) -F(p, y) = h(p) + h * (y + ) -y + , p -h(p) -h * (y) + y, p = h * (y + ) -h * (y) -y + -y, p = h * (y + ) -y, Q(y) + h(Q(y)) -y + -y, p = h * (y + ) -y, x + h(x) -y + -y, p = h * (y + ) + y + -y, x -y + , x + h(x) -y + -y, p = F(x, y + ) + y + -y, x -p (4.23)
and our proof is complete.

bregman first order methods

Armed with the mirror and prox-mappings presented in Section 4.1, we are now in the position to revisit the Sections 2.5 and 2.6. More precisely, we start with the Bregman version of (GD); widely known as mirror descent (MD) algorithm.

The template upon which the template of MD hinges is the following recursion: In the above, the standard notation of Section 2.5 is preserved. In particular, X t ∈ dom ∂h denotes the current state of the algorithm, V t ∈ V * denotes a generic search direction, γ t > 0 is a step-size parameter, and X t+1 is the new state generated after taking a Bregman proximal step from x along -γ t V t . (MD) closely resembles the projected gradient update (GD) and, indeed, (GD) is recovered if we take h(x) = (1/2) x 2 2 (cf. Example 3.8). In addition, the abstact template:

Mirror Descent X t+1 = P X t (-γ t V t ) (MD) X ⊆ V Y = V * Q Y 1 Y 2 Y 3 -γ 1 V 1 -γ 2 V 2 X 1 X 2 X 3
x + = P x (-γw) (4.24)
is well-posed in our setting. Therefore, its allows us to iterate (MD) in perpetuity. Formally, we have the following result:

Proposition 4.7. The abstract recursion,

x + = P x (-γw) (4.25) satisfies x + ∈ dom ∂h for all x ∈ dom ∂h and all V ∈ V * .

Proposition 4.7 is a direct corollary of Lemma 3.1, so we omit its proof. Now we turn our attention towards generalizing the primal-dual methods described in (LGD) and (DA); more precisely, we shall revisit these methods under the lens of the mirror mapping (4.2). In particular, the lazy version of (MD) is given by the following:

Lazy Mirror Descent Y t+1 = Y t -γ t V t X t+1 = Q(Y t+1 ) (LMD)
whereas the (DAvg) is given by:

Mirror Dual Averaging Y t+1 = Y t -V t X t+1 = Q(γ t+1 Y t+1 ) (DAvg)
Now, building on these templates we are in a position to generalize the method of Section 2.6.1 along with its primal-dual counterpart. In particular, the (EG) template is extended by applying the prox-mapping (4.3). More precisely, Mirror-Prox is derived by applying the following recursion:

Mirror-Prox X t+1/2 = P X t (-γ t V t ) X t+1 = P X t (-γ t V t+1/2 ) (MP)
As it becomes apparent from (MP), the method consists of two (MD) steps. Therefore,if we take h(x) = (1/2) x 2 2 then (MP) boils down to the (EG) template. On the other hand, the respective primal-dual version of (MP); namely that of the Bregman generalization of (DualX). In particular, this given by the following recursion:

X t+1/2 = P X t (-γ t V t ) Y t+1 = Y t -V t+1/2 X t+1 = Q(γ t+1 Y t+1 ) (DualX)
Once more, if the optimizer chooses the euclidean regularizer h

(x) = 1/2 x 2 2
Mirror Dual Extrapolation then (DualX) boils down directly to (DualX).

Having described the crucial algorithmic methods in what follows we shall exploit their adaptivity to the particular geometrical features of problems which exhibit "gradient" singularities.

Part II P R O P O S E D M E T H O D S A N D T H E I R G U A R A N T E E S

# This section incorporates material from the paper [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF] O ur first set of results concerns the generic framework of OCO problems. In particular, in Section 5.1 we present an optimal regret minimization result for convex losses which satisfy (RLC) instead of the traditional (euclidean based) Lipschitz continuity condition. Moving forward, in Section 5.2 we apply our regret minimization result to the particular case of stochastic non-smooth convex minimization and provide the respective convergence rates for such problems. In addition, as an extra feature we also provide an almost sure convergence result towards the problem's set of f minimizers.

regret minimization

Throughout this section, we make the following blanket assumptions:

Online Convex Optimization Setting 1. The t-th stage loss function f t : X → R is convex and satisfies (MB) with constant G t .

2. The optimizer's aggregate loss ∑ T t=1 f t attains its minimum value at some x * ∈ X .

The purpose of the last assumption is to avoid cases where the infimum of a loss function is not attained within the problem's feasible region (such as e -x over R + ).

Moreover, from an algorithmic point of view we consider the (LMD) template

Online Lazy Mirror

Descent Y t+1 = Y t -γ t V t X t+1 = Q(Y t+1 ) (5.1)
which satisfies the following blanket assumption:

Blanket Assumptions 1.
The underlying regularizer h is a Bregman-Finsler function, i.e., it satisfies Definition 3.3.

2.

The algorithm is initialized at the "prox-center" x c = arg min h of X and is run with (constant) step-size α/T 1/2 for some α > 0 chosen by the optimizer.

3. Finally, for (SFO). we make the following assumptions:

a) Unbiasedness: E[ ĝt | F t ] = ∇ f t (X t ). (5.2a) b) Finite mean square: E[ ĝt 2 * | F t ] ≤ M 2 t .
(5.2b)

Having established the algorithmic framework, we are in the position to introduce our first result:

NoLips Regret Guarantees

Theorem 5.1 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF]). Let Reg(T)

≡ Reg x * (T) with x * ∈ arg min x∈X ∑ T t=1 f (x) and M 2 T = T -1 ∑ T t=1 M 2 t .
Then, (LMD) algorithm with noisy feedback of the form (SFO) enjoys the mean regret bound:

E[Reg(T)] ≤ D(x * , x c ) α + αM 2 T 2α √ T (5.3)
In particular, if sup tN M t < +∞ then the method guarantees O( √ T) regret.

The main idea behind the proof of Theorem 5.1 is to relate the Finslerian structure of X to the Bregman regularization framework underlying (LMD). A first such link is provided by the Bregman divergence; however, because of the primal-dual interplay between X t ∈ X and Y t ∈ V * , the Bregman divergence is not sufficiently adapted. At this point is were the Fenchel coupling, defined in (4.18), comes in handy. Armed with toolkit, we are able to establish the main "energy" inequality for this section. Formally, we have the following result.

Regret Template Inequality

Proposition 5.2 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF]). Let h be a Bregman-Finsler regularizer on X with convexity modulus α, fix some p ∈ X , let x = Q(y) for some y ∈ Y. Then, for all w ∈ Y, we have:

F(p, y + w) ≤ F(p, y) + w, x -p + 1 2α w 2
x, *

(5.4)

Proof. By the three-point identity (4.21), we get

F(p, y) = F(p, y + w) + F(Q(y + w), y) + y -(y + w), Q(y + w) -p (5.5)
and hence, after rearranging:

F(p, y + w) = F(p, y) -F(Q(y + w), y) + w, Q(y + w) -p = F(p, y) -F(Q(y + w), y) + w, x -p + w, Q(y + w) -x (5.6)
By Young's inequality [START_REF] Tyrrell | Convex Analysis[END_REF], we also have

w, Q(y + w) -x ≤ α 2 Q(y + w) -x 2 x + 1 2α w 2 x, * (5.7)
Our claim then follows by the fact that

F(Q(y + w), y) ≥ α 2 Q(y + w) -x 2 x (cf. Lemma 4.5).
Proof of Theorem 5.1. Now, applying Proposition 5.2 to (LMD), we get:

F(x * , Y t+1 ) ≤ F(x * , Y t ) -γ ĝt , X t -x * + γ 2 2α ĝt 2 X t , * = F(x * , Y t ) + γ ∇ f t (X t ), x * -X t -γ U t+1 , X t -x * + γ 2 2α ĝt 2 X t , * . (5.8)
Hence, after rearranging and telescoping, we obtain

Reg(T) ≤ T ∑ t=1 ∇ f t (X t ), X t -x * ≤ D(x * , x c ) γ + T ∑ t=1 ξ t+1 + γ 2α T ∑ t=1 ĝt 2 X t , * (5.9)
where, in the last line, we used the definition of the Finsler dual norm • * ≡ • x * , * , and we set ξ t+1 = U t+1 , x * -X t . By taking expectations on both sides, we have:

E [Reg(T)] ≤ D(x * , x c ) γ + T ∑ t=1 E [ξ t+1 ] + T ∑ t=1 E ĝt 2 X t , * (5.10)
We examine each (RHS) term individually. In particular, we have:

• For the term ∑ T t=1 E [ξ t+1 ] we have:

E [ξ t+1 ] = E [ U t+1 , x * -X t ] = E [E [ U t+1 , x * -X t |F t ]] = E [ E [U t+1 |F t ] , x * -X t ] = 0
with the last equality being obtained by the unbiasedness of (SFO)

• For the term ∑ T t=1 E ĝt 2 X t , * we have by the finite mean square assumption of (SFO):

T ∑ t=1 E ĝt 2 X t , * = O( √ T) (5.11)
Finally the result follows by combining the above.

Remark 5.1. We emphasize here that the O( √ M) is known to the optimizer, (5.3) can be optimized further by tuning α.

T) regret bound above is achieved even if X is unbounded or if the range of h H ≡ sup x∈X D(x, x c ) = sup h -inf h of X is infinite.To see this, simply note that D(x, x c ) = h(x) -h(x c ) -∇h(x c ), x -x c < ∞ for all x ∈ X = dom h (recall also that, since x c = arg min h, we have 0 ∈ ∂h(x c ) so x c ∈ dom ∂h). Of course, if H < ∞ and G (or
Our analysis hinges on controlling the second-order error term in (5.2) by means of the (MB) continuity assumption. It is precisely this primal-dual inequality which allows us to go beyond the standard Lipschitz framework: compared to (primal-primal) inequalities of a similar form for global norms [START_REF] Balandat | Minimizing regret on reflexive Banach spaces and Nash equilibria in continuous zero-sum games[END_REF][START_REF] Krichene | Continuous and discrete dynamics for online learning and convex optimization[END_REF][START_REF] Arkadi Semen Nemirovski | Robust stochastic approximation approach to stochastic programming[END_REF][START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF][START_REF] Zhou | Stochastic mirror descent for variationally coherent optimization problems[END_REF], the distinguishing feature of (5.2) is the advent of the Finsler induced norm w x, * . Thanks to the intricate connection between the Finsler norm and h, the secondorder term in (5.2) can be controlled even when the received gradient is unbounded relative to any global norm, i.e., even if the objective is singular.

The main obstacle to achieve this is that the underlying local norm, the Fenchel coupling F and the Bregman divergence D (all state-dependent notions of distance) need not be compatible with one another. That this is indeed the case is obtained by Lemma 4.5; what plays a crucial role in deriving (5.2) is to introduce the local strong convexity with respect to the the Finsler norm to the second argument of the Bregman divergence instead of the first (or any other point in-between). Any other relation between the local norms and h along these lines is not amenable to analyzing (LMD) in this framework.

application to stochastic non-smooth minimization

As announced, the second part of our analysis focuses on the application of the above regret analysis on stochastic non-smooth optimization problems of the form:

Non-Smooth Stochastic Minimization minimize f (x) = E[F(x; ω)] subject to x ∈ X (Opt)
with the expectation taken over some model sample space Ω. Our first result here is as follows:

NonLips Non-Smooth Guarantees Theorem 5.3. Assume that f is convex and satisfies (MB) in mean square, i.e.,

sup x E[ ∇F(x; ω) 2 x, * ] ≤ M 2 (5.12)
for some M > 0. If (LMD) is run for T iterations with a constant step-size of the form α/ √ T and stochastic gradients ĝt = ∇F(X t ; ω t ) generated by an i.i.d. sequence ω t ∈ Ω, we have

E[ f ( XT )] ≤ min f + D c α + αM 2 2α 1 √ T (5. 13
)
where XT = (1/T) ∑ T t=1 X t is the "ergodic average" of X t and

D c = inf x * ∈arg min f D(x * , x c ) < ∞ (5.14)
denotes the Bregman distance of the prox-center x c of X to arg min f .

The key novelty in Theorem 5.3 is that the optimal O(T -1/2 ) convergence rate of (LMD) is maintained even if the stochastic gradients of F become singular at residual points x ∈ cl(X ) \ X . The proof of Theorem 5.3 likewise relies on an online-tobatch conversion of the regret guarantees of (LMD) for the sequence of stochastic gradients ∇F(•; ω t ) of f .

To go beyond the ergodic guarantees of Theorem 5.3, we also analyze below the convergence of the "last iterate" of online mirror descent (OMD), i.e., the actual sequence of generated points X t . This is of particular interest for non-convex problems where ergodic convergence results are of limited value (because Jensen's inequality no longer applies). To obtain global convergence results in this setting, we focus on a class of functions which satisfy a weak secant inequality of the form

Weak Secant Inequality inf{ ∇ f (x), x -x * : x * ∈ arg min f , x ∈ K} > 0 (SI)
for every closed subset K of X that is separated by neighborhoods from arg min f . Variants of this condition have been widely studied in the literature and include non-convex functions with complicated ridge structures [START_REF] Bottou | Online learning and stochastic approximations[END_REF][START_REF] Facchinei | Finite-Dimensional Variational Inequalities and Complementarity Problems[END_REF][START_REF] Jiang | Stochastic approximation approaches to the stochastic variational inequality problem[END_REF][START_REF] Karimi | Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition[END_REF][START_REF] Ljung | Strong convergence of a stochastic approximation algorithm[END_REF][START_REF] Nevel | Stochastic Approximation and Recursive Estimation[END_REF][START_REF] Zhang | Gradient methods for convex minimization: Better rates under weaker conditions[END_REF][START_REF] Zhou | Stochastic mirror descent for variationally coherent optimization problems[END_REF].

In this very general setting, we have:

Theorem 5.4. Assume f satisfies (SI) and satisfies (MB) in L 2 . Suppose further that Last Iterate Convergence arg min f is bounded and (LMD) is run with a sequence of stochastic gradients ĝt = ∇F(X t ; ω t ), a Bregman-Finsler regularizer h, and a variable step-size γ t such that

∑ ∞ t=1 γ t = ∞, ∑ ∞ t=1 γ 2 t < ∞.
Then, with probability 1, X t converges to some (possi- bly random) x * ∈ arg min f . We begin by recalling two important results from probability theory. The first is a version of the law of large numbers for martingale difference sequences that are bounded in L 2 [START_REF] Hall | Martingale Limit Theory and Its Application[END_REF]:

Law of Large Numbers for Martingales Theorem 5.5 (Hall and Heyde [47]). Let Y t = ∑ t i=1 ζ i be a martingale and β t a non-decreasing positive sequence such that lim t→∞ β t = ∞. Then,

lim t→∞ Y t /β t = 0 almost surely (5.15) on the set ∑ ∞ t=1 β -2 t E[ζ 2 t | F t-1 ] < ∞.
The second is a convergence result for quasi-supermartingales due to Robbins and Sigmund [START_REF] Robbins | A convergence theorem for nonnegative almost supermartingales and some applications[END_REF]:

Stochastic Quasi-Fejer Sequences Lemma 5.6 (Robbins and Sigmund [START_REF] Robbins | A convergence theorem for nonnegative almost supermartingales and some applications[END_REF]). Let (F t ) t∈N be a non-decreasing sequence of σ-algebras. Let (α t ) t∈N , (θ t ) t∈N non-negative F tmeasurable random variables, (η t ) t∈N is an F tmeasurable non-negative summable random variable and the following inequality holds:

E[α t+1 | F t ] ≤ α t -θ t + η t almost surely (5.16)
Then, (α t ) t∈N converges almost surely towards a [0, ∞)-valued random variable.

An application of this lemma leads us to the following result which is of independent interest: Proposition 5.7 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF]). Let X t be the sequence of iterates generated by (LMD) run with a step-size sequence γ t such that ∑ ∞ t=1 γ 2 t < ∞ and a stochastic oracle as in the statement of Theorems 5.3 and 5.4. Then, for all x * ∈ arg min f , F(x * , Y t ) converges with probability 1.

Proof. Let x * ∈ arg min f . Recalling our main estimation:

F(x * , Y t+1 ) ≤ F(x * , Y t ) -γ t ĝt , X t -x * x + γ 2 t 2α ĝt 2 X t , * (5.17)
and taking conditional expectations on both sides, we get due to F tmeasurability arguments:

E[F(x * , Y t+1 )|F t ] ≤ F(x * , Y t ) -γ t ĝt , X t -x * x + γ 2 t 2α E[ ĝt 2 X t , * |F t ]. (5.18) Since, (2α) -1 ∑ ∞ t=1 γ 2 t E[ ĝt 2 X t , * |F t ] ≤ M(2α) -1 ∑ ∞ t=1 γ 2
t < ∞ by applying the above we get the result.

Having this at hand, we can establish the following proposition:

Almost Sure Boundedness Proposition 5.8. Let X t be the sequence of iterates generated by (LMD) with assumptions as in Theorem 5.4. Then, for all x * ∈ arg min f , the sequence X tx * X t is bounded with probability 1.

Proof. Recalling our main estimation and taking condition expectations on both sides, we get:

E[F(x * , Y t+1 ) | F t ] ≤ F(x * , Y t ) -γ t ĝt , X t -x * x + γ 2 t 2α E[ ĝt 2 X t , * |F t ] (5.19)
Hence, by the above corollary, we have that the sequence F(x * , Y t ) converges with probability 1 for all x * ∈ arg min f . Thus, it is also bounded with probability 1 for all x * . We then get

X t -x * 2 X t ≤ 2 α F(x * , Y t ) (5.20)
which concludes our proof.

We continue by showing that X t possesses a subsequence that converges to arg min f :

Existence of Convergent

Sub-sequence Proposition 5.9 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF]). Let X t be the sequence of iterates generated by (LMD) with assumptions as in Theorem 5.4. Then, with probability 1, there exists a (possibly random) subsequence of X t which converges to arg min f .

Proof. Assume to the contrary that, with positive probability, the sequence X t generated by (LMD) admits no limit points in arg min f . Conditioning on this event, there exists a (nonempty) closed set C ⊂ X which is separated by neighborhoods from arg min f and is such that X t ∈ C for all suffiently large t. Then, by relabeling X t if necessary, we can assume without loss of generality that X t ∈ C for all t ∈ N. Thus, by Proposition 5.2, we get:

F(x * , Y t+1 ) ≤ F(x * , Y t ) -γ t ĝt , X t -x * + γ 2 t 2α ĝt 2 X t , * = F(x * , Y t ) -γ t ∇ f (X t ), X t -x * -γ t U t+1 , X t -x * + γ 2 t 2α ĝt 2 X t , * ≤ F(x * , Y t ) -γ t δ(C) + γ t ξ t+1 + γ 2 t 2α ĝt 2 X t , * (5.21)
where in the last line we set

δ(C) = inf{ ∇ f (x), x -x * : x * ∈ arg min f , x ∈ C} > 0 (by (SI)), U t+1 = ĝt -∇ f (X t ), ξ t+1 = -U t+1 , X t -x * and β t = ∑ t i=1 γ i .
Thus, by telescoping and factorizing we get:

F(x * , Y t+1 ) ≤ F(x * , Y 1 ) -β t δ(C) - ∑ t s=1 γ s ξ s+1 β t - ∑ t s=1 γ 2 s ĝs 2 X s , * 2αβ t (5.22)
By the unbiasedness assumption for U t , we have

E[ξ t+1 | F t ] = E[U t+1 | F t ], X t - x * = 0. Moreover, for all x * ∈ arg min f , we have ∞ ∑ t=1 γ 2 t E[ξ t+1 | F t ] ≤ ∞ ∑ t=1 γ 2 t X t -x * 2 X t E[U t+1 | F t ] ≤ ∞ ∑ t=1 γ 2 t F(x * , Y t ) E[U t+1 | F t ] < ∞ (5.23)
where the last (strict) inequality is obtained due to the finite mean square property, the boundness of F(x * , Y t ) and the fact that ∑ ∞ t=1 γ 2 t < ∞. Thus, we can apply the law of large numbers for L 2martingales stated above and conclude that β -1 t ∑ t s=1 γ s ξ s+1 converges to 0 almost surely. On the other hand, for the term S t+1 = ∑ t s=1 γ 2 s ĝs 2 X t , * , since ĝs+1 is F s -measurable for all s = 1, 2 . . . , t -1 we have:

E[S t+1 | F t ] = E t-1 ∑ i=1 γ 2 t ĝi 2 x i , * + γ 2 t ĝt 2 X t , * F t = S t + γ 2 t E ĝt 2 X t , * F t ≥ S t
(5.24) so S t is a submartingale with respect to F t . Furthermore, by the law of total expectation, we also get:

E[S t+1 ] = E[E[S t+1 | F t ]] ≤ σ 2 t ∑ i=1 γ 2 i ≤ σ 2 ∞ ∑ t=1 γ 2 t < ∞, (5.25)
implying that S t is bounded in L 1 . Thus, due to Doob's submartingale convergence theorem [START_REF] Hall | Martingale Limit Theory and Its Application[END_REF], we coclude that S t converges to some (almost surely finite) random variable S ∞ so lim t→∞ S t+1

β t = 0 with probability 1. Now, by letting t → ∞ in (5.22), we get F(x * , Y t ) → -∞, a contradiction. Going back to our original assumption, this shows that there exists a subsequence of X t which converges to arg min f with probability 1, as claimed.

With all this at hand, we proceed to the proof of our convergence result: Proof of Theorem 5.4. By the boundedness (and hence compactness) of arg min f , Proposition 5.9 implies that, with probability 1, there exists some x * ∈ arg min f such that X t k → x * for some (possibly random) subsequence X t k of X t . By the Riemann-Legendre property of h, it follows that

F(x * , Y t k ) = D(x * , X t k ) → 0 as k → ∞, implying in turn that lim t→∞ D(x * , X t ) = 0 (by Proposition 5.7). Since D(x * , X t ) ≥ α X t -x * 2 X t ≥ µ X t -x * 2
, we conclude that X t → x * , and our proof is complete.

The above result is a first step towards establishing convergence rate guarantees for NoLips (non-smooth) convex minimization problems. In what will follow, we dive into more detail regarding the interplay between the different NoLips regularity conditions.

numerical evaluation in poisson inverse problems

For the purposes of validation, we proceed with an application of our algorithmic results to a broad class of Poisson inverse problems that arise in tomography problems; the objective of interest here is the Poisson likelihood loss (generalized Kullback-Leibler divergence):

������� ���� ��� ��������������                                   • • • • • • ••• • •• • •• • • • • • • • • • • • • • • • • • • • • • • • ◼ ◼ ◼ ◼ ◼ ◼◼◼◼ ◼◼◼ ◼ ◼◼ ◼◼ ◼◼ ◼ ◼ ◼ ◼◼ ◼ ◼◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◻ ◻ ◻ ◻ ◻ ◻◻ ◻◻ ◻◻◻ ◻ ◻◻ ◻◻ ◻◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◻ ◆ ◆ ◆ ◆ ◆◆◆ ◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◆◆ ◇ ◇ ◇ ◇ ◇ ◇◇ ◇◇ ◇ ◇ ◇ ◇ ◇ ◇◇ ◇ ◇ ◇◇ ◇ ◇◇ ◇ ◇◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇  �� (
f (x) = ∑ N j=1 x j log x j (Hx) j + (Hx) j -x j (5.26)
where x ∈ R N + is a vector of Poisson data observations (e.g., pixel intensities) and H ∈ R N×n is an ill-conditioned matrix representing the data-gathering protocol. From an algorithmic point of view we will restrict ourselves to the Riemannian framework. Since the generalized KL objective of (5.26) exhibits an O(1/x) singularity at the boundary of the orthant, we consider the Poincaré metric g(x) = diag(1/x 1 , . . . , 1/x n ) under which the KL divergence is Riemann-Lipschitz continuous. ( Going back to PIP, a suitable Riemannian regularizer for this metric is h(x) = ∑ N i=1 1/x 2 i , which is 1-strongly convex relative to g. We then run the induced mirror descent algorithm with an online-to-batch conversion mechanism as described in Section 5.2. For reference purposes, we call the resulting process Riemannian mirror descent (RMD).

Subsequently, we ran RMD on a Poisson denoising problem for a 384 × 384 test image contaminated with Poisson noise (so n ≈ 10 5 in this case). For benchmarking, we also ran a fast variant of the widely used Lucy-Richardson (LR) algorithm [START_REF] Bertero | Image deblurring with Poisson data: from cells to galaxies[END_REF], and the recent composite mirror prox (CMP) method of [START_REF] He | Fast and simple optimization for Poisson likelihood models[END_REF]; all methods were run with stochastic gradients and the same minibatch size. Because of the "dark area" gradient singularities when [Hx] j → 0, Euclidean stochastic gradient methods oscillate without converging, so they are not reported. As we see in Fig. 5.1, the RMD process provides the sharpest reconstruction of the original. In particular, after an initial warm-up phase, the last iterate of Riemannian mirror descent consistently outperforms the LR algorithm by 7 orders of magnitude, and CMP by 3. We also note that the Poisson likelihood loss decreases faster under the last iterate of RMD relative to the different algorithmic variants that we tested, exactly because of the hysteresis effect that is inherent to ergodic averaging.

Overall, we note that the introduction of an additional degree of freedom (the choice of Bregman function and that of the local Riemannian norm), makes RMD a particularly flexible and powerful paradigm for loss models with singularities.

We find these results particularly encouraging for further investigations on the interplay between Riemannian geometry and Bregman-proximal methods.

# This section incorporates material from the paper [START_REF] Antonakopoulos | Adaptive first-order methods revisited: Convex minimization without lipschitz requirements[END_REF] I n this chapter, we proceed to treat in depth the specific case of convex minimiza- tion problems. The key feature which differentiates the present analysis with the results of Chapter 5 is that here we aim to establish order optimal convergence rate guarantees for both deterministic and/or stochastic oracle feedback and (RC) and (RS) respectively.

In doing so, a first issue that should be tackled is to determine what optimal interpolation means for the NoLips framework, i.e., to determine the respective worst case lower bounds under (RC) and (RS). To begin with, Theorem 5.3 provides a convergence rate of order O(1/ √ T) under (RC) which matches the worst-case lower bound presented in Section 2.4.2.

Therefore, the most intriguing part is whether an O(1/T 2 ) bound can also be achieved in the (RS) case. This question remained open until [START_REF] Dragomir | Optimal complexity and certification of Bregman first-order merthods[END_REF] established the discrepancy between standard smoothness and (RC). More precisely, they showed that the worst case lower bound under (RS) is Ω(1/T) and hence the optimal rate for relatively smooth problems does not match the O(1/T 2 ) rate for standard Lipschitz smooth problems. 1With all this in hand, in Section 6.1 we begin by describing the respective "universal" step-size policy that we will study to interpolate between (RC) and (RS). Moving forward, in Section 6.2 we present our results with respect to a deterministic oracle. More precisely, we show that the time-averages of the (MD) iterates run with our adaptive step-size policy achieve simultaneously order optimal guarantees for both (RS) and (RC) objectives. As an additional feature we establish that the actual iterates of the method -before any averaging occurs-converge towards the solution set.

We conclude this chapter by providing the respective convergence rate guarantees for stochastic case. More precisely, in Section 6.3 we illustrate the respective stochastic rates under (RC) and (RS) via explicit upper bounds.

a universal step-size

Throughout this chapter, the blanket algorithmic template which we will be focusing on is that of (MD), i.e., For the purposes of this table, (L) refers to "Lipschitz" and (R) to "relative" continuity or smoothness respectively. In the case of AdaProx, "partial" means that the non-Lipschitz conditions under which it guarantees convergence form a subset of (RC) / (RS). Logarithmic factors are ignored throughout; we also note that the O(1/T) rate in the column (RS) is, in general, unimprovable [START_REF] Dragomir | Optimal complexity and certification of Bregman first-order merthods[END_REF].

X + = P X (-γV) (

where P is a (Bregman) proximal operator associated to a Bregman function h as per Definition 3.1 and V ∈ V * . The next important element for our analysis is to define the method's step-size. In the unconstrained case, as we described in Section 2.7 , a popular adaptive choice is the so-called "inverse-sum-of-squares" policy:

γ t = 1 ∑ t s=1 ∇ f (X s ) 2 * , (6.2)
where X t is the series of iterates produced by the algorithm. However, in relatively continuous/smooth problems, this definition encounters two crucial issues. First, because the gradient of f is unbounded (even over a bounded domain), the denominator of (6.2) may grow at an uncontrollable rate, leading to a step-size policy that vanishes too fast to be of any practical use. The second is that, if the problem is constrained, the extra terms entering the denominator of γ t do not vanish as the algorithm approaches a solution, so (6.2) may still be unable to exploit the smoothness of the objective.

We begin by addressing the second issue. In the Euclidean case, the key observation is that the difference x +x must always vanish near a solution (even near the boundary), so we can use it as a proxy for ∇ f (x) in constrained problems. This idea is formalized by the notion of the gradient mapping [START_REF] Nesterov | Introductory Lectures on Convex Optimization: A Basic Course[END_REF] that can be defined here as

Gradient Mapping δ = x + -x γ. (6.3)
On the other hand, in a Bregman setting, the prox-mapping tends to deflate gradient steps, so the norm difference between two successive iterates x + and x of (MD) could be very small relative to the oracle signal that was used to generate the update. As a result, the Euclidean residual (6.3) could lead to a disproportionately large step-size that would be harmful for convergence. For this reason, we consider a gradient mapping that takes into account the Bregman geometry of the method and we set δ = D(x, x + ) + D(x + , x) γ.

(6.4)

Obviously, when h(x) = (1/2) x 2 2 , we readily recover the definition of the

Bregman Gradient Mapping

Euclidean gradient mapping (6.3). In general however, by the strong convexity of h, the value of this "Bregman residual" exceeds the corresponding Euclidean definition, so the induced step-size exhibits smoother variations that are more adapted to the framework in hand.

Having all this hand, we are in a position to put everything together and define our adaptive (MD) method. In this regard, combining the abstract template (MD) with the Bregman residual and "inverse-sum-of-squares" approach discussed above, we will consider the recursive policy AdaMir Algorithm X t+1 = P X t (-γ t V t ) (6.5) with V t , t = 1, 2, . . . , coming from a oracle model of the form (SFO), and with γ t defined as

γ t = 1 ∑ t-1 s=0 δ 2 s with δ 2 s = D(X s , X s+1 ) + D(X s+1 , X s ) γ 2 s . (Adapt)
In the sequel, we will use the term to refer interchangeably to the update X t ← X t+1 and the specific step-size policy used within. The convergence properties of (MD) run with (Adapt); abbreviated as AdaMir are discussed in detail in the next two sections in both deterministic and stochastic problems.

deterministic analysis

We are now in a position to state our main convergence results for our method. We begin with the deterministic analysis (σ = 0), treating both the method's "timeaverage" as well as the induced trajectory of query points; the analysis for the stochastic case (σ > 0) is presented in the next section.

Ergodic convergence and rate interpolation

We begin by showing the convergence rate guarantees of the method's "timeaveraged" state, i.e., X T = (1/T) ∑ T t=1 X t . More precisely, we show that our method simultaneously achieves an O(1/ √ T) value convergence rate under (RC) and O(1/T) under (RS). Moreover, if both regularity conditions are satisfied we are able to obtain a more detailed "any-time" rate. Formally, we have the following result. Theorem 6.1. Let X t , t = 1, 2, . . . , denote the sequence of iterates generated by AdaMir, and let D 1 = D(x * , X 1 ). Then, AdaMir simultaneously enjoys the following guarantees:

1. If f satisfies (RC), we have:

f (X T ) -min f ≤ √ 2G D 1 + 8G 2 /δ 2 0 + 2 log(1 + 2G 2 T/δ 2 0 ) √ T + 3 √ 2G + 4G 2 /δ 2 0 T . (6.6)
2. If f satisfies (RS), we have f (X T )min f = O(D 1 /T).

3. If f satisfies (RS) and (RC), we have:

f (X T ) -min f ≤ f (X 1 ) -min f + 2 + 8G 2 δ 2 0 + 2 log 4β 2 δ 2 0 β 2 D 1 T . (6.7)
Universality Guarantees (Deterministic)

As we already mentioned Theorem 6.1 shows that, up to logarithmic factors, AdaMir achieves the optimal lower bounds for objectives which satisfy either belong to the (RC) oracle complexity class or satisfy (RS). The key element of the proof is to the following regret bound:

∑ t=1 [ f (X t ) -f (x * )] ≤ D 1 γ T + ∑ T t=1 γ 2 t δ 2 t γ T + T ∑ t=1 γ t δ 2 t . (6.8)
Proof. By the convexity of f and the definition of the Bregman proximal step in Proposition 4.3, we have:

f (X t ) -f (x * ) ≤ V t , X t -x * ≤ 1 γ t ∇h(X t ) -∇h(X t+1 ), X t -x * . (6.9)
Hence, by applying again the three-point identity (Lemma 4.2), we obtain:

f (X t ) -f (x * ) ≤ D(x * , X t ) -D(x * , ) γ t + D(X t , ) γ t ≤ D(x * , X t ) -D(x * , ) γ t + D(X t , ) + D(, X t ) γ t = D(x * , X t ) -D(x * , ) γ t + γ t δ 2 t (6.10)
where the last equality follows readily from the definition (6.4) of δ t . Therefore, by summing through t = 1, 2, . . . , T, we obtain:

T ∑ t=1 [ f (X t ) -f (x * )] ≤ D(x * , X 1 ) γ 1 + T ∑ t=2 1 γ t - 1 γ t-1 D(x * , X t ) + T ∑ t=1 γ t δ 2 t . (6.11)
Now, we are left to bound from above the second term on the right-hand side (RHS) of (6.11). By the second part of Proposition 4.3, we have:

D(x * , X s+1 ) ≤ D(x * , X s ) -γ t V t , X t -x * + D(X s , X s+1 ) ≤ D(x * , X s ) + D(X s , X s+1 ) ≤ D(x * , X s ) + D(X s+1 , X s ) + D(X s , X s+1 ) (6.12)
Thus, by telescoping through s = 1, 2, . . . , t, we obtain:

D(x * , X t ) ≤ D(x * , X 1 ) + t ∑ s=1 [D(X s , X s+1 ) + D(X s+1 , X s )] ≤ D(x * , X 1 ) + T ∑ s=1 [D(X s , X s+1 ) + D(X s+1 , X s )] = D(x * , X 1 ) + T ∑ s=1 γ 2 s δ 2 s (6.13)
where the last equality follows from the definition (6.4) of δ t . So, summarizing

T ∑ t=2 1 γ t - 1 γ t-1 D(x * , X t ) ≤ T ∑ t=2 1 γ t - 1 γ t-1 D(x * , X 1 ) + T ∑ s=1 γ 2 s δ 2 s ≤ D(x * , X 1 ) γ T - D(x * , X 1 ) γ 1 + T ∑ s=1 γ 2 s δ 2 s • T ∑ t=1 1 γ t - 1 γ t-1 ≤ D(x * , X 1 ) γ T - D(x * , X 1 ) γ 1 + ∑ T t=1 γ 2 t δ 2 t γ T . (6.14)
Hence, by combining the above with (6.11), our claim follows.

The proof of Proposition 6.2 hinges on the specific definition of the adaptive step-size, and the exact functional form of the regret bound (6.8) plays a crucial role in the sequel. Specifically, under the regularity conditions (RC) and (RS), we respectively obtain the following key lemmas:

Boundedness of the Residuals Under (RC) Lemma 6.3. Under (RC), the sequence of the Bregman residuals δ t of is bounded as δ 2 t ≤ 2G 2 for all t ≥ 1.

Proof. By the definition of the Bregman proximal step in (MD) and Proposition 4.3, we have:

D(X t , X t+1 ) + D(X t+1 , X t ) = ∇h(X t ) -∇h(X t+1 ), X t -X t+1 ≤ γ t V t , X t -X t+1 . (6.15)
Hence, by invoking (RC) we get:

D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ t G 2D(X t+1 , X t ) ≤ γ t G 2 [D(X t+1 , X t ) + D(X t , X t+1 )] (6.16)
We thus get: D(X t , X t+1 ) + D(X t+1 , X t ) ≤ 2γ 2 t G 2 . (6.17)

Hence, by the definition (6.4) of δ 2 t , we conclude that

δ 2 t = D(X t , X t+1 ) + D(X t+1 , X t ) γ 2 t ≤ 2G 2 .
(6.18) Lemma 6.4. Under (RS), the sequence of the Bregman residuals δ t is square-summable, i.e., ∑ t δ 2 t < ∞. Consequently, the method's step-size converges to a positive limit γ ∞ > 0.

Summability of the Residuals Under (RS)

Proof. Since the adaptive step-size policy γ t is decreasing and bounded from below (γ t ) ≥ 0 we get that its limit exists,i.e., lim

t→+∞ γ t = γ ∞ for some γ ∞ ≥ 0 (6.19)
Assume that γ ∞ = 0. By Proposition 3.3, we obtain:

f (X t+1 ) ≤ f (X t ) + ∇ f (X t ), X t+1 -X t + βD(X t+1 , X t ) ≤ f (X t ) - 1 γ t D(X t , X t+1 ) - 1 γ t D(X t+1 , X t ) + β [D(X t , X t+1 ) + D(X t+1 , X t )] (6.20)
whereas by recalling the definition of the residuals (Adapt) the above can be rewritten as follows:

f (X t+1 ) ≤ f (X t ) -γ t δ 2 t + βγ 2 t δ 2 t = f (X t ) - 1 2 γ t δ 2 t - 1 2 γ t δ 2 t + βγ 2 t δ 2 t (6.21)
Moreover, by rearranging and factorizing the common term γ t δ 2 t we get:

1 2 γ t δ 2 t ≤ f (X t ) -f (X t+1 ) + γ t δ 2 t βγ t - 1 2 (6.22)
Now, by the fact that βγ t -1 2 ≤ 0 for γ t ≤ 1/2β and the fact that γ t converges to 0 by assumption, we get that there exists some t 0 ∈ N such that:

βγ t - 1 2
≤ 0 for all t > t 0 (6.23)

Hence, by telescoping for t = 1, 2, . . . , T for sufficiently large T, we have

1 2 T ∑ t=1 γ t δ 2 t ≤ f (X 1 ) -f (X T+1 ) + t 0 ∑ t=1 βγ t - 1 2 γ t δ 2 t ≤ f (X 1 ) -min x∈X f (x) + t 0 ∑ t=1 βγ t - 1 2 γ t δ 2 t (6.24)
Now, by applying the (LHS) of Lemma A.4 we get:

1 2 1 γ T -δ 0 ≤ 1 2 δ 2 0 + T-1 ∑ t=1 γ t δ 2 t ≤ T ∑ t=1 γ t δ 2 t ≤ f (X 1 ) -min x∈X f (x) + t 0 ∑ t=1 βγ t - 1 2 γ t δ 2 t (6.
25) Since γ t → 0 we get that 1/γ t → +∞ and hence the above yields that +∞ ≤

f (X 1 ) -min x∈X f (x) + ∑ t 0 t=1 β K γ t -1 2 γ t δ 2
t , a contradiction. Therefore we get that: lim

t→+∞ γ t = γ ∞ > 0 (6.26)
Moreover, by recalling the definition of the adaptive step-size policy γ t :

γ t = 1 δ 2 0 + ∑ t-1 s=1 δ 2 s (6.27)
and after rearranging we obtain:

t-1 ∑ s=1 δ 2 s = 1 γ 2 t -δ 2 0 (6.28)
and therefore by taking limit on both sides we obtain:

+∞ ∑ t=1 δ 2 t = lim t→+∞ t-1 ∑ s=1 δ 2 s = lim t→+∞ 1 γ 2 t -δ 2 0 = 1 γ 2 ∞ -δ 2 0 < +∞ (6.29)
and hence the result follows.

As we explain below, the boundedness estimate of Lemma 6.3 is necessary to show that the iterates of the method do not explode; however, without further assumptions, it is not possible to sharpen this bound. The principal technical difficulty -and an important novelty of our analysis -is the stabilization of the step-size to a strictly positive limit in Lemma 6.4. This property plays a crucial role because the method is not slowed down near a solution. To the best of our knowledge, there is no comparable result for the step-size of parameter-agnostic methods in the literature. 2Armed with these two lemmas, we will establish below the following series of estimates:

1. Under (RC), the terms in the RHS of (6.8) can be bounded respectively as O(G √ T), O(log(G 2 T) √ T), and O(G √ T). As a result, we obtain an Õ(1/ √ T) rate of convergence.

2. Under (RS), all terms in the RHS of (6.8) can be bounded as O(1), so we obtain an O(1/T) convergence rate for XT .

We formalize all this below:

Proof of Theorem 6.1. Repeating the statement of Proposition 6.2, the iterate sequence X t generated by ADM enjoys the bound:

T ∑ t=1 [ f (X t ) -f (x * )] ≤ D(x * , X 1 ) γ T + ∑ T t=1 γ 2 t δ 2 t γ T + T ∑ t=1 γ t δ 2 t (6.8)
We now proceed to bound each term on the RHS of (6.8) from above. We consider three separate cases, first only under (RC),then under (RS) and finally when both (RC) and (RS) holds.

Under (RC): We begin with problems satisfying (RC).

Analysis Under (RC)

• For the first term, Lemma 6.3 gives:

D(x * , X 1 ) γ T = D(x * , X 1 ) T-1 ∑ t=0 δ 2 t ≤ D(x * , X 1 ) √ 2G 2 T. (6.30)
• For the second term, we have:

T ∑ t=1 γ 2 t δ 2 t ≤ T ∑ t=1 δ 2 t ∑ t-1 s=0 δ 2 s = T ∑ t=1 δ 2 t δ 2 0 + ∑ t-1 s=1 δ 2 s . (6.31)
Hence, by Lemmas 6.3 and A.5, we get:

T ∑ t=1 γ 2 t δ 2 t ≤ 2 + 8G 2 δ 2 0 + 2 log 1 + T-1 ∑ t=1 δ 2 t δ 2 0 = 2 + 8G 2 δ 2 0 + 2 log T-1 ∑ t=0 δ 2 t δ 2 0 ≤ 2 + 8G 2 δ 2 0 + 2 log 2G 2 T δ 2 0 . (6.32)
• Finally, for the third term, we get:

T ∑ t=1 γ t δ 2 t = T ∑ t=1 δ 2 t ∑ t-1 s=0 δ 2 t = T ∑ t=1 δ 2 t δ 2 0 + ∑ t-1 s=1 δ 2 t . ( 6.33)
Hence, Lemmas 6.3 and A.4 again yield:

T ∑ t=1 γ t δ 2 t ≤ 4G 2 δ 0 + 3 √ 2G + 3 δ 2 0 + T-1 ∑ t=1 δ 2 t ≤ 4G 2 δ 0 + 3 √ 2G + 3 T-1 ∑ t=0 δ 2 t ≤ 4G 2 δ 0 + 3 √ 2G + 3 √ 2G 2 T. (6.34)
The claim of Theorem 6.1 then follows by combining the above within the regret bound (6.8).

Under (RS):

We now turn to problems satisfying (RS). Recalling Lemma 6.4, we shall revisit the terms of (6.8). In particular, we have:

Analysis Under (RS)

• For the first term, we have:

D(x * , X 1 ) γ T = D(x * , X 1 ) T-1 ∑ t=0 δ 2 t ≤ D(x * , X 1 ) γ ∞ (6.35)
• For the second term, we have:

T ∑ t=1 γ 2 t δ 2 t ≤ 1 δ 2 0 T ∑ t=1 δ 2 t ≤ 1 δ 2 0 γ 2 ∞ -1 (6.36)
• Finally, for the third term, we get:

T ∑ t=1 γ t δ 2 t ≤ 1 δ 0 T ∑ t=1 δ 2 t ≤ 1 δ 0 γ 2 ∞ -δ 0 (6.37)
Combining all the above, the result follows.

3. Under (RS) and (RC): Finally, we consider objectives where (RC) and (RS) hold simultaneously. Now, by working in the same spirit as in the proof of Lemma 6.4 we get:

Analysis Under (RC) & (RS) 1 2 γ t δ 2 t ≤ f (X t ) -f (X t+1 ) + γ t δ 2 t βγ t - 1 2 (6.38)
which after telescoping t = 1, . . . , T it becomes:

1 2 T ∑ t=1 γ t δ 2 t ≤ f (X 1 ) -min x∈X f (x) + T ∑ t=1 γ t δ 2 t βγ t - 1 2 (6.39)
Now, after denoting:

t 0 = max{t ∈ N : 1 ≤ t ≤ T such that γ t ≥ 1 2β } (6.40)
and decomposing the sum we get:

1 2 T ∑ t=1 γ t δ 2 t ≤ f (X 1 ) -min x∈X f (x) + t 0 ∑ t=1 γ t δ 2 t βγ t - 1 2 + T ∑ t=t 0 +1 γ t δ 2 t βγ t - 1 2 ≤ f (X 1 ) -min x∈X f (x) + t 0 ∑ t=1 γ t δ 2 t βγ t - 1 2 ≤ f (X 1 ) -min x∈X f (x) + β t 0 ∑ t=1 γ 2 t δ 2 t (6.41)
On the other hand, by applying Lemma A.5, we have:

t 0 ∑ t=1 γ 2 t δ 2 t ≤ 2 + 8G 2 δ 2 0 + 2 log 1 + t 0 -1 ∑ t=1 δ 2 t δ 2 0 = 2 + 8G 2 δ 2 0 + 2 log 1 δ 2 0 δ 2 0 + t 0 -1 ∑ t=1 δ 2 t = 2 + 8G 2 δ 2 0 + 2 log 1 δ 2 0 γ 2 t 0 (6.42)
and by the definition of t 0 we get: .43) which yields:

t 0 ∑ t=1 γ 2 t δ 2 t ≤ 2 + 8G 2 δ 2 0 + 2 log 4β 2 δ 2 0 . ( 6
T ∑ t=1 γ t δ 2 t ≤ f (X 1 ) -min x∈X f (x) + β 2 + 8G 2 δ 2 0 + 2 log 4β 2 δ 2 0 (6.44)
The result then follows by plugging in the above bounds in (6.8).

Having established the convergence rate for the time-average iterates as output of our method, we proceed with examining the asymptotic behaviour of the iterates of the method per se.

Other modes of convergence

In complement to the analysis above, we provide below a spinoff result for the method's "last iterate", i.e., the actual trajectory of queried points. In particular, these results become more appealing for the more general non-convex landscapes. Formalizing the blanket assumption in order to get the said last-iterate convergence results we shall assume throughout this section that the underlying objective f satisfies the so-called 'secant condition" [START_REF] Bottou | Online learning and stochastic approximations[END_REF][START_REF] Zhou | On the convergence of mirror descent beyond stochastic convex programming[END_REF]:

Weak Secant Inequality inf{ ∇ f (x), x -x * : x * ∈ arg min f , x ∈ K} > 0 (SI)
for every closed subset K of X that is separated by neighborhoods from arg min f . The formal statement is as follows.

Last Iterate Convergence Theorem 6.5. Suppose that f satisfies (RC) or (RS) along with (SI) condition. Then X t converges to arg min f .

The main idea of the proof consists of two steps. The first key step is to show that, under (RC) ∪ (RS), the iterates have convergent subsequences, i.e., lim inf f (X t ) = min f . In particular, we have the following result.

Extracting a Convergenr

Sub-sequence Proposition 6.6. Assume that f satisfies (RC) or (RS) along with the (SI) and X t are the iterates generated by AdaMir. Then there exists a subsequence X k t which converges to the solution set X * .

Proof. Assume to the contrary that the sequence X t generated by AMD admits no limit points in X * = arg min f . Then there exists a (non-empty) closed set K ⊆ X which is separated by neighborhoods from arg min f and is such that X t ∈ C for all sufficiently large t. Then, by relabelling X t if necessary, we can assume without loss of generality that X t ∈ K for all t ∈ N. Thus, we have:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + D(X t , X t+1 ) ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + [D(X t , X t+1 ) + D(X t+1 , X t )] = D(x * , X t ) -γ t ∇ f (X t ), X t -x * + γ 2 t δ 2 t (6.45)
with the last equality being obtained by the definition of (6.4). Now, applying (SI) we get:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t δ(K) + γ 2 t δ 2 t (6.46) with δ(K) = inf{ ∇ f (x), x -x * : x * ∈ arg min f , x ∈ K} > 0.
Hence, by telescoping t = 1, . . . , T, factorizing and setting β t = ∑ T t=1 γ t we have:

D(x * , X T+1 ) ≤ D(x * , X 1 ) -β t δ(K) - ∑ T t=1 γ 2 t δ 2 t
β t (6.47) (6.47) will be the crucial lemma that will walk throughout our analysis. In particular, we will treat the different regularity conditions of (RC) and (RS) seperately.

1. The (RC) case: Assume that f satisfies (RC). By examining the asymptotic behaviour of each term individually, we obtain:

Sub-sequence Under (RC)
• For the term β T = ∑ T t=1 γ t , we have:

β T = T ∑ t=1 1 δ 2 0 + ∑ t-1 j=1 δ 2 t ≥ T ∑ t=1 1 δ 2 0 + 2G 2 t (6.48)
which yields that β T → +∞ and more precisely β T = Ω( √ T).

• For the term

∑ T t=1 γ 2 t δ 2 t β T
, for the numerator we have:

T ∑ t=1 γ 2 t δ 2 t = T ∑ t=1 δ 2 t δ 2 0 + ∑ t-1 j=1 δ 2 j /δ 2 0 ≤ 2 + 8G 2 /δ 2 0 + 2 log(1 + T-1 ∑ t=1 δ 2 t /δ 2 0 ) ≤ 2 + 8G 2 /δ 2 0 + 2 log(1 + 2G 2 T/δ 2 0 ) (6.49)
which yields that ∑ T t=1 γ 2 t δ 2 t = O(log T), and combined with the fact that β t = Ω( √ T) we readily get:

∑ T t=1 γ 2 t δ 2 t β T → 0 (6.50)
So, combining all the above and letting T → +∞ in (6.47), we get that D(x * , X T+1 ) → -∞, a contradiction. Therefore, the result under (RC) follows.

2. The (RS) case: On the other hand, assume that f satisfies (RS). Recalling Lemma 6.4 and the fact that γ t is decreasing we have:

Sub-sequence Under (RS) T ∑ t=1 γ t δ 2 t ≤ +∞ ∑ t=1 δ 2 t < +∞ (6.51)
which by working as in Lemma 6.4 also yields:

lim t→+∞ γ t = γ ∞ > 0 (6.52)
Additionally, since γ t is decreasing and bounded we also have that γ ∞ = inf t γ t . Now, we shall re-examine the terms of (6.47). More precisely, we have:

• For β T we have:

β T = T ∑ t=1 γ t ≥ γ ∞ T ∑ t=1 1 = γ ∞ T (6.53)
which in turn yields that β T → +∞ and more precisely β T = Ω(T).

• For the term

∑ T t=1 γ 2 t δ 2 t β T
, for the numerator we have by the fact that γ t ≤ 1/δ 0 and Lemma 6.4: 1), which combined with (6.53) gives that:

T ∑ t=1 γ t δ 2 t ≤ 1 δ 0 T ∑ t=1 δ 2 t < +∞ (6.54) which yields that ∑ T t=1 γ 2 t δ 2 t = O(
∑ T t=1 γ 2 t δ 2 t β T → 0 (6.55)
so, again combing the above and letting T → +∞ in (6.47), we get that D(x * , X T+1 ) → -∞, a contradiction. Therefore, the result follows also under (RS). Now, given the existence of a convergent subsequence, the rest of our proof strategy branches out depending on whether f satisfies (RC) or (RS). Under (RS), the analysis relies on arguments that involve a quasi-Fejér argument as in [START_REF] Bottou | Online learning and stochastic approximations[END_REF][START_REF] Patrick | Quasi-Fejérian analysis of some optimization algorithms[END_REF]; this is described by the following lemma.

Quasi-Fejer Seuences Lemma 6.7. Let χ ∈ (0, 1], (α t ) t∈N , (β t ) t∈N non-negative sequences and (ε t ) t∈N ∈ l 1 (N) such that t = 1, 2, . . . :

α t+1 ≤ χα t -β t + ε t (6.56)
Then, α t converges.

Proof. First, one shows that α t∈N is a bounded sequence. Indeed, one can derive directly that:

α t+1 ≤ χ t+1 α 0 + t ∑ k=0 χ t-k ε k (6.57)
Hence, (α t ) t∈N lies in [0, α 0 + ε], with ε = ∑ +∞ t=0 ε t . Now, one is able to extract a convergent subsequence (α k t ) t∈N , let say lim t→+∞ α k t = α ∈ [0, α 0 + ε] and fix δ > 0. Then, one can find some t 0 such that α k t 0 α < δ 2 and ∑ m>t k t 0 ε m < δ 2 . That said, we have:

0 ≤ α t ≤ α k t 0 + ∑ m>t k t 0 ε m < δ 2 + α + δ 2 = α + δ (6.58)
Hence, lim sup t α t ≤ lim inf t α t + δ. Since, δ is chosen arbitrarily the result follows.

However, under (RC), the quasi-Fejér property fails, so we prove the convergence of X t via a novel induction argument that shows that the method's iterates remain trapped within a Bregman neighborhood of x * if they enter it with a sufficiently small step-size. Therefore, we provide the relevant details of Theorem 6.5.

Proof of Theorem 6.5. We will divide our proof in two parts by distinguishing the two different regularity cases.

1. The (RC) case: Given that γ t is decreasing and bounded from below we have that its limit exists, denoted by γ ∞ ≥ 0. We shall consider two cases:

Last Iterate Under (RC)
a) γ ∞ > 0: Following the same reasoning with Lemma 6.4 we get that:

T ∑ t=1 γ 2 t δ 2 t ≤ +∞ ∑ t=1 δ 2 t < +∞ (6.59)
Hence, by recalling the inequality:

D(x * , X t+1 ) ≤ D(x * , X t ) + γ 2 t δ 2
t for all x * ∈ X * (6.60) whereas after taking infima on both sides with respect to X * , we get:

inf x * ∈X * D(x * , X t+1 ) ≤ inf x * ∈X * D(x * , X t ) + γ 2 t δ 2 t (6.61)
and since the sequence γ 2 t δ 2 t is summable we can directly apply Lemma 6.7 which yields that the sequence inf x * ∈X * D(x * , X t ) is convergent. Now, since by Proposition 6.6, AMD possesses a convergent subsequence towards the solution set X * the result follows. b) γ ∞ = 0: Pick some ε > 0 and consider the Bregman zone:

D ε = {x ∈ X : D(X * , x) < ε}. (6.62)
Then, it suffices to show that X t ∈ D ε for all sufficiently large t. In doing so, consider the inequality:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + γ 2 t δ 2 t ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + γ 2 t 2G 2 K (6.63)
with the second inequality being obtained by Lemma 6.3. To proceed, assume inductively that X t ∈ D ε . By the regularity assumptions of the regularizer h, it follows that there exists a δ-neighbourhood contained in the closure of D ε/2 . So, by the (SI) condition we have: f (x), xx * ≥ c > 0 for some c ≡ c(ε) > 0 and for all x ∈ D ε \ D ε/2 and x * ∈ X * (6.64) We consider two cases:

• X t ∈ D ε \ D ε/2 :
In. this case, we have:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + γ 2 t 2G 2 K ≤ D(x * , X t ) -γ t c + γ 2 t 2G 2 K (6.65)
Thus, provided that γ t ≤ cK 2G 2 we get that D(x * , X t+1 ) ≤ D(x * , X t ). Hence, by taking infima on both sides relative to x * ∈ X * , we get that D(X * , X t+1 ) ≤ D(X * , X t ) < ε.

• X t ∈ D ε/2 : In this case, we have:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t ∇ f (X t ), X t -x * + γ 2 t 2G 2 K ≤ D(x * , X t ) + γ 2 t 2G 2 K (6.66)
with the second inequality being obtained by the optimality of x * . Now, provided that γ 2 t ≤ εK 4G 2 or equivalently γ t ≤ √ εK 2G we have:

D(x * , X t+1 ) ≤ D(x * , X t ) + ε 2 (6.67)
whereas again by taking infima on both sides we get that D(X * , X t+1 ) ≤ D(X * , X t ) + ε 2 < ε. Hence, summarizing we have that X t+1 ∈ D ε whenever X t ∈ D ε and

γ t ≤ min{ cK 2G 2 ,
√ εK 2G }. Hence, the result follows by. Proposition 6.6 and the fact that γ t → 0.

The (RS) case Recall that we have the following inequality,

Last Iterate Under (RS) D(x * , X t+1 ) ≤ D(x * , X t ) + γ 2 t δ 2 t for all x * ∈ X * (6.68)
whereas taking infima on both sides relative to X * we readily get:

inf x * ∈X * D(x * , X t+1 ) ≤ inf x * ∈X * D(x * , X t ) + γ 2 t δ 2 t (6.69)
Now, by recalling that by Lemma 6.4, we have γ 2 t δ 2 t is summable. we can apply directly Lemma 7.10. Thus, we have the sequence inf x * ∈X * D(x * , X t ) is convergent. Moreover, Proposition 6.6 guarantees that there a subsequence of inf x * ∈X * Xx * 2 that converges to 0. We obtain that there exists also a subsequence of inf x * ∈X * D(x * , X t ) that converges to 0 and since inf x * ∈X * D(x * , X t ) is convergent, we readily get that:

inf x * ∈X * x * -X t 2 ≤ inf x * ∈X * D(x * , X t ) → 0 (6.70)
and the proof is complete.

Even more generally, Lemma 6.4 also allows us to derive results for general nonconvex problems. Indeed, the proof of Proposition 3.3 shows that min 1≤t≤T δ 2 t = O(1/T) without requiring any properties on f other than (RS). As a result, we conclude that the "best iterate" of the method -i.e., the iterate with the least residual -decays as O(1/ √ T). This fact partially generalizes a similar result obtained in [START_REF] Li | On the convergence of stochastic gradient descent with adaptive stepsizes[END_REF][START_REF] Ward | AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization[END_REF] for AdaGrad applied to non-convex problems; however, an in-depth discussion of this property would take us too far afield, so we do not attempt it.

the stochastic case

In this last section, we focus on the stochastic case (σ > 0). Our main results here are as follows.

Theorem 6.8. Let X t , t = 1, 2, . . . , denote the sequence of iterates generated by AdaMir, and let D 1 = D(x * , X 1 ) and G σ = G + σ/ √ K. Then, under (RC), we have

E [ f ( XT ) -f (x * )] ≤ (D 1 + H) δ 2 0 + 2G 2 σ T (6.71)
where H = 8G 2 σ /δ 2 0 + 2 log(1 + 2G 2 σ T/δ 2 0 ).

Guarantees of AdaMir (Stochastic)

Moreover, if (RS) kicks in, we have the sharper guarantee: Theorem 6.9. With notation as above, if f satisfies (RS), it enjoys the bound

E[ f ( XT ) -f (x * )] ≤ (2 + D 1 + H) A T + Bσ √ T (6.72)
where:

a) A = δ 0 + 2[ f (X 1 ) -min f ] + β 2 + 8G 2 σ δ 2 0 + 2 log(4β 2 /δ 2 0 ) . (6.73a) b) B = (4 + 2H)/K. (6.73b)
The proof of Theorems 6.8 and 6.9 hinges on the following key steps:

Step 1: We first show that, under (RC), the method's residuals are bounded as δ 2 t ≤ 2G 2 σ (a.s.).

Step 2: With this at hand, the workhorse for our analysis is the following boxing bound for the mean "weighted" regret

∑ T t=1 E[γ t ∇ f (X t ), X t -x * ]: E γ T T ∑ t=1 [ f (X t ) -f (x * )] ≤ E T ∑ t=1 γ t ∇ f (X t ), X t -x * ≤ D 1 + E T ∑ t=1 γ 2 t δ 2 t
We prove this bound in the supplement, where we also show that

E[∑ T t=1 γ 2 t δ 2 t ] = O(log T).
At this point the analysis between Theorems 6.8 and 6.9 branches out. First, in the case of Theorem 6.8, we show that the method's step-size is bounded from below as γ t ≥ 1/ (δ 2 0 + 2G 2 σ )t; the guarantee (6.71) then follows by the boxing bound. Instead, in the case of Theorem 6.9, the analysis is more involved and relies crucially on the lower bound γ t ≥ 1/(A + Bσ √ t). The bound (6.72) then follows by combining this lower bound for γ t with the regret boxing bound above. Therefore, we first provide the crucial lemma of almost sure boundedness of the residual. Lemma 6.10. Assume that f satisfies (RC) and X t are the AdaMir iterates run with feedback of the form (SFO). Then, the sequence of the residuals δ 2 t is bounded with probability 1. In particular, we have: Proof. By working in the same spirit, we get that:

δ 2 t ≤ G2 = √ 2G + 2 K σ 2 for all t = 1,
D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ t V t , X t -X t+1 (6.75)
and by recalling that:

V t = ∇ f (X t ) + U t (6.76)
we get with probability 1:

D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ t [ ∇ f (X t ), X t -X t+1 + U t , X t -X t+1 ] ≤ γ t G 2D(X t+1 , X t ) + U t * X t -X t+1 (6.77)
with the second inequality being obtained by (RC). Now, by invoking the strong convexity assumption of K, the (LHS) of the above becomes:

γ t G 2D(X t+1 , X t ) + U t * X t -X t+1 ≤ γ t [G 2(D(X t+1 , X t ) + D(X t , X t+1 )) + U t * 2 K (D(X t+1 , X t ) + D(X t , X t+1 ))] (6.78)
which in turn yields:

D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ t D(X t+1 , X t ) + D(X t , X t+1 ) √ 2G + 2 K U t *
(6.79) Therefore, we get:

D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ 2 t √ 2G + 2 K U t * 2 (6.80)
and by stochastic first-order oracle (SFO) we get with probability 1:

D(X t , X t+1 ) + D(X t+1 , X t ) ≤ γ 2 t √ 2G + 2 K σ 2 (6.81)
or equivalently,

δ 2 t = D(X t , X t+1 ) + D(X t+1 , X t ) γ 2 t ≤ √ 2G + 2 K σ 2 (6.82)
and the result follows.

Armed with the above we are ready to provide the detailed of Theorem 6.8 and Theorem 6.9. In particular, we have

Stochastic Analysis Under (RC)

Proof of Theorem 6.8. By the second part of Proposition 4.3, we have:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t V t , X t -x * + D(X t , X t+1 ) ≤ D(x * , X t ) -γ t V t , X t -x * + D(X t+1 , X t ) + D(X t , X t+1 ) ≤ D(x * , X t ) -γ t V t , X t -x * + γ 2 t δ 2 t (6.83)
which yields after rearranging and summing t = 1, . . . , T:

T ∑ t=1 γ t V t , X t -x * ≤ D(x * , X 1 ) + T ∑ t=1 γ 2 t δ 2 t (6.84)
and by recalling that V t = ∇ f (X t ) + U t and taking expectations on both sides we get:

E T ∑ t=1 γ t ∇ f (X t ), X t -x * ≤ D(x * , X 1 ) + E T ∑ t=1 γ t U t , X t -x * + E T ∑ t=1 γ 2 t δ 2 t (6.
85) First, we shall the (LHS) from below. In particular, we have by convexity:

E T ∑ t=1 γ t ∇ f (X t ), X t -x * ≥ E T ∑ t=1 γ t ( f (X t ) -f (x * )) (6.86) Moreover, by denoting G2 = √ 2G + 2 K σ 2
we have with probability 1:

T ∑ t=1 γ t ( f (X t ) -f (x * ) = T ∑ t=1 1 δ 2 0 + ∑ t-1 s=1 δ 2 s ( f (X t ) -f (x * ) ≥ T ∑ t=1 1 δ 2 0 + G2 t ( f (X t ) -f (x * )) ≥ T ∑ t=1 1 (δ 2 0 + G2 )t ( f (X t ) -f (x * ) ≥ 1 (δ 2 0 + G2 )T T ∑ t=1 ( f (X t ) -f (x * ) (6.87)
with the second inequality being obtained by Lemma 6.10. Hence, we get:

E T ∑ t=1 γ t ∇ f (X t ), X t -x * ≥ 1 (δ 2 0 + G2 )T E T ∑ t=1 ( f (X t ) -f (x * )) (6.88)
We now turn our attention towards to the (LHS). In particular, we shall bound each term individually from above.

• For the term E ∑ T t=1 γ t U t , X tx * :

E T ∑ t=1 γ t U t , X t -x * = T ∑ t=1 E [γ t U t , X t -x * ] = T ∑ t=1 E [E [γ t U t , X t -x * |F t ]] = T ∑ t=1 E [γ t E [ U t , X t -x * |F t ]] = T ∑ t=1 E [γ t E[U t |F t ], X t -x * ] = 0 (6.89)
with the third and the fourth equality being obtained by the fact that γ t and X t are F tmeasurable.

• For the term E ∑ T t=1 γ 2 t δ 2 t : By applying Lemma A.5 and Lemma 6.10, we have with probability 1:

T ∑ t=1 γ 2 t δ 2 t ≤ 2 + 4 G2 δ 2 0 + 2 log(1 + T ∑ t=1 δ 2 t δ 2 0 ) ≤ 2 + 4 G2 δ 2 0 + 2 log(1 + G2 Kδ 2 0 T) (6.90)
Therefore we get:

E T ∑ t=1 γ 2 t δ 2 t ≤ 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T) (6.91)
Thus, combining all the above we obtain:

1

(δ 2 0 + G2 )T E T ∑ t=1 ( f (X t ) -f (x * )) ≤ D(x * , X 1 ) + 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T) (6.92) and hence, E T ∑ t=1 ( f (X t ) -f (x * )) ≤ (δ 2 0 + G2 )T D(x * , X 1 ) + 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 )T) (6.93)
The result follows by dividing both sides by T.

Stachastic Analysis Under (RS)

Proof of Theorem 6.9. By Proposition 3.3, we have:

f (X t+1 ) ≤ f (X t ) + ∇ f (X t ), X t+1 -X t + βD(X t+1 , X t ) ≤ f (X t ) + ∇ f (X t ), X t+1 -X t + β [D(X t+1 , X t ) + D(X t , X t+1 )] = f (X t ) + V t , X t+1 -X t + U t , X t -X t+1 + βγ 2 t δ 2 t ≤ f (X t ) - 1 γ t [D(X t+1 , X t ) + D(X t , X t+1 )] + U t * X t -X t+1 + βγ 2 t δ 2 t = f (X t ) -γ t δ 2 t + U t * X t -X t+1 + βγ 2 t δ 2 t (6.94)
Now, since h is Kstrongly convex we have that:

X t -X t+1 ≤ 2 K [D(X t+1 , X t ) + D(X t , X t+1 )] = 2 K γ t δ t (6.95)
and using the fact that the noise U t * ≤ σ almost surely, we have:

f (X t+1 ) ≤ f (X t ) -γ t δ 2 t + 2 K γ t δ 2 t + βγ 2 t δ 2 t (6.96)
Therefore, after rearranging and telescoping we get:

T ∑ t=1 γ t δ 2 t ≤ 2 f (X 1 ) -min x∈X f (x) + T ∑ t=1 γ t δ 2 t (βγ t - 1 2 ) + σ 2 K T ∑ t=1 γ t δ t (6.97)
Now, let us bound each term of the (RHS) of the above individually:

• For the term ∑ T t=1 γ t δ 2 t (βγ t -1 2 ) we first set:

t 0 = max{1 ≤ t ≤ T : γ t ≥ 1 2β } (6.98)
Then, by decomposing the said sum we get:

T ∑ t=1 γ t δ 2 t (βγ t - 1 2 ) = t 0 ∑ t=1 γ t δ 2 t (βγ t - 1 2 ) + T ∑ t=t 0 +1 γ t δ 2 t (βγ t - 1 2 ) ≤ t 0 ∑ t=1 γ t δ 2 t (βγ t - 1 2
)

≤ β t 0 ∑ t=1 γ 2 t δ 2 t (6.99)
with the second inequality being obtained by the definition of t 0 . Now, due to the fact that δ 2 t ≤ G2 almost surely (by invoking Lemma 6.10) we have:

β t 0 ∑ t=1 γ 2 t δ 2 t = β t 0 ∑ t=1 δ 2 t δ 2 0 + ∑ t-1 s=1 δ 2 s ≤ β 2 + 4 G2 δ 2 0 + 2 log(1 + 1 δ 2 0 t 0 -1 ∑ t=1 δ 2 t ) ≤ β 2 + 4 G2 δ 2 0 + 2 log 1 δ 2 0 (δ 2 0 + t 0 -1 ∑ t=1 δ 2 t ) ≤ β 2 + 4 G2 δ 2 0 + 2 log 1 δ 2 0 γ 2 t 0 (6.100)
Therefore, by the definition of t 0 we finally get with probability 1:

T ∑ t=1 γ t δ 2 t (βγ t - 1 2 ) ≤ β 2 + 4 G2 δ 2 0 + 2 log 4β 2 δ 2 0 (6.101)
• For the term σ 2 K ∑ T t=1 γ t δ t we have:

σ 2 K T ∑ t=1 γ t δ t = σ 2 K T ∑ t=1 γ 2 t δ 2 t ≤ σ 2 K √ T T ∑ t=1 γ 2 t δ 2 t (6.102)
Therefore, by working in the same spirit as above we get:

σ 2 K T ∑ t=1 γ t δ t ≤ σ 2 K 2 + 4 G2 δ 2 0 + 2 log(1 + 1 δ 2 0 T ∑ t=1 δ 2 t ) ≤ σ 2 K √ T 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T) (6.103)
On the other hand, we may the (LHS) from below as follows:

T ∑ t=1 γ t δ 2 t ≥ γ T T ∑ t=1 δ 2 t ≥ γ T δ 2 0 -δ 2 0 + T ∑ t=1 δ 2 t = γ T γ 2 T+1 -δ 2 0 γ T = 1 γ T -δ 2 0 γ T (6.104)
So, combining the above:

1 γ T -δ 2 0 γ T ≤ 2( f (X 1 ) -min x∈X f (x) + β 2 + 4 G2 δ 2 0 + 2 log 4β 2 δ 2 0 + σ 2 K √ T 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T)) (6.105)
which finally yields with probability 1:

1 γ T ≤ δ 0 + 2( f (X 1 ) -min x∈X f (x) + β 2 + 4 G2 δ 2 0 + 2 log 4β 2 δ 2 0 + σ 2 K √ T 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T)) (6.106)
and hence with probability 1:

γ T ≥ δ 0 + 2( f (X 1 ) -min x∈X f (x) + β 2 + 4 G2 δ 2 0 + 2 log 4β 2 δ 2 0 + σ 2 K √ T 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T)) -1
Therefore, by setting:

A = δ 0 + 2( f (X 1 ) -min x∈X f (x) + β 2 + 4 G2 δ 2 0 + 2 log 4β 2 δ 2 0 (6.107)
and

B = σ 2 K 2 + 4 G2 δ 2 0 + 2 log(1 + G2 δ 2 0 T)) (6.108)
we get that:

E T ∑ t=1 ( f (X t ) -f (x * ))γ T ≥ A + B √ T -1 E T ∑ t=1 ( f (X t ) -f (x * )) (6.109)
Moreover, working in the same spirit as in Theorem 6.8 we have:

A + B √ T -1 E T ∑ t=1 ( f (X t ) -f (x * )) ≤ E T ∑ t=1 ( f (X t ) -f (x * ))γ T ≤ D 1 + E T ∑ t=1 γ 2 t δ 2 t (6.110
) which in turn yields:

E T ∑ t=1 ( f (X t ) -f (x * )) ≤ D 1 + E T ∑ t=1 γ 2 t δ 2 t A + B √ T (6.111)
The result then follows by dividing both sides by T and by the fact that E ∑ T t=1 γ 2 t δ 2 t = O(log T).

6.4 fisher markets: a case study

The Fisher market model

We now proceed to illustrate the convergence properties of AdaMir in a Fisher equilibrium problem with linear utilities -both stochastic and deterministic. Following [START_REF] Nisan | Algorithmic Game Theory[END_REF], a Fisher market consists of a set N = {1, . . . , N} of N buyers -or players -that seek to share a set A = {1, . . . , n} of n perfectly divisible goods (ad space, CPU/GPU runtime, bandwidth, etc.). The allocation mechanism for these goods follows a proportionally fair price-setting rule that is sometimes referred to as a Kelly auction [START_REF] Kelly | Rate control for communication networks: shadow prices, proportional fairness and stability[END_REF]: each player i = 1, . . . , N bids x ia per unit of the a-th good, up the player's individual budget; for the sake of simplicity, we assume that this budget is equal to 1 for all players, so ∑ n a=1 x ia ≤ 1 for all i = 1, . . . , N. The price of the a-th good is then set to be the sum of the players' bids, i.e., p a = ∑ i∈N x ia ; then, each player gets a prorated fraction of each good, namely Now, if the marginal utility of the i-th player per unit of the a-th good is θ ia , the agent's total utility will be

w ia = x ia /p a . ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇◇ ◇◇ ◇◇◇ ◇◇◇ ◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇◇◇◇◇◇◇◇◇ □ □ □ □ □ □ □ □ □□ □□ □□ □□ □□ □□ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□□ □□□□ □□□□ □□□□ □□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□□□ □□□ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ �������� ������� (�������) □ ������������ �������� ○ ������ � ���� ���� ���� ���� �� ��� �� -�� �� -� �� -� ����� � ����� ����������� (�������������� ���� �������) (a) Last-iterate convergence ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇◇◇◇◇◇◇◇◇
○ ○ ○ ○ ○ ○ ○○ ○○ ○○ ○○ ○ ○ ○ ○ ○○ ○ ○○ ○ ○ ○ ○ ○ ○○ ○○ ○ ○ ○○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ �������� ������� (�������) □ ������������ �������� ○ ������ � �� ��� ���� �� � ����� ����� ����� � �� ��� ����� ����������� (�������������� ������� �������) (b) Ergodic convergence
u i (x i ; x -i ) = ∑ a∈A θ ia w ia = ∑ a∈A θ ia x ia ∑ j∈N x ja , (6.112)
where x i = (x ia ) a∈A denotes the bid profile of the i-th player, and we use the shorthand (x i ; x -i ) = (x 1 , . . . , x i , . . . , x N ). A Fisher equilibrium is then reached when the players' prices bids follow a profile x * = (x * 1 , . . . , x * N ) such that

u i (x * i ; x * -i ) ≥ u i (x i ; x * -i ) (Eq)
for all i ∈ N and all x i = (x ia ) a∈A such that x ia ≥ 0 and ∑ a∈A x ia = 1. 3As was observed by Shmyrev [START_REF] Vadim | An algorithm for finding equilibrium in the linear exchange model with fixed budgets[END_REF], the equilibrium problem (Eq) can be rewritten equivalently as

minimize F(x; θ) ≡ ∑ a∈A p a log p a -∑ i∈N ∑ a∈A x ia log θ ia subject to p a = ∑ i∈N x ia , ∑ a∈A x ia = 1, and x ia ≥ 0 for all a ∈ A, i ∈ N , (Opt)
with the standard continuity convention 0 log 0 = 0. In the above, the agents' marginal utilities are implicitly assumed fixed throughout the duration of the game. On the other hand, if these utilities fluctuate stochastically over time, the corresponding reformulation instead involves the mean objective

f (x) = E[F(x; ω)]. (6.113)
Because of the logarithmic terms involved, F (and, a fortiori, f ) cannot be Lipschitz continuous or smooth in the standard sense. However, as was shown by Birnbaum et al. [START_REF] Birnbaum | Distributed algorithms via gradient descent for Fisher markets[END_REF], the problem satisfies (RS) over X = {x ∈ R Nn + : ∑ a∈A x ia = 1} relative to the negative entropy function h(x) = ∑ ia x ia log x ia . As a result, mirror descent methods based on this Bregman function are natural candidates for solving (6.113).

In more detail, following standard arguments [START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF], the general mirror descent template (MD) relative to h can be written as

[ + x ia ] =
x ia exp(-γg ia ) ∑ a ∈A x ia exp(-γg ia ) (6.114) where the (stochastic) gradient vector g ≡ g(x; θ) is given in components by

g ia = 1 + log p a -log θ ia . (6.115)
Explicitly, this leads to the entropic gradient descent algorithm

X ia,t+1 = X ia,t (θ ia /p a ) γ t ∑ a ∈A X ia ,t (θ ia /p a ) γ t (EGD)
In particular, as a special case, the choice γ = 1 gives the proportional response (PR) algorithm of Wu and Zhang [START_REF] Wu | Proportional response dynamics leads to market equilibrium[END_REF], namely

X ia,t+1 = θ ia w ia,t ∑ a ∈A θ ia w ia ,t , (PR)
where w ia,t = X ia,t ∑ j∈N X ja,t . As far as we aware, the PR algorithm is considered to be the most efficient method for solving deterministic Fisher equilibrium problems [START_REF] Birnbaum | Distributed algorithms via gradient descent for Fisher markets[END_REF].

Experimental validation and methodology

For validation purposes, we ran a series of numerical experiments on a synthetic Fisher market model with N = 50 players sharing n = 5 goods, and utilities drawn uniformly at random from the interval [START_REF] Absil | Optimization Algorithms on Matrix Manifolds[END_REF][START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]. For stationary markets, the players' marginal utilities were drawn at the outset of the game and were kept fixed throughout; for stochastic models, the parameters were redrawn at each stage around the mean value of the stationary model (for consistency of comparisons). All experiments were run on a MacBook Pro with a 6-Core Intel i7 CPU clocking in at 2.6GHZ and 16 GB of DDR4 RAM at 2667 MHz. The Mathematica notebook used to generate the raw data and run the algorithms is included as part of the supplement (but not the entire sequence of random seed used in the stochastic case, as this would exceed the OpenReview upload limit).

In each regime, we tested three algorithms, all initialized at the barycenter of X : a) an untuned version of (EGD); b) the proportional response algorithm (PR); and c) AdaMir. For stationary markets, we ran the untuned version of (EGD) with a step-size of γ = .1; (PR) was ran "as is", and AdaMir was run with δ 0 determined by drawing a second initial condition from X . In the stochastic case, following the theory of Lu [START_REF] Lu | Relative-continuity" for non-Lipschitz non-smooth convex optimization using stochastic (or deterministic) mirror descent[END_REF] and Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF], the updates of (EGD) and (PR) were modulated by a √ t factor to maintain convergence; by contrast, AdaMir was run unchanged to test its adaptivity properties.

The results are reported in Figs. 6.1-6.3. For completeness, we plot the evolution of each method in terms of values of f , both for the "last iterate" X t and the "ergodic average" Xt . The results for the deterministic case are presented in Fig. 6.1. For stochastic market models, we present a sample realization in Fig. 6.2, and a statistical study over S = 50 sample realizations in Fig. 6

○ ○ ○ ○ ○ ○○ ○○○○ ○○○ ○ ○ ○ ○ ○○ ○ ○○○ ○ ○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ ���������� �������� ������� □ ���������� ������������ �������� ○ ������ � �� ��� ���� �� � �� -� ����� ����� �� ����� ����������� (����������� ���� �������) (a) Last-iterate convergence ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇◇◇◇◇◇◇◇◇◇◇
○ ○ ○ ○ ○ ○ ○ ○○ ○○ ○○ ○ ○ ○ ○ ○ ○○ ○ ○○ ○ ○ ○ ○ ○ ○○ ○○ ○ ○ ○○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ ���������� �������� ������� □ ���������� ������������ �������� ○ ������ � �� ��� ���� �� � ���� ���� � �� ����� ����������� (����������� ������� �������) (b) Ergodic convergence Figure 6.2:
The convergence speed of (EGD), (PR) and AdaMir in a stochastic Fisher market, with marginal utilities drawn i.i.d. at each epoch. outperforms both (EGD) and (PR), in terms of both last-iterate and time-average guarantees.

◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇ ◇◇◇◇◇◇◇◇◇◇◇ ◇◇ ◇ ◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇ □ □ □ □ □ □ □ □ □□ □□ □□ □□ □□ □□ □ □ □□ □ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □ □ □□ □□ □ □ □ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □ ○ ○ ○ ○ ○ ○ ○ ○○ ○○ ○○ ○○ ○ ○ ○ ○ ○○ ○ ○○ ○ ○○ ○ ○ ○○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ �������� ��������
□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ◇ �������� ��������
An interesting observation is that each method's last iterate exhibits faster convergence than its time-average, and the convergence speed of the methods' time-averaged trajectories is faster than our worst-case predictions. This is due to the specific properties of the Fisher market model under consideration: more often than not, players tend to allocate all of their budget to a single good, so almost all of the problem's inequality constraints are saturated at equilibrium. Geometrically, this means that the problem's solution lies in a low-dimensional face of X , which is identified at a very fast rate, hence the observed accelerated rate of convergence. However, this is a specificity of the market model under consideration and should not be extrapolated to other convex problems -or other market equilibrium models to boot.

VA R I AT I O N A L I N E Q U A L I T I E S B E Y O N D L I P S C H I T Z C O N T I N U I T Y

# This section incorporates material from the papers [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF][START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF] I n this chapter we proceed by illustrating our contributions concerning the NoLips (VI) framework; in particular in what follows we consider operators that satisfy either (MB) or (MS). In a nutshell, Chapter 7 collects three types of results:

1. Establish methods that achieve optimal rates given that the optimizer has a prior knowledge on the regularity class of the associated operator.

2. We proceed by deriving adaptive methods relative to the smoothness modulus of the respective operator.

3. Finally, we provide our fully adaptive method for non-smooth/smooth and stochastic cases. More precisely, we provide a method which does not require any prior knowledge of Lipschitz conditions and/or the type of the oracle's feedback (deterministic or stochastic).

In what follows, we illustrate the above contributions in detail.

In Section 7.1 we start by presenting the non adaptive case for operators that transcend the typical Lipschitz regularity conditions. Having this in hand we show that the traditional optimal rates are recovered also for (VI) associated with operators with possible singularities. Our next step is to explore various adaptivity aspects. In doing so, we start given that the associated operator satisfies the respective smoothness like condition. To that end, in Section 7.2 we derive an adaptive mirror-prox algorithm which attains the optimal O(1/T) rate of convergence in problems with possibly singular operators, without any prior knowledge of the degree of smoothness (the Bregman analogue of the Lipschitz constant). Subsequently, in Section 7.3 we introduce a novel adaptive step-size policy with mirror prox as the underlying template. The combination of these ingredients will allow us to automatically exploit the geometry of the gradient data observed at earlier iterations to perform more informative extra-gradient steps in later ones. Thanks to this adaptation mechanism, the proposed method automatically detects whether the problem is smooth or not, without requiring any prior tuning by the optimizer. As a result, the algorithm simultaneously achieves order-optimal convergence rates, i.e., it converges to an O(1/T) rate for smooth problems, and O(1/ √ T) for non-smooth ones. Importantly, these guarantees do not require any EG [START_REF] Korpelevich | The extragradient method for finding saddle points and other problems[END_REF] Graal [START_REF] Malitsky | Golden ratio algorithms for variational inequalities[END_REF] GMP [START_REF] Stonyakin | Generalized mirror prox for monotone variational inequalities: Universality and inexact oracle[END_REF] AMP [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF] BL [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] AdaProx [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF] Param. Agnostic Partial Partial Universality Unbounded Singularities Table 7.1: Overview of related adaptive methods for solving variational inequalities. For the purposes of this table, "parameter-agnostic" means that the method does not require prior knowledge of the parameters of the problem it was designed to solve (Lipschitz modulus, domain diameter, etc.); "rate interpolation" means that the algorithm's convergence rate is O(1/T) or O 1/ √ T in smooth / non-smooth problems respectively; "unbounded domain" is self-explanatory; and, finally, "singularities" means that the problem's defining vector field may blow up at a boundary point of the problem's domain.

of the standard boundedness or Lipschitz continuity conditions that are typically assumed in the literature. That said, the set of results presented in Section 7.3 requires perfect oracle feedback. On that account, in Section 7.4 we present the full potency of our results by being able to treat also stochastic settings. In particular, we employ an adaptive learning rate combined with the Dual Extrapolation algorithmic template. This combination allow us to achieve optimal convergence rates for both deterministic and stochastic settings without any prior knowledge over the boundedness, smoothness and/or the level of noise.

non adaptive case

We first start by presenting a family of non-adaptive methods and their respective rates for NoLips operators. To that end we will make some preliminary assumptions. In particular throughout this section, we assume that the following blanket assumptions hold: Blanket Assumptions Assumption 7.1. The solution set X * ≡ Sol(X , A) of (VI) is nonempty. Assumption 7.2. A is monotone and β-Bregman continuous, i.e.,

A(x) -A(x ) x, * ≤ β 2D(x, x ) for all x, x ∈ X (7.1)
with D being the Bregman divergence defined in Definition 3.3.

In addition to the above, in terms of the oracle's feedback structure we assume that the optimizer gains access to an (SFO) mechanism where U t ∈ V * is an additive noise variable. The two cases of interest that we consider here are (i) when U t = 0 for all t; and (ii) when U t satisfies the statistical hypotheses:

Statistical Assumptions a) Zero-mean: E[U t | F t ] = 0. (7.2a) b) Finite variance: E[ U t
with F t denoting the history (natural filtration) of X t . Finally, we assume for the moment that the smoothness parameter β is known a priori.

Having all this in hand, we can now extend the (optimal) standard convergence rates of the Euclidean setting for the general class of (7.1). Formally, this is stated by the following result.

NoLips Guarantees Theorem 7.1 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF]). Assume that A satisfies Assumptions 7.1 and 7.2, and let Gap H denote the restricted gap function for the Bregman zone C H = {x ∈ X : D(x, x c ) ≤ H}. Suppose further that (MP) is run with an α-strongly convex Bregman function and oracle feedback of the form (SFO). Then, for all H > 0, the averaged sequence XT = ∑ T t=1 γ t X t+1/2 ∑ T t=1 γ t enjoys the following gap bounds: a) If σ 2 = 0 and the algorithm's step-size satisfies

0 < γ min ≡ inf t γ t ≤ sup t γ t ≡ γ max ≤ √ α/β, ( 7.3)
we have

Gap H ( XT ) ≤ H γ min 1 T (7.4) b) Otherwise, if σ 2 > 0 and γ t ≤ √ α/2/β, we have E[Gap H ( XT )] = O H+σ 2 ∑ T t=1 γ 2 t ∑ T t=1 γ t (7.5)
In particular, if

γ t ∝ 1/ √ T, we get E[Gap H ( Xt )] = O(1/ √ T).
For convenience we divide the proof of Theorem 7.1 into the deterministic and stochastic part. In doing so, the main ingredient of the proof of the deterministic case is the following energy inequality:

Method's Template Inequality D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p -1 - β 2 γ 2 t α D(X t+ 1 2
, X t ).

(7.6) is obtained directly by the connection established by Proposition 4.4 between two prox-steps combined with the (7.1). Formally, we show the following result.

Proposition 7.2 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF]). Assume that A satisfies Assumption 7.2 and (MP) is run with perfect oracle feedback. Then, for all p ∈ X , we have:

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p -1 - β 2 γ 2 t α D(X t+ 1 2 , X t ). Proof. By setting x = X t , y 1 = -γ t A(X t ), x + 1 = X t+ 1 2 , y 2 = -γ t A(X t+ 1 2
) and

x + 2 = X t+1 in Proposition 4.4, we readily obtain:

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p -γ t A(X t+ 1 2 ) -A(X t ), X t+1 -X t+ 1 2 -D(X t+1 , X t+ 1 2 ) -D(X t+ 1 2 , X t ). (7.6)
Proceeding line-by-line, the Fenchel-Young inequality applied to the function

φ(x) = x 2 X t+ 1 2 further gives A(X t+ 1 2 ) -A(X t ), X t+1 -X t+ 1 2 ≤ α 2γ t X t+1 -X t+ 1 2 2 X t+ 1 2 + γ t 2α A(X t+ 1 2 ) -A(X t ) X t+ 1 2 , * . (7.7)
Thus, by substituting in (7.6), we get

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p + α 2 X t+1 -X t+ 1 2 2 X t+ 1 2 + γ 2 t 2α A(X t+ 1 2 ) -A(X t ) 2 X t+ 1 2 , * -D(X t+1 , X t+ 1 2 ) -D(X t+ 1 2 , X t ). (7.8)
and hence, by Lemma 3.2, we obtain:

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p + γ 2 t 2α A(X t+ 1 2 ) -A(X t ) 2 X t+ 1 2 , * -D(X t+ 1 2 , X t ). (7.9)
However, the Bregman continuity of A also yields

A(X t+ 1 2 ) -A(X t ) 2 X t+ 1 2 , * ≤ 2β 2 D(X t+ 1 2 , X t ) (7.10)
so our claim follows by combining Eqs. (7.9) and (7.10).

Having established the template inequality in Proposition 7.2 we are now in a position to illustrate the proof for the O(1/T) convergence rate of the restricted merit function (2.10) for deterministic problems. Formally, we have: Proof of Theorem 7.1 -deterministic case. Fix some p ∈ C H . Since γ t ≤ 1/β by assumption, a slight rearrangement of Proposition 7.2 readily yields:

γ t A(X t+ 1 2 ), X t+ 1 2 -p ≤ D(p, X t ) -D(p, X t+1 ) (7.11)
Moreover, by the monotonicity of A, we also have:

A(p), X t+ 1 2 -p ≤ A(X t+ 1 2 ), X t+ 1 2 -p . ( 7.12)
Thus, combining the two inequalities above, we get

γ t A(p), X t+ 1 2 -p ≤ D(p, X t ) -D(p, X t+1 ) (7.13)
and, proceeding to telescope from t = 1 to T, we obtain:

T ∑ t=1 γ t A(p), X t+ 1 2 -p ≤ D(p, X 1 ) -D(p, X t+1 ) ≤ D(p, x c ) (7.14)
Then, dividing by ∑ T t=1 γ t finally yields

A(p), XT -p ≤ D(p, x c ) ∑ T t=1 γ t ≤ D(p, x c ) γ min T , (7.15)
so our result follows by taking the supremum over all p ∈ X such that D(p, x c ) ≤ H (i.e., over all p ∈ C H ).

We now turn our attention towards the stochastic part of Theorem 7.1. In a nutshell, we emphasize that the main building block for deriving this result is the inequality obtained in Proposition 4.4. More precisely, we have the following:

Proof of Theorem 7.1 -stochastic case. Working in the same spirit as for the deterministic case, let x = X t ,

y 1 = -γ t V t , x + 1 = X t+ 1 2 , y 2 = -γ t V t+ 1 2 and x + 2 = X t+1
in the first part of Proposition 4.4. We then get:

D(p, X t+1 ) ≤ D(p, X t ) -γ t V t+ 1 2 , X t+ 1 2 -p + γ t V t+ 1 2 , X t+1 -X t+ 1 2 -D(X t+1 , X t ) ≤ D(p, X t ) -γ t A(X t+ 1 2 ), X t+ 1 2 -p -γ t ξ t+ 1 2 + γ t V t+ 1 2 , X t+1 -X t+ 1 2 -D(X t+1 , X t ) (7.16)
where we used the feedback decomposition

V t+ 1 2 = A(X t+ 1 2 ) + U t+ 1 2 for V t+ 1 2 and we set ξ t+ 1 2 = U t+ 1 2 , X t+ 1 2
p in the last line. By the second part of Proposi- tion 4.4, we also have

γ t V t+ 1 2 , X t+1 -X t+ 1 2 -D(X t+1 , X t ) ≤ γ t V t -V t+ 1 2 , X t+1 -X t+ 1 2 -D(X t+1 , X t+ 1 2 ) -D(X t+ 1 2 , X t ) (7.17)
Now, by applying the Fenchel-Young inequality to the duality pairing in the above inequality, we get

γ t V t -V t+ 1 2 , X t+1 -X t+ 1 2 ≤ γ 2 t 2α V t -V t+ 1 2 2 X t+ 1 2 , * + α 2 X t+1 -X t+ 1 2 2 X t+ 1 2 .
(7.18) On the other hand, by the stochastic oracle assumption (SFO), we have:

γ 2 t 2α V t -V t+ 1 2 2 X t+ 1 2 , * ≤ γ 2 t α A(X t ) -A(X t+ 1 2 ) 2 X t+ 1 2 , * + γ 2 t α U t -U t+ 1 2 2 X t+ 1 2 , * ≤ 2β 2 γ 2 t α D(X t+ 1 2 , X t ) + γ 2 t µα U t -U t+ 1 2 2 * . (7.19)
where the last line follows from the Bregman continuity of A (Assumption 7.2) and the fact that • x ≥ µ • for some µ > 0 and all x ∈ X (implying in turn that • x, * ≤ µ -1 • * for all x ∈ X ). We thus get:

γ t V t+ 1 2 , X t+1 -X t+ 1 2 -D(X t+1 , X t ) ≤ 2β 2 γ 2 t α -1 D(X t+ 1 2 , X t ) + γ 2 t µα U t -U t+ 1 2 2 * (7.20)
Since γ 2 t ≤ α/(2β 2 ) by assumption, substituting (7.20) in (7.16) and rearranging yields

γ t A(X t+ 1 2 ), X t+ 1 2 -p ≤ D(p, X t ) -D(p, X t+1 ) -γ t ξ t+ 1 2 + γ 2 t µα U t -U t+ 1 2
2 * (7.21) which in turn yields:

γ t A(X t+ 1 2 ), X t+ 1 2 -p ≤ D(p, X t ) -D(p, X t+1 )) -γ t ξ t+ 1 2 + 2γ 2 t µα U t 2 * + U t+ 1 2 2 * . (7.22)
In order to bound ξ t+ 1 2 , we will need to introduce the auxilliary process

Z t+1 = arg min x∈X { U t+ 1 2 , Z t -x + µ γ t D(x, Z t )} (7.23)
with Z 1 = x c . We then have

-γ t ξ t+1 = γ t U t+ 1 2 , p -X t+ 1 2 = γ t U t+ 1 2 , Z t -X t+ 1 2 + γ t U t+ 1 2
, p -Z t (7.24) In order to bound the term which depends on p, we have the following:

γ t U t+ 1 2 , p -Z t = γ t U t+ 1 2 , p -Z t+1 + γ t U t+ 1 2 , Z t+1 -Z t ≤ µ ∇h(Z t+1 ) -∇h(Z t ), p -Z t+1 + γ 2 t 2α U t+ 1 2 2 * ,Z t + α 2 Z t+1 -Z t 2 Z t (7.25)
and so,

γ t U t+ 1 2 , p -Z t ≤ µ ∇h(Z t+1 ) -∇h(Z t ), p -Z t+1 + γ 2 t 2µα U t+ 1 2 2 * + αµ 2 Z t+1 -Z t 2 . (7.26)
Hence, by the three-point identity, we obtain:

γ t U t+ 1 2 , p -Z t ≤ µ[D(p, Z t ) -D(p, Z t+1 )] -µD(Z t+1 , Z t ) + γ 2 t 2µα U t+ 1 2 2 * + αµ 2 Z t+1 -Z t 2 ≤ µ[D(p, Z t ) -D(p, Z t+1 )] + γ 2 t 2µα U t+ 1 2 2 * (7.27)
where the last inequality is a consequence of the strong convexity of h. Thus, combining all these with the fact that A is monotone, we can telescope and obtain

T ∑ t=1 γ t A(p), X t+ 1 2 -p ≤ (1 + µ)D(p, x c ) + T ∑ t=1 γ t U t+ 1 2 , Z t -X t+ 1 2 + 1 µα T ∑ t=1 γ 2 t 2 U t 2 * + 5 2 U t+ 1 2 2 * .
Hence, after dividing by ∑ T t=1 γ t and taking the supremum over p ∈ C H , by setting

λ t = U t+ 1 2 , Z t -X t+ 1 2 we get: Gap H ( XT ) ≤ (1 + µ)H + ∑ T t=1 γ t λ t + 1 µα ∑ T t=1 γ 2 t 2 U t 2 * + 5 2 U t+ 1 2 2 * ∑ T t=1 γ t . (7.28) Since E[ U t+ 1 2 , Z t -X t+ 1 2 ] = E[E[ U t+ 1 2 , Z t -X t+ 1 2 ] | F t+ 1 2 ] = 0, taking expecta- tions yields E[Gap H ( Xt )] ≤ (1 + µ)D + 9σ 2 2µα ∑ T t=1 γ 2 t ∑ T t=1 γ t , (7.29)
which proves our claim. Finally, the RHS of this last inequality is Õ

(1/T 1/2 ) if γ t ∝ 1/ √ t, so the Õ(1/ √ T) result follows.
Theorem 7.1 relies crucially on prior knowledge of the following key factors:

1. That the associated operator satisfies the respective Bregman smoothness regularity condition.

2.

A fortiori, in order to properly tune the method's step-size policy the optimizer needs to be able to estimate the precise (Bregman) Lipschitz constant.

As a prelude of the analysis to come the following section will be focusing on relaxing different aspects of these elements.

adaptivity to the smoothness modulus

As we already mentioned, a crucial assumption underlying the analysis of the previous section is that the optimizer must know in advance -or be otherwise able to estimate -the Bregman constant β. In practice, this can be difficult to achieve, so it is important to be able to run (MP) with an adaptive step-size policy. Therefore our first step towards adaptivity is to design methods for solving Bregman smooth (VI)'s where the respective "smoothness" parameter is unknown a priori.

Our starting point is the observation that, with perfect oracle feedback, one can estimate β by setting Estimator of Smoothness Constant

β t = A(X t+1/2 ) -A(X t ) X t+1/2 , * 2D(X t+1/2 , X t ) (7.30)
whenever X t+1/2 = X t ; obviously, if A is β-Bregman continuous, we have β t ≤ β. 1However, the fact that the Bregman constant is being under-estimated means that a step-size policy of the form γ t ∝ √ α/β t would over-estimate the inverse Bregman constant 1/β, so the resulting step-size policy would have no reason to satisfy (7.3).

To overcome this obstacle, we introduce the following comparison mechanism: first, at each t = 1, 2, . . . , we use the estimation (7.30) to test the step-size γt = √ α/β t . Then, to avoid the growth phenomenon outlined above, we shrink γt by a constant factor of θ and, to avoid running into vanishing step-size issues, we take the previous step-size employed if the shrunk one would be smaller. Formally, we consider the adaptive step-size policy:

Step-Size Adaptive to the Lipschitz Constant

γ t+1 =    min{γ t , θ √ α/β t } if X t = X t+1/2 , γ t otherwise, (7.31)
with β t defined as in (7.30) and θ ∈ (0, 1) chosen arbitrarily.

Guarantees under

Adaptivity to the Lipschitz Constant Theorem 7.3 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF]). Assume that the monotone operator A satisfies Assumptions 7.1 and 7.2, and (MP) is run with perfect oracle feedback and the adaptive step-size policy (7.31). Then, with notation as in Theorem 7.1, the algorithm's ergodic average

XT = T ∑ t=1 γ t X t+1/2 T ∑ t=1 γ t (7.32) enjoys the gap bound Gap H ( XT ) = O(1/T). (7.33)
Proof. We begin with an induction argument to show that the adaptive step-size policy γ t+1 = min{γ t , θ √ α/β t } is lower bounded as

γ t ≥ min{γ 1 , θ √ α/β}. (7.34)
Indeed, assuming this bound for γ t , we have either a) γ t+1 = γ t ≥ θ √ α/β by the inductive assumption; or b) γ t+1 = θ √ α/β t ≥ θ √ α/β by the fact that β t is an under-estimate of β. Thus, with β t (weakly) decreasing, it follows that γ t converges to some well-defined limit value

γ ∞ ≥ θ √ α/β < √ α/β.
To proceed, given that β t ≤ β, working in the same spirit as we did to obtain the basic energy inequality (7.6) in the previous section, we get:

D(p, X t+1 ) ≤ D(p, X t ) + γ t g t+1/2 , X t+1/2 -p -1 -θ 2 γ 2 t γ 2 t+1 D(X t+1/2 , X t ) (7.35)
leading to the estimate for all t greater than some (finite) t 0 . Accordingly, summing and telescoping as in the analysis of the previous section, we get

γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X t ) -D(p, X t+1 ) -1 -θ 2 γ 2 t γ 2 t+1 D(X t+1/2
T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, x c ) + t 0 ∑ t=1 1 -θ 2 γ 2 t γ 2 t+1 D(X t+1/2 , X t )
< +∞ whenever T > t 0 . Our result then follows by dividing both sides of this last inequality by ∑ T t=1 γ t and recalling the fact that γ t ≥ θ √ α/β > 0 for all t.

As we mentioned the contributions of Theorem 7.3 hinge on the fact that the optimizer knows in advance that associate operator satisfies (7.1). In what follows, we shall tackle this drawback by developing methods that are agnostic to the respective regularity condition at hand.

the deterministic case

Moving forward we define the appropriate adaptive step-size policy that will allow the (MP) method to exhibit "regime-agnostic" optimal convergence rates, i.e., adjust optimally its performance without any prior knowledge of the underlying regularity condition. Our starting point for designing such methods is by considering first the deterministic case. More precisely, throughout this section we assume the following blanket conditions:

1. X is a regular Finsler space (cf. Section 3.2.2).

2.

Regarding the NoLips condition we will assume that the respective operator satisfies either (MB) or (MS)

3. The associated regularizer h is in the sense of Definition 3.3.

With all this is in place, the deterministic version (MP) method, defined by the following recursion:

X t+1/2 = P X t (-γ t V t ) X t+1 = P X t (-γ t V t+1/2 ) (7.39)
can be adapted to our current setting as follows:

Universal Step-Size (Deterministic)

γ t = 1 1 + ∑ t-1 j=1 V j+1/2 -V j 2 X j+1/2 , * (Adapt)
with V t = A(X t ), t = 1, 1/2, . . . . In words, this method builds on the template of (MP) by replacing the global norm with a dual Finsler norm evaluated at the algorithm's leading state X j+1/2 combined with the respective adaptive step-size policy. We conclude this section by providing an intuitive explanation for the step-size (Adapt). In particular, under (MB) we get that:

Intuition Behind Universality V j -V j+1/2 2 X j+1/2 , * ≈ "constant"
and hence γ t ∝ 1/ √ t. Hence, we have that:

T ∑ t=1 γ t = Ω( √ T) (7.40)
On the other hand under (MS) we show that γ t stabilizes to some strictly positive value which in turn yields:

T ∑ t=1 γ t = Ω(T) (7.41)
and thus leads to a faster convergence rate. In the forthcoming analysis, we shall explain this behaviour of the adaptive step-size in detail.

Optimal rate interpolation

With all this in hand, our main result for our method can be stated as follows:

Universality Guarantees (Deterministic)

Theorem 7.4 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Suppose A is a monotone operator, let C be a compact neighborhood of a solution of (VI), and set H = sup x∈C D(x, X 1 ) Then, (MP) run with the adaptive step-size (Adapt) enjoys the guarantees:

1. If A satisfies (MB):

Gap C ( XT ) = O H + G 3 (1 + 1/K) 2 + log(1 + 4G 2 (1 + 2/K) 2 T) √ T . (7.42) 2. If A satisfies (MS): Gap C ( XT ) = O H T . (7.43)
In a nutshell we mention here that its key element is the determination of the asymptotic behavior of the adaptive step-size policy γ t in the non-smooth and smooth regimes, i.e., under (MB) and (MS) respectively. At a very high level, (MB) guarantees that the difference sequence A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * is bounded, which implies in turn that ∑ T t=1 γ t = Ω( √ T) and eventually yields the bound (7.42) for the algorithm's ergodic average XT . This is accomplished formally in the following lemma.

Boundness of the Residual Under (MB)

Lemma 7.5 (Antonakopoulos et al. [8]). Suppose that the monotone operator A satisfies (MB). Then, the sequence A(X t+1/2 ) -A(X t ) 2 X t , * is bounded. In particular, the following inequality holds:

A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ C 2 (7.44) with C = 2G + β 4G K .
Proof. It suffices to show that: A(X t+1/2 ) -A(X t ) X t+1/2 , * is bounded. More precisely, by the triangle inequality we have:

A(X t+1/2 ) -A(X t ) X t+1/2 , * ≤ A(X t+1/2 ) X t+1/2 , * + A(X t ) X t+1/2 , * (7.45)
We shall bound the (RHS) part of (7.45) term by term. In particular, we have:

• For the first term A(X t+1/2 ) X t+1/2 , * we readily get by (MB):

A(X t+1/2 ) X t+1/2 , * ≤ G (7.46)

• For the second term A(X t ) X t+1/2 , * , we have:

A(X t ) X t+1/2 , * ≤ A(X t ) X t , * + β X t -X t+1/2 X t + X t -X t+1/2 X t+1/2 ≤ G + β X t -X t+1/2 X t + X t -X t+1/2 X t+1/2 (7.47)
Therefore, it suffices to show that the quantity X t -X t+1/2 X t + X t -X t+1/2 X t+1/2 is bounded from above. Indeed, we have:

D(X t , X t+1/2 ) + D(X t+1/2 , X t ) = ∇h(X t ) -∇h(X t+1/2 ), X t -X t+1/2 ≤ γ t A(X t ), X t -X t+1/2 ≤ Gγ t X t -X t+1/2 X t
where the last inequality is obtained by (MB). Moreover, by Definition 3.3 we get:

D(X t , X t+1/2 ) + D(X t+1/2 , X t ) ≤ γ t G 2 K D(X t+1/2 , X t ) ≤ G 2 K [D(X t , X t+1/2 ) + D(X t+1/2 , X t )] which yields D(X t , X t+1/2 ) + D(X t , X t+1/2 ) ≤ 2G 2 K (7.48)
Hence, by the local strong convexity in Definition 3.3 of h, we get:

K 2 X t -X t+1/2 2 X t + X t -X t+1/2 2 X t+1/2 ≤ 2G 2 K (7.49)
which in turn implies that:

X t -X t+1/2 X t ≤ 2G K and X t -X t+1/2 X t+1/2 ≤ 2G K (7.50)
and so,

X t -X t+1/2 X t + X t -X t+1/2 X t+1/2 ≤ 4G K (7.51)
Moreover, by combining (7.47) and (7.51) we get:

A(X t ) X t+1/2 , * ≤ G + β 4G K (7.52)
Summarizing, (7.45) combined with (7.47) and (7.52) yields:

A(X t+1/2 ) -A(X t ) X t+1/2 , * ≤ 2G + β 4G K (7.53)
and hence the result follows.

On the other hand, if (MS) kicks in, we have the following finer result:

Summability of the Residual Under (MS) Lemma 7.6 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Assume the monotone operator A satisfies (MS). Then, 1. γ t decreases monotonically to a strictly positive limit γ ∞ = lim t→∞ γ t > 0;

2. The sequence A(X t+1/2 ) -A(X) X t+1/2 , * is square summable: in particular,i.e.,

∞ ∑ t=1 A(X t+1/2 ) -A(X) 2 X t+1/2 , * = 1/γ 2 ∞ -1. (7.54)
Proof. Since γ t is decreasing and bounded from below (γ t ≥ 0), then we readily obtain that its limit exists and more precisely we have:

lim t→+∞ γ t = inf t∈N γ t = γ ∞ ≥ 0 (7.55)
We now assume that γ ∞ = 0. Then, by recalling (7.6):

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+1/2 ), X t+1/2 -p + γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 -D(X t+1/2 , X t ) -D(X t+1 , X t+1/2 ) (7.56)
By rearranging the above and telescoping t = 1, . . . , T we get:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + T ∑ t=1 γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 - T ∑ t=1 D(X t+1/2 , X t ) - T ∑ t=1 D(X t+1 , X t+1/2 ) (7.57)
whereas, by applying Fenchel-Young inequality to the above we readily get:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + K 2 T ∑ t=1 X t+1 -X t+1/2 2 X t+1/2 - T ∑ t=1 D(X t+1/2 , X t ) - T ∑ t=1 D(X t+1 , X t+1/2 ) (7.58)
and by considering that the local-strong convexity of Definition 3.3:

K 2 T ∑ t=1 X t+1 -X t+1/2 2 X t+1/2 - T ∑ t=1 D(X t+1 , X t+1/2 ) ≤ 0 (7.59)
we finally obtain:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * - T ∑ t=1 D(X t+1/2 , X t ) (7.60)
Therefore, by the definition (MS) we have:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * - K 2β 2 T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.61)
which becomes:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + T ∑ t=1 γ 2 t 2K - K 4β 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * - K 4β 2 T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.62)
Now, by setting p = x * with x * being a solution of (VI) and using the fact that A(X t+1/2 ), X t+1/2x * ≥ 0 and D(x * , X 1 ) ≤ D (by the compatibility of h), we obtain:

K 4β 2 T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ D + T ∑ t=1 γ 2 t 2K - K 4β 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.63)
Moreover, by observing that the quantity

γ 2 t 2K -K 4β 2 ≤ 0, whenever γ t ≤ √
2K/2β and since we assumed that γ t → 0, there exists some t 0 ∈ N such that:

γ 2 t 2K - K 4β 2 ≤ 0 for all t ≥ t 0 (7.64)
Therefore, (7.63) becomes:

1

γ 2 T+1 -1 = T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ D + t 0 ∑ t=1 γ 2 t 2K - K 4β 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2
, * (7.65)

In addition, since 1/γ T+1 → +∞, by the fact that γ t → 0, this yields that:

+∞ ≤ D + t 0 ∑ t=1 γ 2 t 2K - K 4β 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.66)
which is a contradiction. Hence, we get that:

lim t→+∞ γ t = inf t∈N γ t = γ ∞ > 0 (7.67)
In order to prove our second claim, we first recall the definition of γ t :

γ t = 1 1 + ∑ t-1 j=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2, * (7.68)
whereas by developing and rearranging we have:

t-1 ∑ j=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2, * = 1 γ 2 t -1 (7.69)
Hence, by taking limits on both sides we get:

+∞ ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2, * = lim t→+∞ t-1 ∑ j=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2, * = 1 γ 2 ∞ -1 where 0 ≤ 1 γ 2 ∞ -1 < +∞, since 0 < γ ∞ ≤ 1 and therefore the result follows.
By means of this lemma, it follows that ∑ T t=1 γ t ≥ γ ∞ T = Ω(T); hence it ultimately follows that (MP) run with (Adapt) enjoys an O(1/T) rate of convergence under (MS).

Proof of Theorem 7.4. By recalling (7.6) we have:

D(p, X t+1 ) ≤ D(p, X t ) -γ t A(X t+1/2 ), X t+1/2 -p + γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 -D(X t+1/2 , X t ) -D(X t+1 , X t+1/2 ) (7.70)
We start our analysis rearranging (7.6). In particular, by telescoping t = 1, . . . , T we get:

T ∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + T ∑ t=1 γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 - T ∑ t=1 D(X t+1/2 , X t ) - T ∑ t=1 D(X t+1 , X t+1/2 ) (7.71)
On the other hand, since A is monotone, we readily get:

γ t A(p), X t+1/2 -p ≤ γ t A(X t+1/2 ), X t+1/2 -p (7.72)
Thus, combining (7.72) and (7.71), dividing by ∑ T t=1 γ t and setting

XT = T ∑ t=1 γ t -1 T ∑ t=1 γ t X t+1/2 (7.73)
we get:

A(p), XT -p ≤ T ∑ t=1 γ t -1 (D(p, X 1 ) + T ∑ t=1 γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 - T ∑ t=1 D(X t+1/2 , X t ) - T ∑ t=1 D(X t+1 , X t+1/2 )) (7.74)
whereas, by applying Fenchel-Young inequality to the above we readily get:

A(p), XT -≤ T ∑ t=1 γ t -1 (D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + K 2 T ∑ t=1 X t+1 -X t+1/2 2 X t+1/2 - T ∑ t=1 D(X t+1/2 , X t ) - T ∑ t=1 D(X t+1 , X t+1/2 )) (7.75)
Thus, if C is a compact neighbourhood of the solution set X * , considering that by Definition 3.3:

K 2 T ∑ t=1 X t+1 -X t+1/2 2 X t+1/2 - T ∑ t=1 D(X t+1 , X t+1/2 ) ≤ 0 (7.76)
and taking suprema on both sides, yields:

Gap C ( XT ) ≤ T ∑ t=1 γ t -1 (sup p∈C D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * - T ∑ t=1 D(X t+1/2 , X t )) (7.77)
1. Case 1: Convergence under (MB): Therefore, in order to determine the convergence speed of X T under (MB), we shall examine the asymptotic Analysis Under (MB)

behaviour of each term of the nominator on the (RHS) of (7.87). In particular, we have the following:

• For the first term: we readily get by the compactness of C, • For the second term: ∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * , we have:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * = T ∑ t=1 (γ 2 t -γ 2 t+1 ) A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + T ∑ t=1 γ 2 t+1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.79)
Since, γ t is non-increasing and therefore (γ 2 tγ 2 t+1 ≥ 0), and γ t ≤ 1 the above becomes:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ C 2 + T ∑ t=1 γ t+1 A(X t+1/2 ) -A(X t ) 2
X t+1/2 , * (7.80) and by the definition of γ t we get:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * = C 2 + T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * 1 + ∑ t j=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.81)
and finally,

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ C 2 + 1 + log(1 + T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ) (7.82)
with the last inequality being obtained by Lemma A.2 which combined with (MB) yields:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ C 2 + 1 + log(1 + C 2 T) (7.83)
Finally, for ∑ T t=1 γ t , we have the following lower-bound

T ∑ t=1 γ t = T ∑ t=1 1 1 + ∑ t-1 j=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≥ T ∑ t=1 1 √ 1 + tC 2 (7.
∑ t=1 γ t A(X t+1/2 ), X t+1/2 -p ≤ D(p, X 1 ) + 1 2K T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * - T ∑ t=1 D(X t+1/2 , X t ) (7.87)
by examining the asymptotic behaviour term by term, we get:

• For the first term D(x * , X 1 ), since x * ∈ dom A = dom h and X 1 ∈ dom ∂h, we have: D(x * , X 1 ) < +∞ (7.88)

• For the second term ∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * we have:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ T ∑ t=1 A(X t+1/2 ) -A(X t )
∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ 1 γ 2 ∞ -1 (7.90) with γ ∞ = inf t γ t > 0.
Finally, by applying Lemma 7.6 once more by considering γ ∞ = inf t∈N γ t > 0 we have:

T ∑ t=1 γ t ≥ γ ∞ T ∑ t=1 1 = γ ∞ T (7.91)
which yields:

T ∑ t=1 γ t = Ω(T) (7.92)
and the result follows.

Having established optimal convergence rate interpolation guarantees for the ergodic average of the (MP) iterates, a natural question that arises what the asymptotic behaviour of the iterates themselves,i.e., before any average occurs. This problem is treated in the next section.

Trajectory convergence

Throughout this section we will provide a trajectory convergence result that governs the actual iterates of the adaptive (MP) algorithm. Formally, we have the following: Theorem 7.7 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Suppose that A(x), xx * < 0 whenever x * is a solution of (VI) and x is not. If, A satisfies (MB) or (MS), the iterates X t of (MP) run with the adaptive step-size (Adapt) converge to a solution of (VI).

Last Iterate Convergence

The importance of this result is that, in many practical applications (especially in non-monotone problems), it is more common to harvest the "last iterate" of the method (X t ) rather than its ergodic average ( XT ); as such, Theorem 7.7 provides a certain justification for this design choice.

Structurally, the first step is to show that X t visits any neighborhood of a solution point x * ∈ X * infinitely often (this is where the coherence assumption A(x), xx * is used). The second is to use this trapping property in conjunction with a suitable "energy inequality" to establish convergence via the use of a quasi-Fejér technique as in [START_REF] Patrick | Quasi-Fejérian analysis of some optimization algorithms[END_REF].

Vanishing Residuals Lemma 7.8 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Suppose that A satisfies (MB) (respectively (MS)) and X t , X t+1/2 are the iterates of (MP) run with the adaptive step-size (Adapt). Then, the following hold:

1. X t+1/2 -X t → 0 while t → +∞ 2. max{D(X t+1/2 , X t ), D(X t , X t+1/2 )} ≤ 2G 2 K γ 2 t Proof.
For the proof of the first claim, we shall treat the cases of (MB) and (MS) individually.

1. Under (MB) condition: Since γ t is decreasing and bounded from below, then we readily obtain that its limit exists and more precisely:

lim t→+∞ γ t = γ ∞ ≥ 0 (7.93)
We shall distinguish two individual cases:

• γ ∞ > 0: By recalling the definition of the adaptive step-size:

γ t = 1 1 + ∑ t-1 j=1 A(X j+1/2 ) -A(X j ) 2 X j+1/2 , * (7.94)
whereas by rearranging and developing we have:

t-1 ∑ j=1 A(X j+1/2 ) -A(X j ) 2 X j+1/2 , * = 1 γ 2 t -1 (7.95)
Therefore, by taking limits on both sides:

+∞ ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * = lim t→+∞ 1 γ 2 t -1 = 1 γ 2 ∞ -1 ≥ 0 (7.96)
Hence, by recalling (7.6) we have:

T ∑ t=1 D(X t+1/2 , X t ) ≤ D(x * , X 1 ) + T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ D(x * , X 1 ) + T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * which in turn yields, lim inf t A(X t+1/2 ), X t+1/2 -x * = c > 0 (7.112)
Now, by setting p = x * for some x * ∈ X * in (7.6), we get:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 ≤ D(x * , X t ) -cγ t + γ 2 t A(X t+1/2 ) -A(X) 2 X t+1/2
whereas by telescoping t = 1, . . . , T we obtain:

D(x * , X T ) ≤ D(x * , X 1 ) - T ∑ t=1 γ t c - ∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ∑ T t=1 γ t (7.113)
Having this established this general setting, we shall examine the asymptotic behaviour term by term for each regularity case individually, which in both cases shall lead to a contradiction.

Under (MB) condition:

• For the first term: ∑ T t=1 γ t , we have by (7.85) that:

T ∑ t=1 γ t → +∞ and T ∑ t=1 γ t = Ω( √ T) (7.114)
• For the second term

∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ∑ T t=1 γ t
, we first examine the denominator. In particular, by the definition of (Adapt) we get:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * = T ∑ t=1 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * 1 + ∑ t-1 j=1 A(X j+1/2 ) -A(X j ) 2 X j+1/2 , * (7.115
) which by recalling (7.83) we obtain:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * = O(log T) (7.116)
So, by combining (7.114) and (7.116) we readily obtain:

∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ∑ T t=1 γ t → 0 while T → +∞ (7.117)
Therefore, by letting T → +∞, the inequality (7.113) yields D(x * , X T ) → -∞, contradiction.

2. Under (MS) condition: Examining the asymptotic behavior of (7.113) term by term under (MS) we get the following:

• For ∑ T t=1 γ t , (MS) guarantees by (7.92):

T ∑ t=1 γ t = Ω(T) and T ∑ t=1 γ t → +∞ (7.118) • For ∑ T t=1 γ 2 t A(X t+1/2 )-A(X t ) 2 X t+1/2, *
∑ T t=1 γ t , (7.6) guarantees:

T ∑ t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2, * = O(1) (7.119)
which combined with (7.92) gives us:

∑ T t=1 γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2, * ∑ T t=1 γ t → 0 (7.120)
Therefore, y letting T → +∞, the inequality (7.113) yields that D(x * , X T ) → -∞, a contradiction.

Having all this at hand, we are finally in the position to prove the main result of this section; namely the convergence of the actual iterates of the method. For that we will need an intermediate lemma that shall allow us to pass from a convergent subsequence to global convergence (see also [START_REF] Patrick | Quasi-Fejérian analysis of some optimization algorithms[END_REF], [START_REF] Teodorovich | Introduction to Optimization[END_REF]).

Quasi-Fejer Sequences Lemma 7.10. Let χ ∈ (0, 1], (α t ) t∈N , (β t ) t∈N non-negative sequences and (ε t ) t∈N ∈ l 1 (N) such that t = 1, 2, . . . :

α t+1 ≤ χα t -β t + ε t (7.121)
Then, α t converges.

Proof. First, one shows that α t∈N is a bounded sequence. Indeed, one can derive directly that:

α t+1 ≤ χ t+1 α 0 + t ∑ k=0 χ t-k ε k (7.122) Hence, (α t ) t∈N lies in [0, α 0 + ε], with ε = ∑ +∞ t=0 ε t . Now, one is able to extract a convergent subsequence (α k t ) t∈N , let say lim t→+∞ α k t = α ∈ [0, α 0 + ε] and fix δ > 0. Then, one can find some t 0 such that α k t 0 -α < δ 2 and ∑ m>t k t 0 ε m < δ 2 .
That said, we have:

0 ≤ α t ≤ α k t 0 + ∑ m>t k t 0 ε m < δ 2 + α + δ 2 = α + δ (7.123)
Hence, lim sup t α t ≤ lim inf t α t + δ. Since, δ is chosen arbitrarily the result follows.

Proof of Theorem 7.7. Once more, we shall treat each regularity class individually.

1. Under (MB) condition: For the (MB), by denoting lim t→+∞ γ t = γ ∞ case we shall consider two cases for the asymptotic behaviour of the step-size γ t .

• γ ∞ > 0: By recalling the definition of γ t :

γ t = 1 1 + ∑ t-1 j=1 A(X j+1/2 ) -A(X j ) 2 X j+1/2 (7.124)
whereas by rearranging we get:

t-1 ∑ j=1 A(X j+1/1 ) -A(X j ) 2 X j+1/2 = 1 γ 2 t -1 (7.125)
and hence:

+∞ ∑ t=1 A(X t+1/1 ) -A(X t ) 2 X t+1/2 = 1 γ 2 ∞ -1 < +∞ (7.126)
Therefore, by recalling (7.6), we have for solution of (VI),

x * ∈ X D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + γ 2 t A(X t+1/2 ) -A(X t ) 2 t+1/2, * (7.127)
which enables us to directly apply Lemma 7.10 for α t = D(x * , X t ), β t = γ t A(X t+1/2 ), X t+1/2x * and ε t = γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * .

• γ ∞ = 0: Fix an equilibrium x * ∈ X * and consider the "Bregman zone":

D ε = {x ∈ X : D(x * , x) < ε} (7.128)
By the assumption for the regularizer h, it follows that there exists some δ > 0 such that:

B δ = {x ∈ X : x * -x < δ} (7.129)
is contained in D ε . Hence, by regularity assumption for the (2.10), it follows that:

A(x), xx * ≥ c > 0 for some c ≡ c(ε) > 0 and for all x / ∈ D ε , (7.130) in particular, for all x ∈ D 2ε \ D ε . Assume now that x * is a limit point of X t , i.e., X t ∈ D 2ε for infinitely many t ∈ N. Now, by the prox-step, we get:

γ t A(X t ), X t -x * ≤ ∇h(X t ) -∇h(X t+1/2 ), X t -x * (7.131)
whereas by Lemma 4.2 and after rearranging we get:

D(x * , X t+1/2 ) ≤ D(x * , X t ) -γ t A(X t ), X t -x * + D(X t , X t+1/2 ) ≤ D(x * , X t ) -γ t A(X t ), X t -x * + max{D(X t , X t+1/2 ), D(X t , X t+1/2 )}
Therefore, by Lemma 7.8 we obtain:

D(x * , X t+1/2 ) ≤ D(x * , X t ) -γ t A(X t ), X t -x * + 2G 2 K γ 2 t (7.132)
We consider two cases:

a) X t ∈ D 2ε \ D ε : Then, A(X t ), X t -x * ≥ c > 0. So, D(x * , X t+1/2 ) ≤ D(x * , X t ) -cγ t + 2G 2 K γ 2 t (7.133) Now, provided that 2G 2 γ 2 t K ≤ cγ t or equivalently γ t ≤ cK 2G 2 . we get: D(x * , X t+1/2 ) ≤ 2ε. b) X t ∈ D ε : Then, in this case we have: D(x * , X t+1/2 ) ≤ D(x * , X t ) + 2G 2 K γ 2 t (7.134) Again, provided that 2G 2 K γ 2 t ≤ ε or equivalently γ t ≤ √ 2εK 2G we get D(x * , X t+1/2 ) ≤ 2ε
Therefore, by summarizing the above we get that if

γ t ≤ min{ √ 2εK 2G , cK 2G 2 }
, we have that X t+1/2 ∈ D 2ε whenever X t ∈ D 2ε . Going further, due to Proposition 4.4 by setting p = x * , x 1 = X t+1/2 , x + 2 = X t+1 , x = X t , w 1 = -γ t A(X t+1/2 ) and w 2 = -γ t A(X t+1/2 ) we get:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + γ t A(X t+1/2 ) -A(X t ), X t+1 -X t+1/2 -D(X t+1 , X t+1/2 ) -D(X t+1/2 , X t ) (7.135)
whereas by applying Fenchel's inequality we obtain:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + γ 2 t 2K A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + K 2 X t+1 -X t+1/2 2 X t+1/2 -D(X t+1 , X t+1/2 ) -D(X t+1/2 , X t ) (7.136) Now, since K 2 X t+1 -X t+1/2 2 X t+1/2 -D(X t+1 , X t+1/2 ) ≤ 0 by Defini- tion 3.3 we get: D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + γ 2 t 2K A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * (7.137)
which, in turn, by (7.53) the above yields:

D(x * , X t+1 ) ≤ D(x * , X t ) -γ t A(X t+1/2 ), X t+1/2 -x * + C 2 2K γ 2 t (7.138) with C = 2G + β 4G K .
Recall that X t+1/2 ∈ D 2ε by our previous claim. We now consider the following two cases:

a) X t+1/2 ∈ D 2ε \ D ε : In this case: A(X t+1/2 ), X t+1/2 -x * ≥ c > 0, so, D(x * , X t+1 ) ≤ D(x * , X t ) -cγ t + C 2 2K γ 2 t (7.139) which holds provided that C 2 γ 2 t 2K ≤ cγ t or equivalently γ t ≤ 2cK C 2 , b) X t+1/2 ∈ D ε : First recall that: D(X t+1/2 , X t+1 ) + D(X t+1 , X t+1/2 ) ≤ 2γ 2 t K A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ 2γ 2 t K C 2
Therefore, we get that:

X t+1 -X t+1/2 2 ≤ 4µ 2 C 2 K 2 γ 2 t (7.140)
Now, let us define the following:

D ε (α) = max{D(x * , x) : dist(x, D ε (x * )) < α} (7.141) Clearly, D ε (α) is continuous relative to α and lim α→0 + D ε (α) = ε.
Therefore, we have:

D ε (α) ≤ ε for all α ≤ α * with α * sufficiently small. (7.142)

Moreover, due to (7.140), we conclude that D(x * , X t+1 ) ≤ 2ε, provided that γ t ≤ α * 2µC K.

We conclude that X t+1 ∈ U 2ε provided that X t ∈ D 2ε and

γ t ≤ min{ 2cK G 2 , √ 2εK 2G , α * 2µC K} (7.143)
Since, γ t → 0 and X t ∈ D 2ε infinitely often (due to Proposition 7.9) we conclude that X t ∈ D 2ε for all sufficiently large t. With ε > 0 being arbitrary, the result follows.

Under (MS) condition: By plugging in α

t = D(x * , X t ), β t = γ t A(X t+1/2 , X t+1/2 - x * and ε t = γ 2 t A(X t+1/2 ) -A(X t ) 2 X t+1/2
, * in Lemma 7.10 and combine it with Lemma 7.6, we get inf x * ∈X * x * , X t converges. Thus, the result follows by applying Proposition 7.9.

Having established the last iterate convergence of the adaptive method we proceed to evaluate numerically the performance of the method. The figure on the left shows the methods' convergence in a 100 × 100 bilinear game; the one on the right shows the methods' convergence in a non-convex/nonconcave covariance learning problem. In both cases, the parameters of the EG and BL algorithms have been tuned with a grid search (AdaProx has no parameters to tune). All curves have been averaged over S = 100 sample runs, and the 95% confidence interval is indicated by the shaded area.

○ ○ ○ ○ ○ ○ ○○○○○○○○ ○ ○ ○ ○ ○ ○○ ○ ○○○ ○ ○ ○ ○ ○○ ○ ○ ○ ○ ○○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ □ □ □ □ □ □ □ □ □□□□□ □□□□ □□□ □ □ □□ □ □□ □□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □□ □ □ □ □ □ □ □□□□□ □ □ □ □ □ □ □□□ □ □ □□□□ □ □ □ □ □□ □□□□ □ □ □□ □□□□□ □ □ □ □□ □ □ □ □ □ □ □□ □ □ □ □□ □ □ □□ □ □ □□ □□ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △△ △△△△△ △ △ △ △ △ △ △ △ △ △△ △△△ △△ △ △ △ △ △△ △ △△△ △ △ △ △ △△ △ △ △ △ △△ △ △ △ △ △ △△ △ △ △ △ △ △ △ △ △ △ △△ △ △△ △ △ △△ △ △ △ △ △ △ △ △ △ △ △△ △ △ △△ △ △ △△ △△△△ △ △△ ○ �����-�������� (������ �������) �����-�������� (����� ��������) □ ����-���� (������ �������) ����-���� (����� ��������) △ ������� (������ �������) ������� (����� ��������) � �� ��� ���� �� � ������� ������ ����� ���� ��� �� ��� ����������� ����� (���������� ���������) ○ ○ ○ ○ ○ ○ ○ ○○○○ ○○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ □ □ □ □ □ □ □ □ □□□ □□□ □□□ □□□ □ □ □□ □ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□□ □□□ □□□ □□□ □□□□ □□□□□□□□□□□□□□ □□ □□ □□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □□□ □ □ □ □ □ □ □ □□ □□ □ △ △ △ △ △ △ △ △ △△△△ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △△△ △ △ △ △ △ △ △ △△ △ △△△△△ △ △ △△△△△△△△△△ △△△△ △△ △△ △△ △△ △△ △△ △△ △ △△ △△ △△ △△ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △ △△ △△ △△ △△ △ ○ �����-�������� (������ �������) �����-�������� (����� ��������)

Numerical evaluation

We conclude in this section with a numerical illustration of the convergence properties of AdaProx in two different settings: a) bilinear min-max games; and b) a simple Wasserstein GAN in the spirit of Daskalakis et al. [START_REF] Daskalakis | Training GANs with optimism[END_REF] with the aim of learning an unknown covariance matrix.

bilinear min-max games. For our first set of experiments, we consider a minmax game of the form of the form Φ(x 1 , x 2 ) = (x 1x * 1 ) A(x 2x * 2 ) with x 1 , x 2 ∈ R 100 and A ∈ R 100 × R 100 (drawn i.i.d. component-wise from a standard Gaussian). To test the convergence of AdaProx beyond the "full gradient" framework, we ran the algorithm with stochastic gradient signals of the form V t = A(X t ) + U t where U t is drawn i.i.d. from a centered Gaussian distribution with unit covariance matrix. We then plotted in Fig. 7.1 the squared gradient norm A( XT ) 2 of the method's ergodic average XT after T iterations (so values closer to zero are better). For benchmarking purposes, we also ran the extra-gradient (EG) and Bach-Levy (BL) algorithms [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF] with the same random seed for the simulated gradient noise. The step-size parameter of the EG algorithm was chosen as γ t = 0.025/ √ t, whereas the BL algorithm was run with diameter and gradient bound estimation parameters D 0 = .5 and M 0 = 2.5 respectively (both determined after a hyper-parameter search since the only theoretically allowable values are D 0 = M 0 = ∞; interestingly, very large values for D 0 and M 0 did not yield good results). The experiment was repeated S = 100 times, and AdaProx gave consistently faster rates. covariance matrix learning. Going a step further, consider the covariance learning game

Φ(x 1 , x 2 ) = E x∼N (0,Σ) [x x 1 x] -E z∼N (0,I) [z x 1 x 2 x 1 z], x 1 , x 2 ∈ R n × R n . (7.
144) The goal here is to generate data drawn from a centered Gaussian distribution with unknown covariance Σ; in particular, this model follows the Wasserstein GAN formulation of Daskalakis et al. [START_REF] Daskalakis | Training GANs with optimism[END_REF] with generator and discriminator respectively given by G(z) = x 1 z and D(x) = x x 2 x (no clipping). For the experiments, we took n = 100, a mini-batch of m = 128 samples per update, and we ran the EG, BL and AdaProx algorithms as above, tracing the square norm of A as a measure of convergence. Since the problem is non-monotone, there are several disjoint equilibrium components so the algorithms' behavior is considerably more erratic; however, after this initial warm-up phase, AdaProx again gave the faster convergence rates.

universality in the presence of noise

In order to derive our general universality result, we change gears from the (MP) template. In particular, we shall adopt a primal-dual approach; more precisely, our focal point is that of the dual extrapolation template presented in (DualX). We recall that the said method is defined by the following recursion:

Universal Dual Extrapolation X t+1/2 = P X t (-γ t V t ) Y t+1 = Y t -V t+1/2 X t+1 = Q(γ t+1 Y t+1 )
Throughout this section, given the dual extrapolation method run once more with the adaptive learning rate (Adapt):

Universal Step-Size γ t = 1 1 + ∑ t-1 j=1 V j+1/2 -V j 2 X j+1/2 , * (Adapt)
In addition, we assume that the optimizer has access to a first order oracle of the form (SFO) which satisfies the following statistical assumptions: Furthermore, concerning the regularity conditions assumed for the associated operators and the ambient space the same assumptions hold as in Section 7.3, i.e., 1. X is a regular Finsler space (cf. Section 3.2.2).

2. The respective generalizations of the standard Lipschitz regularity are: given a family of local norms • x with x ∈ X , the respective monotone operators under study will be that satisfying (MB) and/or (MS)

3. The associated regularizer h is a Bregman-Finsler function, i.e., satisfies Definition 3.3.

Having all this at hand, we are in position to present the main result of this section; namely we present optimal convergence rate guarantees for both deterministic and stochastic settings. Formally, we have the following theorem.

Universality Guarantees (Stochastic & Deterministic)

Theorem 7.11 (Antonakopoulos and Mertikopoulos [START_REF] Antonakopoulos | Universal methods for variational inequalities with divergent operators[END_REF]). Assume that X t+1/2 , X t are the (DualX) iterates run with the adaptive step-size policy (Adapt) and a (SFO) satisfying (7.145b). Then, the following hold:

1. If A satisfies (MB), then, E [Gap C ( XT )] = O(1/ √ T) (7.146) 2. If A satisfies (MS), then, E [Gap C ( XT )] = O A T + Bσ √ T (7.147)
In order to prove Theorem 7.11 we will use extensively a key template which connects iterates after the. respective prox and mirror steps. In what follows we will illustrate this in a detailed manner.

Template inequalities

The proof of Theorem 7.11 hinges again on a primal-dual type template inequality which involves Fenchel couplings instead of Bregman divergences as in (MP) setting. Namely, we seek to prove an inequality of the form:

T ∑ t=1 V t+1/2 , X t+1/2 -x ≤ h(x) -min h γ T+1 + T ∑ t=1 V t+1/2 -V t , Q(γ t Y t+1 ) -X t+1/2 - T ∑ t=1 1 γ t D(Q(γ t Y t+1 ), X t+1/2 ) - T ∑ t=1 1 γ t D(X t+1/2 , X t ) (7.148)
In doing so we will need the following result.

Template Inequality for Universality Lemma 7.12 (Antonakopoulos and Mertikopoulos [START_REF] Antonakopoulos | Universal methods for variational inequalities with divergent operators[END_REF]). If X t+1/2 , X t are the iterates of (DualX) run with a decreasing learning rate γ t , then the following inequality holds for all x ∈ X :

1 γ t F(x, γ t Y t+1 ) ≤ 1 γ t F(x, γ t Y t ) -V t+1/2 , X t+1/2 -x + V t+1/2 -V t , Q(γ tY t+1 ) -X t+1/2 - 1 γ t D(Q(γ t Y t+1 ), X t+1/2 ) - 1 γ t D(X t+1/2 , X t ) (7.149)
Proof. For all x ∈ X we have:

V t+1/2 , Q(γ t Y t+1 ) -x = 1 γ t γ t Y t -γ t Y t+1 , Q(γ t Y t+1 ) -x = 1 γ t F(x, γ t Y t ) - 1 γ t F(x, γ t Y t+1 ) - 1 γ t F(Q(γ t Y t+1 ), γ t Y t )
Therefore, by rearranging we get: and the result follows.

1 γ t F(x, γ t Y t+1 ) = 1 γ t F(x, γ t Y t ) -V t+1/2 , Q(γ t Y t+1 ) -x - 1 γ t F(Q(γ t Y t+1 ), γ t Y t ) = 1 γt F(x, γ tY t ) -V t+1/2 , X t+1/2 -x + V t+1/2 , Q(γ t Y t+1 ) -X t+1/2 - 1 γ t F(Q(γ t Y t+1
Now armed Lemma 7.12, we are in the position to prove our main "energy" inequality (7.148).

Regret Inequality Proposition 7.13. Assume that X t+1/2 , X t are the iterates of (DualX) run with a decreasing learning rate γ t . Then, for all x ∈ X , the following "regret" estimation holds: • For the second term 1 γ t F(x, γ t Y t+1 ) -1 γ t F(x, γ t Y t ) we readily get by Lemma 7.12:

T ∑ t=1 V t+1/2 , X t+1/2 -x ≤ h(x) -min h γ T+1 + T ∑ t=1 V t+1/2 -V t , Q(γ t Y t+1 ) -X t+1
1 γ t F(x, γ t Y t+1 ) - 1 γ t F(x, γ t Y t ) ≤ V t+1/2 , X t+1/2 -x + V t+1/2 -V t , Q(γ tY t+1 ) -X t+1/2 - 1 γ t D(Q(γ t Y t+1 ), X t+1/2 ) - 1 γ t D(X t+1/2 , X t ) (7.160)
Hence, combining all this and after telescoping through t = 1, . . . , T we get: Moving forward, the second crucial ingredient for establishing our main result we will need a result for martingale differences, introduced by [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF][START_REF] Kakade | Lecture notes in multivariate analysis, dimensionality reduction, and spectral methods[END_REF]. Proposition 7.15 (Bach and Levy [14], Kakade [START_REF] Kakade | Lecture notes in multivariate analysis, dimensionality reduction, and spectral methods[END_REF]). Let C ⊂ R n and h : X → R be a Bregman function. Also assume that for all x ∈ C we have: where after taking expectations on both sides we get:

T ∑ t=1 V t+1/2 , X t+1/2 -x ≤ 1 γ 1 F(x, Y 1 ) + 1 γ T+1 - 1 γ 1 (h(x) -min h) + T ∑ t=1 V t+1/2 -V t , Q(γ t Y t+1 ) -X t+1/2 - T ∑ t=1 1 γ t D(Q(γ t Y t+1 ), X t+1/2 ) - T ∑ t=1 1 γ t D(X t+1/2
E Gap C (X T ) ≤ 1 T D E 1 γ T+1 + 1 2K E T ∑ t=1 γ t V t+1/2 -V t 2 X t+1/2 , * + sup x∈C E T ∑ t=1 U t+1/2 , x -X t+1/2 - K 2 E T ∑ t=1 1 γ t X t+1/2 -X t 2 X t (7.185)
Now, we shall bound from above each (RHS) term individually:

• For the term D E 1 γ T+1

we have:

D E 1 γ T+1 = D E   1 + T ∑ t=1 V t+1/2 -V t 2 X t+1/2 , *   ≤ D 1 + T ∑ t=1 E V t+1/2 -V t 2
X t+1/2 , * and hence by applying Lemma 7.14 we get:

D E 1 γ T+1 ≤ D 1 + C 2 T (7.186)
with C 2 being the constant obtained in Lemma 7.14.

• For the term 1 2K E ∑ T t=1 γ t V t+1/2 -V t 2 X t+1/2 , * : For the second term of the expression we have:

1 2K E T ∑ t=1 γ t V t+1/2 -V t 2 X t+1/2 , * = 1 2K E T ∑ t=1 (γ t -γ t+1 ) V t+1/2 -V t 2 X t+1/2 , * + 1 2K E T ∑ t=1 γ t+1 V t+1/2 -V
E T ∑ t=1 U t+1/2 , X t+1/2 = T ∑ t=1 E [ E [U t+1/2 |F t+1/2 ] , X t+1/2 ]
= 0 with the last equality being obtained by the zero mean assumption for the noise U t for t = 1, 1/2, . . . Now, the tricky part is dealing with the first term; at this point we shall apply Proposition 7.15. In particular, we have:

sup x∈C E T ∑ t=1 U t , x = max x∈C E T ∑ t=1 U t , x ≤ D 2 T ∑ t=1 E [ U t+1/2 2 * ] ≤ Dσ √ T 2
with the last inequality obtained by the (almost sure) boundedness of the noise.

Therefore, summarizing we get that:

E Gap C (X T ) = O(1/ √ T) (7.190)
and hence the first result follows.

2. Now we turn our attention towards the (MS) case. In particular, working in the same spirit as in (MB) we have:

Analysis Under (MS) Having all this at hand, we revisit (7.199). In particular, we have:

E Gap C (X T ) ≤ 1 T D E 1 γ T+1 + 1 2K E T ∑ t=1 γ t V t+1/2 -V
E T ∑ t=1 γ t V t+1/2 -V t 2 X t+1/2 , * ≤ C 2 + E   1 + T ∑ t=1 V t+1/2 -V t
E T ∑ t=1 1 2K γt - K 2β 2 γt B 2 t ≤ 1 2K E t 0 ∑ t=1 γt B 2 t = 1 2K E t 0 ∑ t=1 ( γt -γt+1 )B 2 t + 1 2K E t 0 ∑ t=1 γt+1 B 2 t ≤ C 2 2K + 1 2K E   1 + t 0 ∑ t=1 B 2 t   = C 2 2K + 1 2K E   1 + t 0 -1 ∑ t=1 B 2 t   + 1 2K E [B t 0 ] ≤ C 2 2K + C 2K + 1 2K E 1 γt 0 ≤ C 2 2K + C 2K + β 2K 2
Hence summarizing the above bounds, we conclude that:

E Gap C ((X) T ) = O A T + Bσ √ T (7.205)
and so the result follows.

P E R S P E C T I V E S W e conclude this thesis by providing some research perspectives for future work.

These directions concern the two key optimization scenarios treated in this thesis. We shall illustrate these for each framework individually.

minimization settings

Our theoretical analysis confirms that Mirror Descent methods concurrently achieve optimal rates of convergence in relatively continuous and relatively smooth problems, both stochastic or deterministic, constrained or unconstrained, and without requiring any prior knowledge of the problem's smoothness/continuity parameters. These appealing properties open the door to the following future research directions:

• NoLips Acceleration:

One important question that remains is whether the O(1/T) rate can be improved to O(1/T 2 ) for relative smooth problems. Assuming boundedness, we know that this is possible in the Euclidean case: AcceleGrad and UnixGrad already achieve an accelerated rate [START_REF] Kavis | UnixGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization[END_REF][START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF]. On the other hand, for problems that are h-smooth in the sense of [START_REF] Heinz | A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications[END_REF], the very recent paper of [START_REF] Dragomir | Optimal complexity and certification of Bregman first-order merthods[END_REF] showed that the O(1/T) rate is, in general, unimprovable.

One idea is to substitute relative smoothness by (MS) conditions. In particular, as we discussed above, one may show that metrically smooth problems are also h-smooth for a suitable choice of h, suggesting that the O(1/T) rate may also be optimal in this problem class. We conjecture that this indeed the case; at the same time, there is no evidence to suggest that an accelerated rate cannot be obtained for real-world singular problems like D-optimal design or PIP.

• Adaptation between smooth and relative smooth objectives:

Another linked question is whether we could design a method which is able to adapt its performance optimally between the classes of smooth and relative smooth functions. More precisely, is it possible to have a method which exhibits a generic rate of O(1/T) for relative smooth objectives and automatically adjusts its performance to O(1/T 2 ) if the respective function is smooth in the ordinary sense. An intuitive approach is to examine whether a Bregman based variant of AcceleGrad can satisfy this type of adaptivity.

We defer both these questions to future work.

variational inequality setting

Our main goal in Chapter 7 was to design a universal, regime-agnostic first-order method for variational inequality problems with possibly unbounded domains and/or divergent operators for both deterministic and stochastic settings. By leveraging a suitable Finsler regularity framework and a compatible Bregman toolkit, adaptive Mirror-Prox/ Dual Extrapolation algorithms achieve the above desiderata, and their rates interpolates sharply between O(1/ √ T) and O(1/T) for metrically bounded/ stochastic and smooth problems respectively. This leaves open several questions such as:

• (VI) to (Opt) adaptation:

As we already described throughout there is a convergence rate discrepancy between min-min and min-max problems: if the underlying operator is smooth, it is possible to achieve a O(1/T 2 ) value convergence rate for (Opt); by contrast for (VI) the best attainable rate is O(1/T). However, in order to apply the appropriate algorithm the optimizer should know in advance whether the associated operator is a smooth gradient field or not. Therefore, a natural open question which arises in this context is whether it is possible to design an accelerated regime-agnostic method that achieves a O(1/T) rate for general variational inequalities -i.e., those for which A is not a gradient field -and a O(1/T 2 ) when the underlying operator is a gradient. The key difficulty in order to establish such a method is to be able to provide simultaneously:

1. an acceleration mechanism which is activated whenever the operator is a gradient field.

2. An extra-gradient type template to ensure optimal rates for (VI)

An obvious candidate seems to be UniXGrad; however it remains unclear whether it exhibits such a behavior.

• Last iterate convergence rates for adaptive methods convergence rate for (VI):

As we already mentioned the best attainable rate for (VI) relative to the restricted merit function is O(1/T); as we discussed this rate is achieved by the ergodic and/or time average of extra-gradient methods. This rises the question of what is the asymptotic behaviour of the last iterate of (MP) method, i.e., before any type of average occurs. In a recent paper, Golowich et al. [START_REF] Golowich | Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems[END_REF] showed that the last iterate of (MP) is actually slower, if run with a constant step-size ≤ 1/β, showing that it exhibits a rate of the order O(1/ √ T). Furthermore, Yoon and Ryu [START_REF] Yoon | Accelerated algorithms for smooth convex-concave minimax problems with O(1/k 2 ) rate on squared gradient norm[END_REF] showed that O(1/T) can be recovered for the last iterate of a (MP) variants which incorporates a so-called anchoring mechanism. That said, their still requires a prior knowledge of the Lipschitz constant. Thus, the question of what is performance of (MP) run with an adaptive step-size remains open. More precisely, can an adaptive stepsize ensure that the last iterate of (MP) exhibits optimal speed of convergence? We defer this question for future work. Then, dividing the sum by T 0 , we get:

T ∑ t=1 α t α 0 + ∑ t-1 i=1 α i ≤ T 0 -1 ∑ t=1 α t α 0 + ∑ t-1 i=1 α i + T ∑ t=T 0 α t α 0 + ∑ t-1 i=1 α i ≤ 1 α 0 T 0 -1 ∑ t=1 α t + T ∑ t=T 0 α t 1/2α 0 + 1/2α + 1/2 ∑ t-1 j=1 α j ≤ α α 0 + 2 T ∑ t=T 0 α i /α 0 1 + ∑ t j=T 0 α j /α 0 ≤ 2α α 0 + 2 + 2 log 1 + T ∑ t=T 0 α i /α 0 ≤ 2α α 0 + 2 + 2 log 1 + T ∑ t=1 α i /α 0 (A.7)
where we used the fact that ∑ T 0 -2 j=1 α j ≤ α as well as for all t ≥ T 0 , ∑ t-1 j=1 α j ≥ α (both follow from the definition of T 0 ) and Lemma A.2. colophon This manuscript was typeset with L A T E X 2 ε using Hermann Zapf's Palatino type face (the actual Type 1 PostScript fonts used were URW Palladio L and FPL). The monospaced text (hyperlinks, etc.) was typeset in Bera Mono, originally developed by Bitstream, Inc. as "Bitstream Vera" (with Type 1 PostScript fonts by Malte Rosenau and Ulrich Dirr).

The typographic style of this dissertation was inspired by the authoritative genius of Bringhurst's Elements of Typographic Style, ported to L A T E X by André Miede, the original designer of the classicthesis template. Any unsightly deviations from these works should be attributed solely to the author's (not always successful) efforts to conform to the awkward A4 paper size.

  viii A C K N O W L E D G M E N T S

2 1 . 3 1 . 4

 21314 Main objective and contributions of this thesis 5 Diagrammatic outline 6 Notational conventions 7

Figure 2 . 1 14 Figure 2 . 2 Schematic representation of gradient descent 20 Figure 2 . 3 20 Figure 2 . 4 23 Figure 4 . 1 Schematic representation of lazy mirror descent 48 Figure 5 . 1 60 Figure 6 . 1 84 Figure 6 . 2 86 Figure 6 . 3 Statistics of adaptive mirror descent 86 Figure 7 . 1

 21142220232024234148516061846286638671 Figure 2.1 Schematic representation of a VI problem 14 Figure 2.2 Schematic representation of gradient descent 20 Figure 2.3 Lazy vs. ordinary gradient descent 20 Figure 2.4 Schematic representation of the extra-gradient algorithm 23 Figure 4.1 Schematic representation of lazy mirror descent 48 Figure 5.1 Reconstruction of the Lena test image 60 Figure 6.1 Convergence of AdaMir in a Fisher market model 84 Figure 6.2 Convergence of AdaMir in a stochastic Fisher market 86 Figure 6.3 Statistics of adaptive mirror descent 86 Figure 7.1 Comparison of adaptive methods for VI problems 112

1 I

 1 N T R O D U C T I O N

2 .

 2 Interpolation of convergence rates between different problem classes (e.g., O(1/ √ T) for non-smooth vs. O(1/T) or O(1/T 2 ) for smooth, etc.).

Figure 2 . 1 :

 21 Figure 2.1: Schematic representation of a variational inequality problem: the defining vector field A at a solution x * of (VI) belongs to the polar cone PC(x * ) to X at x * .

Figure 2 . 3 :

 23 Figure 2.3: Lazy vs. ordinary gradient descent.

Figure 2 . 4 :

 24 Figure 2.4: Schematic representation of the extra-gradient algorithm.

Figure 4 . 1 :

 41 Figure 4.1: Schematic representation of lazy mirror descent.

Figure 5 . 1 :

 51 Figure 5.1: Reconstruction of the Lena test image from a sample contaminated with Poisson noise. Left to right: (a) the contaminated sample; (b) RMD reconstruction; and (c) Poisson likelihood loss at each iteration. The RMD process provides a sharper definition of image features relative to the CMP algorithm (which is the secondbest).

Figure 6 . 1 :

 61 Figure 6.1: The convergence speed of (EGD), (PR) and AdaMir in a stationary Fisher market.

Figure 6 . 3 :

 63 Figure 6.3: Statistics for the convergence speed of (EGD), (PR) and AdaMir in a stochastic Fisher market, with marginal utilities drawn i.i.d. at each epoch. The marked lines are the observed means from S = 50 realizations, whereas the shaded areas represent a 95% confidence interval.

  sup p∈C D(p, X 1 ) ≤ D for some constant D > 0. (7.78) by the Definition 3.3 of the regularizer h.

Figure 7 . 1 :

 71 Figure 7.1: Numerical comparison between the extra-gradient (EG), Bach-Levy (BL) and AdaProx algorithms (red circles, green squares and blue triangles respectively).The figure on the left shows the methods' convergence in a 100 × 100 bilinear game; the one on the right shows the methods' convergence in a non-convex/nonconcave covariance learning problem. In both cases, the parameters of the EG and BL algorithms have been tuned with a grid search (AdaProx has no parameters to tune). All curves have been averaged over S = 100 sample runs, and the 95% confidence interval is indicated by the shaded area.

bibliography Lemma A. 5 .

 5 For all non-negative numbers: α 1 , . . . α t ∈ [0, α], α 0 ≥ 0, we have: Let us denoteT 0 = min t ∈ [T] : ∑ t-1 j=1 α j ≥ α (A.6)

Table 6 . 1

 61 Overview of adaptive methods for convex optimization 64

Table 7 . 1

 71 Overview of adaptive methods for VI problems 88

  al. (2019): (Antonakopoulos et al.[START_REF] Kimon Antonakopoulos | An adaptive mirror-prox algorithm for variational inequalities with singular operators[END_REF][START_REF] Kimon Antonakopoulos | Online and stochastic optimization beyond Lipschitz continuity: A Riemannian approach[END_REF][START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Let • x be a local norm. We say V → R is a Bregman-Finsler function on X if: h is a Bregman function in the sense of Definition 3.1 and h is strongly convex relative to the underlying local norm, i.e., h(p)

	Definition 3.3 Bregman-Finsler
	that h :	Functions

  Inspired by the above toy-example a robust theoretical is provided via a local norm induced by Definition 3.2. Formally, we propose the following definition for a generic operator A; firstly introduced in Antonakopoulos et al. (2021). (Antonakopoulos et al.[START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Let • x , x ∈ X be a local norm. We say that A is metrically bounded relative to • x , if there exists some G > 0 such that:

	1 for all x > 0	(3.21)
		Metric Boundedness
	Definition 3.6

A(x)

x, * ≤ G for all x ∈ X (MB) Remark 3.1. Of course, the standard Euclidean (Bd) is directly recovered by considering • x = • . Moreover, in Antonakopoulos et al. (2020) a Riemann-Lipschitz continuity condition is introduced and extends (LC) for the OCO setting as follows. Let • x be a family of local norms on X , induced by an appropriate Riemannian metric, and let w x, * = max x x ≤1 w, x denote the corresponding dual norm. Then, f is Riemann-Lipschitz continuous relative to • x if there exists some G > 0 such that:

Table 6 . 1 :

 61 Overview of related adaptive methods for convex optimization.

	Mirror Descent
	Template

  2, . . . almost surely (6.74)

	Almost Sure
	Boundedness of the
	Residual Under (RC)

  ◇◇◇ ◇◇ ◇ ◇◇◇ ◇◇◇◇◇ ◇ ◇◇◇◇ ◇◇ ◇◇◇ ◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇

	□ ○	□	◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ ◇◇ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □ □□ □□ □□ □□ □□ □□ □□□ □□□ □ □□ □□□ □ □ □□ □ □□ □□□ □□□ □□ □□ □□□ □ □ □ □ □ □ □□

  .[START_REF] Alvarez | Hessian Riemannian gradient flows in convex programming[END_REF]. In all cases, AdaMir

	◇ □ ○	◇ □	◇ ◇ ◇ ◇ ◇ ◇◇◇◇◇◇◇◇◇◇◇◇ ◇◇ ◇ ◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇ ◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ □□□ □□□ □□□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□□ □□ □□□ □□□ □□□ □□□ □□□ □□□□ □□□ □□ □□□ □ □ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□ □□□□ □ □ □ □ □ □ □□ □□□

  ◇◇ ◇ ◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇◇◇◇◇ ◇◇◇◇◇◇◇ ◇◇◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ ◇◇◇ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ □□ ○

	□	□	□ □ □ □ □ □ □□ □□□ □□□ □□□ □□ □ □□ □ □□□ □□□ □□□ □□□ □□□ □□□ □□ □□ □□	◇◇◇ ◇◇◇ ◇◇◇

  , X t ).(7.36) Since γ t converges, it follows that lim t→∞ γ 2

				t /γ 2 t+1 = 1, so we get
	lim t→∞	1 -θ 2 γ 2 t t+1 γ 2	= 1 -θ 2 > 0,	(7.37)
	implying in turn that
	1 -θ 2 γ 2 t t+1 γ 2	D(X t+1/2 , X t ) > 0	(7.38)

  Case 2: Convergence under (MS) We now suppose that A satisfies (MS) condition. By applying Lemma 7.6 along with :

	2. Analysis Under (MS)
	T
								84)
	which yields:
	T ∑ t=1	γ t = Ω(	√	T) and	T t=1 ∑	γ t → +∞	(7.85)
	Now, by combining (7.78), (7.83) and (7.85) we readily get that under (MB)
	we get that:	Gap C ( XT ) = O(1/	√	T).	(7.86)

  Boundedness with probability 1, i.e., there exists some σ 2 > 0 such that

	U t	2 * ≤ σ 2 almost surely for all t = 1, 2, . . .	(7.145b)

Blanket Assumptions Assumption 7.3. 1. Zero-mean noise, i.e., E[U t |F t ] = 0 for all t = 1, 2, . . . . (7.145a) 2.

  ), γ t Y t ) and since F(Q(γ t Y t+1 ), γ t Y t ) ≥ D(Q(γ t Y t+1 ), X t ) the above becomes:γ t Y t ) -V t+1/2 , X t+1/2x + V t+1/2 ), Q(γ t Y t+1 ) -X t+1/2 Y t+1 ), X t ) (7.150)On the other hand, by the prox-step we have:V t , X t+1/2x ≤ 1 γ t ∇h(X t ) -∇h(X t+1/2 ), X t+1/2x = 1 γ t ∇h(X t+1/2 ) -∇h(X t ), x -X t+1/2 , X t )where the last equality is obtained by Lemma 4.2. Hence, by rearranging and settingx = Q(γ t Y t+1 ) we get: , X t ) + V t , X t+1/2 -Q(γ t Y t+1 ) ≤ 1 , γ t Y t ) -V t+1/2 , X t+1/2x + V t+1/2 , Q(γ t Y t+1 ) -X t+1/2 + V t , X t+1/2 -Q(γ t Y t+1 ) -, γ t Y t ) -V t+1/2 , X t+1/2x + V t+1/2 -V t , Q(γ t Y t+1 ) -X t+1/2 -1

	1 γ t	F(x, γ t Y t+1 ) ≤	1 γ t	F(x1 γ t	D(Q(γ t Y t+1 ), X t+1/2 ) -	1 γ t	D(X t+1/2 , X t )
								(7.152)
	So, finally we have:
	1 γ t	F(x, γ t Y t+1 ) ≤	1 γ t	F(xγ t	D(Q(γ t Y t+1 ), X t+1/2 ) -	1 γ t	D(X t+1/2 , X t )
								(7.153)
	1 γ t D(X t+1/2 1 F(x, γ t Y t+1 ) ≤ 1 γ t F(x, -1 γ t D(Q(γ = 1 γ t D(x, X t ) -1 γ t D(x, X t+1/2 ) -1 γ t γ t D(Q(γ t Y t+1 ), X t+1/2 ) + 1 γ t D(X t+1/2 γ t	D(Q(γ t Y t+1 ), X t )
						(7.151)
	Thus, by combining the above inequalities we obtain:

t

  Proof. By setting E t = 1 γ t F(x, γ t Y t ), we have:[ γw, Q(γw)h * (γw) + min h] = 1 γ 2 [h(Q(γw))min h] ≥ 0Therefore, we get that φ is non-decreasing function and since γ t+1 ≤ γ t we have φ(γ t+1 ) ≤ φ(γ t ), i.e.,: (γ t+1 Y t+1 ) + min h] ≤ 1

	Now, considering the function:
					φ(γ) =	1 γ	[h * (γw) + min h]	(7.156)
	By taking the derivative (with respect to γ) we get:
		φ (γ) =	1 γ		w, Q(γw) -	1 γ 2 [h * (γw) + min h]
	1 γ 2 1 = γ t+1 [h γ t	[h * (γ t Y t+1 ) + min h]	(7.157)
	where after rearranging we obtain:
	1 γ t+1	h * (γ t+1 Y t+1 ) -	1 γ t	h * (γ t Y t+1 ) ≤	1 γ t	-	1 γ t+1	min h	(7.158)	/2
	-Hence, by combining all the above we get: T ∑ t=1 1 γ t D(Q(γ t Y t+1 ), X t+1/2 ) -1 γ t+1 F(x, γ t+1 Y t+1 ) -1 γ t F(x, γ t Y t+1 ) ≤ 1 γ t+1 -1 γ t h(x) -min h (7.159) T ∑ t=1 1 γ t D(X t+1/2 , X t ) (7.154)
	E t+1 -E t =	1 γ t+1	F(x, γ t+1 Y t+1 ) -	1 γ t	F(x, γ t Y t )
		=	1 γ t+1	F(x, γ t+1 Y t+1 ) -	1 γ t	F(x, γ t Y t+1 ) +	1 γ t	F(x, γ t Y t+1 ) -	1 γ t	F(x, γ t Y t )
	Now let us deal with each bracket individually.
	• For the first term 1 γ t+1	F(x, γ t+1 Y t+1 ) -1 γ t F(x, γ t Y t+1 ) we have:
		1 γ t+1	F(x, γ t+1 Y t+1 ) -	1 γ t	F(x, γ t Y t+1 ) =	1 γ t+1	h(x) +	1 γ t+1	h * (γ t+1 Y t+1 ) -Y t+1 , x
											-	1 γ t	h(x) -	1 γ t	h * (γ t Y t+1 ) + Y t+1 , x
		and hence we have:
		1 γ t+1	F(x, γ t+1 Y t+1 ) -	1 γ t	F(x, γ t Y t+1 ) =	1 γ t+1	-	1 γ t	h(x) +	1 γ t+1	h * (γ t+1 Y t+1 )
											-	1 γ t	h * (γ t Y t+1 ) (7.155)

*

  , X t ) ≤ 4 G 2 + µ 2 σ 2 + 2 2G 2 + 2µ 2 σ 2 +

	Finally, by combining (7.166) with (7.168) and (7.174) we get:
	V t -V t+1/2	2 X t+1/2 2βγ 1 K	2G 2 + 2µ 2 σ 2	2
			(7.175)
	and the proof is complete.
			(7.161)

  Then, for any martingale difference sequence (ζ t ) ∈ R n and any random vector x defined Armed with these tools we are in position to prove Theorem 7.11.Proof of Theorem 7.11. For the sake of convenience we shall present the analysis under (MB) and (MS) separately.1.For the (MB) case we have the following: By recalling Proposition 7.[START_REF] Auslender | Interior gradient and proximal methods for convex and conic optimization[END_REF] we have for all x ∈ X : Moreover, since A is monotone we have for all x ∈ X : A(x), X t+1/2x ≤ A(X t+1/2 ), X t+1/2x

			+	T ∑ t=1	U t+1/2 , x -X t+1/2 -	K 2	T ∑ t=1	1 γ t	X t+1/2 -X t	2 X t (7.180)
												(7.181)
	which in turn yields:
	T ∑ t=1	A(x), X t+1/2 -x ≤	h(x) -min h γ T+1	+	1 2K	T ∑ t=1	γ t V t+1/2 -V t	2 X t+1/2 , *
	h(x) -min x∈C U t+1/2 , x -X t+1/2 -K 2 T ∑ t=1 and so, by dividing both sides by T and exploiting convexity we have: h(x) ≤ 1 + T ∑ t=1 1 γ t X t+1/2 -X t 2 X t (7.182) D 2 2	(7.176)
	Martingale Difference Estimation	over C, we have: A(x), X T -x ≤ 1 T + T ∑ t=1 U t+1/2 , x -X t+1/2 -E T ∑ t=1 ζ t , x ≤ h(x) -min h γ T+1 + 1 2K T ∑ γ t V t+1/2 -V t D 2 T ∑ t=1 E [ ζ t 2 * ] 2 X t+1/2 , * t=1 K 2 T ∑ t=1 1 γ t X t+1/2 -X t 2 X t	(7.183)	(7.177)
	Now, by considering a compact neighbourhood C of a solution and taking
	suprema we have:
	Gap C (X T ) ≤	1 T		D γ T+1	+	1 2K	T ∑ t=1	γ t V t+1/2 -V t	2 X t+1/2 , *
	Analysis Under (MB)		+	T ∑ t=1 T ∑ t=1	V t+1/2 , X t+1/2 -x ≤ U t+1/2 , x -X t+1/2 -K h(x) -min h γ T+1 2 T ∑ t=1 1 γ t X t+1/2 -X t + 1 2K ∑ t=1 T 2 X t γ t V t+1/2 -V t (7.184)	X t+1/2 , * 2
					+	K 2	T ∑ t=1	1 γ t	Q(γ t Y t+1 ) -X t+1/2	2 X t+1/2 -	K 2	T ∑ t=1	1 γ t	X t+1/2 -X t	2 X t
												T
												-	t=1 ∑	D(Q(γ t Y t+1 ), X t+1/2 ) (7.178)
			Thus by applying Definition 3.3 the above becomes:
				T ∑ t=1		V t+1/2 , X t+1/2 -x ≤	h(x) -min h γ T+1	+	1 2K	T ∑ t=1	γ t V t+1/2 -V t	2 X t+1/2 , *
												-	K 2	T ∑ t=1	1 γ t	X t+1/2 -X t	2 X t (7.179)
			Now by the definition of the oracle's feedback, we have:
												γ T+1	+	1 2K	T ∑ t=1	γ t V t+1/2 -V t	2 X t+1/2 , *

T ∑ t=1 A(X t+1/2 ), X t+1/2x ≤ h(x)min h

  γ t+1 ) V t+1/2 -V t For the term sup x∈C E ∑ T t=1 U t+1/2 , x -X t+1/2 we have: , x -X t+1/2 = sup

								t	2 X t+1/2 , *	(7.187)
	Now, we have by applying Lemma 7.14:
		1 2K	E		2 X t+1/2 , * ≤	1 2K	C 2	(7.188)
	• sup x∈C	E	T ∑ t=1	U t+1/2 x∈C	E	T t=1 ∑	U t+1/2 , x
					T
				-E	t=1 ∑	U t+1/2 , X t+1/2	(7.189)

T ∑ t=1 (γ t

  = min{ A(X t ) -A(X t+1/2 ) 2 X t+1/2 , * , V t+1/2 -V t Moreover by denoting ξ t = [V t+1/2 -V t ] -[A(X t+1/2 ) -A(X t )] we have: , * ≤ 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + 2 ξ t 2 X t , * ≤ 2 A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * + 2µ ξ t -min{ A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * , V t+1/2 -V twith the last inequality being obtained by the fact that ifV t+1/2 -V t 2 X t+1/2 , * ≥ A(X t+1/2 ) -A(X t ) 2 , * -A(X t+1/2 ) -A(X t ) 2 X t+1/2, * ≤ B 2 t + 2µ ξ t 2 * (7.197)

										X t+1/2 , *	(7.196)
				then it yields:
					V t+1/2 -V t	2 X t+1/2
										t	2 X t+1/2 , *
	+ sup x∈C	E	T ∑ t=1	U t+1/2 , x -X t+1/2 -	K 2	E	T ∑ t=1	1 β 2 γ t	A(X t+1/2 ) -A(X t ) 2 X t+1/2 , *
										(7.191)
	Now, set:
	B 2 t 2 X t+1/2 , * }	(7.192)
	and the respective auxiliary learning rate:
								γt =	1 1 + ∑ t-1 j=1 B 2 j	(7.193)
	By definition of B 2 t , we have 1 γt ≤ 1 γ t and hence:
					-	1 γ t	A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * ≤ -	1 γt	B 2 t	(7.194)
		V t+1/2 -V t	2 X t+1/2 2 *
	Moreover, we have:
	V t+1/2 -V t	2 X t+1/2 , * ≤ B 2 t + V t+1/2 -V t	2 X t+1/2 , *
										2 X t+1/2 , * } (7.195)
	which in turn yields:
	V t+1/2 -V t	2 X t+1/2 , * ≤ B 2 t + max{0, V t+1/2 -V t	2 X t+1/2 , * -A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * }
							≤ B 2 t + B 2 t + 2µ ξ t	2 *
							= 2B 2 t + 2 ξ t	2 *
							≤ B 2 t + max{0, V t+1/2 -V t	2 X t+1/2 , * -A(X t+1/2 ) -A(X t ) 2 X t+1/2 , * }
							≤ B 2 t + B 2 t + 2µ ξ t	2 *
							= 2B 2 t + 2µ ξ t	2 *

  2 denoting the constant derived in Lemma 7.14. Now by combining the inequalities we have:

	• For the term 1 K E	2µ ∑ T t=1 ξ t 2 * we have by Jensen's inequality:
			1 K	E	  2µ	T ∑ t=1	ξ t 2 *	  ≤	1 K	2µ	T t=1 ∑	E [ ξ t 2 * ]	(7.202)
	Moreover we have that:
				E[ ξ t	2 * ] = E[ U t+1/2 -U t	2 * ]
							≤ 2 E[ ξ t+1/2 ≤ 4σ 2	2 * ] + 2 E[ ξ t	2 * ]	2 X t+1/2 , *	 
	which in turn implies:							(7.198)
			with C E T ∑ t=1	γ t V t+1/2 -V t 1 K E   2µ ∑ 2 X t+1/2 , * ≤ C 2 + E  T t=1 ξ t 2 *  ≤ 2σ K 2µT  1 + 2 	∑ t=1 T	B 2 t + 2 (7.203) ∑ t=1 T	α 2 t ξ t 2 *	 
	• For the term E ∑ T t=1 This yields:	1 2K γt -K 2β 2 γt		≤ C 2 + E ≤ C 2 + E K }   1 + 2   T ∑ t=1 γt B 2 t + 2 2 T ∑ t=1 B 2 t + 2µ T µ ξ t 2 T ∑ t=1 * (7.204) β ∑ t=1	ξ t 2 *  	 
	E	T ∑ t=1	So, (7.199) becomes: E Gap C (X T ) ≤ 1 2K γt -K 2β 2 γt B 2 t = E 1 T + E T ∑ t=1 1 2K γt -D E t 0 ∑ t=1 K 2β 2 γt 1 2K 1 γ T+1 γt -+ sup K 2β 2 γt x∈C E B 2 t + 1 K E  B 2 t + E T ∑ t=1 U t+1/2 , x -X t+1/2 T ∑ 1 2K γt -t=t 0 +1  2µ t=1 ∑ ξ t 2 * (7.199) K 2β 2 γt  + C 2 T  ≤ E t 0 ∑ t=1 1 2K γt -K 2β 2 γt B 2	B 2 t
			Now let us bound each term individually:
			• For the term D E 1 γ T+1	we have by Lemma 7.14:
											D E	1 γ T+1	≤ D 1 + C 2 T	(7.200)
			• For the term sup x∈C E ∑ T t=1 U t+1/2 , x -X t+1/2 working in the same
			spirit as in we have:
							sup x∈C	E	T ∑ t=1	U t+1/2 , x -X t+1/2 ≤	Dσ 2 √	T	(7.201)

B

2

t we have: First we set:

t 0 = max{1 ≤ t ≤ T : γt ≥ t

with the last inequality being obtained by the definition of t 0 . Moreover, we have:

Non smooth and smooth objectives are included as extreme cases for the Hölderian exponent q = 0, 1.

1.4 notational conventions

In the literature, this formulation of the problem is sometimes referred to as a Stampacchia[START_REF] Facchinei | Finite-Dimensional Variational Inequalities and Complementarity Problems[END_REF] or "strong" variational inequality[START_REF] Juditsky | Solving variational inequalities with stochastic mirror-prox algorithm[END_REF][START_REF] Nesterov | Dual extrapolation and its applications to solving variational inequalities and related problems[END_REF]. This is to distinguish with the Minty or "weak" variational inequality; these two formulations are equivalent when A is monotone, so we will not distinguish between them in the sequel.

Recent developments on convergence results and rates can be found in[START_REF] Golowich | Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems[END_REF][START_REF] Yoon | Accelerated algorithms for smooth convex-concave minimax problems with O(1/k 2 ) rate on squared gradient norm[END_REF] and references therein.

For a panoramic overview of cocoercive operators we refer the reader to[START_REF] Heinz | Convex Analysis and Monotone Operator Theory in Hilbert Spaces[END_REF]

The choice of such a step-size assumes a prior knowledge of the horizon of the process and is also referred to as "doubling" trick[START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF]

Proposition 2.10 hinges on the methodology of. the Dual-Extapolation method of[START_REF] Nesterov | Primal-dual subgradient methods for convex problems[END_REF]. However, we are not aware of a specific paper which provides a proof for it. We revisit this from a more general point of view in Chapter 7.

Of course(2.44) includes the classes of (Bd) and (LC) as extreme cases for q = 0 and q = 1 respectively.

Local for this context refers to the fact that the said family of norms dependents on the point upon which it is evaluated.

In the above, we are ignoring background emission noise which, in many applications, can be eliminated by pre-processing the detected image[START_REF] Bertero | Image deblurring with Poisson data: from cells to galaxies[END_REF].

For posterity, note here that X is convex but it is not necessarily closed.

It is trivial to see that, in this market problem, all users would saturate their budget constraints at equilibrium, i.e., ∑ a∈A x ia = 1 for all i ∈ N .

The latter holds because h is strongly convex relative to • x , and • x has been tacitly assumed bounded from below by a multiple µ • of • .

In[START_REF] Hanzely | Accelerated Bregman proximal gradient methods for relatively smooth convex optimization[END_REF] proposed a tentative path towards faster convergence in certain beyond Lipschitz problems. However, in doing so they require some strict regularity conditions.

In more detail,[START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF],[START_REF] Li | On the convergence of stochastic gradient descent with adaptive stepsizes[END_REF] and[START_REF] Kavis | UnixGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization[END_REF] establish the summability of a suitable residual sequence to sharpen the O(1/ √ T) rate in their respective contexts, but this does not translate to a step-size stabilization result. Under (RC)/(RS), controlling the method's step-size is of vital importance because the gradients that enter the algorithm may be unbounded even over a bounded domain; this crucial difficulty does not arise in any of the previous works on adaptive methods for ordinary Lipschitz problems.

It is trivial to see that, in this market problem, all users would saturate their budget constraints at equilibrium, i.e., ∑ a∈A x ia = 1 for all i ∈ N .

In a Euclidean setting, similar ideas can be found in, e.g.,[START_REF] Radu Ioan Boţ | The forward-backwardforward method from continuous and discrete perspective for pseudo-monotone variational inequalities in Hilbert spaces[END_REF][START_REF] Malitsky | Golden ratio algorithms for variational inequalities[END_REF]. We ignore the origins of this technique.

which in turn by (7.96) yields: +∞ ∑ t=1 D(X t+1/2 , X t ) < +∞ (7.97) and hence D(X t+1/2 , X t ) → 0. Moreover, by considering Definition 3.3:

Now, by recalling µ • ≤ • x , we get:

and the result follows.

• γ ∞ = 0: By the prox-step, we get:

∇h(X t ) -∇h(X t+1/2 ), X t -X t+1/2 ≤ γ t A(X t ), X t -X t+1/2 ≤ γ t A(X t ) X t , * X t -X t+1/2 X t (7.100)

On the other hand, we have:

∇h(X t ) -∇h(X t+1/2 ), X t -X t+1/2 = D(X t , X t+1/2 ) + D(X t+1/2 , X t ) (7.101) Thus, we get by Definition 3.3:

where the last inequality is obtained due to (MB). This in turn yields:

So, a fortiori we have:

Moreover, by Definition 3.3:

Now, by recalling µ • ≤ • x , we get:

and the result follows since we assumed that γ t → 0.

2. Under (MS) condition: Following similar reasoning as above, we have:

which by taking limits on both sides and by applying Lemma 7.6 we get that:

Therefore, D(X t+1/2 , X t ) → 0, whereas by applying Definition 3.3 we obtain:

Now, by recalling µ • ≤ • x , we get:

and the result follows.

On the other hand, for the second claim, we have by the prox-step:

Therefore, by following the same reasoning with the first claim, we get:

and hence since D(•, •) ≥ 0, we have:

and so the result follows Remark 7.1. We shall point out that (1) in Lemma 7.8 establishes the convergence with respect to the global ambient reference norm of R n .

Extracting a Convergent Sub-sequence Proposition 7.9 (Antonakopoulos et al. [START_REF] Kimon Antonakopoulos | Adaptive extra-gradient methods for min-max optimization and games[END_REF]). Suppose that A satisfies (MB) (respectively (MS)). Then, the iterates X t , X t+1/2 of (MP) run with the adaptive step-size (Adapt) possess convergent subsequences towards the equilibrium set X * .

Proof. By Lemma 7.8, it suffices to show that X t+1/2 possesses such a subsequence. Assume to the contrary that it does not. That implies that:

Moreover, by setting Y 1 = 0 we get:

1

So, finally we get:

and the result follows.

Optimal rate interpolation analysis

In order to proceed to the particular analysis of Theorem 7.11 we shall some additional results. The first concerns the almost sure boundedness of V t+1/2 -V t 2 X t+1/2 , * which is vital for establishing the stochastic rates under (MB). Formally, we. have the following result.

Almost Sure Boundedness of the Residual Lemma 7.14 (Antonakopoulos and Mertikopoulos [START_REF] Antonakopoulos | Universal methods for variational inequalities with divergent operators[END_REF]). Assume that X t , X t+1/2 are the iterates of (DualX) run with a non-increasing step-size γ t . Moreover, assume that the oracle satisfies the mean square boundedness condition:

Then, the sequence V t+1/2 -V t 2 X t+1/2 , * is bounded almost surely. In particular, the following inequality holds with probability 1:

Proof. We have:

Now, let us bound each term of the above individually. For the term V t+1/2

we have:

Moreover, we have:

with the last inequality being obtained by applying Jensen's inequality. Hence, since the oracle satisfies the mean square boundedness condition we get:

Therefore, summarizing: V t+1/2 2 X t+1/2 , * is upper bounded almost surely by:

Now, we turn our attention to V t 2 X t+1/2 , * , recalling the regularity of X we have for some β ≥ 0 such that:

or equivalently:

Now, by the definition of (DualX) we have:

with the last inequality being obtained by applying Definition 3.3. So, by applying Cauchy-Shwartz inequality on the (LHS) we get:

and so,

So, combining (7.170) and (7.171) we get:

In this appendix, we provide some necessary inequalities on numerical sequences that we require for the convergence rate analysis of the previous sections. Most of the lemmas presented below already exist in the literature, and go as far back as Auer et al. [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] and McMahan and Streeter [START_REF] Mcmahan | Adaptive bound optimization for online convex optimization[END_REF]; when appropriate, we note next to each lemma the references with the statement closest to the precise version we are using in our analysis. These lemmas can also be proved by the general methodology outlined in Gaillard et al. [START_REF] Gaillard | A second-order bound with excess losses[END_REF]Lem. 14], so we only provide a proof for two ancillary results that would otherwise require some more menial bookkeeping.

Lemma A.1 [START_REF] Mcmahan | Adaptive bound optimization for online convex optimization[END_REF][START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF]. For all non-negative numbers α 1 , . . . α t , the following inequality holds:

Lemma A.2 [START_REF] Kfir | Online adaptive methods, universality and acceleration[END_REF]. For all non-negative numbers α 1 , . . . α t , the following inequality holds: The following set of inequalities are due to [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF]. For completeness, we provide a sketch of their proof. Lemma A.4 [START_REF] Bach | A universal algorithm for variational inequalities adaptive to smoothness and noise[END_REF]. For all non-negative numbers: α 1 , . . . α t ∈ [0, α], α 0 ≥ 0, the following inequality holds: