Je souhaite tout d'abord remercier Emilie Kaufmann et Panayotis Mertikopoulos d'avoir accepté d'être rapporteurs de ma thèse ainsi que Vianney Perchet, Gilles Stoltz et Marta Soare d'avoir accepté de faire partie du jury de ma soutenance de thèse. Vos retours et les discussions que nous avons pu avoir durant ma soutenance de thèse ont été très enrichissants.

Durant ces quatre dernières années, j'ai eu la chance de pouvoir être encadré par Yann Chevaleyre, Rida Laraki, Igor Colin et Albert Thomas qui ont su me guider tout au long de cette thèse et m'inculquer le métier de chercheur. Pour cela, je souhaiterai leur témoigner ma profonde gratitude et les remercier de toute l'aide qu'ils ont pu m'apporter, tant sur le plan professionnel que personnel.

Cette thèse effectuée d'une part au sein de l'équipe MILES du laboratoire LAMSADE de l'Université Paris Dauphine et d'autre part au sein de l'équipe Noah's Ark Paris de Huawei a représenté pour moi un cadre de travail propice à l'épanouissement scientifique et personnel, et cela a été possible grâce aux doctorants et membres permanents de ces deux équipes. C'est pourquoi je souhaite chaleureusement les remercier et leur témoigner que tous les moments (scientifiques comme moins scientifiques) partagés avec eux resteront un très bon souvenir. Enfin, j'ai eu la chance d'être entouré par ma famille et mes amis qui ont su me supporter et m'épauler aux moments où j'en avais le plus besoin. Pour cela, je les remercie infiniment. Finalement, une partie de ce doctorat vous revient aussi.

Résumé

Nous introduisons un nouveau modèle appelé Bandits Bilinéaires Graphiques où un apprenant (ou une entité centrale) alloue des bras aux noeuds d'un graphe et observe pour chaque arête une récompense bilinéaire bruitée représentant l'interaction entre les deux noeuds associés. Dans cette thèse, nous étudions le problème d'identification du meilleur bras et la maximisation des récompenses cumulées. Pour le premier, un apprenant veut trouver l'allocation du graphe maximisant la somme des récompenses bilinéaires obtenues à travers le graphe. Pour le second problème, au cours du processus d'apprentissage, l'apprenant doit faire un compromis entre l'exploration des bras pour acquérir une connaissance précise de l'environnement et l'exploitation des bras qui semblent être les meilleurs pour obtenir la récompense la plus élevée. Quel que soit l'objectif de l'apprenant, le modèle de bandits bilinéaires graphiques révèle un problème combinatoire sousjacent qui est NP-Dur et qui empêche l'utilisation de tout algorithme existant pour l'identification du meilleur bras (BAI) ou pour la maximisation des récompenses cumulées. Pour cette raison, nous proposons tout d'abord un algorithme d'α-approximation pour le problème NP-Dur sousjacent, puis nous nous attaquons aux deux problèmes mentionnés ci-dessus. En exploitant efficacement la géométrie du problème du bandit, nous proposons une stratégie d'échantillonnage aléatoire pour le problème BAI avec des garanties théoriques. En particulier, nous caractérisons l'influence de la structure du graphe (par exemple, étoile, complet ou cercle) sur le taux de convergence et proposons des expériences empiriques qui confirment cette dépendance. Pour le problème de la maximisation des récompenses cumulées, nous présentons le premier algorithme basé sur le regret pour les bandits bilinéaires graphiques utilisant le principe d'optimisme face à l'incertitude. L'analyse théorique de la méthode présentée borne l'α-regret par Õ( √ T ) et souligne l'impact de la structure du graphe sur le taux de convergence. Enfin, nous démontrons par diverses expériences la validité de nos approches. ix Set of d-dimensional real-valued vectors

List of Figures and Tables

R d×d ′ Set of d × d ′ -dimensional real-valued matrices [v] i the i-th element of a vector v ∈ R d [A] ij
The element at the i-th row and j-th

column of A ∈ R d×d ′ S + d
The cone of all positive semi-definite matrices in R d×d ⟨u, v⟩

The scalar product between u and v ∈ R d : ⟨u, v⟩ ≜ u ⊤ v ∥v∥ p The ℓ p norm of a vector v ∈ R d :

∥v∥ p = d i=1 |[v] i | p 1/p ∥v∥ A
The Mahanalobis norm of v ∈ R d with a matrix A ∈ S + d :

∥v∥ A ≜ √ v ⊤ Av ∥A∥
The spectral norm of a matrix A ∈ R d×d : ∥A∥ ≜ sup x:∥x∥ 2 =1 ∥Ax∥ 2 ∥A∥ F

The Frobenius norm of a matrix A:

∥A∥ F = d i=1 d j=1 |[A] ij | 2 |X | Cardinality of the finite set X S X |X |-dimensional simplex: S X ≜ {γ ∈ R |X | | x∈X γ x = 1} I d d × d identity matrix N (•, •)
The Gaussian distribution P(•)

The probability of a random event

E[•]
The expectation of a random event 1 [•] The indicator function:

1(E) = 1 if E is true, 0 otherwise O(•)
The Landau notation: f (T ) = O(g(T )) ⇔ lim sup T →∞ f (T )

g(T ) < ∞ o(•)
The Landau notation:

f (T ) = o(g(T )) ⇔ lim T →∞ f (T ) g(T ) = 0 ∧
The logical AND ∨

The logical OR ⊗

The Kronecker product vec (A) The vector in R

Context & motivations

This thesis aims at solving centralized multi-agent problems that involve pairwise interactions between agents. Configuring antennas in a wireless cellular network [START_REF] Siomina | Automated optimization of service coverage and base station antenna configuration in UMTS networks[END_REF] is an example of those problems: the choice of a parameter for an antenna has an impact on both its own signal quality and that of each of its neighboring antennas due to signal interference. Likewise, in a wind farm, the adjustment of a turbine blade not only impacts its own energy collection efficiency but also that of its neighbors' due to wind turbulence [START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF][START_REF] Van Dijk | Yaw-misalignment and its impact on wind turbine loads and wind farm power output[END_REF]. By considering each antenna or turbine blade as an agent, these problems can be modeled as a multi-agent multi-armed bandit problem (MA-MAB) [START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF] with the knowledge of a coordination graph [START_REF] Guestrin | Coordinated Reinforcement Learning[END_REF] where each node represents an agent and each edge represents an interaction between two agents. A multi-armed bandit problem (MAB) is a sequential decision problem where a learner must take an action (also called arm) at each iteration and gets a (possibly perturbed) associated reward that informs about the quality of the chosen action. Naturally, the learner does not know the distribution of the reward for each possible action. The learner may have very different goals, such as maximizing the rewards accumulated during the process, or in a minimum number of tries and regardless of the accumulated rewards, inferring which action is the best to choose i.e., the most rewarding. Hence, a multi-agent multi-armed bandit is the setting where several agents face a multi-armed bandit problem. In the bandit literature, one can distinguish unstructured and structured bandits. While the unstructured bandit considers that playing an action and getting the associated reward does not allow to deduce anything about the distribution of rewards of other actions, the structured one includes the bandit settings where the rewards of the different actions share a common parameter [START_REF] Lattimore | Bandit Algorithms[END_REF]. For instance, a popular structured bandit setting is the linear bandit [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF] where the reward associated with any action is linearly dependent on an unknown parameter vector θ. Hence at a given time, choosing an action and receiving its associated reward gives information about θ and by definition also about the rewards of all other actions. Here, we are interested in such structured environments and on that matter we present a novel multi-agent structured bandit called

Graphical Bilinear Bandits. The specificity of this environment lies in the interdependence of the rewards obtained by the neighboring agents in the graph and in the assumption that these rewards are bilinear, which appears to us as the natural extension of linear rewards when agents are pairwise dependent. Indeed, while MA-MAB problems have been studied in the setting of unstructured bandits with independent and dependent agents (see e.g., [START_REF] Agarwal | Multi-agent multi-armed bandits with limited communication[END_REF][START_REF] Amin | Graphical Models for Bandit Problems[END_REF][START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF][START_REF] Besson | Multi-player bandits revisited[END_REF][START_REF] Bistritz | Distributed multi-player bandits-a game of thrones approach[END_REF][START_REF] Heliou | Learning with bandit feedback in potential games[END_REF][START_REF] Landgren | Distributed cooperative decision making in multi-agent multi-armed bandits[END_REF][START_REF] Sankararaman | Social learning in multi agent multi armed bandits[END_REF][START_REF] Shahrampour | Multi-armed bandits in multi-agent networks[END_REF]101]), only the setting of structured bandits with independent agents has been explored (see e.g., [START_REF] Amani | Decentralized multi-agent linear bandits with safety constraints[END_REF][START_REF] Cesa-Bianchi | A gang of bandits[END_REF][START_REF] Chan | Parallelizing contextual linear bandits[END_REF]). Through this thesis and the papers it refers to, we want to lay a first stone to the building.

Problem setting 1.2.1 Stochastic Graphical Bilinear Bandits

Let G = (V, E) be the directed graph defined by V the finite set of nodes representing the agents and E the set of edges representing the agent interactions. We assume that if (i, j) ∈ E then (j, i) ∈ E. The graph could be considered as undirected but we assume that the interactions between two neighbors are not necessarily symmetrical with respect to the obtained rewards, so we choose to keep the directed graph to emphasize this potential asymmetry. For all agent i ∈ V , we denote N i the set of its neighboring agents. Let n = |V | denote the number of nodes, m = |E| the number of edges and let X ⊂ R d be a finite arm set where K = |X | denote the number of arms. The graphical bilinear bandit with a graph G and an arm set X consists in the following sequential decision problem:

Stochastic Graphical Bilinear Bandits

For each round t > 0,

1. Each agent i ∈ V chooses an arm x (i) t in X 2. Then, each agent i ∈ V receives a noisy bilinear reward for each of its neighbors j ∈ N i y

(i,j) t = x (i)⊤ t M ⋆ x (j) t + η (i,j) t , (1.1)
where M ⋆ ∈ R d×d is an unknown matrix, and η (i,j) t a zero-mean σ-sub-Gaussian random variable.

The reward y (i,j) t reflects the quality of the interaction between the neighboring nodes i and j when pulling respectively the arm x (i) t and x (j) t at time t. The bilinear setting appears as a natural extension of the commonly studied linear setting to model the interaction between two agents.

Note that this setting can be considered either in a decentralized setting where agents take actions without consultation with others agents or in the centralized setting where a central entity chooses the arms of all the agents as well as aggregates the obtained rewards and designs a global strategy for the agents in the graph.

In this thesis, we only consider the centralized setting where a central entity manages all the agents, chooses at each time t the joint arm (x (1) t , . . . , x (n) t ) and then receives the associated rewards y (i,j) t for all (i, j) ∈ E. We illustrate the sequential decision problem at a given round t in Figure 1.1. The learner chooses (x

(1) t , x (2) t , x (3)
t )

The learner receives y (i,j) t , ∀(i, j) ∈ E Figure 1.1: Illustration of the learner's decision process at a given round t for a simple graph of three nodes

Objectives

As briefly mentioned earlier, there are two different main goals that a learner (here the central entity) may want to achieve in a bandit problem.

Identifying the best joint arm. The first objective that we want to deal with in this thesis is where the learner is interested in finding within a minimum of rounds the best joint arm (x (1) ⋆ , . . . , x (n) ⋆ ) that maximizes the expected global reward over the graph:

(x (1)
⋆ , . . . , x

(n) ⋆ ) =
arg max (x (1) ,...,x (n) )∈X n (i,j)∈E

x (i)⊤ M ⋆ x (j) . This objective implies that the central entity do not mind choosing a suboptimal joint arm (x (1) t , . . . , x (n) t ) at each time t as long as it gives enough information on the unknown parameter M ⋆ in order to construct an accurate estimate M. This objective is known as pure exploration or best arm identification [START_REF] Audibert | Best arm identification in multi-armed bandits[END_REF][START_REF] Bubeck | Pure exploration in multi-armed bandits problems[END_REF].

Maximizing the cumulative rewards. The second objective is the most commonly considered in the bandit literature where the learner wishes to maximise the sum of the (expected) rewards obtained over the rounds. In our setting, the central entity wants to maximize the cumulative expected global rewards given by T t=1 (i,j)∈E

x (i)⊤ t M ⋆ x (j) t .
While the first goal allows the learner to be in a pure exploration setting, regardless of the rewards obtained throughout the process, the goal of maximizing the cumulative rewards requires a trade-off between exploring the different possible arms to have an accurate estimate M of M ⋆ and exploiting the arms that seems to be the most optimal given M in order to obtain the maximum cumulative rewards.

In both objectives (i.e., the best arm identification or the maximization of the cumulative rewards) and given an estimate M, the learner will have to solve at some point the following optimization problem max (x (1) ,...,x (n) )∈X n (i,j)∈E

x (i)⊤ Mx (j) .

(1.2)

Indeed, for the best arm identification, this optimization problem must be solved at the end when the learner wants to return the best joint arm given the estimate M constructed during the learning procedure. For the maximization of the cumulative rewards, this optimization problem may need to be solved during the learning procedure when the learner wants to exploit and return the best estimated joint arm given its current knowledge of the environment which is the constructed estimate M.

Solving this optimization problem is not trivial, so for both objectives we consider the common underlying objective of solving this problem.

Outline of the thesis and contributions

In Chapter 2, we introduce and formalize the stochastic multi-armed bandit problem and more particularly the stochastic linear bandit problem with the algorithms and guarantees that exist for the best arm identification problem and the maximization of the cumulative rewards. Indeed, many tools developed in the corresponding literature will be used to solve the problems related to the graphical bilinear bandits setting. Then we put our graphical bilinear bandits model in perspective with some multi-agent bandit models that use structured bandits and bandits in graphs.

In Chapter 3, we tackle the underlying objective of solving the optimization problem given in (1.2). For this part we consider that the learner already has the best estimate M = M ⋆ . We show that the problem is NP-Hard and we give two α-approximation algorithms with α ≥ 1/2.

In Chapter 4, we formalize the best arm identification problem relative to the graphical bilinear bandits. By efficiently exploiting the geometry of this bandit problem, we propose an allocation strategy based on randomized sampling with theoretical guarantees. In particular, we characterize the influence of the graph structure (e.g. star, complete or circle) on the convergence rate and propose empirical experiments that confirm these dependencies.

In Chapter 5, we present a regret-based algorithm (i.e., an algorithm that aims to maximize the cumulative rewards) for graphical bilinear bandits using the principle of optimism in the face of uncertainty. Theoretical analysis of this new method yields an upper bound of Õ( √ T ) on the α-regret (a useful measure that we introduce in chapter 2) and evidences the impact of the graph structure on the rate of convergence. We show through various experiments the validity of our approach.

Finally, in Chapter 6 we present the conclusion of this thesis and discuss the different research perspectives that the graphical bilinear bandits model offers. In this chapter, we first present the basics of the stochastic multi-armed bandit and the stochastic linear bandit. The different notions and algorithms that appear in the following sections do not cover the whole field and do not necessarily include the most recent or the most optimal ones since we only want to (i) introduce the reader to this domain and give the tools that allow a good understanding of the following chapters of this thesis and (ii) explain why the existing algorithms cannot be straightforwardly applied to the graphical bilinear bandit setting. For a more in-depth view of the field, we refer the reader to the book [START_REF] Lattimore | Bandit Algorithms[END_REF] which provides a detailed and comprehensive overview of bandit problems.

Besides, in the last section of this chapter, we present some specific works that model structured multi-agent bandits. Since the studied models are very different from ours, a direct comparison of the methods would not be appropriate, however they bring perspective to our work. Furthermore some of the methods and tools used in the cited papers may still be useful for graphical bilinear bandits problems.

An introduction to the stochastic multi-armed bandit problem 2.1.1 Motivations and formalization

The bandit problem was first introduced in [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] to model sequential clinical trials where a learner chooses for each patient a drug and then observes the associated effects. Then, the stochastic 7 multi-armed bandit problem was formalized in [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF] to present the general sequential decision problem: it considers a learner who has access to several actions (most often called arms) and during a given number of rounds, the learner has to choose an arm at each round that will reveal an associated perturbed reward. The expected reward of each arm is unknown to the learner. One of the most popular cases illustrating this situation is that of a gambler who is in a casino and has access to slot machines with different expected payoffs. Given his or her budget, the learner has a finite number of tries and must choose which slot machine to play at each time and then obtain the associated payout. The common goal of the player is to maximize the cumulative payoffs, but other goals such as identifying the best slot machine in a minimum number of tries can be interesting.

Consider a finite set of arms X with K = |X | the number arms and a collection of distributions ν = {P x : x ∈ X }. Given a time horizon T > 0, the learner faces the following sequential decision problem: Stochastic Multi-Armed Bandit For each round t = 1, . . . , T , 1. the learner chooses an arm x t in a finite arm set X 2. the environment samples a reward y t ∈ R from P xt and reveals y t to the learner.

In the next section, we present an algorithm that solves the problem of maximizing the cumulative rewards obtained during the learning procedure. The algorithms that solves the problem of identifying the best arm for multi-armed bandits use very different techniques than those used for linear bandits and by extension those we use for graphical bilinear bandits. For this reason and to maintain a clearer narrative, we do not present algorithms that tackle the best-arm identification problem for multi-armed bandits.

Maximizing the cumulative rewards

As we briefly mentioned in the previous section, a natural goal the learner might have is to maximize the cumulative rewards obtained during the learning process. We recall that the learner does not know the expected reward of each arm. Hence it has to explore and try out the different arms to have an estimate of their associated rewards, but also exploit the arm that appear to have the highest reward. These two subgoals are complementary: on the one hand, by exploring, the learner gets to correctly estimate the different expected rewards associated with all arms; however, this implies pulling suboptimal arms, which have bad impacts on the cumulative rewards. On the other hand, by exploiting, the learner gets to pull arms that appears to give the greatest reward; however, since other arms are disregarded, this could lead to less accurate estimation of the different expected rewards and thus to over exploiting arms that are actually suboptimal. Hence the learner has to do a tradeoff between exploration and exploitation in order to maximise the cumulative rewards.

Wanting to maximize cumulative noisy rewards is the same as wanting to pull at each round the arm that gives the best expected reward µ ⋆ where µ ⋆ = max x∈X ∞ -∞ udP x (u). Indeed, the 2.1 An introduction to the stochastic multi-armed bandit problem learner does not control the randomness coming from the environment, but pulling the arms with the highest expected reward would give him, in expectation, the highest cumulative rewards. By defining µ x = ∞ -∞ udP x (u) the expected reward when pulling arm x, we have more formally that maximizing the cumulative rewards is equivalent to minimizing the pseudo-regret

R(T ) = T µ ⋆ - T i=1 µ xt ,
The notion of pseudo-regret simply describes the difference between what the learner would have done with full information, i.e., pull the best arm during the T rounds, and what is actually done without initial information on the expected rewards of the arms. In [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], the authors show that asymptotically a regret of order log(T ) is unavoidable. The objective for the learner is to have a pseudo-regret R(T ) such that lim

T →∞ R(T ) T = 0 .
This ensures that the learner chooses the optimal arm almost all the time when T tends to infinity. One of the most popular algorithm for stochastic multi-armed bandit that does the explorationexploitation tradeoff is the Upper Confidence Bound (UCB) algorithm [START_REF]Sample mean based index policies by o (log n) regret for the multi-armed bandit problem[END_REF][START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] using the principle of Optimism in the Face of Uncertainty (OFU). The idea of the algorithm is to select at each time t the arm that seems to be the most optimistically optimal. The notion of optimism takes into account the value of the estimated reward of the arms but also the number of samples used for the estimations, in other words the precision of the estimations. Indeed, at each time t and for each arm x ∈ X , the learner has an estimate μx of the reward of x. Hence, instead of only pulling the arm that has the highest estimated reward max x∈X μx (i.e., only exploiting), it uses the upper confidence bound (UCB) on the estimate μx and chooses the arm with the highest UCB. The less an arm has been pulled the higher the UCB. Intuitively it means that at each round, the learner either exploits with high confidence or explores other arms that have been less pulled and that might give (optimistically) a better reward. A lot of versions exist for the UCB algorithm, we recall here the original method presented in [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] in Algorithm 1.

Algorithm 1: UCB1

Input : arm set X = {1, . . . , K} Pull each arm x ∈ X once and set μx the obtained reward and n x = 1;

for t = K + 1 to T do
The learner pulls the arm x t = arg max x∈X μx + We state the guarantee of the algorithm on the pseudo-regret in the following proposition that we borrow from [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. Proposition 2.1 (Theorem 1 in [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]). For any K ≥ 1 and arm set X = {1, . . . , K}, if the policy UCB1 is run on X with the associated reward distribution ν = (P x : x ∈ X ) with support in [0, 1], then, for any number of plays T , its pseudo-regret is such that

R(T ) ≤     8 x∈X µx<µ ⋆ ln T µ ⋆ -µ x     + 1 + π 2 3 x∈X µ ⋆ -µ x , (2.1)
where for all x ∈ X ,

µ x = ∞ -∞ udP x (u)
This proposition tells us that, given the sublinear bound on the regret, the learner is constantly improving his choice, otherwise the regret would have been of order T with the learner remaining stuck and drawing a suboptimal arm. The reader can refer to [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] for instance for an improved and asymptotically optimal algorithm.

As stated in the introduction, in bandit theory, one can distinguish unstructured bandits from structured bandits. A bandit is called unstructured when it is not possible for the learner to learn information about one arm by drawing another. In other words, we can define an unstructured bandit as one where ν is a product of distributions (that may be of different classes), so that drawing an arm gives a reward from the associated distribution without helping the learner understand the other distributions. In contrast, structured bandits such as linear bandits allow the learner to pull an arm and infer information about the rewards of other arms.

The stochastic linear bandit problem

Formalization

We present the sequential decision problem for the linear bandit in the following:

Stochastic Linear Bandit

For each round t = 1, . . . , T , 1. the learner chooses an arm x t in a finite arm set X ⊂ R d with d > 1 2. the learner obtains from the environment the associated reward

y t = ⟨x t , θ ⋆ ⟩ + η t (2.2)
where θ ⋆ ∈ R d is an unknown parameter vector, and η t is a random variable sampled from a certain distribution When the learner draws an arm x ∈ X and receives a noisy linear reward ⟨x, θ ⋆ ⟩ + η t , it gives information about θ ⋆ and by extension about the other expected reward ⟨x ′ , θ ⋆ ⟩ for any x ′ ∈ X . This setting is particularly interesting because it might be enough for the learner to draw d arms that span R d to start having a reasonable estimate of θ ⋆ . Therefore, when the number of arms is large, the learner does not necessarily have to draw all the arms to get a good estimate of θ ⋆ and thus the estimated rewards for all the arms. Notice that this bandit can be formulated for instance with ν = {N (⟨x, θ ⋆ ⟩, σ) : x ∈ X } if we consider that the noise is a gaussian random variable. For the rest of this chapter, we consider that the noise terms are σ-sub-Gaussian random variables.

Experimental designs serving the pure exploration setting

When the objective of the learner is to identify the best arm x ⋆ = arg max x∈X ⟨x, θ ⋆ ⟩ within a minimum number of rounds, it is equivalent to look for the arm x ∈ X such that for all x ′ ∈ X , ⟨x -x ′ , θ ⋆ ⟩ ≥ 0 .

(2.3) However, one does not have access to θ ⋆ , so we have to use its empirical estimate. For t > 0, we consider a sequence of arms x t = (x 1 , . . . , x t ) ∈ X t and the corresponding noisy rewards (y 1 , . . . , y t ). We assume that the noise terms in the rewards are i.i.d., following a σ-sub-Gaussian distribution. Let θt = A -1 t b t ∈ R d be the solution of the ordinary least squares problem with

A t = t s=1 x s x ⊤ s ∈ R d×d and b t = k s=1 x s y s ∈ R d .
We suppose that A t is nonsingular for all t > 0. We first recall the following property. [START_REF] Soare | Best-arm identification in linear bandits[END_REF]). Let c = 2σ √ 2. For every fixed sequence x t , with probability 1 -δ, for all t > 0 and for all x ∈ X , we have

Proposition 2.2 (Proposition 1 in

x ⊤ θ ⋆ -x ⊤ θt ≤ c∥x∥ A -1 t log 6t 2 K δπ .
As it is done in [START_REF] Soare | Best-arm identification in linear bandits[END_REF], let us consider a confidence set Ŝ(x t ) centered at θt ∈ Ŝ(x t ) and such that P θ ⋆ / ∈ Ŝ(x t ) ≤ δ, for some δ > 0. Since θ ⋆ belongs to Ŝ(x t ) with probability at least 1 -δ, one can stop pulling arms when an arm has been found, such that the condition (2.3) is verified for any θ ∈ Ŝ(x t ). More formally, the best arm identification task will be considered successful when an arm x ∈ X verifies the following condition for any x ′ ∈ X and any θ ∈ Ŝ(x t ):

⟨x -x ′ , θt -θ⟩ ≤ ∆t (x, x ′ ) ,
where ∆t (x, x ′ ) = (x -x ′ ) ⊤ θt is the empirical gap between x and x ′ .

Corollary 2.1. For all t > 0, let Ŝ(x t ) be such that

Ŝ(x t ) = θ ∈ R d : ∀x ∈ X , ∀x ′ ∈ X , ⟨x -x ′ , θt -θ⟩ ≤ c∥x -x ′ ∥ A -1 t log 6t 2 K 2 δπ . With probability 1 -δ, θ ⋆ is in Ŝ(x t ).
Proof. Using the upper bound in Proposition 2.2 and replacing x by (x -x ′ ), we get the result.

Then, the stopping condition can be reformulated as follows:

∃x ∈ X , ∀x ′ ∈ X , c∥x -x ′ ∥ A -1 t log 6t 2 K 2 δπ ≤ ∆t x, x ′ . (2.4)
As mentioned in [START_REF] Soare | Best-arm identification in linear bandits[END_REF], by noticing that max

(x,x ′ )∈X 2 ∥x -x ′ ∥ A -1 t ≤ 2 max x∈X ∥x∥ A -1 t
, an admissible strategy is to pull arms minimizing max x∈X ∥x∥ A -1 t in order to satisfy the stopping condition as soon as possible. More formally, one wants to find the sequence of arms x ⋆ t = (x ⋆ 1 , . . . , x ⋆ t ) such that:

x ⋆ t ∈ arg min (x 1 ,...,xt) max x ′ ∈X x ′ ⊤ t i=1 x i x ⊤ i -1 x ′ . (G-opt-X )
This is known as G-allocation (see e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]) and is NP-hard to compute [START_REF] Çivril | On selecting a maximum volume sub-matrix of a matrix and related problems[END_REF]104]. One way to find an approximate solution is to rely on a convex relaxation of the optimization problem (G-opt-X ) and first compute a real-valued allocation γ ⋆ ∈ S X such that

γ ⋆ ∈ arg min γ∈S X max x ′ ∈X x ′ ⊤ x∈X γ x xx ⊤ -1 x ′ . (G-relaxed-X )
One could either use random sampling to draw arms as i.i.d.samples from the γ ⋆ distribution or rounding procedures to efficiently convert each component in γ ⋆ into an integer and thus constructed the optimal matrix A -1 t . Another way to visualize the effect of the constructed covariance matrix A -1 t is through the confidence ellipsoids which are of the form

E = {θ ∈ R d , ( θt -θ) ⊤ A -1
t ( θt -θ) ≤ τ } where τ depends on the confidence level. The ellipsoid associated with a confidence parameter δ ∈ (0, 1) represents the region that contains the true parameter θ ⋆ with probability 1 -δ (see Figure 2.1). For a fixed confidence parameter δ, one would want to minimize the region covered by the ellipsoid to ensure that the approximated parameter θt is as close as possible to the real parameter θ ⋆ . Since the ellipsoid depends on A -1 t , one way to do so, instead of estimating θt by choosing all the arms x ∈ X (also called experiments), is to select the one that are the most statistically efficient. This problem is known as experimental design [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF] and one criterion that has been studied is the G-optimal design that minimises max x∈X x ⊤ A -1 t x. The G-optimal design minimizes the worst possible predicted variance and one can see that this objective coincides exactly with the one formulated in (G-opt-X ).

To approximate the solution of (G-opt-X ), the authors of [START_REF] Soare | Best-arm identification in linear bandits[END_REF] give a greedy strategy that at each time t chooses the arm x t ∈ X such that

θt θ ⋆ [θ] 1 [θ] 2 { θ ∈ R 2 | ( ˆθt - θ ) ⊤ A -1 t ( ˆθt - θ ) ≤ τ } Figure 2.1:
The δ-confidence level ellipsoid

x t = arg min x∈X max x ′ ∈X x ′⊤ A t-1 + xx ⊤ -1 x ′ . (2.5)
The greedy strategy appears to be a fine way to approximate this problem because the number of pulls to satisfy the stopping condition is not known in advance, hence, converting directly the distribution γ to integers is not relevant. Moreover finding respectively the optimal sequence (x 1 , . . . , x t ) and (x 1 , . . . , x t+1 ) by rounding procedure and for a certain t with respect to the G-allocation strategy gives the same sequence modulo the extra x t+1 arm.

In Algorithm 2, we share the method.

Algorithm 2: Best-arm Identification in Linear Bandit : greedy G-Allocation strategy

Input : arm set X ⊂ R d , confidence δ > 0 Set t = 0; A 0 = I d ; b 0 = 0; while (2.4) is not true do t = t + 1; x t = arg min x∈X max x ′ ∈X x ′⊤ A t-1 + xx ⊤ -1 x ′ The learner observes y t = ⟨x t , θ ⋆ ⟩ + η t ; θt = A -1 t b t ; end return arg max x∈X ⟨x, θt ⟩;
The authors give a guarantee on the sample complexity of any algorithm that gives a β-approximation of the solution of (G-opt-X ): 13 Proposition 2.3 ([90], Theorem 1). If the G-allocation strategy is implemented with a β-approximate method and the stopping condition (2.4) is used, then with probability at least 1-δ, arg max x∈X ⟨x, θt ⟩ = x ⋆ and t ≤

16c 2 d(1 + β) log 6t 2 K 2 δπ ∆ 2 min , where ∆ min = min x∈X \{x⋆} ⟨x ⋆ -x, θ ⋆ ⟩ and c = 2σ √ 2
For Algorithm 2, the greedy algorithm gives a β that depends on t, we note it β t and is equal to

d+d 2 +2 2t
.

Optimism in the face of uncertainty for linear bandits (OFUL)

For the objective of maximizing the cumulative rewards with a budget of T rounds, the existing methods use the same idea of optimism in the face of uncertainty but adapted to the linear setting. Indeed, at each time t we saw in the previous section that the learner can build an estimate θt . Although the objectives are completely different, one can nevertheless consider again the confidence ellipsoids of the form

E = {θ ∈ R d , ( θt -θ) ⊤ A -1 t ( θt -θ) ≤ τ } as shown in Figure 2.1.
Given a confidence level δ, one can construct this ellipsoid and tell with probability 1 -δ that the true parameter vector θ ⋆ is in it. Hence, a strategy using the optimism in the face of uncertainty would be to select the arm that gives the best reward with respect to the best θ ∈ E. More formally at each time t the learner would select x t = arg max x∈X max θ∈E ⟨x, θ⟩. This method has been presented in [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF] and we recall their approach in the following.

Let us define

θt = A -1 t b t , (2.6)
where,

A t = λI d + t s=1 x s x ⊤ s ,
with λ > 0 a regularization parameter and

b t = t s=1
x s y s .

We also define the confidence set

C t (δ) = θ : ∥θ -θt ∥ A -1 t ≤ σ d log 1 + tL 2 /λ δ + √ λS ,
where we assume that for any x ∈ X , ∥x∥ 2 ≤ L and ∥θ ⋆ ∥ 2 ≤ S. We know from Theorem 2 in [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF] that with probability 1 -δ, θ ⋆ is in C t (δ) for all t ∈ {1, . . . , T }, and δ ∈ (0, 1].

Algorithm 3: OFUL Algorithm

Input : arm set X for t = 1 to T do

x t , θt-1 = arg max (x,θ)∈X ×C t-1 ⟨x, θ⟩;
Obtain the rewards y t ; Compute θt as in (2.6) end return θt

The pseudo-regret in the linear setting can be formulated as follows:

R(T ) = T t=1 ⟨x ⋆ , θ ⋆ ⟩ -⟨x t , θ ⋆ ⟩ = T t=1 ⟨x ⋆ -x t , θ ⋆ ⟩ , (2.7)
where we recall that x ⋆ = arg max x∈X ⟨x, θ ⋆ ⟩ Proposition 2.4 ([2], Theorem 3). Assume that for all t and all x ∈ X , ⟨x,

θ ⋆ ⟩ ∈ [-1, 1].
Then with probability 1 -δ, the pseudo-regret of the OFUL algorithm satisfies

R(T ) ≤ 4 T d log(λ + T L/d) √ λS + σ 2 log(1/δ) + d log(1 + T L/(λd)) ,
where for any x ∈ X , ∥x∥ 2 ≤ L and ∥θ ⋆ ∥ 2 ≤ S

In the next section, we present the bilinear bandit which appears as the natural extension of the linear bandit that models the interaction between two agents in the obtained rewards.

Bilinear bandits are linear bandits in a higher dimensional space

In [START_REF] Jun | Bilinear Bandits with Low-rank Structure[END_REF], the authors introduce the Bilinear Bandit model where at each round t a learner chooses an arm x t from a finite arm set X ⊂ R d that contains K arms and a second arm x ′ t from another finite arm set X ′ ⊂ R d ′ that contains K ′ arms, and obtains an associated reward that is bilinear with respect to the two chosen arms and an unknown parameter matrix M ⋆ ∈ R d×d ′ . More formally, this sequential decision problem is defined as follows:

Stochastic Bilinear Bandit

For each round t = 1, . . . , T , 1. the learner chooses an arm x t ∈ X and x ′ t ∈ X ′ 2. the learner obtains from the environment the associated reward

y t = x ⊤ t M ⋆ x ′ t + η t (2.8)
where M ⋆ ∈ R d×d ′ is an unknown parameter matrix, and η t is a random variable sampled from a certain distribution This kind of setting can model different real life applications, such as drug discovery applications [START_REF] Luo | A network integration approach for drug-target interaction prediction and computational drug repositioning from heterogeneous information[END_REF] or in the context of recommender systems as explained in [START_REF] Jun | Bilinear Bandits with Low-rank Structure[END_REF].

The bilinear reward can be written as a linear reward in a higher dimensional space:

y t = vec x t x ′⊤ t , vec (M ⋆ ) + η t , (2.9)
where for any matrix A ∈ R d×d , vec (A) denotes the vector in R d 2 which is the concatenation of all the columns of A.

Therefore, for both the best arm identification problem and the cumulative rewards maximization problem, solving this bilinear bandit problem is equivalent to solving a linear bandit problem of dimension d × d ′ with an arm set

Z = {vec (xx ′⊤ )|(x, x ′ ) ∈ X × X ′ } of K × K ′ arms.
To directly apply the existing linear bandit algorithms to the bilinear bandit model, the learner must coordinate the choices of the two chosen arms (x t , x ′ t ) at time t, which is equivalent to choosing an arm z t = vec (x t x ′⊤ t ) ∈ Z. Although approaching a bilinear reward from a linear angle is useful, it is less trivial to use linear bandit algorithms for more complex models such as graphical bilinear bandits. Indeed, our model exposed in section 1.2 can be viewed as a bilinear bandit problem between each pair of neighbors, where each agent chooses an arm from a set of arms X (in our framework X = X ′ ). Thus, if the learner coordinates the choice of two neighboring agents and chooses the pair (x, x ′ ) ∈ X and thus its associated arm in Z, it constrains the choices related to all the pairs of neighbors (j, k) ∈ E since it is already composed of the arm x ′ associated with agent j. Due to the interdependencies of the bilinear bandit problems, it is not possible to consider the graphical bilinear bandits as simple linear bandits in parallel and directly use the linear bandit algorithms present in the literature.

Another idea that one could have is to notice that since the unknown parameters matrix M ⋆ is common to all the edges (i, j) of the graph, the expected global reward at time t can also be written as the scalar product (i,j)∈E vec x

(i) t x (j)⊤ t
, vec (M ⋆ ) . Therefore, solving the best-arm identification problem or maximizing the cumulative rewards in the described graphical bilinear bandits reduces to solving the same problems in a global linear bandit. Although this trick allows the use of classical algorithms in linear bandits, the number of joint arms grows exponentially with the number of nodes, which makes these methods impractical. Of course, some tools are still very useful and relevant to solve the presented problems for the graphical bilinear bandits model and we use them in this thesis.

Multi-agent bandits and combinatorial bandits

Parallelizing contextual linear bandits

Centralized multi-agent bandit problems where the learner has to choose the actions of all the agents at each round implies to parallelize the learning process on the agents. In the context of linear rewards where all the agents share the same reward function (i.e., , the same parameter θ ⋆ ), the authors in [START_REF] Chan | Parallelizing contextual linear bandits[END_REF] give a detailed analysis of the problem of maximizing the cumulative rewards and show that a sublinear regret in T can be reached but with an additional cost specific to the parallelization.

More formally, they consider P agents, and at each round t, a context

X (i)
t ⊂ R is revealed to agent i and a central entity has to choose the arm

x (i) t ∈ X (i)
t for each agent i ∈ {1, . . . , P }. Then the learner receives for all agent i ∈ {1, . . . , P } the rewards y

(i) t = ⟨x (i) t , θ ⋆ ⟩ + η (i) t where η (i)
t is a σ-sub-gaussian random variable.

Here the pseudo-regret is formulated as follows:

R(T ) = T t=1 P p=1 ⟨x (i) ⋆,t , θ ⋆ ⟩ -⟨x (i) t , θ ⋆ ⟩ (2.10)
where

x (i) ⋆,t = arg max x∈X (i) t ⟨x, θ ⋆ ⟩.
On can construct the estimate θt as follows:

θt = A -1 t b t ,
where

A t = λI d + t s=1 P i=1 x (i) s x (i)⊤ s ,
with λ > 0 a regularization parameter and

b t = t s=1 P i=1 x (i) s y (i) s .
We define also the confidence set

C t (δ) = θ : ∥θ -θt ∥ A -1 t ≤ σ d log 1 + tP L 2 /λ δ + √ λS .
The authors show that applying the OFUL algorithm on each agent at time t where θt-1 and C t-1 (δ) aggregate the information of all the draws and rewards of the previous rounds, gives the following bound on the regret: R(T ) ≤ Õ(d

√ T P ) + O(dP log(T P )) (2.11)
where Õ hides logarithmic factors.

Their analysis shows that parallelizing agents that play the same contextual bandit and applying the OFUL algorithm for each of them with an aggregation of the information at the end of each round to construct θt and C t (δ) give an upper bound on the regret that is the sum of the nearoptimal regret of a single agent pulling T P arms plus a second term that represents the cost of parallelizing.

The similarities between the graphical bilinear bandits and their setting is that if m = |E| is the number of edges in our considered graph, the central entity in our model plays m bilinear bandits in parallel that can be seen as m linear bandits in parallel. However, the main difference is that they consider independent agents whereas we assume interactions between the agents. In particular, the arm associated with each (bi-)linear bandit problem cannot be chosen independently at each round since some of them share a mutual information. So we have to deal with both the parallel aspect and the dependent aspect at the same time.

Bandit problems in graphs

Graphs are often used to bring structure to a bandit problem. But one can distinguish two main representations.

Single agent. In [START_REF] Valko | Spectral Bandits for Smooth Graph Functions[END_REF] and [START_REF] Mannor | From bandits to experts: On the value of side-observations[END_REF] for instance, the arms are the nodes of a graph and pulling an arm gives information on the rewards of the neighboring arms. This kind of setting considers only one agent which is out of the scope of this thesis. The reader can also refer to [START_REF] Valko | Bandits on graphs and structures[END_REF] for an account on such problems.

Multi-agents.

As in [START_REF] Cesa-Bianchi | A gang of bandits[END_REF], each node is an instance of a linear bandit and the neighboring nodes are assumed to have similar unknown regression coefficients.

More precisely, they consider an undirected graph G = (V, E) where each node i ∈ G has an associated parameter θ

(i)
⋆ where the authors make the assumption that

(i,j)∈E ∥θ (i) ⋆ -θ (j) ⋆ ∥ 2 2 (2.12) is small compared to i∈V ∥θ (i) ⋆ ∥ 2 2 .
At each time t, the learner receives the index I t of one of the nodes in V as well as a set of contexts X (It) t

where it has to choose a context x t ∈ X (It) t and then receives an associated linear reward of the form ⟨x t , θ (i) ⋆ ⟩ + η t . Note that at each round, only the instance of the linear bandit of node I t is used, the learner does not choose an arm for each node of the graph. However, given the assumption that near-by nodes have similar associated parameter vectors θ (i) ⋆ , the learner may still get some information on the behaviors of other nodes.

Again, the main difference with our model is that the rewards of the nodes are independent, and although playing an arm at a given node may give information about its neighbor's rewards, the reward is not directly affected by the possible choices of its neighbors.

Link with unstructured multi-agents bandits

In the same way that a linear bandit with canonical arms can be seen as a classical multi-armed bandit 1 , hence loosing its structured property, the graphical bilinear bandit can also be seen as an unstructured graphical multi-agent bandit when the arm set X is the canonical basis X = (e 1 , . . . , e d ).

As mentioned in the introduction, some works have studied this unstructured setting. In particular, the authors in [START_REF] Amin | Graphical Models for Bandit Problems[END_REF] consider a graph G = (V, E) where at each round t and for each node i in V = {1, . . . , n}, a learner receives a context c (i) t , then has to choose an arm x (i) t and finally gets a global reward F (x t , c t ) where x t = (x

(1) t , . . . , x (n) t ) and c t = (c (1) t , . . . , c (n)
t ). They assume that the reward can be decomposed as a sum of subfunctions that depend only on subset of neighboring nodes. More formally, given a collection of subset P ⊂ 2 V , we have

F (x t , c t ) = P ∈P f P (x P , c P ) (2.13)
where x P = (x

(i) t ) i∈P , c P = (c (i)
t ) i∈P and f P are unknown functions for any P ∈ P. Note that a subset P ∈ P contains only nodes that are neighbors one to another. The main similarity with our framework is that the global function can be decomposed as the sum of local rewards that depend on the arms of neighboring nodes, which highlights the dependencies between the nodes. The reader can also refer to [START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF] for such works. However, all the algorithms that we presented for the graphical bilinear bandit leverage the structured aspect where pulling an arm informs about the rewards of the other arms through the unknown parameter matrix M ⋆ , which is not done in the unstructured setting.

Combinatorial bandits

A combinatorial bandit consists of a sequential decision problem where a learner has access to a set of K arms, at each round t selects a subset of arms under some constraints and then receives the associated reward. More formally, consider the arm set X = {0, 1} d where an arm x ∈ X is often called a super-arm. When the i-th coordinate of x ∈ X equals 1 it means that the learner selects the i-th arm. An example of a constraint that the learner can have is to select an arm x ∈ X such that ∥x∥ 1 ≤ m for m > 0, which means that the learner can at most select a subset of m arms per round. The type of rewards associated with a super arm varies from one setting to another. For instance, let consider ν = {P 1 , . . . P K } where P i is the distribution associated with the i-th arm. At each round, let us denote X i,t the random variable drawn at time t from P i , and denote X t the vector in R K containing in its coordinates all the X i,t for all i ∈ {1, . . . , K}. The reward at time t associated with the super arm x t ∈ X is given by

y t = ⟨x t , X i,t ⟩ .
In this particular example, we can notice that the reward is linear i.e., the reward y t is the sum of all the rewards associated with the selected arms by the learner, but other forms of reward can be considered and the learner may not even know how it is calculated.

Semi-bandit feedback.

In what is called the semi-bandit feedback, the learner receives each of the rewards X i,t if the i-th coordinate of x t is equal to 1.

The number of super-arms being exponential in K and when we do not have the knowledge on how the reward is computed, these kinds of problems can be hard to solve and the knowledge of an oracle is often assumed to return the optimal super-arm to play at a round t according to the estimates of the learner. More precisely, given the estimates μ = (μ 1 , . . . , μK ) of the expected reward associated with each arm, the learner asks the oracle which super-arm to play at time t. A relaxation in [START_REF] Chen | Combinatorial multi-armed bandit: General framework and applications[END_REF] considers an (α, β)-Approximation-oracle that returns with probability β, the α-optimal super-arm given the estimates. More formally, if opt μ is the value of the best superarm given μ, the (α, β)-Approximation-oracle returns a super arm that gives at least an expected reward equals to α • opt μ.

Given that (α, β)-Approximation-oracle, the authors used the αβ-pseudo-regret [START_REF] Kakade | Playing games with approximation algorithms[END_REF] which is defined as follows :

R(T ) = E T t=1 αβopt ⋆ -y t ,
where opt ⋆ is the optimal expected reward, and where the expectation is on the randomness of both the environment and the learner's policy.

The combinatorial bandit setting can be viewed as a similar model to ours in that we solve a bandit problem with an exponential number of arms and the underlying problem of returning an estimated best joint-arm (which can be viewed as a super arm under some particular constraint) at each round is NP-Hard and may require knowledge of an oracle. Although we do not assume knowledge of an oracle, we instead design an α-approximation algorithm to solve the underlying problem (see Chapter 3). And as is the case in the combinatorial bandit literature, the use of α-approximation algorithms makes the notion of α-regret a relevant measure of the performance for our algorithms. 2 Moreover, we present algorithms that takes into account the graph structure, which has not been considered in the combinatorial framework. In this chapter, we focus on the optimization problem of finding the joint arm x (1) , . . . , x (n) that maximizes (i,j)∈E x (i)⊤ M ⋆ x (j) while knowing the parameter matrix M ⋆ . This problem being non-trivial, it is natural to understand the guarantees on the solutions with the full information of the matrix M ⋆ (we relax this assumption in the next chapters where the matrix M ⋆ will be considered as unknown). Hence the objective of this chapter is the following:

Objective: Given the parameter matrix M ⋆ , design an algorithm that returns the allocation of arms (x

(1) ⋆ , . . . , x (n) ⋆ ) that maximises (i,j)∈E x (i)⊤ M ⋆ x (j)
We follow the notations we established in section 1.2.1.

An NP-Hard problem

Reduction to the max-cut problem

We address the problem of finding the best joint arm given M ⋆ and we denote it as follows:

(x (1) ⋆ , . . . , x (n) ⋆ ) = arg max (x (1) ,...,x (n) )∈X n (i,j)∈E x (i)⊤ M ⋆ x (j) . (3.1) Notice that if the couple (x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ is such that x ⋆ = x ′
⋆ then finding the best joint arm is trivial and the solution is to assign x ⋆ to all nodes. Conversely, 23 if x ⋆ ̸ = x ′ ⋆ , the problem may be harder: according to the graph G, the optimal joint arm could either be composed exclusively of the couple (x ⋆ , x ′ ⋆ ) or be composed of other arms in X . One might want to use dynamic programming as in [START_REF] Amin | Graphical Models for Bandit Problems[END_REF] to solve this optimization problem, however in this particular setting, it would lead to use a non-polynomial time algorithm. Indeed, the following theorem states that, even with the knowledge of the true parameter M ⋆ , identifying the best join-arm (x

(1) ⋆ , . . . , x (n)
⋆ ) is NP-hard with respect to the number of nodes n. Theorem 3.1. Consider a given matrix M ⋆ ∈ R d×d and a finite arm set X ⊂ R d . Unless P=NP, there is no polynomial time algorithm guaranteed to find the optimal solution of max (x (1) ,...,x (n) )∈X n (i,j)∈E

x (i)⊤ M ⋆ x (j) .
Proof. We prove the statement by reduction to the Max-Cut problem that is NP-Hard itself.

Let G = (V, E) be a graph with V = {1, . . . , n}. Let X = {e 0 , e 1 }, where e 0 = (1, 0) ⊤ and e 1 = (0, 1) ⊤

. Let M ⋆ = 0 1 1 0 . For any joint arm assignment x (1) . . . x (n) ∈ X n , let F ⊆ E be defined as F = i :

x (i) = e 1 . Note that (i,j)∈E x (i)⊤ M ⋆ x (j) = (i,j)∈E 1 x (i) ̸ = x (j) = 2 × (i,j)∈E 1[i ∈ F, j / ∈ F ],
where 1[•] is the indicator function.

The assignment x (1) , . . . , x (n) induces a cut (F, V \F ), and the value of the assignment is precisely twice the value of the cut. Thus, if there were a polynomial time algorithm solving our problem, this algorithm would also solve the Max-Cut problem.

Hence, given the true parameter matrix M ⋆ , the learner is not guaranteed to find in polynomial time the joint arm x (1) ⋆ , . . . , x (n) ⋆ maximizing the expected global reward. In the next sections, we give polynomial time approximation algorithms that have guarantees on the returned expected global reward with respect to the optimal one.

Approximation algorithm and guarantees

Given the true parameter M ⋆ , the objective is to design an algorithm that returns a joint arm x (1) , . . . , x (n) such that its associated expected global reward y = (i,j)∈E x (i)⊤ M ⋆ x (j) has the guarantee to be close to the optimal expected global reward y ⋆ = (i,j)∈E x

(i)⊤ ⋆ M ⋆ x (j) ⋆ .
In other words, we want to find a approximation parameter 0 < α ≤ 1 such that, y ≥ αy ⋆ . Although the optimization problem we seek to solve can be found in the literature on Markov Random Fields when dealing with a multi-labeled graph (see e.g., [START_REF] Ajanthan | Optimization of Markov random fields in computer vision[END_REF]), to the best of our knowledge, algorithms that give an approximation ratio on the optimal solution have not been explored.

Assumption 3.1 (Positive rewards).

A classical assumption in the linear bandit literature is that expected rewards are positive. For any (x, x ′ ) ∈ X 2 , since the bilinear reward x ⊤ M ⋆ x ′ can be formulated as a linear reward ⟨vec (xx ′⊤ ), vec (M ⋆ )⟩ and that in the rest of this thesis we will use tools from the linear bandit literature, we make the same assumption. We consider that for any (x, x ′ ) ∈ X 2 , the associated expected reward

x ⊤ M ⋆ x ′ is positive, x ⊤ M ⋆ x ′ ≥ 0.
The approach we present in this section is first to consider the problem locally, i.e., at the edge level. Indeed, let us consider two neighboring nodes i and j in V and only the expected rewards related to these nodes, which are x (i)⊤ M ⋆ x (j) and x (j)⊤ M ⋆ x (i) . By summing those two quantities, 1 we get j) which represents the expected reward between the two neighbors (i) and (j). A local strategy that the central entity should carry out is thus to allocate (x

x (i)⊤ M ⋆ x (j) + x (j)⊤ M ⋆ x (i) = x (i)⊤ M ⋆ + M ⊤ ⋆ x (
(i) , x (j) ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ = (x ⋆ , x ′ ⋆ )
. Naturally, while this local strategy is easy to apply for a couple of neighbors (i, j), it becomes infeasible to simultaneously extend it to all the other couples in the graph since some of them share the same nodes. However, one can learn something from this strategy, which is that given the optimal joint arm (x Hence, instead of looking for the optimal joint arm (which is NP-Hard), one can alternatively aim at seeking the allocation that, for any edge (i, j) ∈ E, constructs as many pairs (x (i) , x (j) ) = (x ⋆ , x ′ ⋆ ) as possible. Assigning x ⋆ to a subset of nodes and x ′ ⋆ to the complementary is equivalent to cutting the graph into two pieces and creating two distinct sets of nodes V 1 and V 2 such that

V = V 1 ∪ V 2 and V 1 ∩ V 2 = ∅.
Thus, the described strategy boils down to finding a cut passing through the maximum number of edges.

This problem is known as the Max-Cut problem (see e.g., [START_REF] Goemans | 879-approximation algorithms for max cut and max 2sat[END_REF][START_REF] Sahni | P-complete approximation problems[END_REF]), which is also NP-Hard. However, the extensive attention this problem has received allows us to use one of the many approximation algorithms (see, e.g., Algorithm 4) which are guaranteed to yield a cut passing through at least a given fraction of the edges in the graph. Most of the guarantees for the approximation of the Max-Cut problem are stated with respect to the optimal Max-Cut solution, which is not exactly the guarantee we are looking for: we need a guarantee as a proportion of the total number of edges. We thus have to be careful on the algorithm we choose.

From Algorithm 4, one can have a guarantee on the proportion of cut edges with respect to the total number of edges m = |E|. We state this guarantee in the following Proposition.

Proposition 3.1. Given a graph G = (V, E), Algorithm 4 returns a couple (V 1 , V 2 ) such that |{(i, j) ∈ E | (i ∈ V 1 ∧ j ∈ V 2 ) ∨ (i ∈ V 2 ∧ j ∈ V 1 )}| ≥ m 2 .
Algorithm 4: Approx-MAX-CUT [START_REF] Sahni | P-complete approximation problems[END_REF] Input :

G = (V, E) Set V 1 = ∅, V 2 = ∅ for i in V do n 1 = |{(i, j) ∈ E | j ∈ V 1 }|; n 2 = |{(i, j) ∈ E | j ∈ V 2 }|; if n 1 > n 2 then V 2 ← V 2 ∪ {i} else V 1 ← V 1 ∪ {i}; end return (V 1 , V 2 )
Proof. At each iteration, we take a node i that is neither in V 1 nor in V 2 , count its neighbors already in V 1 and V 2 and save the results respectively in n 1 and n 2 . For the sake of simplicity in the proof, we will denote them n

(i) 1 and n (i)
2 to distinguish from one node to the other. Since n

(i) 1 represents the number of neighbors of i already assigned to V 1 , if the node i is added to V 2 , 2 × n (i)
1 edges would be cut (the factor 2 comes from the fact that between two nodes i and j, there are the edges (i, j) and (j, i)). Similarly, since n

(i) 2 represents the number of neighbors already assigned in V 2 , if the node i is added to V 1 , 2 × n (i)
2 edges would be cut. In the algorithm, the node i is added to V 1 or V 2 such that we cut the most edges, hence by denoting m i the number of additional cut edges implied by the assignment of node i in V 1 or V 2 , we have:

m i = max 2n (i) 1 , 2n (i) 2 ≥ 2n (i) 1 + 2n (i) 2 2 = n (i) 1 + n (i) 2 .
By summing for all the nodes in the graph :

n i=1 m i ≥ n i=1 n (i) 1 + n (i) 2 = m 2 .
By definition n i=1 m i is the total number of edges that are cut which also means that

n i=1 m i = |{(i, j) ∈ E | (i ∈ V 1 ∧ j ∈ V 2 ) ∨ (i ∈ V 2 ∧ j ∈ V 1 )}| .
Given this guarantee with respect to the total number of edges, it only remains to present the full strategy that is to allocate to the nodes in V 1 the arm x ⋆ and to the nodes in V 2 the arm x ′ ⋆ . We give the strategy in Algorithm 5.

Algorithm 5: Approximation algorithm of our NP-Hard problem

Input :

Graph G = (V, E), arm set X , parameter matrix M ⋆ (V 1 , V 2 ) = Approx-MAX-CUT(G); Find (x ⋆ , x ′ ⋆ ) ∈ arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ ; for i in V 1 do x (i) t =
x ⋆ ; // Can be done in parallel end for i in V 2 do x (i) t = x ′ ⋆ ; // Can be done in parallel end return (x (1) , . . . , x (n) )

With this algorithm, given the returned allocation (x (1) , . . . , x (n) ), for some edges (i, j) ∈ E, the associated allocated arms will be the optimal couples (x ⋆ , x ′ ⋆ ) or (x ′ ⋆ , x ⋆ ) and for other edges (i, j) ∈ E the associated allocated arms will be the suboptimal and unwanted couples (x ⋆ , x ⋆ ) or (x ′ ⋆ , x ′ ⋆ ). Before we state the guarantee of this algorithm with respect to the optimal global reward, let us introduce m 1 (respectively m 2 ) the number of edges that go from nodes in V 1 (respectively V 2 ) to nodes in V 1 as well (respectively V 2 ) and m 1→2 (respectively m 2→1 ) the number of edges that goes from nodes in V 1 (respectively V 2 ) to nodes in V 2 (respectively V 1 ). Notice that the total number of edges m = m 1→2 + m 2→1 + m 1 + m 2 and that by definition of the edge set E and using Proposition 3. ⋆ ) be the optimal joint arm as defined in (3.1) and let 0 ≤ ξ ≤ 1 be a problem-dependent parameter defined by

ξ = min x∈X x ⊤ M ⋆ x
Proof. Given the allocation (x (1) , . . . , x (n) ) return by Algorithm 5, the associated reward y can be written as

y = m 1→2 × x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 × x ′⊤ ⋆ M ⋆ x ⋆ (a) + m 1 × x ⊤ ⋆ M ⋆ x ⋆ + m 2 × x ′⊤ ⋆ M ⋆ x ′ ⋆ (b)
Let us analyse (a):

(a) = m 1→2 × x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 × x ′⊤ ⋆ M ⋆ x ⋆ = m 1→2 + m 2→1 2 × x ⊤ ⋆ M ⋆ + M ⊤ ⋆ x ′ ⋆ (because m 1→2 = m 2→1 ) = m 1→2 + m 2→1 m n i=1 j∈N i j>i x ⊤ ⋆ M ⋆ + M ⊤ ⋆ x ′ ⋆ ≥ m 1→2 + m 2→1 m n i=1 j∈N i j>i x (i)⊤ ⋆ M ⋆ + M ⊤ ⋆ x (j) ⋆ (3.3) = m 1→2 + m 2→1 m (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ = m 1→2 + m 2→1 m y ⋆ ,
where (3.3) comes from Equation (3.2). Now, we analyse (b) by using the definition of ξ where for any x ′ ∈ X ,

x ′⊤ M ⋆ x ′ ≥ min x∈X x ⊤ M ⋆ x (3.4) = 1 m ξ M ⋆ x (j) ⋆ . (3.5)
Hence we have,

(b) ≥ m 1 m ξ (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ + m 2 m ξ (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ = m 1 + m 2 m ξy ⋆ .
Thus,

y = (a) + (b) ≥ m 1→2 + m 2→1 m y ⋆ + m 1 + m 2 m ξy ⋆ = m 1→2 + m 2→1 m + m 1 + m 2 m ξ y ⋆ (3.6) = m 1→2 + m 2→1 m + 1 - m 1→2 + m 2→1 m ξ y ⋆ = m 1→2 + m 2→1 + mξ -(m 1→2 + m 2→1 )ξ m y ⋆ = m 1→2 + m 2→1 m (1 -ξ) + ξ y ⋆ ≥ 1 2 (1 -ξ) + ξ y ⋆ = 1 + ξ 2 y ⋆ .
Moreover, the Approx-Max-Cut Algorithm has a complexity in O(n 2 ), then we do K 2 estimations to find the best couple (x ⋆ , x ′ ⋆ ) ∈ X 2 , and each round in the first and second for loop of the algorithm is in O(1) and there are

|V 1 | + |V 2 | = n rounds.
Hence the complexity is in

O(K 2 + n 2 ).

What is ξ and what value can we expect?

This parameter measures what minimum gain with respect to the optimal reward one could get by allocating the unwanted and suboptimal couple of arms of the form (x, x) for two neighbors. For example, if there exists x 0 ∈ X such that x ⊤ the m 2 rewards of the form x ′⊤ ⋆ M ⋆ x ′ ⋆ that one gets when allocating x ⋆ to the nodes in V 1 and x ′ ⋆ to the nodes in V 2 .

Here, the improvement we can make is to include them in the optimization problem and to weight the different rewards obtained through the graph using the proportions m 1→2 , m 2→1 , m 1 and m 2 . By denoting (x ⋆ , x′ ⋆ ) the solution of the following optimization problem, max

(x,x ′ )∈X 2 m 1→2 • x ⊤ M ⋆ x ′ + m 2→1 • x ′⊤ M ⋆ x + m 1 • x ⊤ M ⋆ x + m 2 • x ′⊤ M ⋆ x ′ , (3.7)
we are optimizing the total global reward that one would obtain when allocating only two arms (x, x ′ ) ∈ X 2 in the graph. This strategy is described in Algorithm 6.

Algorithm 6: Improved approximation algorithm of our NP-Hard problem

Input :

Graph G = (V, E), arm set X , parameter matrix M ⋆ (V 1 , V 2 ) = Approx-MAX-CUT(G); m 1→2 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 2 }|; m 2→1 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 1 }|; m 1 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 1 }|; m 2 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 2 }|; Find (x ⋆ , x′ ⋆ ) solution of (3.7); for i in V 1 do x (i) t = x⋆ ; // Can be done in parallel end for i in V 2 do x (i) t =

x′

⋆ ; // Can be done in parallel end return (x (1) , . . . , x (n) )

To understand and analyse this new algorithm, let us define ∆ ≥ 0 the global reward difference of allocating (x ⋆ , x′ ⋆ ) instead of (x ⋆ , x ′ ⋆ ) as follows:

∆ = m 1→2 x⊤ ⋆ M ⋆ x′ ⋆ -x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 x′⊤ ⋆ M ⋆ x⋆ -x ′⊤ ⋆ M ⋆ x ⋆ + m 1 x⊤ ⋆ M ⋆ x⋆ -x ⊤ ⋆ M ⋆ x ⋆ + m 2 x′⊤ ⋆ M ⋆ x′ ⋆ -x ′⊤ ⋆ M ⋆ x ′ ⋆ .
The new guarantees that we get on the reward of the allocation obtained by Algorithm 6 are stated in the following theorem. Theorem 3.3. Let us consider the graph G = (V, E), a finite arm set X ⊂ R d and the matrix M ⋆ ∈ R d×d given as input to Algorithm 6. Let (x

(1) ⋆ , . . . , x (n)
⋆ ) be the optimal joint arm as defined in (3.1) and let 0 ≤ ξ ≤ 1 be defined as in Theorem 3.2. Let 0 ≤ ϵ ≤ 1 2 be a problem dependent parameter that measures the relative gain of optimizing on the suboptimal rewards defined as:

α = m 1→2 + m 2→1 m + m 1 + m 2 m ξ + ϵ .
Proof. The proof follows exactly the one of Theorem 3.2 where we stop the reasoning at Equation 3.6 and then plug the result into the proof of Theorem 3.3

This corollary is useful to understand in practice the kind of guarantees we can have depending on the graph structure and the approximation algorithm we use to solve the Max-Cut problem. For instance, in the most favorable graphs, which are bipartite graphs (i.e., graphs where all the m edges goes from nodes in V 1 to nodes in V 2 or vice versa), we have m 1→2 + m 2→1 = m and m 2 + m 1 = 0. Also it implies that (x ⋆ , x ′ ⋆ ) = (x ⋆ , x′ ⋆ ), so ϵ = 0 which gives α = 1 and makes the Algorithm 10 find the optimal solution of the problem. What may also be of interest is to understand how α varies with respect to ξ, ϵ and the quantity m 1 and m 2 for graphs that are between a complete graph (that is the worst case scenario in terms of constraints) and a bipartite graph. We investigate experimentally this dependency in Section 3.2.

Numerical experiments: influence of the parameters on the solution

In this section, we give some insights on the problem-dependent parameters ξ and ϵ and the corresponding α. Let α 1 and α 2 be the α stated respectively in Theorem 3.2 and Corollary 3.1. In the first experiment, we show the dependence of α 1 and α 2 on the graph type and the chosen approximation algorithm for the max-cut problem with respect to ξ and ϵ. We also highlight the differences between the two parameters α 1 and α 2 as well as the significant improvement in guarantees that one can obtain using Algorithm 6 depending on the type of the graph. The results are presented in Table 3.1. One can notice that the complete graph seems to give the worst guarantee on the α-approximation with respect to ϵ and ξ. Thus, we conducted a second experiment where we consider the worst case scenario in terms of the graph type -e.g., the complete graph-and where there are n = 10 agents. This second experiment studies the variation of ϵ and ξ with respect to the unknown parameter matrix M ⋆ . To design such experiment, we consider the arm-set X as the vectors (e 1 , . . . , e d ) of the canonical base in R d . We generate the matrix M ⋆ randomly in the following way: first, all elements of the matrix are drawn i.i.d.from a standard normal distribution, and then we take the absolute value of each of these elements to ensure that the matrix only contains positive numbers. The choice of the vectors of the canonical base as the arm-set allows us to modify the matrix M ⋆ and to illustrate the dependence on ξ and ϵ in a simple way. Consider the best couple (i ⋆ , j ⋆ ) = arg max (i,j)∈{1,...,d} 2 e ⊤ i M ⋆ e j , we want to see how the rewards of the suboptimal couples of arms (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ) impact the values of ξ, ϵ and thus α. Notice that the reward associated with the couple of arms (e i ⋆ , e i ⋆ ) (respectively (e j ⋆ , e j ⋆ )) is

[M ⋆ ] i ⋆ i ⋆ (respectively [M ⋆ ] j ⋆ j ⋆ ). Hence we define 0 ≤ ζ < 1 and set [M ⋆ ] i ⋆ i ⋆ = [M ⋆ ] j ⋆ j ⋆ = ζ × 1 2 ([M ⋆ ] i ⋆ j ⋆ + [M ⋆ ] j ⋆ i ⋆ ).
We study the variation of ξ, ϵ, α 1 and α 2 with respect to ζ. The results are presented in Figure 3.1. One can see that when the associated rewards of (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ) are low (thus ξ is low and ϵ high), Algorithm 6 gives a much better guarantees than Algorithm 5 since it focuses on other arms than e i ⋆ and e j ⋆ that give a higher global reward. Moreover, even when the unwanted couples (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ) give high rewards, the guarantees on the regret of Algorithm 10 are still stronger because it takes into consideration the quantities m 1 and m 2 of the constructed suboptimal couple of arms (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ).

Conclusion and perspectives

In this chapter, we showed that the underlying optimization problem is NP-hard and that one has to rely on approximation algorithms. To that matter, we designed a first α-approximation algorithm based on the max-cut problem, with α ≥ 1/2. We showed that by exploiting the graph structure and the typology of the problem, one can both improve the performance in practice and have a better theoretical guarantee on α. The method presented in this chapter can be extended in many ways, especially in the choice of the max-cut approximation algorithm. For instance, one can consider cutting the graph into 3 or more pieces, which is equivalent to approximating the problem of a Max-k-Cut [START_REF] Frieze | Improved approximation algorithms for max k-cut and max bisection[END_REF] with k ≥ 3. With the knowledge of such a partition of nodes V 1 , . . . , V k , one may want to look for a k-tuple of arms maximizing the optimistic allocated reward rather than a pair, therefore introducing an elegant tradeoff between the optimality of the solution and the computational complexity of the arms allocation. In this chapter, we assume that we do not know the parameter matrix M ⋆ and as described in the problem setting in Section 1.2, a central entity faces a graphical bilinear bandits problem where at each round it chooses an arm for each node of the graph and observes a bilinear reward for each edge of the graph. In this chapter, we will focus on the best-arm identification objective that reads as follows:

Objective: Find, within a minimum number of rounds, the joint arm (x

(1) ⋆ , . . . , x (n) ⋆ ) such that the expected global reward (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j)
⋆ is maximized, where M ⋆ is unknown to the learner.

We follow the notations we established in section 1.2.1.

Preliminaries

A two-stage algorithm template

For simplicity, we consider for now that the unknown parameter M ⋆ is symmetric, which greatly simplifies the reasoning, and we will relax this assumption in Section 4.2.3. In Chapter 3, we designed polynomial-time algorithms that allow us to compute an α-approximation solution to the NP-Hard problem of finding the best joint arm given M ⋆ . Notice that in the α-approximation Algorithm 5, M ⋆ is only used to identify the best pair (x ⋆ , x ′ ⋆ ) as follows:

x ⋆ , x ′ ⋆ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ x ′ . (4.1)
Thus, using an estimate M of M ⋆ having the following property:

arg max (x,x ′ )∈X 2 x ⊤ Mx ′ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ x ′ = x ⋆ , x ′ ⋆ , (4.2)
allows us to identify the pair (x ⋆ , x ′ ⋆ ), and thus gives us the same guarantees as the ones presented in Theorem 3.2. We thus address the problem of computing M such that, in a minimum number of rounds, with high probability, we are able to identify the pair (x ⋆ , x ′ ⋆ ) and apply the Algorithm 5.

Our general algorithm can thus be thought of as a two-part algorithm where, in the first instance, the central entity applies a pure exploration algorithm and draws the arms of the nodes at each round to compute the best possible estimate M (in a certain sense), and then, in the second instance, uses M to apply the α-approximation-Algorithm 5 described in chapter 3. We describe the general procedure in Algorithm 7.

Algorithm 7: General framework for BAI in GBB : a two-stage algorithm

Input : graph G = (V, E), arm set X M = Pure-Exploration-Algorithm(G,X ); // This chapter x (1) , . . . , x (n) = α-Approximation-Algorithm(G, X , M); // Done in chapter 3 return x (1) , . . . , x (n)

Stopping condition

Notice that the optimization problem (4.1) is equivalent to the following optimization problem

(x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X vec xx ′⊤ , vec (M ⋆ ) .
Let us simplify the notations and denote θ ⋆ ≜ vec (M ⋆ ) the vectorized version of the unknown matrix M ⋆ . Let us also use the notation z xx ′ ≜ vec xx ′⊤ , and define Z = {z xx ′ |(x, x ′ ) ∈ X 2 } the set containing such vectors. Then looking for the couple (x ⋆ , x ′ ⋆ ) is the same as looking for the vector z ⋆ ∈ Z where

z ⋆ = arg max z∈Z ⟨z, θ ⋆ ⟩ .
In other words, we want to find an arm z ∈ Z, such that for all z ′ ∈ Z, (z -z ′ ) ⊤ θ ⋆ ≥ 0. However, one does not have access to θ ⋆ , so we have to use its empirical estimate.

Hence at each round t, the central entity can choose for each couple of neighbors (i, j) an arm z ∈ Z and get a noisy linear reward of the form ⟨z, θ ⋆ ⟩ + η where η is a σ-subgaussian random variable, that can be used to compute an estimate θt .

For more clarity, we refer to any x ∈ X as a node-arm and any z ∈ Z will be referred as an edge-arm. If x (i) t ∈ X represents the node-arm allocated to the node i ∈ V at time t, for each edge (i, j) ∈ E we will denote the associated edge-arm by z

(i,j) t ≜ vec x (i) t x (j)⊤ t ∈ Z.
Assumption 4.1 (Bounded edge-arm norm). We consider that there exists L > 0, for all edge-arm z ∈ Z, such that ∥z∥ 2 2 ≤ L.

Assumption 4.2 (Positive rewards).

We consider that for any z ∈ Z, the associated expected reward ⟨z, θ ⋆ ⟩ is such that ⟨z, θ ⋆ ⟩ ≥ 0 Assumption 4.3 (Spanning the action space). We consider that X spans R d .

The goal here is to define the optimal sequence (z 1 , . . . , z mt ) ∈ Z mt that should be pulled in the first t rounds so that (4.2) is reached as soon as possible. A natural approach is to rely on classical strategies developed for best arm identification in linear bandits. We define (y 1 , . . . , y mt ) the corresponding noisy rewards of the sequence (z 1 , . . . , z mt ). We assume that the noise terms in the rewards are i.i.d., following a σ-sub-Gaussian distribution. Let θt = A -1 t b t ∈ R d 2 be the solution of the ordinary least squares problem with

A t = mt i=1 z i z ⊤ i ∈ R d 2 ×d 2 and b t = mt i=1 z i y i ∈ R d 2 .
We first recall the Proposition 2.2 with the notation of our problem.

Proposition 4.1 (Proposition 1 in [START_REF] Soare | Best-arm identification in linear bandits[END_REF]). For every fixed sequence (z 1 , . . . , z mt ) ∈ Z mt , with probability 1 -δ, for all t > 0 and for all z ∈ Z, we have

z ⊤ θ ⋆ -z ⊤ θt ≤ 2σ √ 2∥z∥ A -1 t log 6m 2 t 2 K 2 δπ .
Following the steps of [START_REF] Soare | Best-arm identification in linear bandits[END_REF] and the ones developed in Section 2.2.2, we can show that if there exists z ∈ Z such that for all z ′ ∈ Z the following holds:

∥z -z ′ ∥ A -1 t 8σ 2 log 6m 2 t 2 K 4 δπ 2 ≤ ∆t z, z ′ , (4.3)
where ∆t (z, z ′ ) = (z -z ′ ) ⊤ θt is the empirical gap between z and z ′ , then with probability at least 1-δ, the OLS estimate θt leads to the best edge-arm z ⋆ , which means that arg max z∈Z ⟨z, θt ⟩ = arg max z∈Z ⟨z, θ ⋆ ⟩. Therefore, when the Equation (4.3) is true, the learner can stop pulling arms, we call it the stopping condition.

As mentioned in [START_REF] Soare | Best-arm identification in linear bandits[END_REF], by noticing that max

(z,z ′ )∈Z 2 ∥z -z ′ ∥ A -1 t ≤ 2 max z ∈ Z ∥z∥ A -1 t
, an admissible strategy is to pull edge-arms minimizing max z∈Z ∥z∥ A -1 t in order to satisfy the stopping condition as soon as possible.

A Constrained G-Allocation

Given the stopping condition (4.3) derived in the previous section, one wants to find the sequence of edge-arms z ⋆ mt = (z ⋆ 1 , . . . , z ⋆ mt ) such that:

z ⋆ mt ∈ arg min (z 1 ,...,zmt)∈Z mt max z ′ ∈Z z ′ ⊤ mt i=1 z i z ⊤ i -1 z ′ . (G-opt-Z)
This is known as G-allocation (see e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]) and is NP-hard to compute ( [START_REF] Çivril | On selecting a maximum volume sub-matrix of a matrix and related problems[END_REF]104]). One way to find an approximate solution is to rely on a convex relaxation of the optimization problem (G-opt-Z) and first compute a real-valued allocation Γ ⋆ ∈ S Z such that

Γ ⋆ ∈ arg min Γ ∈S Z max z ′ ∈Z z ′ ⊤ z∈Z Γ z zz ⊤ -1 z ′ . (G-relaxed-Z)
One could either use random sampling to draw edge-arms as i.i.d. samples from the Γ ⋆ distribution or rounding procedures to efficiently convert each Γ ⋆ z into an integer. However, these methods do not take into account the graphical structure of the problem. Indeed, at a given round, the chosen edge-arms may result in two different assignments for the same node, we call this phenomenon a collision. In Figure 4.1, we characterise a collision that occurs when the central entity allocates respectively two edge-arms z and z ′ to two edges that share the same node. Therefore, random sampling or rounding procedures cannot be straightforwardly used to select edge-arms in Z. Nevertheless, (G-relaxed-Z) still gives valuable information on the number of times, in proportion, each edge-arm z ∈ Z must be allocated to the graph. In the next section, we present an algorithm satisfying both the proportion requirements and the graphical constraints.

i j k z = v e c x x ′ ⊤ z ′ = v e c x ′ ′ x ′ ′ ′ ⊤ x (i) = x x (j) = x ′ x (k) = x ′′′ x (j) = x ′′ Collision if x ′ = x ′′

Algorithm and guarantees

Random Allocation over the Nodes

Our algorithm is based on a randomized method directly allocating node-arms to the nodes and thus avoiding the difficult task of choosing edge-arms and trying to allocate them to the graph while ensuring that every node has an unique assignment. The validity of this random allocation is based on Theorem 4.1 below showing that one can draw node-arms in X and allocate them to the graph such that the associated edge-arms follow the probability distribution Γ ⋆ solution of (G-relaxed-Z).

Theorem 4.1. Let γ ⋆ be a solution of the following optimization problem:

min γ∈S X max x ′ ∈X x ′ ⊤ x∈X γ x xx ⊤ -1 x ′ . (G-relaxed-X ) Let Γ ⋆ ∈ S Z be defined for all z = vec xx ′⊤ ∈ Z by Γ ⋆ z = γ ⋆ x γ ⋆ x ′ . Then, Γ ⋆ is a solution of (G-relaxed-Z).
To prove Theorem 4.1, we first state a useful lemma. For any finite set

X ⊂ R d and γ ∈ S X , let Σ X (γ) = x∈X γ x xx ⊤ . We define the function h X : S X → R ∪ {+∞} as follows: for any γ ∈ S X , h X (γ) = max x ′ ∈X x ′⊤ Σ X (γ) -1 x ′ if Σ X (γ) is invertible +∞ otherwise . Lemma 4.1. Let X ⊂ R d be a finite set spanning R d and let Z = {vec (xx ′⊤ ) | (x, x ′ ) ∈ X 2 }. If γ ⋆ ∈ S X is a minimizer of h X , then γ ⋆ is a solution of min γ∈S X max z∈Z z ⊤ x∈X x ′ ∈X γ x γ x ′ vec xx ′⊤ vec xx ′⊤ ⊤ -1 z .
Proof. First, let us notice that, for any

X ⊂ R d , one has h X ≥ 0. Thus, γ ⋆ is also a minimizer of h 2 X . In addition, X is spanning R d so h X (γ ⋆ ) < +∞. Developing h X (γ ⋆ ) 2 yields: h X (γ ⋆ ) × h X (γ ⋆ ) = max x∈X x ⊤ Σ X (γ ⋆ ) -1 x × max x∈X x ⊤ Σ X (γ ⋆ ) -1 x = max x∈X max x ′ ∈X x ⊤ Σ X (γ ⋆ ) -1 xx ′⊤ Σ X (γ ⋆ ) -1 x ′ = max x∈X max x ′ ∈X vec xx ′⊤ ⊤ vec Σ X (γ ⋆ ) -1 xx ′⊤ Σ X (γ ⋆ ) -1 = max x∈X max x ′ ∈X vec xx ′⊤ ⊤ Σ X (γ ⋆ ) -1 ⊗ Σ X (γ ⋆ ) -1 vec xx ′⊤ = max z∈Z z ⊤ Σ X (γ ⋆ ) -1 ⊗ Σ X (γ ⋆ ) -1 z ,
where ⊗ denotes the Kronecker product. We can now focus on the central term:

Σ X (γ ⋆ ) -1 ⊗ Σ X (γ ⋆ ) -1 = x∈X γ ⋆ x xx ⊤ -1 ⊗ x∈X γ ⋆ x xx ⊤ -1 = x∈X γ ⋆ x xx ⊤ ⊗ x∈X γ ⋆ x xx ⊤ -1 = x∈X x ′ ∈X γ ⋆ x γ ⋆ x ′ xx ⊤ ⊗ x ′ x ′⊤ -1 = x∈X x ′ ∈X γ ⋆ x γ ⋆ x ′ vec xx ′⊤ vec xx ′⊤ ⊤ -1
, and the result holds.

Proof of Theoreme 4.1. From [START_REF] Kiefer | The Equivalence of Two Extremum Problems[END_REF], we know that min Γ ∈S Z h Z (Γ ) = d 2 and min γ∈S X h X (γ) = d. Then, using Lemma 4.1, one has

d 2 = h X (γ ⋆ ) × h X (γ ⋆ ) = max z ′ ∈Z z ′⊤ x∈X x ′ ∈X γ ⋆ x γ ⋆ x ′ vec xx ′⊤ vec xx ′⊤ ⊤ -1 z ′ = max z ′ ∈Z z ′⊤ z∈Z Γ ⋆ z zz ⊤ -1 z ′ . This result implies that h Z (Γ ⋆ ) = d 2 . Since min Γ ∈S Z h Z (Γ ) = d 2 , Γ ⋆ is a minimizer of h Z .
This theorem implies that, at each round t > 0 and each node

i ∈ V , if x (i)
t is drawn from γ ⋆ , then for all pairs of neighbors (i, j) ∈ E the probability distribution of the associated edgearms z (i,j) t follows Γ ⋆ . Moreover, as γ ⋆ is a distribution over the node-arm set X , Γ ⋆ is a joint (product) probability distribution on X 2 with marginal γ ⋆ .

On the computation of γ ⋆ . Let us first state the following proposition:

Proposition 4.2. Let d > 0, for any set X ⊂ R d , h X is convex. Proof. Let (γ, γ ′ ) ∈ S 2 X be two distributions in S X . If either Σ X (γ) or Σ X (γ ′ ) are not invertible, then for any t ∈ [0, 1] one has h X (tγ + (1 -t)γ ′ ) ≤ th X (γ) + (1 -t)h X (γ ′ ) = +∞ .
Otherwise, for t ∈ [0, 1], we define the positive definite matrix Z(t) ∈ R d×d as follows:

Z(t) = tΣ X (γ) + (1 -t)Σ X (γ ′ ) .
Simple linear algebra [START_REF] Petersen | The matrix cookbook, nov 2012[END_REF] yields

∂Z(t) -1 ∂t = Z(t) -1 ∂Z(t) ∂t Z(t) -1 .
Using this result and the fact that ∂ 2 Z(t)/∂t 2 = 0, we obtain

∂ 2 Z(t) -1 ∂t 2 = 2Z(t) -1 ∂Z(t) ∂t Z(t) -1 ∂Z(t) ∂t Z(t) -1 .
Therefore, for any x ∈ X,

∂ 2 x ⊤ Z(t) -1 x ∂t 2 = 2x ⊤ Z(t) -1 ∂Z(t) ∂t Z(t) -1 ∂Z(t) ∂t Z(t) -1 x = 2 ∂Z(t) ∂t Z(t) -1 x ⊤ Z(t) -1 ∂Z(t) ∂t Z(t) -1 x ≥ 0 ,
which shows convexity for any fixed x ∈ X. The final results yields from the fact that h X is a maximum over convex functions.

Although we face a min-max optimization problem and given the convexity of h X , we apply the Frank-Wolfe algorithm [START_REF] Frank | An algorithm for quadratic programming[END_REF] to compute the solution γ ⋆ of (G-relaxed-X ), as it is more suited to optimization tasks on the simplex than projected gradient descent. The convergence of the algorithm has been proven in [START_REF] Damla Ahipasaoglu | Linear convergence of a modified Frank-Wolfe algorithm for computing minimum-volume enclosing ellipsoids[END_REF].

Given the characterization in Theorem 4.1 and our objective to verify the stopping condition in (4.3), we present our sampling procedure in Algorithm 8. We also note that at each round the sampling of the node-arms can be done in parallel.

This sampling procedure implies that each edge-arm follows the optimal distribution Γ ⋆ . However, if we take the number of times each z ∈ Z appears in the m pulled edge-arms of a given round, we might notice that the observed proportion is not close to Γ ⋆ z , regardless of the size of m. This is due to the fact that the m edge-arms are not independent because of the graph structure Algorithm 8: Pure-Exploration-Algorithm : Randomized G-Allocation for GBB

Input : graph G = (V, E), arm set X Set A 0 = I ; b 0 = 0 ; t = 1;
Apply the Frank-Wolfe algorithm to find γ ⋆ solution of (G-relaxed-X ). while stopping condition (4.3) is not verified do // Sampling the node-arms Draw x

(1) t , . . . , x

(n) t iid ∼ γ ⋆ and obtain for all (i, j) in E the rewards y (i,j) t ; // Estimating θt with the associated edge-arms

A t = A t-1 + (i,j)∈E z (i,j) t z (i,j)⊤ t ; b t = b t-1 + (i,j)∈E z (i,j) t y (i,j) t ; θt = A -1 t b t t ← t + 1; end return θt (cf. Section 4.3.1)
. Conversely, since each group of m edge-arms are independent from one round to another, the proportion of each z ∈ Z observed among the mt pulled edge-arms throughout t rounds is close to Γ ⋆ z .

One may wonder whether deterministic rounding procedures could be used instead of random sampling on γ ⋆ , as it is done in many standard linear bandit algorithms [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]. Applying rounding procedure on γ ⋆ gives the number of times each node-arm x ∈ X should be allocated to the graph. However, it does not provide the actual allocations that the learner must choose over the t rounds to optimally pull the associated edge-arms (i.e., pull edge-arms following Γ ⋆ ). Thus, although rounding procedures give a more precise number of times each node-arm should be pulled, the problem of allocating them to the graph remains open, whereas by concentration of the measure, randomized sampling methods imply that the associated edge-arms follow the optimal probability distribution Γ ⋆ . In this thesis, we present a simple and standard randomized G-allocation strategy, but other more elaborated methods could be considered, as long as they include the necessary randomness.

On the choice of the G-allocation problem.

We have considered the G-allocation optimization problem (G-opt-Z), however, one could want to directly minimize max (z,z ′ )∈Z 2 ∥z -z ′ ∥ A -1 t , known as the XY-allocation [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]. Hence, one may want to construct edge-arms that follow the distribution Γ ⋆ XY solution of the relaxed XY-allocation problem:

min Γ ∈S Z max (z ′ ,z ′′ )∈Z 2 z ′ -z ′′ ⊤ z∈Z Γ z zz ⊤ -1 z ′ -z ′′ .
Although efficient in the linear case, this approach outputs a distribution Γ ⋆ XY which is not a joint probability distribution of two independent random variables, and so cannot be decomposed as the product of its marginals. Hence, there is no algorithm that allocates identically and indepen-dently the nodes of the graph to create edge-arms following Γ ⋆ XY . Thus, we will rather deal with the upper bound given by the G-allocation as it allows sampling over the nodes.

Static design versus adaptive design. Adaptive designs as proposed for example in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] and [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF] provide a strong improvement over static designs in the case of linear bandits. In our particular setting however, it is crucial to be able to adapt the edge-arms sampling rule to the node-arms, which is possible thanks to Theorem 4.1. This result requires a set of edge-arms Z expressed as a product of node-arms set X . Extending the adaptive design of [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF] to our setting would eliminate edge-arms from Z at each phase, without trivial guarantees that the newly obtained edge-arms set Z ′ ⊂ Z could still be derived from another node-arms set X ′ ⊂ X . An adaptive approach is definitely a natural and promising extension of our method, and is left for future work.

Convergence Analysis

We now prove the validity of the random sampling procedure detailed in Algorithm 8 by controlling the quality of the approximation max z∈Z z ⊤ A -1 t z with respect to the optimum of the G-allocation optimization problem

max z ′ ∈Z z ′ ⊤ mt i=1 z ⋆ i z ⋆⊤ i -1 z ′ described in (G-opt-Z).
As is usually done in the optimal design literature (see e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Sagnol | Optimal design of experiments with application to the inference of traffic matrices in large networks: second order cone programming and submodularity[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]) we bound the relative error β t :

max z∈Z z ⊤ A -1 t z ≤ (1 + β t ) max z ′ ∈Z z ′ ⊤ mt i=1 z ⋆ i z ⋆⊤ i -1 z ′ .
Our analysis relies on several results from matrix concentration theory. One may refer for instance to [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF] and references therein for an extended introduction on that matter. We first introduce a few additional notations. Let f Z be the function such that for any non-singular matrix

Q ∈ R d 2 ×d 2 , f Z (Q) = max z∈Z z ⊤ Q -1 z and for any distribution Γ ∈ S Z we recall that Σ Z (Γ ) ≜ z∈Z Γ z zz ⊤
is the associated covariance matrix. Finally let A ⋆ t = mt i=1 z ⋆ i z ⋆⊤ i be the G-optimal design matrix constructed during t rounds. Theorem 4.2. Let Γ ⋆ be a solution of the optimization problem (G-relaxed-Z). Let 0 < δ ≤ 1 and let t 0 be such that

t 0 = 2Ld 2 log 2d 2 /δ /λ min ,
where λ min is the smallest eigenvalue of the covariance matrix 1 K 2 z∈Z zz ⊤ . Then, at each round t ≥ t 0 with probability at least 1 -δ, the randomized G-allocation strategy for graphical bilinear bandits in Algorithm 8 produces a matrix A t such that:

f Z (A t ) ≤ (1 + β t )f Z (A ⋆ t ) ,
where

β t = Ld 2 mλ 2 min 2v t log 2d 2 δ + o 1 √ t ,
and v ≜ E (A 1 -EA 1 ) 2 .
To prove this confidence bound, we need the two following propositions.

Proposition 4.3 ([96]

, Chapter 5 and 6). Let Z 1 , . . . , Z t be i.i.d. positive semi-definite random matrices in R d 2 ×d 2 , such that there exists L > 0 verifying 0 ⪯ Z 1 ⪯ mLI. Let A t be defined as A t ≜ t s=1 Z s . Then, for any 0 < ε < 1, one can lowerbound λ min (A t ), the minimum eigenvalue of A t , as follows:

P(λ min (A t ) ≤ (1 -ε)λ min (EA t )) ≤ d 2 e - tε 2 λ min (EZ 1 )
2mL .

If in addition, there exists some v > 0, such that ∥E (Z 1 -EZ 1 ) 2 ∥ ≤ v, then for any u > 0, one has

P(∥S t ∥ ≥ u) ≤ 2d 2 e - u 2 2mLu/3 + 2tv ,
From the second inequality, [START_REF] Rizk | Refined bounds for randomized experimental design[END_REF] derived a slightly different inequality that we use here :

Proposition 4.4 ([78], Appendix A.3). Let Z 1 , . . . , Z t be t i.i.d. random symmetric matrices in R d 2 ×d 2 such that there exists L > 0 such that ∥Z 1 ∥ ≤ mL, almost surely. Let A t ≜ t i=1 Z i .
Then, for any u > 0, one has:

P ∥A t -EA t ∥ ≥ √ 2tvu + mLu 3 ≤ d 2 e -u .
where

v ≜ E (Z 1 -EZ 1 ) 2 .
Finally, to prove our main theorem, we need the following lemma.

Lemma 4.2. One has

Σ Z (Γ ⋆ ) -1 ≤ d 2 λ min
, where λ min is the smallest eigenvalue of the covariance matrix 1

K 2 z∈Z z ⊤ z. Proof. Define B = z ∈ R d 2 : ∥z∥ = 1 . First, for any semi-definite matrix A ∈ R d 2 ×d 2 ,
we have ∥A∥ = max z∈B z ⊤ Az. Because Σ Z (Γ ⋆ ) -1 is positive definite and symmetric, and by Rayleigh-Ritz theorem,

Σ Z (Γ ⋆ ) -1 = max z∈B z ⊤ Σ Z (Γ ⋆ ) -1 z z ⊤ z = max z∈B z ⊤ Σ Z (Γ ⋆ ) -1 z .
Let Z ∈ R K 2 ×d 2 be the matrix whose rows are vectors of Z in an arbitrary order. Notice that

Z spans R d 2 , since X spans R d . Now for any z ∈ B, define β (z) ∈ R K 2 as a vector such that z = Z ⊤ β (z) . Then, Σ Z (Γ ⋆ ) -1 = max z∈B β (z) ⊤ ZΣ Z (Γ ⋆ ) -1 Z ⊤ β (z) = max z∈B d 2 i=1 d 2 j=1 β (z) i β (z) j z ⊤ i Σ Z (Γ ⋆ ) -1 z j ≤ max z∈B β (z) 2 1 × max i,j z ⊤ i Σ Z (Γ ⋆ ) -1 z j . Define zi = Σ Z (Γ ⋆ ) -1 2 z i . Clearly, max i,j z ⊤ i Σ Z (Γ ⋆ ) -1 z j = max i,j z⊤ i zj = max i z2 i . So we have Σ Z (Γ ⋆ ) -1 ≤ max z∈B β (z) 2 1 × max z ′ ∈Z z ′⊤ Σ Z (Γ ⋆ ) -1 z ′ ≤ max z∈B β (z) 2 1 d 2 .
The last inequality comes from Kiefer and Wolfowitz equivalence theorem [START_REF] Kiefer | The Equivalence of Two Extremum Problems[END_REF]. Now observe that β (z) can be obtained by least square regression :

β (z) = ZZ ⊤ -1 Zz = Z ⊤ † z where (•) †
is the Moore-Penrose pseudo-inverse. Note that ZZ ⊤ is a Gram matrix. It is known that for a matrix having singular values {σ i } i , its pseudo-inverse has singular values

1 σ i if σ i ̸ = 0 0 otherwise
for all i. So for z ∈ B, we have:

β (z) 2 1 ≤ K 2 β (z) 2 2 ≤ K 2 Z ⊤ † 2 ≤ K 2 σ min (Z) 2 ,
where σ min (•) refers to the smallest singular value. Let λ min (•) refer to the smallest eigenvalue. Noting that

σ min (Z) 2 = λ min Z ⊤ Z = K 2 λ min 1 K 2 z∈Z zz ⊤ ,
yields the desired result.

We can now prove the main theorem.

Proof of Theorem 4.2. Let (X

s ) s=1,...,t , . . . , (X

(n) s ) s=1,...,t be nt i.i.d. random vectors in R d such that for all x ∈ X , P X (1) 1 = x = γ ⋆ x . For (i, j) ∈ E and 1 ≤ s ≤ t, we define the random matrix Z (i,j) s by Z (i,j) s = vec X i s X j⊤ s vec X i s X j⊤ s ⊤ .
Finally, let us define for all 1 ≤ s ≤ t, the edge-wise sum

Z s ∈ R d 2 ×d 2 , that is Z s = (i,j)∈E Z (i,j) s .
One can easily notice that Z 1 , . . . , Z t are i.i.d. random matrices. We define the overall sum

A t = t s=1 Z s and our goal is to measure how close f Z (A t ) is to f Z (mt × Σ Z (Γ ⋆ ))
, where mt corresponds to the total number of sampled arms z ∈ Z during the t rounds of the learning procedure. By definition of A t , one has

max z∈Z z ⊤ (EA t ) -1 z = max z∈Z z ⊤   t s=1 (i,j)∈E E Z (i,j) s   -1 z = max z∈Z z ⊤   t s=1 (i,j)∈E x,x ′ ∈X γ ⋆ x γ ⋆ x ′ vec (xx ′⊤ ) vec (xx ′⊤ ) ⊤   -1 z = max z∈Z z ⊤   t s=1 (i,j)∈E z ′ ∈Z Γ ⋆ z ′ z ′ z ′⊤   -1 z = f Z (mtΣ Z (Γ ⋆ )) .
This allows us to bound the relative error as follows:

β t = f Z (A t ) f Z (mt × Σ Z (Γ ⋆ )) -1 = max z∈Z z ⊤ A -1 t -(EA t ) -1 + (EA t ) -1 z f Z (mt × Σ Z (Γ ⋆ )) -1 ≤ max z∈Z z ⊤ A -1 t -(EA t ) -1 z f Z (mt × Σ Z (Γ ⋆ ))
.

Using the fact that f Z (mtΣ Z (Γ ⋆ )) = d 2 /mt [START_REF] Kiefer | The Equivalence of Two Extremum Problems[END_REF], we obtain

β t ≤ mt d 2 × max z∈Z z ⊤ A -1 t -(EA t ) -1 z ≤ mt d 2 × max z∈Z ∥z∥ 2 ∥A -1 t -(EA t ) -1 ∥ ≤ mtL d 2 × ∥A -1 t -(EA t ) -1 ∥ .
Therefore, controlling the quantity ∥A -1 t -(EA t ) -1 ∥ will allow us to provide an upper bound on the relative error. Notice that

∥A -1 t -(EA t ) -1 ∥ = ∥A -1 t (EA t -A t )(EA t ) -1 ∥ ≤ ∥A -1 t ∥ ∥EA t -A t ∥ ∥(EA t ) -1 ∥ .
Using Proposition 4.3, we know that for any d 2 e -tλ min (EZ 1 ) mL < δ h < 1, the following holds:

∥A -1 t ∥ ≤ ∥(EA t ) -1 ∥ 1 -2mL t ∥(EZ 1 ) -1 ∥ log(d 2 /δ h )
, with probability at least 1 -δ h . Similarly, using Proposition 4.4, for any 0 < δ b < 1, we have

∥A t -EA t ∥ ≤ mL 3 log d 2 δ b + 2tv 2 log d 2 δ b ,
with probability at least 1 -δ b . Combining these two results with a union bound leads to the following bound, with probability 1 -(δ b + δ h ):

A -1 t -(EA t ) -1 ≤ (EA t ) -1 2 (mL/3) log d 2 /δ b + 2tv log(d 2 /δ b ) 1 -(2mL/t) (EZ 1 ) -1 log(d 2 /δ h )
.

In order to obtain a unified bound depending on one confidence parameter 1 -δ, one could optimize over δ b and δ h , subject to δ b + δ h = δ. This leads to a messy result and a negligible improvement. One can use simple values δ b = δ h = δ/2, so the overall bound becomes, with probability 1 -δ:

∥A -1 t -(EA t ) -1 ∥ ≤ 1 tm 2 Σ Z (Γ ⋆ ) -1 2 2v t log 2d 2 δ   1 + m 2 L 2 log(2d 2 /δ) 18vt 1 -2L∥Σ Z (Γ ⋆ ) -1 ∥ log(2d 2 /δ) t   .
This can finally be formulated as follows:

A -1 t -(EA t ) -1 ≤ 1 tm 2 Σ Z (Γ ⋆ ) -1 2 2v t log 2d 2 δ + o 1 t √ t .
Using the obtained bound on

∥A -1 t -E(A t ) -1 ∥ yields f Z (A t ) f Z (mt × Σ Z (Γ ⋆ )) -1 ≤ mtL d 2 × 1 tm 2 Σ Z (Γ ⋆ ) -1 2 2v t log 2d 2 δ + o 1 t √ t ≤ L md 2 Σ Z (Γ ⋆ ) -1 2 2v t log 2d 2 δ + o 1 √ t , By noticing that f Z (mt × Σ Z (Γ ⋆ )) ≤ f Z (A ⋆ t )
and by using Lemma 4.2, the result holds.

We have just shown that the approximation value max z∈Z z ⊤ A -1 t z converges to the optimal value with a rate of O √ v/(m √ t) . In Section 4.3.1, we show that the best case graph implies

a v = O(m) matching the convergence rate O 1/
√ mt of a linear bandit algorithm using randomized sampling to pull mt edge-arms without (graphical) constraints. Moreover, we will see that the worst case graph implies that v = O m 2 .

Since we filled the gap between our constraint objective and the problem of best arm identification in linear bandits, thanks to Theorem 4.1 and 4.2, we are able to extend known results for best arm identification in linear bandits on the sample complexity and its associated lower bound.

Corollary 4.1 ([90], Theorem 1). If the G-allocation is implemented with the random strategy of Algorithm 8, resulting in an β t -approximation, then with probability at least 1 -δ, the best arm obtained with θt is z ⋆ and

t ≤ 128σ 2 d 2 (1 + β t ) log 6m 2 t 2 K 4 δπ m∆ 2 min , where ∆ min = min z∈Z\{z⋆} (z ⋆ -z) ⊤ θ ⋆ .
Moreover, let τ be the number of rounds sufficient for any algorithm to determine the best arm with probability at least 1 -δ. A lower bound on the expectation of τ can be obtained from the one derived for the problem of best arm identification in linear bandits (see e.g., Theorem 1 in [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF]):

E[τ ] ≥ min Γ ∈S Z max z∈Z\{z⋆} log 1 2.4δ 2σ 2 ∥z ⋆ -z∥ 2 Σ Z (Γ ) -1 m (z ⋆ -z) ⊤ θ ⋆ 2 .
As observed in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] this lower bound can be upper bounded, in the worst case, by 4σ 2 d 2 /(m∆ 2 min ) which matches our bound up to log terms and the relative error β t . Note, however, that since we borrow this lower bound from the standard linear bandit literature, it does not take into account the graph constraint, so it may never be reached.

Case where M ⋆ is not symmetric

Consider the relaxation of the assumption we made at the beginning of the chapter, namely that M ⋆ is symmetric. We now consider that M ⋆ is not necessarily symmetric. We recall here that in the graph G = (V, E) associated with the graphical bilinear bandits framework, (i, j) ∈ E if and only if (j, i) ∈ E. Therefore, for a given allocation (x (1) , . . . , x (n) ) ∈ X n , we can write the associated expected global reward as follows:

(i,j)∈E x (i)⊤ M ⋆ x (j) = n i=1 j∈N (i) j>i x (i)⊤ M ⋆ x (j) + x (j)⊤ M ⋆ x (i) = n i=1 j∈N (i) j>i x (i)⊤ M ⋆ x (j) + x (j)⊤ M ⋆ x (i) ⊤ = n i=1 j∈N (i) j>i x (i)⊤ M ⋆ x (j) + x (i)⊤ M ⊤ ⋆ x (j) = n i=1 j∈N (i) j>i x (i)⊤ M ⋆ x (j) + M ⊤ ⋆ x (j) = n i=1 j∈N (i) j>i x (i)⊤ M ⋆ + M ⊤ ⋆ x (j) . Let us denote M⋆ = M ⋆ + M ⊤ ⋆ . One can notice that M⋆ is symmetric. Consider the edge set Ẽ = {(i, j) ∈ E | i ∈ V, j ∈ V, j > i},
there are exactly m/2 edges in Ẽ and the objective of the central entity is to find, within a minimum number of rounds, the joint arm that maximises

(i,j)∈ Ẽ x (i)⊤ M⋆ x (j) . (4.4)
Every time the central entity chooses a joint arm (x

(1) t , . . . , x (n)
t ) at time t, it receives for each edge (i, j) ∈ Ẽ the rewards y

(i,j) t = x (i)⊤ t Mx (j) and y (j,i) t = x (j)⊤ t
Mx (i) . By summing y (i,j) t and y (j,i) , one has,

y (i,j) t + y (j,i) t = vec x (i) t x (j)⊤ t , vec (M ⋆ ) + z x (j) t x (i) t , vec (M ⋆ ) + η (i,j) t + η (j,i) t = vec x (i) t x (j)⊤ t , vec (M ⋆ ) + vec x (i) t x (j)⊤ t , vec M ⊤ ⋆ + η (i,j) t + η (j,i) t = vec x (i) t x (j)⊤ t , vec M ⋆ + M ⊤ ⋆ + η (i,j) t + η (j,i) t = vec x (i) t x (j)⊤ t , vec M⋆ + η (i,j) t + η (j,i) t = z (i,j) t , θ⋆ + η (i,j) t + η (j,i) t √ 2σ-sub-gaussian random variable
, where θ⋆ = vec M⋆ is the vectorized version of the true parameter matrix M⋆ .

Hence at each round t, and for each edge (i, j) ∈ Ẽ, the central entity aggregate the reward y (i,j) t and y (j,i) t to get a reward of the form ⟨z, θ⋆ ⟩ + η with z ∈ Z and where η is a √ 2σsubgaussian random variable, that can be used to compute an estimate θt of θ⋆ . Notice that the amount of rewards y

(i,j) t
obtained during a round is m, but since the central entity needs to sum y (i,j) t + y (j,i) t to compute θt , we thus consider only m/2 obtained information per round. Solving this best arm identification problem for the graphical bilinear bandits with a nonsymmetric matrix M ⋆ and an edge set E is exactly solving the best arm identification problem for graphical bilinear bandits with the symmetric matrix M⋆ and the edge set Ẽ. The solution of this problem is exactly what we proposed throughout the previous sections.

Influence of the graph structure on the convergence rate 4.3.1 Characterization of the variance associated with the randomized strategy

The convergence bound in Theorem 4.2 depends on v = ∥E (A 1 -EA 1 ) 2 ∥. In this section, we characterize the impact of the graph structure on this quantity and, by extension, on the convergence rate.

For i ∈ {1, . . . , n} and s ∈ {1, . . . , t}, let X (i)

s be i.i.d. random vectors in X such that for all x ∈ X , P X (1) 1
= x = γ ⋆ x . Each X (i)
s is to be viewed as the random arm pulled at round s for the node i. Hence one can write

A 1 = (i,j)∈E vec X (i) 1 X (j)⊤ 1 vec X (i) 1 X (j)⊤ 1 ⊤ . Let denote A (i,j) 1 = vec (X (i) 1 X (j)⊤ 1 ) vec (X (i) 1 X (j)⊤ 1 ) ⊤ such that A 1 = (i,j)∈E A (i,j) 1
and let define for any random matrices A and B the operators

Var(A) ≜ E (A -E[A]) 2 and Cov(A, B) ≜ E (A -E[A])(B -E[B]
) . We can derive the variance of A 1 as follows:

Var(A 1 ) = (i,j)∈E Var A (i,j) 1 + (i,j)∈E (k,l)∈E (k,l)̸ =(i,j) Cov A (i,j) 1 , A (k,l) 1
.

One can decompose the sum of the covariances into three groups: a first group where k ̸ = i, j and l ̸ = i, j which means that the two edges do not share any node and Cov(A

(i,j) 1 , A (k,l) 1
) = 0, and two other groups where the edges share at least one node. For all edges (i, j) ∈ E we consider either the edges (i, k) ∈ E where k ̸ = j, yielding Cov(A

(i,j) 1 , A (i,k) 1 ) or the edges (j, k) ∈ E, yielding Cov(A (i,j) 1 , A (j,k) 1
).

Hence, one has

Var(A 1 ) = (i,j)∈E Var A (i,j) 1 + n i=1 j∈N (i) k∈N (i) k̸ =j Cov A (i,j) 1 , A (i,k) 1 + n i=1 j∈N (i) k∈N (j) Cov A (i,j) 1 , A (j,k) 1
.

Let P ≥ 0 be such that for all (i, j) ∈ E, Var A (i,j) 1

⪯ P × I and M, N ≥ 0 such that for all (i, j) ∈ E:

∀k ∈ N (i), Cov A (i,j) 1 , A (i,k) 1 ⪯ M × I , ∀k ∈ N (j), Cov A (i,j) 1 , A (j,k) 1 ⪯ N × I .
We want to compare the quantity ∥Var(A 1 )∥ for different types of graphs: star, complete, circle and a matching graph. To have a fair comparison, we want graphs that reveal the same number of rewards at each round of the learning procedure. Hence, we denote respectively n S , n Co , n Ci and n M the number of nodes in a star, complete, circle and matching graph of m edges and get:

Star graph. We have,

Var(A 1 ) ⪯ m × P • I + (n S -1)(n S -2)M • I + n S (n S -1)N • I .
Since the star graph of m edges has a number of nodes n S = m/2 + 1, we have

∥Var(A 1 )∥ ≤ m × P + (M + N ) × O m 2 .

Complete graph. As for the star graph,

Var(A 1 ) ⪯ m × P • I + n Co (n Co -1)(n Co -2)M • I + n Co (n Co -1)(n Co -1)N • I .

Since the complete graph of m edges has a number of nodes n

Co = 1 + √ 4m + 1 /2, we have ∥Var(A 1 )∥ ≤ m × P + (M + N ) × O m √ m .
Circle graph. Again,

Var(A 1 ) ⪯ m × P • I + 2n Ci M • I + 4n Ci N • I .
Since the circle graph of m edges has a number of nodes n Ci = m/2, we have

∥Var(Z 1 )∥ ≤ m × P + (M + N ) × O(m) .
Matching graph. Finally,

Var(A 1 ) ⪯ m × P • I + n M N • I .
Since the matching graph of m edges has a number of nodes n M = m, we have

∥Var(A 1 )∥ ≤ m × P + m × N .
We thus obtain the bounds stated in Table 4.2.

Graph

Upper bound on ∥Var(A 1 )∥ These four examples evidence the strong dependency of the variance on the structure of the graph. The more independent the edges are (i.e., with no common nodes), the smaller the quantity ∥Var(A 1 )∥ is. For a fixed number of edges m, the best case is the matching graph where no edge share the same node and the worst case is the star graph where all the edges share a central node.

β t Star mP + (M + N )O m 2 O 1/ √ t Complete mP + (M + N )O(m √ m) O 1/ m 1 4 √ t Circle mP + (M + N )O(m) O 1/ √ mt Matching mP + mN O 1/ √ mt

Experimental results validating the dependence on the graph

In this section, we consider the modified version of a standard experiment introduced by [START_REF] Soare | Best-arm identification in linear bandits[END_REF] and used in most papers on best arm identification in linear bandits [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF][START_REF] Tao | Best Arm Identification in Linear Bandits with Linear Dimension Dependency[END_REF]106,109] to evaluate the sample complexity of our algorithm on different graphs. We consider d+1 node-arms in X ⊂ R d where d ≥ 2. This node-arm set is made of the d vectors (e 1 , . . . , e d ) forming the canonical basis of R d and one additional arm x d+1 = (cos(ω), sin(ω), 0, . . . , 0) ⊤ with ω ∈]0, π/2]. Note that by construction, the edge-arm set Z contains the canonical basis

(e ′ 1 , . . . , e ′ d 2 ) of R d 2 .
The parameter matrix M ⋆ has its first coordinate equals to 2 and the others equal to 0 which makes

θ ⋆ = vec (M ⋆ ) = (2, 0, . . . , 0) ⊤ ∈ R d 2
. The best edge-arm is thus z ⋆ = z (1,1) = e ′ 1 . One can note that when ω tends to 0, it is harder to differentiate this arm from z (d+1,d+1) = vec x (d+1) x ⊤ (d+1) than from the other arms. We set η (i,j) t

∼ N (0, 1), for all edges (i, j) and round t.

We consider two cases where ω = 0.1 which makes the edge-arms z (1,1) and z (d+1,d+1) difficult to differentiate, and ω = π/2 which makes the edge-arm z (1,1) easily identifiable as the optimal edge-arm. For each of these two cases, we evaluate the influence of the graph structure, the number of edges m and the edge-arm space dimension d 2 on the sampling complexity. Results are shown in Figure 4.3.

When ω = 0.1, the type of the graph does not impact the number of rounds needed to verify the stopping condition. This is mainly due to the fact that the magnitude of its associated variance is negligible with respect to the number of rounds. Hence, even if we vary the number of edges or the dimension, we get the same performance for any type of graph including the matching graph. This implies that our algorithm performs as well as a linear bandit that draws m edgearms in parallel at each round. When ω = π/2, the number of rounds needed to verify the stopping condition is smaller and the magnitude of the variance is no longer negligible. Indeed, when the number of edges or the dimension increases, we notice that the star graph takes more times to satisfy the stopping condition. Moreover, note that the sample complexities obtained for the circle and the matching graph are similar. This observation is in line with the dependency on the variance shown in Table 4.2.

Conclusion & Perspectives

We provided an algorithm based on the G-allocation strategy that uses randomized sampling over the nodes to return a good estimate M that can be used instead of M ⋆ to identify the couple (x ⋆ , x ′ ⋆ ). Moreover, we highlighted the impact of the graph structure on the convergence rate of our algorithm and validated our theoretical results with experiments.

While the estimate M is constructed in order to identify the couple (x ⋆ , x ′ ⋆ ) that is used in algoriothm 5 and gives a 1+ξ 2 -approximation solution, a perspective of improvement can be to construct the estimate M that identifies with high probability the couple (x ⋆ , x′ ⋆ ) that is used in Algorithm 6 that gives the better 1+ξ 2 + ϵ -approximation guarantee. We let this extension for future work. 1 Moreover, in this chapter we based our algorithm on the G-allocation strategy that minimises max zZ 2∥z∥ A -1

t that is upperbound of max (z,z ′ )Z 2 ∥z -z ′ ∥ A -1 t
, objective of XYallocation strategy. An algorithm based on the XY -allocation represents a promising extensions for our model. In this chapter, we also assume that we do not know the parameter matrix M ⋆ and as described in the problem setting in Section 1.2, a central entity faces the graphical bilinear bandits problem where at each round it chooses an arm for each node of the graph and observes a bilinear reward for each edge of the graph. The objective of the central entity is the following:

Objective: Design an algorithm that maximizes the expected cumulative global reward obtained during T rounds T t=1

(i,j)∈E x (i)⊤ t M ⋆ x (j) t .
We will naturally rely on some ideas and results presented in Chapter 3 where the matrix was assumed to be known by the central entity. We follow the notations we established in section 1.2.1.

Optimism in the face of uncertainty for graphical bilinear bandits

Preliminaries

Let us recall that maximizing the cumulative rewards is equivalent to minimizing its associated regret. We thus define the global pseudo-regret over T rounds as follows:

R(T ) = T t=1   (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ - (i,j)∈E x (i)⊤ t M ⋆ x (j) t   .

55

We recall that the objective of the learner is to have a pseudo-regret R(T ), such that lim

T →∞ R(T ) T = 0
We know from Theorem 3.1 that finding the best joint arm

x

⋆ , . . . , x

(n) ⋆ = arg max (x (1) ,...,x (n) )∈X (i,j)∈E x (i)⊤ M ⋆ x (j)
is NP-Hard with respect to the number of agents n. We extend this result in the following corollary.

Corollary 5.1. There does not exist a polynomial time algorithm in

n such that lim T →∞ R(T ) T = 0 , (5.1)
for any instance of the graphical bilinear bandits described in Section 1.2, unless P = N P .

Proof. Suppose that there exists a polynomial time algorithm in n such that lim T →∞ R(T ) T = 0 for any instance of the graphical bilinear bandits described in Section 1.2. In particular consider the class of instances where the graph G is of degree 3 (i.e., each agent has 3 neighbors), where X = {e 0 , e 1 } is the canonical basis in R 2 and the parameter matrix

M ⋆ = 0 1 1 0 . The pseudo-regret is such that ∃f : R → R with lim T →∞ f (T ) = 0, such that R(T ) = T × f (T )
with a computational time in poly(n) per iteration.

Let us consider the best case scenario for the learner and assume that the matrix M ⋆ is known.

Then, for t = arg max t=1,...,T (i,j)∈E x

(i)⊤ t M ⋆ x (j)⊤ t
we have

T (i,j)∈E x (i)⊤ t M ⋆ x (j)⊤ t ≥ T t=1 (i,j)∈E x (i)⊤ t M ⋆ x (j)⊤ t ≥ T (i,j)∈E
which gives,

(i,j)∈E x (i)⊤ t M ⋆ x (j)⊤ t ≥ (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j)⊤ ⋆ -f (T ) .
As T → ∞, f (T ) → 0, hence consider the cases where

f (T ) ≤ c × (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j)⊤ ⋆ , for a certain constant c > 0.
It gives that

(i,j)∈E x (i)⊤ t M ⋆ x (j)⊤ t ≥ (i,j)∈E x M ⋆ x (j)⊤ ⋆ , (5.2)
with a computational time in poly(n) per iteration. However, we know from the proof of Theorem 3.1 that solving the optimization problem max (x (1) ,...,x (n) ) (i,j)∈E x (i)⊤ M ⋆ x (j)⊤ is the same as solving the max-cut problem. Moreover, we know from Theorem 1 in [START_REF] Berman | On some tighter inapproximability results[END_REF] that there does not exist a polynomial time algorithm in the number of nodes that has an approximation ratio better than 330 331 ≈ 0.997 of the optimal solution of the max-cut problem for any graphs of degree 3, unless P = N P . By taking c = 0.002, Eq. (5.2) gives an approximation ratio of 0.998. This concludes the proof.

Hence, the objective of designing an algorithm with a sublinear regret in T is not feasible in polynomial time with respect to n. However, some NP-hard problems are α-approximable (for some α ∈ (0, 1]), which means that there exists a polynomial-time algorithm guaranteed to produce solutions whose values are at least α times the value of the optimal solution. We refer the reader to chapter 3 for more information on approximating the optimal solution of our problem. For these kind of problems, it makes sense to consider the α-pseudo-regret as in [START_REF] Chen | Combinatorial multi-armed bandit: General framework and applications[END_REF][START_REF] Kakade | Playing games with approximation algorithms[END_REF] which is defined for all α ∈ (0, 1] as follows

R α (T ) ≜ T t=1   α (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ - (i,j)∈E x (i)⊤ t M ⋆ x (j) t   ,
and set the objective of designing an algorithm with a sublinear α-regret.

57

Finally, as we did in chapter 4, let us recall that the reward obtained for each edge of the graph at each round can be seen as a noisy linear reward in higher dimension [START_REF] Jun | Bilinear Bandits with Low-rank Structure[END_REF] with

y (i,j) t = vec x (i) t x (j)⊤ t , vec (M ⋆ ) + η (i,j) t .
To simplify the notation, let us refer to any x ∈ X as a node-arm. Let us use the notation z xx ′ ≜ vec xx ′⊤ , and define the arm set Z = {z xx ′ |(x, x ′ ) ∈ X 2 } where any z ∈ Z will be referred as an edge-arm. If the arm x (i) t ∈ X represents the node-arm allocated to the node i ∈ V at time t, for each edge (i, j) ∈ E we will denote the associated edge-arm by z

(i,j) t ≜ vec x (i) t x (j)⊤ t
∈ Z and define θ ⋆ = vec (M ⋆ ) the vectorized version of the unknown matrix M ⋆ . With those notations, the (now) linear reward can be rewritten as follows:

y (i,j) t = z (i,j) t , θ ⋆ + η (i,j) t .
(5.3) Assumption 5.1 (Bounded edge-arm norm and parameter norm). We consider that there exists L > 0, for all edge-arm z ∈ Z, such that ∥z∥ 2 ≤ L. Moreover, for some S > 0, the norm of the true parameter θ ⋆ is such that ∥θ ⋆ ∥ 2 ≤ S.

Assumption 5.2 (Positive and bounded rewards). We consider that for any z ∈ Z, the associated expected reward ⟨z,

θ ⋆ ⟩ is such that 0 ≤ ⟨z, θ ⋆ ⟩ ≤ LS
In this chapter, we choose to design an algorithm based on the principle of optimism in the face of uncertainty [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF], and in the case of a linear reward [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF][START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF], we need to maintain an estimator of the true parameter θ ⋆ . To do so, let us define for all rounds t ∈ {1, . . . , T } the OLS estimator of θ ⋆ as follows:

θt = A -1 t b t , (5.4)
where,

A t = λI d 2 + t s=1 (i,j)∈E z (i,j) s z (i,j)⊤ s ,
with λ > 0 a regularization parameter and

b t = t s=1 (i,j)∈E z (i,j) s y (i,j) s .
We define also the confidence set

C t (δ) = θ : ∥θ -θt ∥ A -1 t ≤ σ d 2 log 1 + tmL 2 /λ δ + √ λS ,
where with probability 1 -δ, we have that θ ⋆ ∈ C t (δ) for all t ∈ {1, . . . , T }, and δ ∈ (0, 1].

Algorithm and analysis of the regret

In chapter 3, we presented two algorithms (Algorithm 5 and 6) that use the true parameter matrix M ⋆ to return an allocation of arm that achieve an α-approximation solution of maximizing the global reward. Naturally, since we do not have access to M ⋆ , we cannot use it directly at each iteration to maximise the cumulative global reward (and thus minimize the associated regret).

Nevertheless, one can use the constructed estimator θt and the principal of optimism in the face of uncertainty to overcome the fact that M ⋆ is unknown. Indeed, we recall that in Algorithm 5, the couple (x ⋆ , x ′ ⋆ ) is chosen as follows,

(x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ (5.5) = arg max (x,x ′ )∈X 2 ⟨z xx ′ + z x ′ x , θ ⋆ ⟩ , (5.6)
and is used to create as much as possible edge arms of the form z xx ′ and z x ′ x in the graph. Here instead, at each round t, the central entity chooses optimistically the couple (x t , x ′ t ) as follows,

(x t , x ′ t ) = arg max (x,x ′ )∈X 2 max θ∈C t-1 (δ) ⟨z xx ′ + z x ′ x , θ⟩ ,
and then allocates the node-arms to maximize the number of locally-optimal edge-arms z xtx ′ t and z x ′ t xt . The method is presented in Algorithm 9 Theorem 5.1. Given the Graphical Bilinear Bandits problem defined in Section 1.2.1, let 0 ≤ ξ ≤ 1 be a problem-dependent parameter defined by

ξ = min x∈X ⟨z xx , θ ⋆ ⟩ 1 m (i,j)∈E z (i,j) ⋆ , θ ⋆ ≥ 0 , and set α = 1+ξ 2 , then the α-regret of Algorithm 9 satisfies R α (T ) ≤ Õ σd 2 + S √ λ T m max(2, (LS) 2 ) + LSm d 2 log 2 T mL 2 /λ δ ,
where Õ hides the logarithmic factors.

Algorithm 9: Adaptation of OFUL algorithm for Graphical Bilinear Bandits

Input : graph G = (V, E), node-arm set X (V 1 , V 2 ) = Approx-MAX-CUT(G) for t = 1 to T do // Find the optimistic best couple of node-arms x t , x ′ t , θt-1 = arg max (x,x ′ ,θ)∈X 2 ×C t-1 ⟨z xx ′ + z x ′ x , θ⟩; // Allocate x t and x ′ t in the graph x (i) t = x t for all i in V 1 ; x (i) t = x ′
t for all i in V 2 ; Obtain for all (i, j) in E the rewards y (i,j) t ; Compute θt as in (5.4) end return θt Proof. To properly derive the regret bounds, we will have to make connections between our setting and a standard linear bandit that chooses sequentially T m arms. For that matter, let us consider an arbitrary order on the set of edges E and denote E[i] the i-th edge according to this order with i ∈ {1, . . . , m}. We define for all t ∈ {1, . . . , T } and p ∈ {1, . . . , m} the OLS estimator

θt,p = A -1 t,p b t,p ,
where

A t,p = λI d 2 + t-1 s=1 m b=1 z E[b] s z E[b]⊤ s + p k=1 z E[k] t z E[k]⊤ t
, with λ a regularization parameter and

b t,p = t-1 s=1 m b=1 z E[b] s y E[b] s + p k=1 z E[k] t y E[k] t . (5.7)
We define also the confidence set

C t,p (δ) = θ : ∥θ -θt,p ∥ A -1 t,p ≤ σ d 2 log 1 + tmL 2 /λ δ + √ λS , (5.8)
where with probability 1 -δ, we have that θ ⋆ ∈ C t,p (δ) for all t ∈ {1, . . . , T }, p ∈ {1, . . . , m} and δ ∈ (0, 1].

Notice that the confidence set C t (δ) defined in Section 5.1.1 is exactly the confidence set C t,m (δ) defined here. The definitions of the matrix A t,m and the vector b t,m follow the same reasoning.

Recall that (x

(1) ⋆ , . . . , x (n) ⋆ ) = arg max (x (1) ,...,x (n) ) (i,j)∈E x (i)⊤ M ⋆ x (j)
is the optimal joint arm, and we define for each edge (i, j) ∈ E the optimal edge arm z

(i,j) ⋆ = vec (x (i) ⋆ x (j)⊤ ⋆ ).
Let us borrow the notion of Critical Covariance Inequality introduced in [START_REF] Chan | Parallelizing contextual linear bandits[END_REF], that is for a given round t ∈ {1, . . . , T } and p ∈ {1, . . . , m}, the expected covariance matrix A t,p satisfies the critical covariance inequality if

A t-1,m ≼ A t,p ≼ 2A t-1,m .
(5.9)

Let us now define the event D t as the event where at a given round t, for all p ∈ {1, . . . , m}, A t,p satisfies the critical covariance inequality (CCI).

We can write the pseudo-regret as follows:

R(T ) = T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ + T t=1 1[D c t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ ≤ T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ (a) + LSm T t=1 1[D c t ] (b)
.

We know that the approximation Max-CUT algorithm returns two subsets of nodes V 1 and V 2 such that there are at least m/2 edges between V 1 and V 2 , and to be more precise: at least m/4 edges from V 1 to V 2 and at least m/4 edges from V 2 to V 1 . Hence at each time t, if all the nodes of V 1 pull the node-arm x t and all the nodes of V 2 pull the node-arm x ′ t , we can derive the term (a) as follows:

(a) = T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -⟨z (i,j) t , θ ⋆ ⟩ = T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -1[i ∈ V 1 ∧ j ∈ V 2 ]⟨z (i,j) t , θ ⋆ ⟩ -1[i ∈ V 2 ∧ j ∈ V 1 ]⟨z (i,j) t , θ ⋆ ⟩ 61 -1[i ∈ V 1 ∧ j ∈ V 1 ]⟨z (i,j) t , θ ⋆ ⟩ -1[i ∈ V 2 ∧ j ∈ V 2 ]⟨z (i,j) t , θ ⋆ ⟩ . Notice that (i,j)∈E z (i,j) ⋆ = (i,j)∈E 1 m (k,l)∈E z (k,l) ⋆
, so one has

(a) = T t=1 1[D t ] (i,j)∈E 1[i ∈ V 1 ∧ j ∈ V 2 ]   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -⟨z (i,j) t , θ ⋆ ⟩   (a 1 ) + T t=1 1[D t ] (i,j)∈E 1[i ∈ V 2 ∧ j ∈ V 1 ]   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -⟨z (i,j) t , θ ⋆ ⟩   (a 2 ) + T t=1 1[D t ] (i,j)∈E 1[i ∈ V 1 ∧ j ∈ V 1 ]   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -⟨z (i,j) t , θ ⋆ ⟩   (a 3 ) + T t=1 1[D t ] (i,j)∈E 1[i ∈ V 2 ∧ j ∈ V 2 ]   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -⟨z (i,j) t , θ ⋆ ⟩   (a 4 )
.

Let us analyse the first term. By using the fact that 1[D t ] ≤ 1, we have

(a 1 ) = T t=1 n i=1 j∈N i j>i 1[i ∈ V 1 ∧ j ∈ V 2 ] 2 m (k,l)∈E z (k,l) ⋆ -z (i,j) t + z (j,i) t , θ ⋆ . By defining (x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X 2 ⟨z xx ′ + z x ′
x , θ ⋆ ⟩, and noticing that in the case where a node i is in V 1 and a neighbouring node j in is V 2 , then z

(i,j) t = z xtx ′ t , we have, 2 m (k,l)∈E z (k,l) ⋆ , θ ⋆ = 2 m n k=1 j∈N k j>k z (k,l) ⋆ + z (l,k) ⋆ , θ ⋆ ≤ 2 m n k=1 j∈N k j>k z x⋆x ′ ⋆ + z x ′ ⋆ x⋆ , θ ⋆ = ⟨z x⋆x ′ ⋆ + z x ′ ⋆ x⋆ , θ ⋆ ⟩ ≤ ⟨z xtx ′ t + z x ′ t xt , θt-1,m ⟩ (optimistic reward) = ⟨z (i,j) t + z (j,i) t , θt-1,m ⟩ .
Plugging this last inequality yields, with probability 1 -δ,

(a 1 ) ≤ T t=1 n i=1 j∈N i j>i 1[i ∈ V 1 ∧ j ∈ V 2 ] z (i,j) t + z (j,i) t , θt-1,m -θ ⋆ = T t=1 (i,j)∈E 1[i ∈ V 1 ∧ j ∈ V 2 ] z (i,j) t , θt-1,m -θ ⋆ .
We define, as in Algorithm 9, 1 z

(i,j) t = z xtx ′ t ≜ 1[i ∈ V 1 ∧ j ∈ V 2 ].
Then, one has, with probability 1 -δ,

(a 1 ) ≤ T t=1 (i,j)∈E 1 z (i,j) t = z xtx ′ t z (i,j) t , θt-1,m -θ ⋆ = T t=1 m k=1 1 z E[k] t = z xtx ′ t z E[k] t , θt-1,m -θ ⋆ = T t=1 m k=1 1 z E[k] t = z xtx ′ t z E[k] t , θt-1,m -θt-1,m + z E[k] t , θt-1,m -θ ⋆ ≤ T t=1 m k=1 1 z E[k] t = z xtx ′ t ∥z E[k] t ∥ A -1 t,k-1 ∥ θt-1,m -θt-1,m ∥ A t,k-1 + 1 z E[k] t = z xtx ′ t ∥z E[k] t ∥ A -1 t,k-1 ∥ θt-1,m -θ ⋆ ∥ A t,k-1 ≤ T t=1 m k=1 1 z E[k] t = z xtx ′ t ∥z E[k] t ∥ A -1 t,k-1 √ 2∥ θt-1,m -θt-1,m ∥ A t-1,m (5.10)
+ 1 z E[k] t = z xtx ′ t ∥z E[k] t ∥ A -1 t,k-1 √ 2∥ θt-1,m -θ ⋆ ∥ A t-1,m ≤ T t=1 m k=1 1 z E[k] t = z xtx ′ t 2 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 (5.11) ≤ T t=1 m k=1 2 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 , (5.12)
with β t (δ) ≤ σ d 2 log 1+tmL 2 /λ δ + √ λS and where (5.10) uses the critical covariance inequality (5.9), (5.11) comes from the definition of the confidence set C t-1,m (δ) (5.8) and (5.12) upper bounds the indicator functions by 1.

Using a similar reasoning, we obtain the same bound for (a 2 ):

(a 2 ) ≤ T t=1 m k=1 2 2β t (δ)∥z E[k] t ∥ A -1 t,k-1
.

Let us bound the terms (a 3 ) and (a 4 ).

(a 3 ) ≤ T t=1 (i,j)∈E 1 z (i,j) t = z xtxt   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -⟨z (i,j) t , θ ⋆ ⟩   .
For all x ∈ X , let ξ x be the following ratio

ξ x = ⟨z xx , θ ⋆ ⟩ 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ , (5.13)
and let ξ be the worst ratio

ξ = min x∈X ⟨z xx , θ ⋆ ⟩ 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ . ( 5.14)
We have

T t=1 (i,j)∈E 1 z (i,j) t = z xtxt   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -ξ xt 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆   ≤ T t=1 (i,j)∈E 1 z (i,j) t = z xtxt   1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ -ξ 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆   = T t=1 (i,j)∈E 1 z (i,j) t = z xtxt (1 -ξ) 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ ≤T m 4 (1 -ξ) 1 m (k,l)∈E z (k,l) ⋆ , θ ⋆ (5.15) = T t=1 (i,j)∈E 1 4 (1 -ξ) z (i,j) ⋆ , θ ⋆ ,
where (5.15) comes from the fact that there is at most m/4 edges that goes from node in

V 1 to other nodes in V 1 .
The derivation of this bound for (a 3 ) gives the same one for (a 4 )

(a 4 ) ≤ T t=1 (i,j)∈E 1 4 (1 -ξ) z (i,j) ⋆ , θ ⋆ .
By rewriting (a), we have :

(a) ≤ T t=1 m k=1 4 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 + 1 2 (1 -ξ)⟨z (i,j) ⋆ , θ ⋆ ⟩ .
In [START_REF] Chan | Parallelizing contextual linear bandits[END_REF], they bounded the term (b) as follows

LSm T t=1 1[D c t ] ≤ LSm d 2 log 2 T mL 2 /λ δ . (5.16)
We thus have the regret bounded by

R(T ) ≤ T t=1 m k=1 4 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 + 1 2 (1 -ξ)⟨z (i,j) ⋆ , θ ⋆ ⟩ + LSm d 2 log 2 T mL 2 /λ δ , which gives us R 1+ξ 2 (T ) ≤ T t=1 m k=1 4 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 + LSm d 2 log 2 T mL 2 /λ δ .
Let us bound the first term with the double sum as it is done in [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF][START_REF] Chan | Parallelizing contextual linear bandits[END_REF]:

T t=1 m k=1 4 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 ≤ T t=1 m k=1 min 2LS, 4 2β t (δ)∥z E[k] t ∥ A -1 t,k-1 ≤ T t=1 m k=1 4 2β t (δ) min LS, ∥z E[k] t ∥ A -1 t,k-1 ≤ T m × 32β T (δ) T t=1 m k=1 min (LS) 2 , ∥z E[k] t ∥ 2 A -1 t,k-1 ≤ 32T mβ T (δ) T t=1 m k=1 max(2, (LS) 2 ) log 1 + ∥z E[k] t ∥ 2 A -1 t,k-1
(5.17)

= 32T mβ T (δ) max(2, (LS) 2 ) T t=1 m k=1 log 1 + ∥z E[k] t ∥ 2 A -1 t,k-1 ≤ 32T mβ T (δ) max(2, (LS) 2 )d 2 log 1 + T mL 2 /λ d 2 (5.18) ≤ 32T md 2 max(2, (LS) 2 ) log 1 + T mL 2 /λ d 2 β T (δ) ,
where (5.17) uses the fact that for all a, x ≥ 0, min(a, x) ≤ max(2, a) log(1 + x),

(5.18) uses the fact that T t=1 m k=1 log 1 + ∥z

E[k] t ∥ 2 A -1 t,k-1 ≤ d 2 log 1 + T mL 2 /λ d 2
from Lemma 19.4 in [START_REF] Lattimore | Bandit Algorithms[END_REF].

We get the final bound by noticing that

β T (δ) ≤ σ d 2 log 1 + T mL 2 /λ δ + √ λS .
One can notice that the first term of the regret-bound matches the one of a standard linear bandit that pulls sequentially T m edge-arms. The second term captures the cost of parallelizing m draws of edge-arms per round. Indeed, the intuition behind this term is that the couple (x t , x ′ t ) chosen at round t is relevant to pull the (tm + 1)-th edge-arm but not necessarily the other (m -1) edge-arms that are pulled in parallel. This is because the reward associated with the (tm + 1)-th edge-arms could have led to change the central entity choice if it had been done sequentially. In [START_REF] Chan | Parallelizing contextual linear bandits[END_REF], they characterize this phenomenon and show that this potential change in choice occurs less and less often as we pull arms and get rewards, hence the dependence in O(log(T m))).

Improved algorithm and analysis of the regret

We address the problem of designing an improved version of the proposed algorithm using the idea presented in Algorithm 6 that improves the approximation ratio.

Let us recall that in the previous section in Algorithm 9, the central entity chooses the couple (x t , x ′ t ) such that

(x t , x ′ t ) = arg max (x,x ′ )∈X 2 max θ∈C t-1 (δ) ⟨z xx ′ + z x ′ x , θ⟩ ,
which maximize the optimistic reward obtained between two nodes if the central entity were able to allocate x t to one node and x ′ t to the second. This strategy being optimal locally but complicated by considering the dependencies between edges, the central entity could take into consideration the edge arms of the form z xx and z x ′ x ′ that are created when allocating the graph node using only two node-arm x and x ′ . This idea follows the one presented in Algorithm 6 where we recall with different notations that the couple (x ⋆ , x′ ⋆ ) chosen to allocate the nodes of the graph are such that

(x ⋆ , x′ ⋆ ) = arg max (x,x ′ )∈X 2 ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x + m 1 z xx + m 2 z x ′ x ′ , θ ⋆ ⟩ .
As in the previous section, we do not have access to θ ⋆ , we use the principle of optimism to find at each round the couple (x t , x′ t ) such that

(x t , x′ t ) = arg max (x,x ′ )∈X 2 max θ∈C t-1 (δ) ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x = m 1 z xx + m 2 z x ′ x ′ , θ⟩ . (5.19)
Here, instead of maximizing the local reward one can get between two nodes, the central entity maximizes the global optimistic reward that one would obtain when allocating only two arms (x, x ′ ) ∈ X 2 in the graph. This strategy is described in Algorithm 10.

Before stating the guarantees on the α-regret, we recall that we defined in Chapter 3 the quantity ∆ ≥ 0 to be the expected reward difference of allocating (x ⋆ , x′ ⋆ ) instead of (x ⋆ , x ′ ⋆ ),

∆ = ⟨m 1→2 z x⋆ x′ ⋆ -z x⋆x ′ ⋆ + m 2→1 z x′ ⋆ x⋆ -z x ′ ⋆ x⋆ + m 1 (z x⋆ x⋆ -z x⋆x⋆ ) + m 2 z x′ ⋆ x′ ⋆ -z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ .
The new guarantees that we get on the α-regret of Algorithm 10 are stated in the following theorem. Theorem 5.2. Given the Graphical Bilinear Bandits problem defined as in Section 1.2, let ξ be defined as in Theorem 5.1, let 0 ≤ ϵ ≤ 1 2 be a problem dependent parameter that measures the gain of optimizing on the suboptimal and unwanted arms defined as:

ϵ = ∆ (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩
, and set α = 1+ξ 2 + ϵ where α ≥ 1/2 by construction, then the α-regret of Algorithm 10 satisfies 67 Algorithm 10: Improved OFUL for Graphical Bilinear Bandits

Input : graph G = (V, E), node-arm set X (V 1 , V 2 ) = Approx-MAX-CUT(G); m 1→2 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 2 }|; m 2→1 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 1 }|; m 1 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 1 }|; m 2 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 2 }|; for t = 1 to T do xt , x′ t , θt-1 = arg max (x,x ′ ,θ)∈X 2 ×C t-1 ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x + m 1 • z xx + m 2 • z x ′ x ′ , θ⟩; x (i) t = xt for all i in V 1 ; x (i) t = x′
t for all i in V 2 ; Obtain for all (i, j) in E the rewards y (i,j) t ; Compute θt as in (5.4)

end return θt R α (T ) ≤ Õ σd 2 + S √ λ T m max(2, (LS) 2 ) + LSm d 2 log 2 T mL 2 /λ δ ,
where Õ hides the logarithmic factors.

Proof. We can write the regret R(T ) as in the proof of Theorem 5.1:

R(T ) = T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ + T t=1 1[D c t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ ≤ T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -z (i,j) t , θ ⋆ (a) + LSm T t=1 1[D c t ] (b)
.

Here, (b) doesn't change, we thus only focus on deriving (a).

(a) = T t=1 1[D t ] (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -⟨z (i,j) t , θ ⋆ ⟩ ≤ T t=1 (i,j)∈E ⟨z (i,j) ⋆ , θ ⋆ ⟩ -⟨z (i,j) t , θ ⋆ ⟩ (where 1[D t ] ≤ 1) = T t=1 (i,j)∈E m 1→2 + m 2→1 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ (a 1 ) + T t=1 (i,j)∈E m 1 + m 2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ - T t=1 (i,j)∈E ⟨z (i,j) t , θ ⋆ ⟩ .
We have

(a 1 ) = T t=1 (i,j)∈E 2m 1→2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ = T t=1 n i=1 j∈N i j>i 2m 1→2 m ⟨z (i,j) ⋆ + z (j,i) ⋆ , θ ⋆ ⟩ ≤ T t=1 n i=1 j∈N i j>i 2m 1→2 m ⟨z x⋆x ′ ⋆ + z x ′ ⋆ x⋆ , θ ⋆ ⟩ = T t=1 n i=1 j∈N i j>i 2 m ⟨m 1→2 • z x⋆x ′ ⋆ + m 2→1 • z x ′ ⋆ x⋆ , θ ⋆ ⟩ = T t=1 n i=1 j∈N i j>i 2 m ⟨m 1→2 • z x⋆x ′ ⋆ + m 2→1 • z x ′ ⋆ x⋆ + m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ - T t=1 n i=1 j∈N i j>i 2 m ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ = T t=1 n i=1 j∈N i j>i - T t=1 n i=1 j∈N i j>i 2 m ∆ - T t=1 n i=1 j∈N i j>i 2 m ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ = T t=1 ⟨m 1→2 • z x⋆ x′ ⋆ + m 2→1 • z x′ ⋆ x⋆ + m 1 • z x⋆ x⋆ + m 2 • z x′ ⋆ x′ ⋆ , θ ⋆ ⟩ -∆ - T t=1 ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ ≤ T t=1 m 1→2 • z xt x′ t + m 2→1 • z x′ t xt + m 1 • z xt xt + m 2 • z x′ t x′ t , θt-1,m -∆ - T t=1 ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ (w.p 1 -δ) = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m ⟩ - T t=1 ∆ - T t=1 ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ .
By plugging the last upper bound in (a) and with probability 1 -δ, we have,

(a) ≤ T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m ⟩ - T t=1 ∆ - T t=1 ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ - T t=1 (i,j)∈E ⟨z (i,j) t , θ ⋆ ⟩ = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ - T t=1 ∆ - T t=1 ⟨m 1 • z x⋆x⋆ + m 2 • z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ - T t=1 ∆ - T t=1 (i,j)∈E m 1 m ξ x⋆ ⟨z (i,j) ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 2 m ξ x ′ ⋆ ⟨z (i,j) ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ ≤ T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ - T t=1 ∆ - T t=1 (i,j)∈E m 1 + m 2 m ξ⟨z (i,j) ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m ⟨z (i,j) ⋆ , θ ⋆ ⟩ = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ - T t=1 ∆ + T t=1 (i,j)∈E m 1 + m 2 m (1 -ξ)⟨z (i,j) ⋆ , θ ⋆ ⟩ = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ - T t=1 (i,j)∈E ϵ⟨z (i,j) ⋆ , θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m (1 -ξ)⟨z (i,j) ⋆ , θ ⋆ ⟩ = T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m (1 -ξ) -ϵ ⟨z (i,j) ⋆ , θ ⋆ ⟩ .
By plugging (a) in the regret and with probability 1 -δ, we have,

R(T ) ≤ T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ + T t=1 (i,j)∈E m 1 + m 2 m (1 -ξ) -ϵ ⟨z (i,j) ⋆ , θ ⋆ ⟩ + LSm T t=1 1[D c t ] ,
which gives,

R(T ) - T t=1 (i,j)∈E m 1 + m 2 m (1 -ξ) -ϵ ⟨z (i,j) ⋆ , θ ⋆ ⟩ ≤ T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ + LSm T t=1 1[D c t ] .
Thus,

R 1-m 1 +m 2 m (1-ξ)-ϵ (T ) ≤ T t=1 (i,j)∈E ⟨z (i,j) t , θt-1,m -θ ⋆ ⟩ + LSm T t=1 1[D c t ] .
The upper bound of the right hand terms follows exactly what we have already done for Theorem 5.1 by applying the upper bounds (5.12) and (5.16).

Moreover,

1 - m 1 + m 2 m (1 -ξ) -ϵ ≥ 1 - (1 -ξ) 2 -ϵ = (1 + ξ) 2 + ϵ .
Here one can see that the improvement happens in the α of the α-regret. In the next section, we confirm this results through experiments.

Numerical experiments

In Chapter 3, we run experiments on α and how it can varying according to the graph and the problem parameters. In this subsection, we only focus on the impact on the regret. We design an experiment that compares in practice the performance of Algorithm 9 and Algorithm 10 with the Explore-Then-Commit (ETC) algorithm by using the exploration strategy designed in Chapter 4 during the exploration phase, and by allocating the nodes in V 1 and V 2 with the best estimated couple (x, x ′ ) = arg max (x,x ′ ) ⟨z xx ′ + z x ′ x , θt ⟩ during the commit phase. However, since the algorithms that we presented in this section have guarantees on α-regrets with different α, we plot the fraction of the optimal global reward for each iteration. As in Chapter 3, we observe a clear improvement when choosing at each round t the couple of arms (x t , x′ t ) instead of (x t , x ′ t ).

Conclusion and perspectives

In this chapter, we presented the first regret-based algorithms for the stochastic graphical bilinear bandits problem with guarantees on the α-regret with α ≥ 1/2. We also showed experimentally that our algorithm achieves a better performance than Explore-Then-Exploit on our synthetic datasets. One could also study this problem in the adversarial setting, in particular adapting adversarial linear bandit algorithms to our case. Finally, our setting could be extended to the case where each agent has its own reward matrix.

Conclusion & perspectives 6.1 Summary of the results

In this thesis we introduced a new model that we named Graphical Bilinear Bandits that models centralized multi-agent problems where pairwise interactions exist between agents.

• In Chapter 3, we have highlighted the fact that the learner faced an underlying optimization problem that is NP-Hard no matter which goal the learner wanted to reach. Hence we have proposed an α-approximation algorithm with α ≥ 1/2 which only required to find the couple of arms (x ⋆ , x ′ ⋆ ) to return the α-approximate solution. Then we have refined this approximation-parameter with respect to problem dependent parameters based on the graph structure and a property of the parameter matrix M ⋆ .

• In Chapter 4, given the α-approximation algorithm designed in Chapter 3, we have presented a pure-exploration algorithm that allowed the learner to construct an estimate M that was statistically efficient in terms of optimal design. Indeed, the problem of finding within a minimum number of rounds the best couple (x ⋆ , x ′ ⋆ ) used in the α-approximation algorithm came down to finding the G-optimal design, also called G-allocation in the bandit literature. Solving this problem in the graphical bilinear bandits implied dealing with an additional constraint. That was why we have presented an algorithm that respected this constraint and that used randomized sampling to construct the estimate M. Our theoretical results have revealed a term that depended on the graph structure, so we showed the impact of the graph in our results.

• Finally, in Chapter 5, we have capitalized on the α-approximation algorithm given in chapter 3 and applied the principle of optimism in the face of uncertainty to design regret-based algorithms that achieved a sublinear α-regret in T where α ≥ 1/2. Furthermore, we have presented experimentally the performance of the proposed algorithms and used compared with an Explore-Then-Commit algorithm relying on the pure-exploration algorithm presented in chapter 4.

Perspective and future works

This thesis aimed to introduce the new graphical bilinear bandit setting and to provide the first solutions to common problems posed in the bandit literature. A lot of other approaches and modifications can be considered. We state two of them in the following.

Different parameter matrices M

(i,j) ⋆

for each edge (i, j) ∈ E. While dealing with a common parameter matrix M ⋆ for all the edges (i, j) ∈ E was convenient for aggregating the rewards and constructing a common estimate M for all the agents, when the rewards y (i,j) t are defined with different matrices M (i,j) ⋆ , the problem becomes more complicated. Indeed consider the setting where the reward y (i,j) t is defined as follows for each (i, j) ∈ E :

y (i,j) t = x (i)⊤ t M (i,j) ⋆ x (j) t + η (i,j) t ,
where M (i,j) ⋆ are unknown parameter matrices and η (i,j) t σ-sub-gaussian random variables. This setting is relevant when the agents do not have the same interactions between each of their neighbors, and thus not the same reward function.

Open questions:

In the context of pure exploration, how does the stopping condition change? Is there a sampling strategy for each agent such that estimates M(i,j) are constructed by satisfying an optimal design criterion? Decentralized setting. When agents are controlled by a central entity, it is possible to aggregate the different rewards and to build a common estimate M of M ⋆ . Moreover, we have seen that the different objectives that appear are relative to the edge-arms and not directly to the node-arms selected by each agent. Indeed, this is due to the fact that we can express the graphical bilinear bandit as linear bandits at the edge level. This particular aspect makes the decentralized framework a bit tricky because coordinating two agents without communication to respectively draw the node-arms that will build the wanted edge-arm becomes even more complicated.

However other problem arise even if the coordination problem is solved. For example, in the best arm identification problem, we have already designed a sampling procedure that can be executed in parallel during a round, hence a decentralized choice for each agent. However, the stopping condition depends on the estimate M constructed with the edge-arms during the learning procedure, but when the agents do not communicate, this estimate cannot be constructed. This is because an agent only knows which node arm it is pulling and observes the reward. However, the reward is linear with respect to the associated edge-arm and the agent does not have access to this edge-arm since it is constructed with its node arm but also with that of its neighbors (to which it does not have access).

Open questions:

In the fully decentralized setting (without communication), what kind of algorithms can we design to take advantage of the (bi-)linear bandit setting? If we allow communication, how can we adapt the proposed algorithms and what is the trade-off between the amount of communications and their performances? which experiments must be chosen in a fixed pool of experiments so as to minimize its variance? In the multi-dimensional case, this is done by minimizing a scalar function of its covariance matrix and several approaches have been considered such as the minimization of the determinant, the trace or the spectral norm, respectively denoted D, A and E-optimal design (see e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Sagnol | Optimal design of experiments with application to the inference of traffic matrices in large networks: second order cone programming and submodularity[END_REF]). (See Appendix 1, for more details on experimental design) E-optimal design has been exploited in practical settings such as for biological experiments [START_REF] Flaherty | Robust design of biological experiments[END_REF] or for treatment versus control comparisons where useful statistical interpretations have been derived, see e.g., [START_REF] Notz | Optimal designs for treatment-control comparisons in the presence of twoway heterogeneity[END_REF][START_REF] Rosa | Optimal approximate designs for comparison with control in dose-escalation studies[END_REF]. Another criterion, known as G-optimal design and which minimizes the worst predicted variance, has recently been investigated in the context of best arm identification in linear bandits [START_REF] Soare | Best-arm identification in linear bandits[END_REF][START_REF] Tao | Best Arm Identification in Linear Bandits with Linear Dimension Dependency[END_REF]106] where one is interested in finding the experiment with maximum linear response.

The optimization problems associated to the aforementioned optimal designs (E, A, D, G) are known to be NP-hard [START_REF] Çivril | On selecting a maximum volume sub-matrix of a matrix and related problems[END_REF]104]. The two common approaches have been to resort to greedy strategies or convex relaxations. A greedy strategy iteratively finds the best experiment whereas solving a convex relaxation returns a discrete probability distribution over the samples. On the one hand, performance guarantees for greedy strategies have been obtained by exploiting supermodularity and approximate supermodularity properties of the different criteria [START_REF] Chamon | Approximate supermodularity bounds for experimental design[END_REF][START_REF] Sagnol | Approximation of a maximum-submodular-coverage problem involving spectral functions, with application to experimental designs[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]. On the other hand, for performance guarantees of randomized optimal designs, only the randomized A-optimal design has been thetoretically studied with bounds on the mean square error of the associated estimator ([103]).

We propose in this paper to fill the gap concerning randomized E and G-optimal designs. More precisely we study their theoretical validity by providing finite-sample confidence bounds and show with experiments that they are worth being considered in practice. The paper is organized as follows. Section A.2 defines the main notations and recalls the problem of experimental design as well as the different optimal criteria. Section A.3 presents the main results of this paper for the random strategies of E and G-optimal designs. Finally, the last section shows empirical results of the studied random strategies and an application to the best arm identification problem in linear bandits.

A.2 Preliminaries

Definitions and notations

Throughout the paper, we use small bold letters for vectors (e.g., x) and capital bold letters for matrices (e.g., X). For any d > 0 and any vector x ∈ R d , ∥x∥ will denote the usual ℓ 2 -norm of x. For any square matrix X ∈ R d×d , we denote as ∥X∥ the spectral norm of X, that is ∥X∥ ≜ sup y:∥y∥=1 ∥Xy∥. We let λ min (X) be the smallest eigenvalue of X. For any 1 ≤ i, j ≤ d, any x ∈ R d and any matrix X ∈ R d×d , [x] i denotes the i-th coordinate of vector x, [X] i the vector of the i-th row and [X] ij the value at the i-th row and j-th column. Finally, we denote by S + d the cone of all d × d positive semi-definite matrices and by

∆ d ≜ {µ ∈ [0, 1] d , d i=1 [µ] i = 1} the simplex in R d .

Experimental design for linear regression

Given X ∈ R K×d a matrix of K experiments1 and y ∈ R K a vector of K measurements, it is assumed that there exists an unknown parameter θ ⋆ ∈ R d such that for all k ∈ {1, . . . , K}, [y] k = θ ⊤ ⋆ x k +ε k where x k = [X] k and ε 1 , . . . , ε K are independent Gaussian random variables with zero mean and variance σ2 . The ordinary least squares (OLS) estimator of the parameter θ ⋆ is given by θ = arg min θ ∥y -Xθ∥ 2 2 = (X ⊤ X) -1 X ⊤ y. 2 This estimator is unbiased and has a covariance matrix Σ -1 = σ 2 (X ⊤ X) -1 .

Experimental design [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF] consists in estimating θ by selecting only the experiments that are the most statistically efficient to reduce the variance.

More formally, let n be the total number of selected experiments and for all k ∈ {1, . . . , K}, let n k be the number of times x k is chosen. We have n k ≥ 0 and K i=1 n k = n. The covariance matrix obtained with such a design can be written as

Σ -1 D = σ 2 ( K k=1 n k x k x ⊤ k ) -1
. The Loewner order on S + d being only a partial order, minimizing Σ -1 D over the cone S + d is an ill-posed problem. An optimal design is thus defined thanks to scalar properties of a matrix in S + d , i.e., as a solution of min n 1 ,...,n K f (Σ -1 D ) where f : S + d → R. The two criterion f we study in this paper are:

• E-optimality : f E (Σ -1 D ) = ∥Σ -1 D ∥.
The E-optimal design minimizes the maximum eigenvalue of Σ -1 D . Geometrically it minimizes the variance ellipsoid in the direction of its diameter only.

• G-optimality : f G (Σ -1 D ) = max x∈X x ⊤ Σ -1 D x.
The G-optimal design minimizes the worst possible predicted variance.

Those two optimality criteria are NP-hard optimization problems [START_REF] Çivril | On selecting a maximum volume sub-matrix of a matrix and related problems[END_REF]104]. However approximate solutions can be found in polynomial time by relying on greedy strategies or by relaxing the problem and looking for proportions µ k ∈ [0, 1] instead of integers n k . By letting µ k = n k /n, the covariance matrix Σ -1 D can be written as

Σ -1 D = σ 2 /n • ( K k=1 µ k x k x ⊤ k ) -1
and it leads us to the convex optimiation problem min µ 1 ,...,µ K ∈[0,1] f (Σ -1 D ) which returns a discrete probability distribution over the samples. For more details on experimental design and optimal design criteria see [START_REF] Boyd | Convex optimization[END_REF][START_REF] Pukelsheim | Optimal Design of Experiments[END_REF].

A.3 Convergence analysis

In this section, we analyze the behavior of random sampling along the distribution associated to the convex relaxation discussed in Section A.2, for E and G-optimal designs. Let X = {x 1 , . . . , x K } ⊆ R d be the set of experiments and let µ ⋆ E and µ ⋆ G be the optimal distributions in ∆ K associated to the convex relaxation of such designs. For any µ ∈ ∆ K , we denote as M(µ) the matrix

M(µ) ≜ K k=1 µ k x k x ⊤ k and f ⋆ G,n ≜ f G (nM(µ ⋆ G )) -1
as the objective at the optimum µ ⋆ G for a sample size n.

Theorem A.1. Let X = {x 1 , . . . , x K } be a set of experiments and let µ ⋆ E be the solution of the relaxation of the E-optimal design. Let 0 ≤ δ ≤ 1 and let n such that

n ≥ 2L M(µ ⋆ E ) -1 log(d/δ),
where L = max x∈X ∥x∥ 2 . Then, with probability at least 1 -δ, one has

f E S -1 E,n ≤     1 + 1 n 2L∥M(µ ⋆ E ) -1 ∥ log(d/δ) -1     f ⋆ E,n ,
where S E,n is the sum of n i.i.d. random matrices drawn from µ ⋆ E . Similarly, let µ ⋆ G be the solution of the relaxed G-optimal design and S G,n the associated sample sum. One has, with probability at least 1 -2δ,

f G S -1 G,n ≤ 1 + L d ∥M(µ ⋆ G ) -1 ∥ 2 2σ 2 n log(d/δ) + o 1 √ n f ⋆ G,n , with σ 2 ≜ L 2 K k=1 [µ ⋆ G ] k (1 -[µ ⋆ G ] k ).
Theses results recover the O(1/

√ n) that one would expect. In addition, this confirms that the randomized approach asymptotically converges toward the true optimum, which is not the casetheoretically-for the greedy strategy. Finally, let us note that the o(1/ √ n) in the G-optimal design rate depends on the interaction between Hoeffding for λ min (S n ) and Bennett for ∥S n -ES n ∥. We refer the reader to the supplementary material in Appendix A.6 for the full bound.

A refined approach of the dimension

In this section, we introduce two quantities, derived from the concept of intrinsic dimension [START_REF] Koltchinskii | Asymptotics and concentration bounds for bilinear forms of spectral projectors of sample covariance[END_REF][START_REF] Minsker | On some extensions of Bernstein's inequality for self-adjoint operators[END_REF], that allow us to refine the convergence rate for G-optimal design. We recall the definition of intrinsic dimension.

Definition A.1 (Intrinsic dimension)

. Let d > 0 and S ∈ R d×d be a positive semi-definite matrix. The intrinsic dimension of S, denoted intdim(S), is defined as follows:

intdim(S) ≜ tr(S) ∥S∥ ≤ d.
Using this definition, one may alter the concentration results on the spectral norm, by replacing the dimension d by 2 × intdim(ES n ). For a matrix with eigenvalues decreasing fast enough, the improvement may be substantial-see [START_REF] Tropp | An Introduction to Matrix Concentration Inequalities[END_REF]Ch. 7] and references therein for more details. The main drawback of this definition is that if eigenvalues are all of the same order of magnitude, one will not notice a sensible improvement; this is typically the case in G-optimal design as eigenvalues are designed to be large overall. We propose a refined version of the intrinsic dimension allowing improvements even with a narrow spectrum, in the form of 2 complementary quantities.

Definition A.2 (Upper and lower intrinsic dimension)

. Let d > 0 and S ∈ R d×d be a positive semi-definite matrix. The upper and lower intrisic dimensions of S, denoted updim(S) and lowdim(S) respectively, are defined as follows:

         updim(S) ≜ tr S -λ min (S)I ∥S∥ -λ min (S) lowdim(S) ≜ tr ∥S∥I -S ∥S∥ -λ min (S) = d -updim(S).
These new quantities use both the largest and the smallest eigenvalues to rescale the spectrum, which is of interest in our setting. Using this definition, one is able to formulate new concentration results on random matrices, including a concentration result on the lowest eigenvalue. In this particular case however, we are more interested in the potential speed up provided for the spectral norm, since it is the value controlling the slowest term in the G-optimal design error.

Theorem A.2. Let X = {x 1 , . . . , x K } be a set of experiments and let µ ⋆ G be the solution of the relaxation of the G-optimal design. Let X 1 , . . . , X n be n i.i.d. random matrices drawn according to µ ⋆ G and S n their sum. Let V be the covariance matrix of X 1 , that is

V ≜ E X 2 1 ] -M(µ ⋆ G ) 2 and let κ be its condition number. Let 0 ≤ δ ≤ 1 and let n such that n ≥ 4L 2 9∥V∥ log d/δ
where L = max x∈X ∥x∥ 2 and d is defined by d ≜ updim(V) + lowdim(V)e -n(1-κ -1 )/16 < d. Then, with probability at 1 -2δ, one has

f G S -1 G,n ≤ 1 + L d ∥M(µ ⋆ G ) -1 ∥ 2 4σ 2 n log d/δ + o 1 √ n f ⋆ G,n .
We refer the reader to the supplementary material in Appendix A.6 for the proof of this result.

A.4 Experiments

In this section we compare the performances of randomized E and G-optimal designs against their greedy counterparts. We first show the behavior of the randomized E-optimal design on a synthetic data set. We then apply the randomized G-optimal design to the problem of best arm identification and compare it to the greedy approach used in [START_REF] Soare | Best-arm identification in linear bandits[END_REF]. We refer the reader to Appendix A.6 and A.6 for more details on best arm identification for linear bandits and on the experiments setting, respectively.

A.5 Conclusion

We have shown the convergence of randomized scheme for G and E-optimal criteria at a rate of O(1/ √ n). We also evidenced the dependence of the rate in a specific characteristic of the covariance matrix for the sampling. Empirically, the random sampling enjoys a favorable comparison with the greedy approach, even in the bandit application. One possible extension of this work could be to investigate the setting of batch or parallel bandits, using a random sampling to select a batch of arms before observing the rewards.

A.6 Proofs and details on experiments

Chernoff inequalities on matrices

Many concentration inequalities have been developped for bounding the deviation of a sum of i.i.d. random variables. In particular, Chernoff inequalities have been extensively studied and derived due to their exponential decay rate on tail distributions. Here we show how these bounds can be extended to random matrices (see e.g., [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF] for an introduction on that matter).

Additional notations

For any Hermitian matrices X, Y, we write X ⪯ Y if and only if the matrix Y -X is positive semidefinite. Recall that for any Hermitian matrix X, there exists a unitary matrix P and a diagonal matrix D such that X = PDP ⊤ . For such a matrix and for any function f : R → R, we denote as f (X) the extension of f to a Hermitian matrix, defined as follows:

f (X) ≜ P     f ([D] 11 ) . . . f ([D] dd )     P ⊤ .
In particular, for any scalar x ∈ R, we define (x) + ≜ max(x, 0) so (X) + is the projection of X onto the positive semidefinite cone. We will use the exponential function for both scalars and matrices: for the sake of clarity, we denote as e x the exponential of a scalar and exp(X) the exponential of a matrix. We will denote as Sp(X) the spectrum of X, that is the set of all eigenvalues associated to X. The identity matrix and the zero matrix in dimension d are denoted I d and 0 d , respectively; when clear from context, we drop the d index.

Useful lemmas

Before stating the concentration inequalities of interest, we need to state several useful lemmas. These lemmas are key for proving concentration of random matrices, as we need similar guarantees for matrix ordering (⪯) than for scalar ordering (≤). We first state two lemmas that will ensure order preserving under basic operations.

Lemma A.1 (Conjugation Rule

). Let M, N ∈ R d×d be two Hermitian matrices, such that M ⪯ N. Let p > 0 and let Q ∈ R p×d . Then, one has

QMQ ⊤ ⪯ QNQ ⊤ .
Proof. The proof is immediate when considering Q(N -M)Q ⊤ and using the definition of a positive semidefinite matrix.

Lemma A.2 (Transfer Rule

). Let M ∈ R d×d be a Hermitian matrix and let f, g : R → R be such that, for any x ∈ Sp(M), f (x) ≤ g(x). Then, one has

f (M) ⪯ g(M).
Proof. Let D be the diagonal matrix in the spectral decomposition of M. Since f ≤ g on Sp(M), one has f (D) ⪯ g(D). The conjugation rule then allows us to conclude.

Finally, we state two lemmas ensuring that two more complex operations (tr exp and log, respectively) preserve the order. Please note that this is usually not the case, even for operators that are monotone on R-e.g., the exponential does not preserve the order. Lemma A.3 (Monotonicity of the trace of the exponential). Let M, N ∈ R d×d be two Hermitian matrices such that M ⪯ N. Then for any non-decreasing function ψ : R → R, one has: tr ψ(M) ≤ tr ψ(N) .

In particular,

tr exp(M) ≤ tr exp(N).

Proof. Let λ 1 (M) ≥ . . . ≥ λ d (M) and λ 1 (N) ≥ . . . ≥ λ d (N) be the sorted eigenvalues of M and N, respectively. Then, for 1 ≤ i ≤ d, one can define an eigenvalue as follows:

λ i (M) = max L⊆R d :dim L=i min u∈L:∥u∥=1 u ⊤ Mu.
Using the fact that M ⪯ N, one can deduce that for any

1 ≤ i ≤ d, λ i (M) ≤ λ i (N).
Since ψ is a non-decreasing function on R, one has that for any 1 ≤ i ≤ d, ψ λ i (M) ≤ ψ λ i (N) . Summing over the dimensions leads to the desired result.

Lemma A.4 (Monotonicity of the logarithm). Let M, N ∈ R d×d be two positive definite matrices such that M ⪯ N. Then one has:

log(M) ⪯ log(N).
Proof. We will first prove that for any γ ∈ R + , (M + γI) -1 ⪰ (N + γI) -1 .

The facts that M ⪯ N and γ ≥ 0 imply that M + γI ⪯ N + γI. Using Lemma A.1, we obtain:

0 ≺ (N + γI) -1/2 (M + γI)(N + γI) -1/2 ⪯ I.
Taking the inverse yields:

(N + γI) 1/2 (M + γI) -1 (N + γI) 1/2 ⪰ I.
Finally, applying again Lemma A.1 with (N + γI) -1/2 yields:

(M + γI) -1 ⪰ (N + γI) -1 .
Let us now focus on the main result. First recall that the logarithm of a positive scalar can be expressed using its integral representation, that is

log x = +∞ 0 1 1 + t - 1 x + t dt,
for any x > 0. Therefore, the logarithm of a matrix X ≻ 0 can be expressed similarly:

log X = +∞ 0 1 1 + t I -(X + tI) -1 dt.
In the beginning of the proof, we have shown that for any γ ≥ 0, (M+tI) -1 ⪰ (N+tI) -1 . Therefore, one has:

1 1 + γ I -(M + γI) -1 ⪯ 1 1 + γ I -(N + γI) -1 ,
and integrating over γ yields the final result.

Chernoff inequalities

Let n > 0 and let X 1 , . . . , X n be i.i.d. positive semidefinite matrices, such that there exists L > 0 verifying: 0 ⪯ X 1 ⪯ LI, almost surely. Let us now consider the random matrix S = n i=1 X i . In what follows, we will develop Chernoff bounds in order to control both ∥S∥/∥ES∥ and ∥S -ES∥.

In the scalar case, Chernoff's bounds for the sum of independent variables are based on the fact that the exponential converts a sum into a products, that is for n i.i.d. random variables X 1 , . . . , X n :

Ee n i=1 X i = E n i=1 e X i ,
and then one uses the independence to pull the product out of the expectation. For two symmetric matrices M, N ∈ R d×d however, the relation exp(M + N) = exp M exp N does not hold in general-it holds if the matrices commute. Hopefully, the following theorem gives us a way to overcome this issue.

Theorem A.3. Let n, d > 0 and let X 1 , . . . , X n be i.i.d. symmetric matrices in R d . Then, for any t ∈ R, one has Proof. We start by the first inequality. Let t ∈ R and let η > 0. As in the scalar case, one has:

P n i=1 X i ≥ t ≤ inf
P n i=1 X i ≥ t = P e η∥ n i=1 X i ∥ ≥ e ηt ≤ e -ηt Ee η∥ n i=1 X i ∥ ,
where the last inequality is an application of Markov's inequality. Using the fact that for a positive semidefinite matrix X, ∥X∥ ≤ tr X, we obtain:

Ee η∥ n i=1 X i ∥ = E exp η n i=1 X i ≤ E tr exp η n i=1 X i .
We will now use Lieb's Theorem [START_REF] Lieb | Convex trace functions and the Wigner-Yanase-Dyson conjecture[END_REF], which states that for any symmetric matrix A ∈ R d×d , the mapping M → tr exp(A + log M) is concave on the cone of positive semidefinite matrices. This allows us to bound the above term as follows:

E tr exp η n i=1 X i = EE tr exp η n i=1 X i F n-1 = EE tr exp η n-1 i=1 X i + log exp(ηX n ) F n-1 ≤ E tr exp η n-1 i=1 X i + log E exp(ηX n ) .
Iterating over n yields:

E tr exp η n i=1 X i ≤ tr exp n i=1 log E exp(ηX i ) ,
hence the result.

The second inequality is a direct consequence from the fact that for any η < 0 and any matrix X, ηλ min (X) = ∥ηX∥.

The formulation of Theorem A.3, although more complicated than is the scalar case, is very helpful for matrix concentration analysis. Indeed, since tr exp and log are both order-preserving operators on positive matrices, a bound on E exp(ηX 1 ) will now be enough to provide an overall bound of the extreme eigenvalues.

Hoeffding's inequality

The bound we develop here ensures that ∥S∥ and λ min (S) do not deviate too much from their counterpart on ES. We are now ready to state the first result.

Theorem A.4. Let X 1 , . . . , X n be i.i.d. positive semidefinite random matrices, such that there exists L > 0 verifying 0 ⪯ X 1 ⪯ LI. Let S be defined as:

S ≜ n i=1 X i .
Then, for any 0 < ε < 1, one can lowerbound λ min (S) as follows:

P(λ min (S) ≤ (1 -ε)λ min (ES)) ≤ d e -ε (1 -ε) 1-ε nλ min (EX 1 ) L .
Similarly, one can upperbound ∥S∥ as follows:

P(∥S∥ ≥ (1 + ε)∥ES∥) ≤ d e ε (1 + ε) 1+ε n∥EX 1 ∥ L .
The following corollary shows an alternate (but slightly weaker) formmulation which is closer to usual concentration results.

Corollary A.1. Let X 1 , . . . , X n and S be defined as above. For any 0 < ε < 1, one can lowerbound λ min (S) as follows:

P(λ min (S) ≤ (1 -ε)λ min (ES)) ≤ d exp - ε 2 λ min (EX 1 ) 2L .
Before proving Theorem A.4, we state a useful lemma for bouding moment generating function of random positive semi-definite matrices.

Lemma A.5. Let t ∈ R and let X be a random matrix such that 0 ⪯ X ⪯ LI almost surely for some L ≥ 0. Then, one has:

E exp(tX) ⪯ I + e tL -1 L EX ⪯ exp e tL -1 L EX .
Proof. Both inequalities are derived from the convexity of the exponential. We will write scalar inequalities based on convexity and then extend them to matrices using the transfer rule in Lemma A.2. Let t ∈ R, for any 0 ≤ x ≤ L, the following holds:

e tx ≤ e 0 + x L (e tL -e 0 ) = 1 + e tL -1 L x.
Since 0 ⪯ X ⪯ LI almost surely, this can be extented to the matrix exponential using the transfer rule in Lemma A.2:

exp(tX) ⪯ I + e tL -1 L X.
Taking the expectation yields the result:

E exp(tX) ⪯ I + e tL -1 L EX.
The second inequality is also an application of Lemma A.2 using the inequality 1 + x ≤ e x for any x ∈ R.

Proof of Theorem A.4. Let t > 0. Combining Lemma A.5 and Theorem A. Since the inequality holds for any η < 0, we can optimize over η. The optimal (lowest) value is reached for η = (L) -1 log(t/λ min (ES)), which is negative if and only if t < λ min (ES).

Let us make the change of variable t = (1 -ε)λ min (ES) for 0 < ε < 1, so the condition holds. Substituting the value of η into (??) yields:

P(λ min (S) ≤ (1 -ε)λ min (ES)) ≤ de (ε-1) log(1-ε)-ε nλ min (EX 1 ) L ,
and the result holds.

Remark A.1. Without additional characterization of the problem, the bound E∥X∥ ≤ tr EX ≤ d∥EX∥ is tight: consider a diagonal random matrix X such that for any 1 ≤ i ≤ d, P(X = E ii ) = 1/d, where (E ii ) 1≤i≤d are the diagonal elements from the canonical base. Then, ∥X∥ = 1 and ∥EX∥ = 1/d, so ∥X∥ = d∥EX∥. If we consider the best arm identification application, this case essentially boils down to the MAB setting and would make the whole linear modeling irrelevant: maybe there is a more subtle way of characterizing linear bandits in order to avoid a brutal d factor in the bound.

Bennett's and Bernstein's inequalities

Using the Chernoff's bound, we were able to prove, with high probability, the following:

x ⊤ n K k=1 [µ ⋆ G ] k x k x ⊤ k -1 x ≤ x ⊤ n i=1 X i -1 x ≤ ∥x∥ 2 (1 -ε) λ max n K k=1 [µ ⋆ G ] k x k x ⊤ k -1
. This is not enough to ensure the convergence of the randomized sampling. Considering again the random matrix

S n = n i=1 X i ,
our goal is to bound the following quantity:

x ⊤ S -1 n -(ES n ) -1 x.
One way to bound the above quantity is to bound the maximum eigenvalue of S -1 n -(ES n ) -1 . One has:

S -1 n -(ES n ) -1 = S -1 n I -S n (ES n ) -1 = S -1 n (ES -S n )(ES n ) -1 .
In Section A.6, we used Hoeffding's inequality to upperbound S -1 n based on (ES n ) -1 value. Therefore, we only need to care about the central term. Since the random matrix ES n -S n is not necessarily positive semi-definite anymore, we cannot use Hoeffding's inequality. We can use Bernstein's inequality however, as stated in the following theorem.

Theorem A.5. Let X 1 , . . . , X n be n i.i.d. random symmetric matrices such that EX 1 = 0 and there exists L > 0 such that ∥X 1 ∥ ≤ L, almost surely. Let S n ≜ n i=1 X i . Then, for any t > 0, one has:

P(∥S n ∥ ≥ t) ≤ de - t 2 2Lt/3 + 2nσ 2 , where σ 2 ≜ E X 2 1 .
As in the scalar case, Bernstein's inequality relies on using the Taylor expansion of the exponential to bound the moment generating function, so we will need the following lemma.

Lemma A.6. Let L > 0 and let X be a random Hermitian matrix such that EX = 0 and X ⪯ LI almost surely. Then, for any 0 < t < 3/L, one has:

E exp(tX) ⪯ exp t 2 /2 1 -tL/3 E X 2 .
Proof. Similarly to the Hoeffding's case, we will show a result for the exponential of a scalar and extend it to a Hermitian matrix. Let L > 0, 0 < x < L and 0 < t < 3/L. Let us define f : [0, L] → R such that for any 0 < y < L, f (y) ≜ e ty -ty -1 y 2 .

In particular, one has e tx = 1 + tx + x 2 f (x). Notice that f is increasing, so e tx ≤ 1 + tx + x 2 f (L). Now, using the Taylor expansion of the exponential, we can write:

f (L) = e tL -tL -1 L 2 = 1 L 2 k≥2 (tL) k k! ≤ t 2 2 k≥2 (tL) k-2 3 k-2 = t 2 /2 1 -tL/3 ,
where the inequality comes from the fact that k! ≥ 2 × 3 k-2 , for any k ≥ 2. Now, using the fact that X ⪯ LI almost surely, we can obtain the following bound:

exp(tX) ⪯ I + tX + X(f (L)I)X = I + tX + f (L)X 2 .
Finally, taking the expectation and combining this result with a common bound of the exponential, we obtain:

E exp(tX) ⪯ I + t 2 /2 1 -tL/3 E X 2 ⪯ exp t 2 /2 1 -tL/3 E X 2 ,
hence the result.

We are now ready to prove Bernstein's inequality for matrices.

Proof of Theorem A.5. Let 0 < η < 3/L, using Markov's inequality one has: P(∥S n ∥ ≥ t) = P e η∥Sn∥ ≥ e ηt ≤ e -ηt Ee η∥Sn∥ .

By Lemma A.6 and the subadditivity of the matrix CGF [96, Lemma 3.5.1, Ch. 3], we obtain

Ee η∥Sn∥ = E∥ exp(ηS n )∥ ≤ tr E exp(ηS n ) ≤ tr exp t 2 /2 1 -tL/3 E S 2 n .
Plugin the trace back into the exponential yields:

Ee η∥Sn∥ ≤ tr exp

η 2 /2 1 -ηL/3 E S 2 n ≤ de η 2 /2 1-ηL/3 E S 2 n .
Optimizing on η would lead to a complicated result, so we use instead η = t/(nσ 2 + tL/3), which verifies the condition η < 3/L and yields the final result.

The relationship between the precision (nt in Theorem A.5) and the confidence level δ b (the RHS of the concentration inequality) is more complicated in Bernstein's inequality than in Hoeffding's. It requires solving a second order polynomial equation and leads to:

t = L 3n log 2d δ b + L 3n log 2d δ b 2 + 2σ 2 n log 2d δ b .
In our case, we will use the bound provided by Bennett's inequality applied to random Hermitian matrices, as it is simpler to derive the precision associated to a confidence level.

Theorem A.6. Let X 1 , . . . , X n be n i.i.d. random Hermitian matrices such that EX 1 = 0 and there exists σ 2 > 0 such that ∥E X 1 2 ∥ ≤ σ 2 . In addition, let us assume that there exists c > 0 such that for any q ≥ 3:

E (X 1 ) q + ≤ q! 2 σ 2 c q-2 ,
where for any symmetric matrix X,(X) + is the orthogonal projection of X onto the semidefinite positive cone. Then, for any t > 0, one has:

P n i=1 X i ≥ √ 2nσ 2 t + ct ≤ de -t .
The proof is very similar to Bernstein's: we need an intermediary result on the moment generating function, as stated in the following lemma.

Lemma A.7. Let σ 2 , c > 0 and let X be a random Hermitian matrix such that EX = 0 and ∥E X 2 ∥ ≤ σ 2 . In addition, we assume that for any q ≥ 3,

E (X) q + ≤ q! 2 σ 2 c q-2 .
Then, for any 0 < t < 1/c, one has:

E exp(tX) ⪯ exp t 2 /2 1 -ct E X 2 .
The proof of this Lemma is ommitted as it is very similar to Lemma A.6.

Proof of Theorem A.6. This proof can be directly adapted from Bernstein's using standard results on concentration (see e.g., [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF] for details).

Proof of Theorem A.1. As mentionned in the beginning of this section, our goal is to bound

∥S -1 n (S n -ES n )(ES n ) -1 ∥.
Let us assume that the batch size n satisfies:

n > 2L log d λ min (EX 1 )
.

Let de -nλ min (EX 1 )

2L

< δ h < 1. Using the Chernoff's bound, we know that with probability at least 1 -δ h , the following holds true:

∥S -1 n ∥ ≤ ∥(ES n ) -1 ∥ 1 -2L n ∥(EX 1 ) -1 ∥ log(d/δ h )
.

Similarly, let 0 < δ b < 1; using Bennett's inequality, with probability at least 1 -δ b , we have:

∥S n -ES n ∥ ≤ L 3 log d δ b + 2nσ 2 log d δ b .
Combining these two results with a union bound leads to the following bound, with probability 1 -(δ b + δ h ):

∥S -1 n -(ES n ) -1 ∥ ≤ ∥(ES n ) -1 ∥ 2 (L/3) log(d/δ b ) + 2nσ 2 log(d/δ b ) 1 -(2L/n)∥(EX 1 ) -1 ∥ log(d/δ h ) ≤ 1 n 2 ∥(EX 1 ) -1 ∥ 2 (L/3) log(d/δ b ) + 2nσ 2 log(d/δ b ) 1 -(2L/n)∥(EX 1 ) -1 ∥ log(d/δ h ) (A.1)
In order to obtain a unified bound depending on one confidence parameter 1 -δ, one could optimize over δ b and δ h , subject to δ b + δ h = δ. This leads to a messy result and a negligible improvement. One can use simple values δ b = δ h = δ/2, so the overall bound becomes, with probability

1 -δ, ∥S -1 n -(ES n ) -1 ∥ is upper bounded by 1 n ∥(EX 1 ) -1 ∥ 2 2σ 2 n log 2d δ 1 + (L 2 /18σ 2 n) log(2d/δ) 1 -(2L/n)∥(EX 1 ) -1 ∥ log(2d/δ) .
This can finally be formulated as follows:

∥S -1 n -(ES n ) -1 ∥ ≤ 1 n ∥(EX 1 ) -1 ∥ 2 2σ 2 n log 2d δ + o 1 n √ n .
The final result yields using the fact that

max x∈X ∥x∥ 2 = L and f ⋆ G,n = f ⋆ D,n = d n . The bound on f E (S -1
n ) is obtained similarly, only using the Hoeffding's result on minimum eigenvalue.

A refined approach of the dimension Intrinsic dimension

Definition A.3 (Intrinsic dimension). Let d > 0 and S n ∈ R d×d be a positive semi-definite matrix. The intrisic dimension of S n , denoted intdim(S n ), is defined as follows:

intdim(S n ) ≜ tr(S n ) ∥S n ∥ . One always has 1 ≤ intdim(S n ) ≤ d.
As for the regular concentration proofs, we will need two useful lemmas: one for deriving a nicer upperbound from Markov's inequality and the other for forcing the intrinsic dimension into the bound.

Lemma A.8. Let Z ∈ R d be a random Hermitian matrix and let ψ : R → R + be non-decreasing and non-negative. Then, for any t ∈ R such that ψ(t) > 0, one has:

P(∥Z∥ ≥ t) ≤ 1 ψ(t)
E tr(ψ(Z)).

Proof. Let t ∈ R. Since ψ is non-decreasing, the event {∥Z∥ ≥ t} contains {ψ(∥Z∥) ≥ ψ(t)}. In addition, using the definition of ψ(Z), one can easily notice that ψ(Z) ⪰ 0 and ∥ψ(Z)∥ ≥ ψ(∥Z∥). Therefore, one can write:

P(∥Z∥ ≥ t) ≤ P(∥ψ(Z)∥ ≥ ψ(t)) ≤ P(tr(ψ(Z)) ≥ ψ(t)),
where we used the fact that ψ(Z) ⪰ 0 in the rightmost inequality. Finally, one can conclude using Markov's inequality.

Lemma A.9. Let φ : R → R be a convex function and let Z be a positive semi-definite matrix.

Then, one has:

tr φ(Z) ≤ intdim(Z)φ ∥Z∥ + (d -intdim(Z))φ(0).
In particular, if φ(0) = 0, one has:

tr φ(Z) ≤ intdim(Z)φ ∥Z∥ .
Proof. Let 0 ≤ x ≤ ∥Z∥. By convexity of φ, we can write:

φ(x) ≤ φ(0) + φ(∥Z∥) -φ(0) x ∥Z∥ .
Using Lemma A.3, we can extend the above inequality to Z:

tr φ(Z) ≤ tr φ(0)I + φ(∥Z∥) -φ(0) ∥Z∥ tr Z ,
which can be rearranged as follows:

tr φ(Z) ≤ intdim(Z)φ ∥Z∥ + (d -intdim(Z))φ(0),
and the result holds.

Using the two previous lemmas, we can adapt the proof in Hoeffding in order to obtain an improved bound with the intrinsic dimension.

Theorem A.7. Let L > 0 and let X 1 , . . . , X n be i.i.d. random matrices such that 0 ⪯ X 1 ⪯ LI. Let S n = n i=1 X i . For any 0 < ε < 1, one can upperbound ∥S n ∥ as follows:

P(∥S n ∥ ≥ (1 + ε)∥ES n ∥) ≤ 2 × intdim(Z) e ε (1 + ε) 1+ε ∥ESn∥ L .
Proof. Let t, η > 0. Using Lemma A.8 with ψ : x ∈ R → (e ηx -1) + yields:

P(∥S n ∥ ≥ t) ≤ 1 e ηt -1 E tr (exp(ηS n ) -I) + = 1 e ηt -1 tr(exp(ηS n ) -I), (A.2)
where we used the fact that S n ⪰ 0 implies exp(ηS n ) ⪰ I. Let 0 ≤ x ≤ L, by convexity of the exponential, one has:

e ηx -1 ≤ e ηL -1 x L .
Once again, we can extend this result to S n and obtain:

tr exp(ηS n ) -I ≤ tr e ηL -1 L S n .
Taking the expectation and using the inequality x ≤ e x -1 yields:

E tr exp(ηS n ) -I ≤ tr e ηnL -1 nL ES n ≤ tr exp e ηnL -1 nL ES n -I
We can now use Lemma A.9 with φ : x ∈ R → e x -1 to obtain a bound depending on ∥ES n ∥:

E tr exp(ηS n ) -I ≤ tr exp e ηnL -1 nL ES n -I ≤ intdim(ES n ) e e ηnL -1 nL ∥ESn∥ -1 .
Combining the previous inequality with (A.2) yields:

P(∥S n ∥ ≥ t) ≤ intdim(ES n ) × e e ηnL -1 nL ∥ESn∥ -1 e ηt -1 ≤ intdim(ES n ) × e ηt e ηt -1 • e -ηt+ e ηnL -1 nL ∥ESn∥ .
The remainder of the proof consists in bounding e ηt /(e ηt -1) by 2 and the rightmost term as in the regular Hoeffding's proof (see [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF] for additional details).

There are two main differences between this version of the Hoeffding's bound and the regular one. First, there is a factor 2 with the intrinsic dimension. This a not necessarily a big deal as we can win on other aspects. Then, we have obtained a bound on the highest eigenvalue, but not on the lowest. This is due to the current definition of the intrinsic dimension: we use this limitation as a motivation for the refinement we propose in the next section.

A refined approach of the intrinsic dimension

Definition A.4 (Upper and lower intrinsic dimension). Let d > 0 and S n ∈ R d×d be a positive semi-definite matrix. The upper and lower intrisic dimensions of S n , denoted updim(S n ) and lowdim(S n ) respectively, are defined as follows:

         updim(S n ) ≜ tr S n -λ min (S n )I ∥S n ∥ -λ min (S n ) lowdim(S n ) ≜ tr ∥S n ∥I -S n ∥S n ∥ -λ min (S n ) = d -updim(S n ). One always has 1 ≤ updim(S n ), lowdim(S n ) ≤ d -1.
This definition brings a different information about the matrix at stake: instead of renormalizing the trace using the spectral norm, we also shift it using the lowest eigenvalue. With these new quantities, we are able to formulate a refined version of Lemma A.9.

Lemma A.10. Let φ : R → R be a convex function and let Z be a positive semi-definite matrix. Then, one has:

tr φ(Z) ≤ updim(Z)φ ∥Z∥ + lowdim(Z)φ(λ min (Z)).
This bound is always tighter than the one with the intrinsic dimension, that is:

updim(Z)φ ∥Z∥ +lowdim(Z)φ(λ min (Z)) ≤ intdim(Z)φ ∥Z∥ +(d-intdim(Z))φ(0).
Proof. To prove both assertions, we will show a more general bound, of the form:

tr φ(Z) ≤ f (l),
for 0 ≤ l ≤ λ min (Z) and then we will show that f is non-increasing. Let 0 ≤ l ≤ λ min (Z) and let l ≤ x ≤ ∥Z∥. Using the convexity of φ, one can write:

φ(x) ≤ φ(l) + (φ(∥Z∥) -φ(l)) x -l ∥Z∥ -l = x -l ∥Z∥ -l φ(∥Z∥) + ∥Z∥ -x ∥Z∥ -l φ(l).
Using Lemma A.3 again, we can extend the above inequality to Z:

tr φ(Z) ≤ tr(Z -lI) ∥Z∥ -l φ(∥Z∥) + tr(∥Z∥I -Z) ∥Z∥ -l φ(l).
It is immediate to see that taking l = 0 leads to Lemma A.9 and l = λ min (Z) shows the first assertion of this lemma. The last assertion of the theorem just comes from the convexity of φ: when applying the convexity bound on two segments I ⊆ J , the bound on I is necessarily tighter than the bound on J .

Theorem A.8. Let X 1 , . . . , X n be i.i.d. positive semidefinite random matrices, such that there exists L > 0 verifying 0 ⪯ X 1 ⪯ LI. Let S n be defined as:

S n ≜ n i=1 X i .
In addition, let κ be the condition number of ES n , that is

κ ≜ ∥ES n ∥ λ min (ES n ) .
Then, for any 0 < ε < 1, one can lowerbound λ min (S n ) as follows:

P(λ min (S n ) ≤ (1 -ε)λ min (ES n )) ≤ dmin e -ε (1 -ε) 1-ε nλ min (EX 1 ) L
, where dmin = lowdim(ES n ) + updim(ES n )e -nελ min (EX 1 )(κ-1)/L .

Similarly, one can upperbound ∥S n ∥ as follows:

P(∥S n ∥ ≥ (1 + ε)∥ES n ∥) ≤ dmax e ε (1 + ε) 1+ε n∥EX 1 ∥ L , where dmax = updim(ES n ) + lowdim(ES n )e -nε∥EX 1 ∥(1-κ -1 )/L .
Proof. The beggining of the proof is similar to the regular Hoeffding's proof so we can directly write, for t, η > 0:

P(∥S n ∥ ≥ t) ≤ e -ηt tr exp e ηL -1 L ES n
Using Lemma A.10 with φ : x ∈ R → e g(η)x and g : η → L -1 (e ηL -1) yields:

tr exp(g(η)ES n ) ≤ updim(ES n )e g(η)∥ESn∥ + lowdim(ES n )e g(η)λ min (ESn) = updim(ES n ) + lowdim(ES n )e g(η)(λ min (ESn)-∥ESn∥) e g(η)∥ESn∥ .
Using the same value for η than in regular Hoeffding's proof thus yields:

P(∥S n ∥ ≥ (1 + ε)∥ES n ∥) ≤ d e ε (1 + ε) 1+ε ∥ESn∥/L where d = updim(ES n ) + lowdim(ES n )e g(η)(λ min (ESn)-∥ESn∥) .
Pluging the value η = L -1 log(1 + ε) into d's expression yields the result. The result on λ min (S n ) is very similar:

P(λ min (S n ) ≤ t) = P(∥ -S n )∥ ≥ -t) ≤ e ηt tr exp e -ηL -1 L ES n .
Using the same reasoning, one obtains:

P(λ min (S n ) ≤ (1 -ε)∥ES n ∥) ≤ d e -ε (1 -ε) 1-ε λ min (ESn)/L where d = lowdim(ES n ) + updim(ES n )e g(-η)(∥ESn∥-λ min (ESn)) ,
which proves the result since η = -L -1 log(1 -ε).

Theorem A.9. Let X 1 , . . . , X n be n i.i.d. random symmetric matrices such that EX 1 = 0 and there exists L > 0 such that

∥X 1 ∥ ≤ L, almost surely. Let S n ≜ n i=1 X i . Let V be the covariance matrix of X 1 , that is V ≜ E X 2 1 ] -M(µ ⋆ G )
2 and let κ be its condition number. Let t verifying

3n∥V∥ 2 L > t > √ n∥V∥ 2 + L 3 √ n .
Then, one has:

P ∥S n ∥ ≥ √ t ≤ de - t 2 4∥V∥ 2 , where d = updim(V) + lowdim(V)e -n 16 (1-κ -1 ) .
Proof. One can combine reasonings of Bernstein's regular proof with the proof of Hoeffding's with updim and lowdim to obtain:

P(∥S n ∥ ≥ √ nt) ≤ updim(V)e - t 2 /2 σ 2 +Lt/3n + lowdim(V)e - t 2 /2 σ 2 +Lt/3n (2-κ -1 ) . Let us assume that 3nσ 2 L > t > √ nσ 2 + L 3 √ n .
Then, the previous result can be bounded as follows:

P(∥S n ∥ ≥ √ nt) ≤ updim(V) + lowdim(V)e -t 2 4σ 2 (1-κ -1 ) e -t 2 4σ 2 ≤ updim(V) + lowdim(V)e -n 16 (1-κ -1 ) e -t 2 4σ 2 ,
and the result holds.

Link with the best arm identification in linear bandits

Let d > 0 and X ⊆ R d a subset of R d , corresponding to the bandit arms. The linear bandit setting assumes that the conditional distribution of the rewards given the arm follows a linear model: there exists an unknown parameter θ ⋆ ∈ R d such that the reward r(x) associated to any action x ∈ X is of the form

r(x) = θ ⊤ ⋆ x + ϵ,
where ϵ is a R-subgaussian noise independent from x. This linear structure implies that some information is shared between arms through the parameter θ ⋆ : an action-reward pair (x, r(x))

gives information about θ ⋆ and thus about the reward distributions of the other actions. This makes this setting very different to the classical multi-armed bandit setting where the reward distributions of each action are assumed to be independent. Whereas multi-armed bandit algorithms mainly focus on the estimation of the mean reward of each action, linear bandit algorithms are mainly interested in the estimation of the parameter θ ⋆ .

Depending on the context, the goal of a bandit algorithm can either be to maximize the cumulated reward (the sum of the rewards collected over several iterations) or to find the arm maximizing the reward, referred to as best arm identification or pure exploration.

We here focus on best arm identification in linear bandits whose objective is to find the arm x ⋆ maximizing the average reward:

x ⋆ = arg max x∈X θ ⊤ ⋆ x .
As the parameter θ ⋆ is unknown the aim is to design a strategy that will sequentially choose t actions x 1 , . . . , x t ∈ X and collect their associated rewards

r i = θ ⊤ ⋆ x i + ϵ i , 1 ≤ i ≤ t,
where ϵ 1 , . . . , ϵ t are independent realizations of ϵ, to obtain an estimate θt of θ ⋆ . To find the best arm, the estimated prediction θ⊤ t x should be close to the real prediction θ ⊤ ⋆ x for all x ∈ X . More precisely, rather than the reward prediction of an action itself, we are interested in comparing the predictions of each pair of arms. We thus want |( θt -θ ⋆ ) ⊤ (x -x ′ )| to be small. Remark A.2. Note that compared to the multi-armed bandit case where known suboptimal arms are no longer played, the situation is different for the best arm identification case as playing suboptimal arms might give information about the parameter θ ⋆ and improve the discrimination of the unknown x ⋆ with other arms.

Most of the designed strategies for best arm identification in linear bandits [START_REF] Soare | Best-arm identification in linear bandits[END_REF][START_REF] Tao | Best Arm Identification in Linear Bandits with Linear Dimension Dependency[END_REF]106] have relied on two concentration inequalities giving high probability bounds on the prediction error |( θt -θ ⋆ ) ⊤ x| of the regression estimator θt obtained from a sequence of action-reward pairs. The first concentration inequality is only valid when the sequence of actions is fixed and hence cannot depend on the observed random rewards. The authors of [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF] derived a concentration inequality which holds when the sequence of actions is adaptive to the observed random rewards. However this concentration inequality offers a looser bound than the one given for fixed sequences. In [START_REF] Soare | Best-arm identification in linear bandits[END_REF] strategies relying on the fixed sequence bound are developed whereas [106] designed a fully adaptive algorithm based on the adaptive bound. These two concentration inequalities are detailed below.

Let θt (λ) denote the ridge estimate of θ ⋆ with a ℓ 2 -penalty λ:

θt (λ) ≜ arg min θ∈R d t s=1 (θ ⊤ x s -r s ) 2 + λ 2 ∥θ∥ 2 .
The ridge estimate θt (λ) can be expressed in closed form:

θt (λ) = Ât (λ) -1 X ⊤ t r t ,
where

X ⊤ t ≜ (x 1 , . . . , x t ), r ⊤ t ≜ (r 1 , . . . , r t ) and Ât (λ) ≜ X ⊤ t X t + λI d .
Fixed design concentration inequality. We assume that there is a finite number of arms |X | = K. If λ = 0 and (x i ) 1≤i≤∞ is a fixed sequence of actions (independent of the random rewards (r i ) 1≤i≤∞ ) we have the following concentration inequality [START_REF] Soare | Best-arm identification in linear bandits[END_REF]: for all δ ∈ (0, 1),

P ∀t ∈ N, ∀x ∈ X , |θ ⊤ ⋆ x -θ⊤ t x| ≤ 2R∥x∥ Â-1 t 2 log 6t 2 K π 2 δ ≥ 1 -δ . (A.3)
One may notice that this result holds over these directions by replacing K by K 2 in the logarithmic term, as there are of the order of K 2 such directions. 3Adaptive design concentration inequality. When the sequence of actions is chosen adaptively of the history, i.e., for all i ∈ N, x i is allowed to depend on (x 1 , r 1 , . . . , x t-1 , r t-1 ), we need to rely on a result established by [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF]: if λ > 0 and ∥x i ∥ ≤ L for all i then for all δ ∈ (0, 1) and all x ∈ R d ,

P |θ ⊤ ⋆ x -θt (λ) ⊤ x| ≤ ∥x∥ Ât(λ) -1 R d log 1 + tL 2 /λ δ + √ λ∥θ ⋆ ∥ ≥ 1 -δ .
(A.4) The reader can refer to [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF]Appendix B] for the proof of this result. The main difference with (A.3) is the presence of an extra √ d factor which cannot be removed and which makes adaptive algorithms suffer more from the dimension than fixed design strategies (see [START_REF] Lattimore | Bandit Algorithms[END_REF]Chapter 20] for a more complete discussion on this aspect). We now omit the dependence of Ât (λ) -1 in λ when it is not relevant for the purpose of the discussion.

Whichever concentration inequality is used, the bound on the prediction error in a direction y = xx ′ , x, x ′ ∈ X depends on the matrix norm ∥y∥ Â-1 t . The goal of a strategy for the problem of best arm identification in linear bandits as formulated in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] is to choose a sequence of actions that reduces this matrix norm as fast as possible for all directions y so as to reduce the prediction error and be able to identify the best arm. This approach thus leads to the following optimization problem:

(x 1 , . . . , x B ) ∈ arg min x 1 ,...,x B ∈X max y∈Y y ⊤ t i=1 x i x ⊤ i -1 y. (A.5) If one upper bounds ∥y∥ Â-1 t by 2∥x∥ Â-1 t
we finally obtain the G-optimal design.

Details on experiment setting and comments

For the randomized strategies we use the cvxopt python package [START_REF] Andersen | CVXOPT: A Python package for convex optimization, version 1.2.0[END_REF] to compute the solution of the semi-definite program associated to the E-optimal design and compute the solution of the convex relaxation of the D-optimal design problem. We recall that as the relaxed G-optimal design problem is equivalent to the relaxed D-optimal design problem we can use the solution of the latter for the former. Finally, for the greedy implementations of E and G-optimal design, when there are ties between several samples at a given iteration we uniformly select one at random.

Randomized strategy versus greedy strategy for E-optimal design

We recall here that the goal of E-optimal design is to choose experiments maximizing

λ min ( K k=1 n k x k x ⊤ k ).
We generate a pool of experiments in R d made of K independent and identically distributed realizations of a standard Gaussian random variable. Figure A.1a shows the performance of the randomized and greedy strategies against the number of selected samples n when K = 500 and d = 10. For very small numbers of selected experiments the performances of the different strategies are equivalent but as the number of experiments increases the randomized E-optimal design outperforms the greedy strategy. Figure A.1b shows the performance of the strategies against the dimension d when K = 500 and the number of selected experiments n is fixed to 500. For small dimensions the randomized E-optimal design achieves a better performance but its superiority decreases when the dimension increases. For both settings we also plot the performance of the random strategy that selects experiments uniformly at random. Furthermore, the results are averaged over 100 random seeds controlling the generation of the dataset as well as the random sampling of the experiments.

Application of randomized G-optimal design to best arm identification in linear bandits

We now compare the randomized G-optimal design with the greedy implementation that has been used for the problem of best arm identification in linear bandits. We note that the objective of this experiment is not to achieve state-of-the-art results for best arm identification in linear bandits but rather to show that the randomized strategy while being easy to implement achieves comparable results as the ones obtained with the greedy strategy.

The underlying model of a linear bandit is the same as the one presented in Section A.2: the relationship between the experiments x, referred to as arms in the bandit literature, and their associated measurements y is assumed to be linear. The goal of best arm identification (see e.g., [START_REF] Soare | Best-arm identification in linear bandits[END_REF][START_REF] Tao | Best Arm Identification in Linear Bandits with Linear Dimension Dependency[END_REF]106]) is to find the arm with maximum linear response among a finite set of arms. We focus on the case where one wants to solve this task with a minimum number of trials for a given confidence level. The core idea of most of the developed strategies is to sequentially choose arms so as to minimize a confidence bound on the prediction error of the linear response. Indeed, the sooner we become confident about the predicted response of each arm the sooner we can identify the best one with high probability.

One would like to take advantage of the past responses y when choosing future arms. However the confidence bound that is available for this adaptive setting has a worse dependence on the dimension d than the confidence bound available for fixed sequences of arm [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF]. The confidence bound for fixed sequences can be stated as follows: for all δ ∈ (0, 1), with probability at least 1 -δ, for all n ∈ N and for all arms x ∈ X ,

|θ ⊤ ⋆ x -θ⊤ n x| ≤ 2c∥x∥ Σ -1 D log 6t 2 |X | π 2 δ , (A.6)
where θn is the OLS estimator obtained with n samples, c is a constant depending on the variance of the Gaussian noise and

Σ D = K k=1 n k x k x ⊤ k .
It can be observed that designing a strategy that minimizes this confidence bound for all arms naturally leads to the G-optimal design optimization problem. The reader can refer to the supplementary material or [START_REF] Soare | Best-arm identification in linear bandits[END_REF] for more details.

To compare the randomized G-optimal design with the greedy implementation used for best arm identification in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] we use the same setting as the one of the experiment presented in Section 6 of [START_REF] Soare | Best-arm identification in linear bandits[END_REF]. More specifically we consider a set of d + 1 arms in R d where d ≥ 2. This set is made of the d vectors (e 1 , . . . , e d ) forming the canonical basis of R d and one additional arm x d+1 = (cos(ω), sin(ω), 0, . . . , 0) ⊤ with ω = 0.1. The true parameter θ ⋆ has all its coordinates equal to 0 except the first one which is set to 2. In this setting, the best arm, i.e., the one with maximum linear response, is e 1 . One can also note that it is much harder to differentiate this arm from x d+1 than from the other arms. The noise of the linear model is a standard Gaussian random variable N (0, 1) and the confidence level in (A.6) is chosen equal to δ = 0.05. We also use the same condition as in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] (equation ( 13) therein) to check when enough arms have been pulled to be able to identify the best arm with high probability. This condition naturally derives from the confidence bound (A.6). As explained in Section A.2 the greedy implementation does not work for the first iterations because the design matrix is singular. As in [START_REF] Soare | Best-arm identification in linear bandits[END_REF] we thus initialize the procedure by choosing once each arm of the canonical basis. Although this would not be required for the randomized strategy as we could start by sampling a given number of experiments, we use the same initialization for the sake of fairness.

The number of samples required to find the best arm are shown in Figure A.1c which summarizes the results obtained over 100 random seeds controlling the Gaussian noise of the linear model and the random selection of the experiments. One can see that the randomized G-optimal design, while being simple to use, achieves similar performances for low dimensions and even better performances on average than the greedy implementation of the G-optimal design as the dimension increases. We note that for all the random repetitions the best arm returned by both strategies is always e 1 .

B.1 Introduction

Adversarial example attacks recently became a major concern in the machine learning community. An adversarial attack refers to a small, imperceptible change of an input that is maliciously designed to fool a machine learning algorithm. Since the seminal work of [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF] and [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] it became increasingly important to understand the very nature of this phenomenon [START_REF] Bubeck | Adversarial examples from computational constraints[END_REF][START_REF] Fawzi | Adversarial vulnerability for any classifier[END_REF][START_REF] Fawzi | Robustness of classifiers: from adversarial to random noise[END_REF][START_REF] Gourdeau | On the Hardness of Robust Classification[END_REF][START_REF] Ilyas | Adversarial Examples Are Not Bugs, They Are Features[END_REF]. Furthermore, a large body of work has been published on designing attacks [START_REF] Athalye | Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples[END_REF][START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF][START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF][START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF] and defenses [START_REF] Cohen | Certified Adversarial Robustness via Randomized Smoothing[END_REF][START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF][START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF][START_REF] Papernot | Distillation as a defense to adversarial perturbations against deep neural networks[END_REF].

Besides, in real-life scenarios such as for an autonomous car, errors can be very costly. It is not enough to just defend against new attacks as they are published. We would need an algorithm that behaves optimally against every single attack. However, it remains unknown whether such a defense exists. This leads to the following questions, for which we provide principled and theoretically-grounded answers.

Q1: Is there a deterministic classifier that ensures optimal robustness against any adversarial attack? A1: To answer this question, in Section B.3, we cast the adversarial examples problem as an infinite zero-sum game between a Defender (the classifier) and an Adversary that produces adversarial examples. Then we demonstrate, in Section B.4, the non-existence of a Nash equilibrium in the deterministic setting of this game. This entails that no deterministic classifier can claim to be more robust than all other classifiers against any possible adversarial attack. Another consequence of our analysis is that there is no free lunch for transferable attacks: an attack that works on all classifiers will never be optimal against any of them.

Q2: Would randomized defense strategies be a suitable alternative to defend against strong adversarial attacks? A2: We tackle this problem both theoretically and empirically. In Section B.5, we demonstrate that for any deterministic defense there exists a mixture of classifiers that offers better worst-case theoretical guarantees. Building upon this, we devise a method that generates a robust randomized classifier with a one step boosting method. We evaluate this method, in Section B.6, against strong adaptive attacks on CIFAR10 and CIFAR100 datasets. It outperforms Adversarial Training against both ℓ ∞ -PGD [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF], and ℓ 2 -C&W [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] attacks. More precisely, on CIFAR10, our algorithm achieves 0.55 (resp. 0.53) accuracy under attack against these attacks, which is an improvement of 0.13 (resp. 0.18) over Adversarial Training.

B.2 Related Work

Many works have studied adversarial examples, in several different settings. We discuss hereafter the different frameworks that we believe to be related to our work, and discuss the aspects on which our contribution differs from them.

Distributionally robust optimization. The work in [START_REF] Sinha | Certifiable Distributional Robustness with Principled Adversarial Training[END_REF] addresses the problem of adversarial examples through the lens of distributionally robust optimization. They study a min-max problem where the Adversary manipulates the test distribution while being constrained in a Wasserstein distance ball (they impose a global constraint on distributions for the Adversary, while we study a local, pointwise constraint, leading to different attack policies). A similar analysis was presented in [START_REF] Lee | Minimax Statistical Learning with Wasserstein distances[END_REF] in a more general setting that does not focus on adversarial examples. Even though our work studies a close problem, our reasoning is very different. We adopt a game theoretic standpoint, which allows us to investigate randomized defenses and endow them with strong theoretical evidences.

Game Theory. Some works have tackled the problem of adversarial examples as a two player game. For example [START_REF] Brückner | Stackelberg Games for Adversarial Prediction Problems[END_REF] views adversarial example attacks and defenses as a Stackelberg game. More recently, [START_REF] Bulò | Randomized Prediction Games for Adversarial Machine Learning[END_REF] and [START_REF] Perdomo | Robust Attacks against Multiple Classifiers[END_REF] investigated zero-sum games. They consider restricted versions of the game where classical theorems apply, such as when the players only have a finite set of possible strategies. We study a more general setting. Finally, [START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] motivates the use of noise injection as a defense mechanism by game theoretic arguments but only present empirical results.

Randomization. Following the work of [START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] and [105], several recent works studied noise injection as a defense mechanism. In particular, [START_REF] Lecuyer | Certified Robustness to Adversarial Examples with Differential Privacy[END_REF], followed by [START_REF] Cohen | Certified Adversarial Robustness via Randomized Smoothing[END_REF][START_REF] Li | Certified Adversarial Robustness with Additive Noise[END_REF][START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF]102] demonstrated that noise injection can, in some cases, give provable defense against adversarial attacks. The analysis and defense method we propose in this paper are not based on noise injection. However, a link could be made between these works and the mixture we propose, by noting that a classifier in which noise is being injected can be seen as an infinite mixture of perturbed classifiers.

Optimal transport. Our work considers a distributionnal setting, in which the Adversary manipulating the dataset is formalized by a push-forward measure. This kind of setting is close to optimal transport settings recently developed by [START_REF] Bhagoji | Lower Bounds on Adversarial Robustness from Optimal Transport[END_REF] and [START_REF] Pydi | Adversarial Risk via Optimal Transport and Optimal Couplings[END_REF]. Specifically, these works investigate classifier-agnostic lower bounds on the risk for binary classification under attack, with some hypothesis on the data distribution. The main differences are that we focus on studying equilibria and not deriving bounds. Moreover, these works do not study the influence of randomization. Finally they express the optimal risk of the Defender in terms of transportation costs between two distributions, whereas we explicitly study the Adversary's behaviour as a transport from one distribution to another. Even though they do not treat the problem from the same prism, we believe that these works are profoundly related and complementary to ours.

Ensemble of classifiers. Some works have been done to improve the robustness of a model by constructing ensemble of classifiers [START_REF] Abbasi | Robustness to adversarial examples through an ensemble of specialists[END_REF][START_REF] Pang | Improving adversarial robustness via promoting ensemble diversity[END_REF][START_REF] Sen | EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks[END_REF]100,107]. However all the defense methods proposed in those papers subsequently proved to be ineffective against adaptive attacks introduced in [START_REF] He | Adversarial example defense: Ensembles of weak defenses are not strong[END_REF][START_REF] Tramer | On adaptive attacks to adversarial example defenses[END_REF]. The main difference with our method is that it is not an ensemble method since it uses sampling instead of voting to aggregate the classifiers' output. Hence in terms of volatility, in voting methods, whenever a majority agrees on an opinion, all others votes will be ignored, whereas here each classifier always contributes according to its probability weights, which do not depend on the others.

B.3 A Game Theoretic point of view.

Initial problem statement

Notations. For any set Z with σ-algebra σ(Z), if there is no ambiguity on the considered σalgebra, we denote P(Z) the set of all probability measures over (Z, σ(Z)), and F Z the set of all measurable functions from (Z, σ(Z)) to (Z, σ(Z)). For µ ∈ P(Z) and ϕ ∈ F Z , the pushforward measure of µ by ϕ is the measure ϕ#µ such that ϕ#µ(B) = µ(ϕ -1 (B)) for any B ∈ σ(Z).

Binary classification task.

Let X ⊂ R d and Y = {-1, 1}. We consider a distribution D ∈ P(X × Y) that we assume to be of support X × Y. The Defender is looking for a hypothesis (classifier) h in a class of functions H, minimizing the risk of h w.r.t. D:

R(h) : = E (X,Y )∼D [1{h(X) ̸ = Y }] = E Y ∼ν E X∼µ Y [1{h(X) ̸ = Y }] . (B.1)
Where H := {h : x → sgn g(x) | g : X → R continuous}, ν ∈ P(Y) is the probability measure that defines the law of the random variable Y , and for any y ∈ Y, µ y ∈ P(X ) is the conditional law of X|(Y = y).

Adversarial example attack (point-wise).

Given a classifier h : X → Y and a data sample (x, y) ∼ D, the Adversary seeks a perturbation τ ∈ X that is visually imperceptible, but modifies x enough to change its class, i.e. h(x + τ ) ̸ = y. Such a perturbation is called an adversarial example attack. In practice, it is hard to evaluate the set of visually imperceptible modifications of an image. However, a sufficient condition to ensure that the attack is undetectable is to constrain the perturbation τ to have a small norm, be it for the ℓ ∞ or the ℓ 2 norm. Hence, one should always ensure that ∥τ ∥ ∞ ≤ ϵ ∞ , or ∥τ ∥ 2 ≤ ϵ 2 , depending on the norm used to measure visual imperceptibility. The choice of the threshold depends on the application at hand. For example, on CIFAR datasets, typical values for ϵ ∞ and ϵ 2 are respectively, 0.031 and 0.4/0.6/0.8. In the remaining of this work, we will define our constraint using an ℓ 2 norm, but all our results are valid for an ℓ ∞ based constraint.

Adversarial example attack (distributional). The Adversary chooses, for every x ∈ X , a perturbation that depends on its true label y. This amounts to construct, for each label y ∈ Y, a measurable function ϕ y such that ϕ y (x) is the perturbation associated with the labeled example (x, y). This function naturally induces a probability distribution over adversarial examples, which is simply the push-forward measure ϕ y #µ y . The goal of the Adversary is thus to find ϕ = (ϕ -1 , ϕ 1 ) ∈ (F X |ϵ 2 ) 2 that maximizes the adversarial risk R adv (h, ϕ) defined as follows:

R adv (h, ϕ) := E Y ∼ν E X∼ϕ Y #µ Y [1{h(X) ̸ = Y }] . (B.2)
Where for any ϵ 2 ∈ (0, 1), F X |ϵ 2 is the set of functions that imperceptibly modifies a distribution:

F X |ϵ 2 := ψ ∈ F X | essup x∈X ∥ψ(x) -x∥ 2 ≤ ϵ 2 .
Adversarial defense, a two-player zero-sum game. With the setting defined above, the adversarial examples problem can be seen as a two-player zero-sum game, where the Defender tries to find the best possible hypothesis h, while a strong Adversary is manipulating the dataset distribution:

inf h∈H sup ϕ∈(F X |ϵ 2 ) 2 R adv (h, ϕ). (B.3)
This means that the Defender tries to design the classifier with the best performance under attack, whereas the Adversary will each time design the optimal attack on this specific classifier. In the game theoretical terminology, the choice of a classifier h (resp. an attack ϕ) for the Defender (resp. the Adversary) is called a strategy. It is crucial to note that the sup-inf and inf-sup problems do not necessarily coincide. In this paper, we mainly focus on the Defender's point of view which corresponds to the inf-sup problem. We will be interested in understanding the behaviour of players in this game, i.e. the best responses they have to a given strategy, and whether some equilibria may arise. This motivates the following definitions.

Definition B.1 (Best Response). Let h ∈ H, and ϕ ∈ F X |ϵ 2 2 . A best response from the Defender to ϕ is a classifier h * ∈ H such that R adv (h * , ϕ) = min h∈H R adv (h, ϕ). Similarly, a best response from the Adversary to h is an attack ϕ * ∈ F X |ϵ 2 2 such that R adv (h, ϕ * ) = max ϕ∈(F X |ϵ 2 ) 2 R adv (h, ϕ).
In the remaining, we denote BR(h) the set of all best responses of the Adversary to a classifier h. Similarly BR(ϕ) denotes the set of best responses to an attack ϕ. Definition B.2 (Pure Nash Equilibrium). In the zero-sum game (Eq. B.3), a Pure Nash Equilib-

rium is a couple of strategies (h, ϕ) ∈ H × F X |ϵ 2 2 such that h ∈ BR(ϕ), and, ϕ ∈ BR(h).
When it exists, a Pure Nash Equilibrium is a state of the game in which no player has any incentive to modify its strategy. In our setting, this simultaneously means that no attack could better fool the current classifier, and that the classifier is optimal for the current attack.

Remark. All the definitions in this section assume a deterministic regime, i.e. that neither the Defender nor the Adversary use randomization, hence the notion of Pure Nash Equilibrium in the game theory terminology. The randomized regime will be studied in Section B.5.

Trivial solution and Regularized Adversary

Trivial Nash equilibrium. Our current definition of the problem implies that the Adversary has perfect information on the dataset distribution and the classifier. It also has unlimited computational power and no constraint on the attack except on the size of the perturbation. Going back to the example of the autonomous car, this would mean that the Adversary can modify every single image that the camera may receive during any trip, which is highly unrealistic. The Adversary has no downside to attacking, even when the attack is unnecessary, e.g. if the attack cannot work or if the point is already misclassified.

This type of behavior for the Adversary can lead to the existence of a pathological (and trivial) Nash Equilibrium as demonstrated in Figure B.1 for the uni-dimensional setting with Gaussian distributions. The unbounded Adversary moves every point toward the decision boundary (each time maximizing the perturbation budget), and the Defender cannot do anything to mitigate the damage. In this case the decision boundary for the Optimal Bayes Classifier remains unchanged, even though both curves have been moved toward the center, hence a trivial equilibrium. In the remaining of this work, we show that such an equilibrium does not exist as soon as there is a small restraint on the Adversary's strength, i.e. as soon as it is not perfectly indifferent to produce unnecessary perturbations. (left) and with three different attacks: no penalty (second drawing), with mass penalty (third) and with norm penalty (fourth). On all figures blue area on the left of the axis is P h (ϵ 2 ) and red area on the right is N h (ϵ 2 ).

Regularized Adversary. To mitigate the Adversary strength, we introduce a penalization term:

inf h∈H sup ϕ∈(F X |ϵ 2 ) 2 [R adv (h, ϕ) -λ Ω(ϕ)] R Ω adv (h, ϕ) . (B.4)
The penalty function Ω represents the limitations on the Adversary's budget, be it because of computational resources or to avoid being detected. λ ∈ (0, 1) is some regularization weight. In this paper, we study two types of penalties: the mass penalty Ω mass , and the norm penalty Ω norm .

From a computer-security point of view, the first limitation that comes to mind is to limit the number of queries the Adversary can send to the classifier. In our distributional setting, this boils down to penalizing the mass of points that the function ϕ moves. Hence we define the mass penalty as:

Ω mass (ϕ) := E Y ∼ν E X∼µ Y [1{X ̸ = ϕ Y (X)}] . (B.5)
The mass penalty discourages the Adversary from attacking too many points by penalizing the overall mass of transported points. The second limitation we consider penalizes the expected norm under ϕ:

Ω norm (ϕ) := E Y ∼ν E X∼µ Y [∥X -ϕ Y (X)∥ 2 ] . (B.6)
This regularization is very common in both the optimization and adversarial example communities. In particular, it is used by Carlini & Wagner [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] to compute the eponymous attack 1 . In the following, we denote BR Ωmass (resp. BR Ωnorm ) the best responses for the Adversary w.r.t the mass (resp. norm) penalty. Section B.4 shows that whatever penalty the Adversary has, no Pure

B.4 Deterministic regime

Nash Equilibrium exists. We characterize the best responses for each player, and show that they can never satisfy Definition B.2.

B.4 Deterministic regime

Notations. Let h ∈ H, we denote P h := {x ∈ X | h(x) = 1}, and

N h := {x ∈ X | h(x) = -1}
respectively the set of positive and negative outputs of h. We also denote the set of attackable points from the positive outputs P h (δ) := {x ∈ P h | ∃z ∈ N h and ∥z -x∥ 2 ≤ δ}, and N h (δ) likewise.

Adversary's best response. Let us first present the best responses of the Adversary under respectively the mass penalty and the norm penalty. Both best responses share a fundamental behavior: the optimal attack will only change points that are close enough to the decision boundary. This means that, when the Adversary has no chance of making the classifier change its decision about a given point, it will not attack it. However, for the norm penalty all attacked points are projected on the decision boundary, whereas with the mass penalty the attack moves the points across the border. Lemma B.1. Let h ∈ H and ϕ ∈ BR Ωmass (h). Then the following assertion holds:

ϕ 1 (x) ∈ (P h ) ∁ if x ∈ P h (ϵ 2 ) ϕ 1 (x) = x otherwise.
Where (P h ) ∁ , the complement of P h in X . ϕ -1 is characterized symmetrically.

Lemma B.2. Let h ∈ H and ϕ ∈ BR Ωnorm (h).

Then the following assertion holds:

ϕ 1 (x) = π(x) if x ∈ P h (ϵ 2 ) x otherwise.
Where π is the orthogonal projection on (P h ) ∁ . ϕ -1 is characterized symmetrically.

These best responses are illustrated in Figure B.1 with two uni-dimensional Gaussian distributions. For the mass penalty, µ 1 is set to 0 in P h (ϵ 2 ), and this mass is transported into N h (ϵ 2 ). The symmetric holds for µ -1 . After attack, we now have µ 1 (P h (ϵ 2 )) = 0, so a small value of µ -1 in P h (ϵ 2 ) suffices to make it dominant, and that zone will now be classified -1 by the Optimal Bayes Classifier. For the norm penalty, the part of µ 1 that was in P h (ϵ 2 ) is transported on a Dirac distribution at the decision boundary. Similarly to the mass penalty, the best response now predicts -1 for the zone P h (ϵ 2 ).

Remark. In practice, it might be computationally hard to generate the exact best response for the norm penalty, i.e. the projection on the decision boundary. That will happen for example if this boundary is very complex (e.g. highly non-smooth), or when X is in a high dimensional space. To keep the attack tractable, the Adversary will have to compute an approximated best response by allowing the projection to reach the point within a small ball around the boundary. This means that the best responses of the norm penalty and the mass penalty problems will often match.

Defender's best response. At a first glance, one would suspect that the best response for the Defender ought to be the Optimal Bayes Classifier for the transported distribution. However, it is only well defined if the conditional distributions admit a probability density function. This might not always hold here for the transported distribution. Nevertheless, we show that there is a property, shared by the Optimal Bayes Classifier when defined, that always holds for the Defender's best response.

Lemma B.3. Let us consider ϕ ∈ F X |ϵ 2
2 . If we take h ∈ BR(ϕ), then for y = 1 (resp. y = -1), and for any B ⊂ P h (resp. B ⊂ N h ) one has

P(Y = y|X ∈ B) ≥ P(Y = -y|X ∈ B)
with Y ∼ ν and for all y ∈ Y, X|(Y = y) ∼ ϕ y #µ y .

In particular, when ϕ 1 #µ 1 and ϕ -1 #µ -1 admit probability density functions, Lemma B.3 simply means that h is the Optimal Bayes Classifier for the distribution (ν, ϕ 1 #µ 1 , ϕ -1 #µ -1 ) 2 . We can now state our main theorem, as well as two of its important consequences.

Theorem B.1 (Non-existence of a pure Nash equilibrium). In the zero-sum game (Eq. B.4) with λ ∈ (0, 1) and penalty Ω ∈ {Ω mass , Ω norm }, there is no Pure Nash Equilibrium.

Consequence 1. (No free lunch for transferable attacks)

To understand this statement, remark that, thanks to weak duality, the following inequality always holds:

sup ϕ∈(F X |ϵ 2 ) 2 inf h∈H R Ω adv (h, ϕ) ≤ inf h∈H sup ϕ∈(F X |ϵ 2 ) 2 R Ω adv (h, ϕ).
On the left side problem (sup-inf), the Adversary looks for the best strategy ϕ against any unknown classifier. This is tightly related to the notion of transferable attacks (see e.g. [START_REF] Tramèr | The Space of Transferable Adversarial Examples[END_REF]), which refers to attacks successful against a wide range of classifiers. On the right side (our) problem (inf-sup), the Defender tries to find the best classifier under any possible attack, whereas the Adversary plays in second and specifically attacks this classifier. As a consequence of Theorem B.3, the inequality is always strict:

sup ϕ∈(F X |ϵ 2 ) 2 inf h∈H R Ω adv (h, ϕ) < inf h∈H sup ϕ∈(F X |ϵ 2 ) 2 R Ω adv (h, ϕ).
This means that both problems are not equivalent. In particular, an attack designed to succeed against any classifier (i.e. a transferable attack) will not be as good as an attack tailored for a given classifier. Hence she has to trade-off between effectiveness and transferability of the attack.

Consequence 2. (No deterministic defense may be proof against every attack) Let us consider the state-of-the-art defense which is Adversarial Training [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF][START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF]. The idea is to compute an efficient attack ϕ, and train the classifier on created adversarial examples, in order to move the decision boundary and make the classifier more robust to new perturbations by ϕ.

B.5 Randomization matters

To be fully efficient, this method requires that ϕ remains an optimal attack on h even after training. Our theorem shows that it is never the case: after training our classifier h to become (h ′ ) robust against ϕ, there will always be a different optimal attack ϕ ′ that is efficient against h ′ . Hence Adversarial Training will never achieve a perfect defense.

B.5 Randomization matters

As we showed that there is no Pure Nash Equilibrium, no deterministic classifier may be proof against every attack. We would therefore need to allow for a wider class of strategies. A natural extension of the game would thus be to allow randomization for both players, who would now choose a distribution over pure strategies, leading to this game:

inf η∈P(H) sup φ∈P (F X |ϵ 2 ) 2 E h∼η ϕ∼φ R Ω adv (h, ϕ) . (B.7)
Without making further assumptions on this game (e.g. compactness), we cannot apply known results from game theory (e.g. Sion theorem) to prove the existence of an equilibrium. These assumptions would however make the problem loose much generality, and do not hold here. Randomization matters. Even without knowing if an equilibrium exists in the randomized setting, we can prove that randomization matters. More precisely we show that any deterministic classifier can be outperformed by a randomized one in terms of the worst case adversarial risk. To do so we simplify Equation B.7 in two ways:

1. We do not consider the Adversary to be randomized, i.e. we restrict the search space of the Adversary to (F X ) 2 instead of P (F X ) 2 . This condition corresponds to the current state-of-the-art in the domain: to the best of our knowledge, no efficient randomized adversarial example attack has been designed (and so is used) yet.

2. We only consider a subclass of randomized classifiers, called mixtures, which are discrete probability measures on a finite set of classifiers. We show that this kind of randomization is enough to strictly outperform any deterministic classifier. We will discuss later the use of more general randomization (such as noise injection) for the Defender. Let us now define a mixture of classifiers.

Definition B.3 (Mixture of classifier).

Let n ∈ N, h = (h 1 , ..., h n ) ∈ H n , and q ∈ P({1, ..., n}).

A mixed classifier of h by q is a mapping m q h from X to P(Y) such that for all x ∈ X , m q h (x) is the discrete probability distribution that is defined for all y ∈ Y as follows:

m q h (x)(y) := E i∼q [1{h i (x) = y}].
We call such a mixture a mixed strategy of the Defender. Given some x ∈ X , this amounts to picking a classifier h i from h at random following the distribution q, and use it to output the predicted class for x, i.e. h i (x). Note that a mixed strategy for the Defender is a non deterministic algorithm, since it depends on the sampling one makes on q. Hence, even if the attacks are defined in the same way as before, the Adversary now needs to maximize a new objective function which is the expectation of the adversarial risk under the distribution m q h . It writes as follows:

E Y ∼ν E X∼ϕ Y #µ Y E Ŷ ∼m q h (X) 1 Ŷ ̸ = Y -λ Ω(ϕ). (B.8)
We also write R Ω adv to mean the left part of Equation (B.8), when it is clear from context that the Defender uses a mixed classifier. Using this new set of strategies for the Defender, we can study whether mixed classifiers outperform deterministic ones, and how to efficiently design them.

Mixed strategy. We demonstrate that the efficiency of any deterministic defense can be improved using a simple mixed strategy. This method presents similarities with the notions of fictitious play [START_REF] Brown | Iterative solution of games by fictitious play[END_REF] in game theory, and boosting in machine learning [START_REF] Freund | A Decision Theoretic Generalization of On-Line Learning and an Application to Boosting[END_REF]. Given a deterministic classifier h 1 , we combine it (via randomization) with the best response h 2 to its optimal attack. The rational behind this idea is that, by construction, efficient attacks on one of these two classifiers will not work on the other. Mixing h 1 with h 2 has two opposite consequences on the adversarial risk. On one hand, where we only had to defend against attack on h 1 , we are now also vulnerable to attacks on h 2 , so the total set of possible attacks is now bigger. On the other hand, each attack will only work part of the time, depending on the probability distribution q. If we can calibrate the weights so that attacks on important zones have a low probability of succeeding, then the average risk under attack on the mixture will be low.

Toy example where a mixture outperforms AT. To better understand how randomization can work, let us look at a simple toy example. Figure B.2 illustrates a binary classification setting between two set of points. Attacking the Optimal Bayes Classifier (bold straight line) consists in moving all the points that lie between the dotted lines to the opposite side of the decision boundary (Figure B.2, left). The general tactic to defend against an attack is to change the classifier's output for points that are too close to the boundary. This can be done all the time, as in Adversar-ial Training (where we move the decision boundary to incorporate adversarial examples), or part of the time as in a randomized algorithm (so that the attack only works with a given probability).

When we use Adversarial Training for the star points (Figure B.2, middle), we change the output on the blue zone, so that 2 of the star (squared) points cannot be successfully attacked anymore. But in exchange, the dilation of the new boundary can now be attacked. For Adversarial Training to work, we need the number of new potential attacks (i.e. the points that are circled, 2 crosses in the dilation and 2 stars that are close to the new boundary) to be smaller than the number of attacks we prevent (the squared points, 2 blue ones that an attack would send in the blue zone, and 3 red points that are far from the new decision boundary). Here we prevent 5 attacks at the cost of 4 new ones, so the Adversarial Training improves the total score from 8 to 7.

Similarly, we observe what happens for the randomized defense (Figure B.2, right). We mix the Optimal Bayes Classifier with the best response to attacking all the points. We get a classifier that is determinsitic outside the gray area, and random inside it 3 . If the first classifier has a weight α = 0.5, 6 of the old attacks now succeed only with probability 0.5 (crosses between the dotted lines), whereas 3 new attacks are created (stars outside of the gray area) that succeed with probability 0.5 also. At the end, the average rate of successful attacks is 6.5, where adversarial training previously achieved 7.

More formally, Theorem B.4 shows that whatever penalty we consider, a deterministic classifier can always be outperformed by a randomized algorithm. We now can state our second main result: randomization matters. andm q h is the mixture of h by q. A similar result holds when Ω = Ω norm (see supplementary materials).

Theorem B.2. (Randomization matters) Let us consider

h 1 ∈ H, λ ∈ (0, 1), Ω = Ω mass , ϕ ∈ BR Ω (h 1 ) and h 2 ∈ BR(ϕ). Then for any α ∈ (max(λ, 1 -λ), 1) and for any ϕ ′ ∈ BR Ω (m q h ) one has R Ω adv (m q h , ϕ ′ ) < R Ω adv (h 1 , ϕ). Where h = (h 1 , h 2 ), q = (α, 1 -α),
Remark Note that depending on the initial hypothesis h 1 and the conditional distributions µ 1 and µ -1 , the gap between R Ω adv (m q h , ϕ ′ )and R Ω adv (h 1 , ϕ) could vary. Hence, with additional conditions on h 1 , µ 1 and µ -1 , we could make the gap appear more explicitly. We keep the formulation general to emphasize that for any deterministic classifier, there exists a randomized one that outperforms it in terms of worst-case adversarial score.

Based on Theorem B.

q ← (1 -α, α) q ← (h 1 , h 2 ) return m q h
Comparison to fictitious play. Contrary to classical algorithms such as Fictitious play that also generates mixtures of classifiers, and whose theoretical guarantees rely on the existence of a Mixed Nash Equilibrium, the performance of our method is ensured by Theorem B.4 to be at least as good as the classifier it uses as a basis. Moreover, the implementation of Fictitious Play would be impractical on the high dimensional datasets we consider, due to its computational costs.

B.7 Discussion & Conclusion

Finally, is there a classifier that ensures optimal robustness against all adversarial attacks? We gave a negative answer to this question in the deterministic regime, but part of the question remains open when considering randomized algorithms. We demonstrated that randomized defenses are more efficient than deterministic ones, and devised a simple method to implement them.

Game theoretical point of view. There remains to study whether an Equilibrium exists in the Randomized regime. This question is appealing from a theoretical point of view, and requires to investigate the space of randomized Adversaries P((F X ) 2 ). The characterization of this space is not straightforward, and would require strong results in the theory of optimal transport. A possible research direction is to quotient the space (F X ) 2 so as to simplify the search in P((F X ) 2 ) and the characterization of the Adversary's best responses. The study of this equilibrium is tightly related to that of the value of the game, which would be interesting for obtaining min-max bounds on the accuracy under attack, as well as certificates of robustness for a set of classifiers.

Advocating for more provable defenses. Although the experimental results show that our mixture of classifiers outperforms Adversarial Training, our algorithm does not provide guarantees in terms of certified accuracy. As the literature on adversarial attacks and defenses demonstrated, better attacks always exist. This is why, more theoretical works need to be done to prove the robustness of a mixture created from this particular algorithm. More generally, our work advocates for the study of mixtures as a provable defense against adversarial attacks. One could, for example, build upon the connection between mixtures and noise injection to investigate a broader range of randomized strategies for the Defender, and devise certificates accordingly.

Improving Boosted Adversarial Training. From an algorithmic point of view, BAT can be improved in several ways. For instance, the weights can be learned while choosing the new classifier for the mixture. This could lead to an improved accuracy under attack, but would lack some theoretical justifications that still need to be set up. Finally, tighter connections with standard boosting algorithms could be established to improve the analysis of BAT.

B.8 Omitted proofs and Additional results

Notations. Let us suppose that (X ,∥.∥) is a normed vector space. B ∥.∥ (x, ϵ) = {z ∈ X | ∥x -z∥ ≤ ϵ} is the closed ball of center x and radius ϵ for the norm ∥.∥. Note that H := {h : x → sgn g(x) | g : X → R continuous}, with sgn the function that outputs 1 if g(x) > 0, -1 if g(x) < 0, and 0 otherwise. Hence for any (x, y) ∼ D, and h ∈ H one has 1{h(x) ̸ = y} = 1{g(x)y ≤ 0}. Finally, we denote ν 1 and ν -1 respectively the probabilities of class 1 and -1.

Introducing remarks.

Let us first note that in the paper, the penalties are defined with an ℓ 2 norm. However, Lemma B.1 and B.2 hold as long as X is an Hilbert space with dot product <|> and associated norm ||.|| = < . | . >. We first demonstrate Lemma B.2 with these general notations. Then we present the proof of Lemma B.1 that follows the same schema. Note that, for Lemma B.1, we do not even need the norm to be Hilbertian, since the core argument rely on separation property of the norm, i.e. on the property ∥x -y∥ = 0 ⇐⇒ x = y. Lemma B.2. Let h ∈ H and ϕ ∈ BR Ωnorm (h). Then the following assertion holds:

ϕ 1 (x) = π(x) if x ∈ P h (ϵ 2 ) x otherwise.
Where π is the orthogonal projection on (P h ) ∁ . ϕ -1 is characterized symmetrically.

Proof. Let us first simplify the worst case adversarial risk for h. Recall that h = sgn(g) with g continuous. From the definition of adversarial risk we have:

sup ϕ∈(F X |ϵ 2 ) 2 R Ωnorm adv (h, ϕ) = sup ϕ∈(F X ) 2 y=±1 ν y E X∼µy 1{h(ϕ y (X)) ̸ = y} -λ∥X -ϕ y (X)∥ -∞1{∥X -ϕ y (X)∥ > ϵ 2 } = sup ϕ∈(F X ) 2 y=±1 ν y E X∼µy 1{g(ϕ y (X))y ≤ 0} -λ∥X -ϕ y (X)∥ -∞1{∥X -ϕ y (X)∥ > ϵ 2 } = y=±1 ν y sup ϕy∈F X E X∼µy 1{g(ϕ y (X))y ≤ 0} -λ∥X -ϕ y (X)∥ -∞1{∥X -ϕ y (X)∥ > ϵ 2 }
Finding ϕ 1 and ϕ 1 are two independent optimization problems, hence, we focus on characterizing ϕ 1 (i.e. y = 1).

sup

ϕ 1 ∈F X E X∼µ 1 1{g(ϕ 1 (X)) ≤ 0} -λ∥X -ϕ 1 (X)∥ -∞1{∥X -ϕ 1 (X)∥ > ϵ 2 } = E X∼µ 1 essup z∈B ∥.∥ (X,ϵ 2 ) 1(g(z) ≤ 0) -λ∥X -z∥ = X essup z∈B ∥.∥ (x,ϵ 2 )
1{g(z) ≤ 0} -λ∥x -z∥ dµ 1 (x).

Let us now consider (H j ) j∈J a partition of X , we can write.

sup

ϕ 1 ∈F X E X∼µ 1 1{g(ϕ 1 (X)) ≤ 0} -λ∥X -ϕ 1 (X)∥ -∞1{∥X -ϕ 1 (X)∥ > ϵ 2 } = j∈J H j essup z∈B ∥.∥ (x,ϵ 2 ) 1{g(z) ≤ 0} -λ∥x -z∥ dµ 1 (x)
In particular, we consider here H 0 = P ∁ h , H 1 = P h \ P h (ϵ 2 ), and H 2 = P h (ϵ 2 ).

For x ∈ H 0 = P ∁ h . Taking z = x we get 1{g(z) ≤ 0} -λ∥x -z∥ = 1. Since for any z ∈ X we have 1{g(z) ≤ 0} -λ∥x -z∥ ≤ 1, this strategy is optimal. Furthermore, for any other optimal strategy z ′ , we would have ∥x -z ′ ∥ = 0, hence z ′ = x, and an optimal attack will never move the points of H 0 = P ∁ h .

For x ∈ H 1 = P h \ P h (ϵ 2 ). We have B ∥.∥ (x, ϵ 2 ) ⊂ P h by definition of P h (ϵ 2 ). Hence, for any z ∈ B ∥.∥ (x, ϵ 2 ), one gets g(z) > 0. Then 1{g(z) ≤ 0} -λ∥x -z∥ ≤ 0. The only optimal z will thus be z = x, giving value 0.

Let us now consider x ∈ H 2 = P h (ϵ 2 ) which is the interesting case where an attack is possible. We know that B ∥.∥ (x, ϵ 2 )∩P ∁ h ̸ = ∅, and for any z in this intersection, 1(g(z) ≤ 0) = 1. Hence :

essup z∈B ∥.∥ (x,ϵ 2 ) 1{g(z) ≤ 0} -λ∥x -z∥ = max(1 -λ essinf z∈B ∥.∥ (x,ϵ 2 )∩P ∁ h ∥x -z∥, 0) (B.9) = max(1 -λπ B ∥.∥ (x,ϵ 2 )∩P ∁ h (x), 0) (B.10)
Where

π B ∥.∥ (x,ϵ 2 )∩P ∁ h is the projection on the closure of B ∥.∥ (x, ϵ 2 ) ∩ P ∁ h . Note that π B ∥.∥ (x,ϵ 2 )∩P ∁ h exists: g is continuous, so B ∥.∥ (x, ϵ 2 ) ∩ P ∁
h is a closed set, bounded, and thus compact, since we are in finite dimension. The projection is however not guaranteed to be unique since we have no evidence on the convexity of the set. Finally, let us remark that, since λ ∈ (0, 1), and

ϵ 2 ≤ 1, one has 1 -λπ B ∥.∥ (x,ϵ 2 )∩P ∁ h (x) ≥ 0 for any x ∈ H 2 .
Hence, on P h (ϵ 2 ), the optimal attack projects all the points on the decision boundary. For simplicity, and since there is no ambiguity, we write the projection π.

Finally. Since H 0 ∪ H 1 ∪ H 2 = X , Lemma B.2 holds. Furthermore, the score for this optimal attack is:

sup ϕ∈(F X |ϵ 2 ) 2 R Ωnorm adv (h, ϕ) = y=±1 ν y j∈J H j essup z∈B ∥.∥ (x,ϵ 2 ) 1{g(z)y ≤ 0} -λ∥x -z∥ dµ y (x)
Since the value is 0 on P h \ P h (ϵ 2 ) (resp. on N h \ N h (ϵ 2 ) ) for ϕ 1 (resp. ϕ -1 ), one gets:

=ν 1     P h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ 1 (x) + P ∁ h 1dµ 1 (x)     + ν -1     N h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ -1 (x) + N ∁ h 1dµ -1 (x)     =ν 1    P h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ 1 (x) + µ 1 (P ∁ h )    + ν -1    N h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ -1 (x) + µ -1 (N ∁ h )    = R(h) + ν 1 P h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ 1 (x) + ν -1 N h (ϵ 2 ) 1 -λ∥x -π(x)∥ dµ -1 (x) (16) holds since R(h) = P(h(X) ̸ = Y )P(g(X)Y ≤ 0) = ν 1 µ 1 (P ∁ h ) + ν -1 µ -1 (N ∁ h
). This provides an interesting decomposition of the adversarial risk into the risk without attack and the loss on the attack zone.

Lemma B.1. Let h ∈ H and ϕ ∈ BR Ωmass (h). Then the following assertion holds:

ϕ 1 (x) ∈ (P h ) ∁ if x ∈ P h (ϵ 2 ) ϕ 1 (x) = x otherwise.
Where (P h ) ∁ , the complement of P h in X . ϕ -1 is characterized symmetrically.

Proof. Following the same proof schema as before the adversarial risk writes as follows:

sup ϕ∈(F X |ϵ 2 ) 2 R Ωmass adv (h, ϕ) = sup ϕ∈(F X ) 2 y=±1 ν y E X∼µy [1{h(ϕ y (X)) ̸ = y} -λ1{X ̸ = ϕ y (X)} -∞1{∥X -ϕ y (X)∥ > ϵ 2 }] = sup ϕ∈(F X ) 2 y=±1 ν y E X∼µy [1{g(ϕ y (X))y ≤ 0} -λ1{X ̸ = ϕ y (X)} -∞1{∥X -ϕ y (X)∥ > ϵ 2 }] = y=±1 ν y sup ϕy∈F X E X∼µy [1{g(ϕ y (X))y ≤ 0} -λ1{X ̸ = ϕ y (X)} -∞1{∥X -ϕ y (X)∥ > ϵ 2 }]
Finding ϕ 1 and ϕ 1 are two independent optimization problem, hence we focus on characterizing ϕ 1 (i.e. y = 1).

sup

ϕ 1 ∈F X E X∼µ 1 1{g(ϕ 1 (X)) ≤ 0} -λ1{X ̸ = ϕ 1 (X)} -∞1{∥X -ϕ 1 (X)∥ > ϵ 2 } = E X∼µ 1 essup z∈B ∥.∥ (X,ϵ 2 ) 1{g(z) ≤ 0} -λ1{X ̸ = z} = X essup z∈B ∥.∥ (x,ϵ 2 ) 1{g(z) ≤ 0} -λ1{x ̸ = z} dµ 1 (x).
Let us now consider (H j ) j∈J a partition of X , we can write.

sup

ϕ 1 ∈F X E X∼µ 1 1{g(ϕ 1 (X)) ≤ 0} -λ1{X ̸ = ϕ 1 (X)} -∞1{∥X -ϕ 1 (X)∥ > ϵ 2 } = j∈J H j essup z∈B ∥.∥ (x,ϵ 2 ) 1{g(z) ≤ 0} -λ1{x ̸ = z} dµ 1 (x)
In particular, we can take H 0 = P ∁ h , H 1 = P h \ P h (ϵ 2 ), and H 2 = P h (ϵ 2 ).

For x ∈ H 0 = P ∁ h or x ∈ H 1 = P h \ P h (ϵ 2 )
. With the same reasoning as before, any optimal attack will choose ϕ 1 (x) = x.

Let x ∈ H 2 = P h (ϵ 2 ). We know that B ∥.∥ (x, ϵ 2 ) ∩ P ∁ h ̸ = ∅, and for any z in this intersection, one has g(z) ≤ 0 and z ̸ = x. Hence essup

z∈B ∥.∥ (x,ϵ 2 ) 1{g(z) ≤ 0} -λ1{z ̸ = x} = max(1 -λ, 0). Since λ ∈ (0, 1) one has 1{g(z) ≤ 0} -λ1{z ̸ = x} = 1 -λ for any z ∈ B ∥.∥ (x, ϵ 2 )∩P ∁ h . Then any function that given a x ∈ X outputs ϕ 1 (x) ∈ B ∥.∥ (x, ϵ 2 )∩P ∁ h is optimal on H 2 .
Finally. Proof. We reason ad absurdum. Let us consider y = 1, the proof for y = -1 is symmetrical.

Since H 0 ∪ H 1 ∪ H 2 = X ,
Let us suppose that there exists

C ⊂ P h such that ν -1 ϕ -1 #µ -1 (C) > ν 1 ϕ 1 #µ 1 (C).
We can then construct h 1 as follows:

h 1 (x) = h(x) if x / ∈ C -1
otherwise.

Since h and h 1 are identical outside C, the difference between the adversarial risks of h and h 1 writes as follows:

R Ωmass adv (h, ϕ) -R Ωmass adv (h 1 , ϕ) = y=±1 ν y C 1{h(x) ̸ = y} -1{h 1 (x) ̸ = y} d(ϕ y #µ y )(x) =ν -1 1{h(x) = 1}ϕ -1 #µ -1 (C) -ν 1 1{h 1 (x) ̸ = 1}ϕ 1 #µ 1 (C) =ν -1 ϕ -1 #µ -1 (C) -ν 1 ϕ 1 #µ 1 (C)
Since by hypothesis ν -1 ϕ -1 #µ -1 (C) > ν 1 ϕ 1 #µ 1 (C) the difference between the adversarial risks of h and h 1 is strictly positive. This means that h 1 gives strictly better adversarial risk than the best response h. Since, by definition h is supposed to be optimal, this leads to a contradiction. Hence Lemma B.3 holds.

Additional Result. Let us assume that there is a probability measure ζ that dominates both ϕ 1 #µ 1 and ϕ -1 #µ -1 . Let us consider ϕ ∈ F X |ϵ 2 2 . If we take h ∈ BR(ϕ), then h is the Bayes Optimal Classifier for the distribution characterized by (ν,

ϕ 1 #µ 1 , ϕ -1 #µ -1 ).
Proof. For simplicity, we denote

f 1 = (dϕ 1 #µ 1 ) dζ and f -1 = d(ϕ -1 #µ -1 )
dζ the Radon-Nikodym derivatives of ϕ 1 #µ 1 and ϕ -1 #µ -1 w.r.t. ζ. The best response h minimizes adversarial risk under attack ϕ. This minimal risk writes:

inf h∈H R Ωmass adv (h, ϕ) = inf h∈H y=±1 ν y E x∼µy [1{h(ϕ y (x)) ̸ = y}] -λ Ω(ϕ).
Since the the penalty function does not depend on h, it suffices to seek inf Finally, since the integral is bounded we get:

inf h∈H X y=±1 ν y 1{h(x) ̸ = y}f y (x) dζ(x)
:= u + v | (u, v) ∈ U × X and ∥v∥ p ≤ ϵ 2 .
We can construct h 2 as follows

h 2 (x) = -h 1 (x) if x ∈ U h 1 (x) otherwise.
This means that h 2 changes the class of all points in U , and do not change the rest, compared to h 1 . Then taking α ∈ (0, 1), we can define m q h , and ϕ ′ ∈ BR Ω (m q h ). We aim to find a condition on α so that the score of m q h is lower than the score of h 1 . Finally, let us recall that

R Ωmass adv (m q h , ϕ ′ ) = ν 1 X essup z∈B ∥.∥ (x,ϵ 2 ) α1{h 1 (z) = -1} + (1 -α)1{h 2 (z) = -1} -λ1{x ̸ = z} dµ 1 (x) + ν -1 X essup z∈B ∥.∥ (x,ϵ 2 ) α1{h 1 (z) = 10} + (1 -α)1{h 2 (z) = 1} -λ1{x ̸ = z} dµ -1 (x).
The only terms that may vary between the score of h 1 and the score of m q h are the integrals on U , U ⊕ϵ 2 ∩P h 1 and ϕ -1 -1 (U ) -inverse image of U by ϕ -1 . These sets represent respectively the points we mix on, the points that may become attacked -when changing from h 1 to m q h -by moving them on U , and the ones that were -for h 1 -attacked before by moving them on U . Hence, for simplicity, we only write those terms. Furthermore, we denote

U + := U ⊕ ϵ 2 ∩ P h 1 \ U, U -:= ϕ -1
-1 (U ) and recall U := P h 1 (ϵ 2 ).

One can refer to Figure B.3 for visual interpretation of this sets. We can now evaluate the worst case adversarial score for h 1 restricted to the above sets. Thanks to Lemma B.1 that characterizes ϕ, we can write

R Ωmass adv (h 1 , ϕ) |U, U + , U - = (1 -λ) × ν 1 µ 1 (U ) + ν -1 µ -1 (U ) + 0 × ν 1 µ 1 U + + ν -1 µ -1 U + + ν 1 µ 1 U -+ (1 -λ) × ν -1 µ -1 U -.
Similarly, we can write the worst case adversarial score of the mixture on the sets we consider. Note that the max operator comes from the fact that the adversary has to make a choice between attacking the zone or just take advantage of the error due to randomization.

R Ωmass adv (m q h , ϕ ′ ) |U, U + , U - = max(1 -α, 1 -λ) × ν 1 µ 1 (U ) + max(α, 1 -λ) × ν -1 µ -1 (U ) + max(0, 1 -α -λ) × ν 1 µ 1 U + + ν -1 µ -1 U + + ν 1 µ 1 U -+ max(0, α -λ) × ν -1 µ -1 U -.
Computing the difference between these two terms, we get the following

R Ωmass adv (h 1 , ϕ) -R Ωmass adv (m q h , ϕ ′ ) (B.11) = (1 -λ -max(1 -α, 1 -λ)) × ν 1 µ 1 (U ) (B.12) + (1 -max(α, 1 -λ)) × ν -1 µ -1 (U ) (B.13) -max(0, 1 -α -λ) × ν 1 µ 1 U + (B.14) +(1 -λ -max(0, α -λ)) × ν -1 µ -1 U - (B.15)
Let us now simplify Equation (B.11) using additional assumptions.

• First, we have that Equation (B.13) is equal to

min(1 -α, λ)µ -1 (U )ν -1 > 0.
Thus, a sufficient condition for the difference between the adversarial scores to be positive is to have the other terms greater or equal to 0.

• To have Equation (B.12) ≥ 0 we can always set max(1 -α, 1 -λ) = 1 -λ. This gives us α ≥ λ.

• Also note that to get (B.14) ≥ 0, we can force max(1 -α -λ, 0) = 0. This gives us α ≥ 1 -λ.

• Finally, since α ≥ λ, we have that 1 -λ -max(0, α -λ) = 1 -α thus Equations (B.15) > 0.

With the above simplifications, we have (B.11) > 0 for any α > max(λ, 1 -λ) which concludes the proof.

Theorem B.5. (Randomization matters) Let us consider h 1 ∈ H, λ ∈ (0, 1), Ω = Ω norm , ϕ ∈ BR Ω (h 1 ) and h 2 ∈ BR(ϕ). Let us take δ ∈ (0, ϵ 2 ), then for any α ∈ (max(1λδ, λ(ϵ 2 -δ)), 1) and for any ϕ ′ ∈ BR Ω (m q h ) one has

R Ωnorm adv (m q h , ϕ ′ ) < R Ωnorm adv (h 1 , ϕ).
Where h = (h 1 , h 2 ), q = (α, 1 -α), and m q h is the mixture of h by q. Proof. Let us take U ⊂ P h 1 (ϵ 2 ) such that

min x∈U ∥x -π P h \P h (ϵ 2 ) (x)∥ = δ ∈ (0, ϵ 2 )
. We construct h 2 as follows.

h 2 (x) = -h 1 (x) if x ∈ U h 1 (x) otherwise.
This means that h 2 changes the class of all points in U , and do not change the rest. Let α ∈ (0, 1), the corresponding mixture m q h , and ϕ ′ ∈ BR Ω (m q h ). We will find a condition on α so that the score of m q h is lower than the score of h 1 . Recall that

R Ωnorm adv (m q h , ϕ ′ ) = ν 1 X essup z∈B ∥.∥ (x,ϵ 2 ) α1{h 1 (z) = -1} + (1 -α)1{h 2 (z) = -1} -λ∥x -z∥ dµ 1 (x) + ν -1 X essup z∈B ∥.∥ (x,ϵ 2 ) α1{h 1 (z) = 1} + (1 -α)1{h 2 (z) = 1} -λ∥x -z∥ dµ -1 (x).
As we discussed in proof of Theorem B.4, the only terms that may vary between the score of h 1 and the score of m q h are the integrals on U , U ⊕ ϵ 2 ∩ P h 1 and ϕ -1 -1 (U ). Hence, for simplicity, we only write those terms. Furthermore, we denote

U + := U ⊕ ϵ 2 ∩ P h 1 \ U, U -:= ϕ -1
-1 (U ) and P ϵ 2 := P h 1 (ϵ 2 ).

One can refer to Figure B.4 for a visual interpretation of this ensembles. We can now evaluate the worst case adversarial score for h 1 restricted to the above sets. Thanks to Lemma B.2 that characterizes ϕ, we can write

R Ωnorm adv (h 1 , ϕ) = ν 1 U 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) + ν -1 µ -1 (U ) + ν 1 U + \Pϵ 2 0 dµ 1 (x) + ν -1 µ -1 U + \ P ϵ 2 + ν 1 U + ∩Pϵ 2 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) + ν -1 µ -1 U + ∩ P ϵ 2 + ν 1 µ 1 U -+ ν -1 U - 1 -λ∥x -π U (x)∥ dµ -1 (x).
Similarly we can evaluate the worst case adversarial score for the mixture,

R Ωnorm adv (m q h , ϕ ′ ) = ν 1 U max 1 -α, 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) + ν -1 U max(α, 1 -λ∥x -π U + (x)∥) dµ -1 (x) + ν 1 U + \Pϵ 2 max(0, 1 -α -λ∥x -π U (x)∥) dµ 1 (x) + ν -1 µ -1 U + \ P ϵ 2 + ν 1 U + ∩Pϵ 2 max 1 -α -λ∥x -π U (x)∥, 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) + ν -1 µ -1 U + ∩ P ϵ 2 + ν 1 µ 1 U - + ν -1 U - max 0, 1 -λ∥x -π N ∁ h 1 \U (x)∥, α -λ∥x -π U (x)∥ dµ -1 (x).
Note that we need to take into account the special case of the points in the dilation that were already in the attacked zone before, and that can now be attacked in two ways, either by projecting on U -but that works with probability α, since the classification on U is now randomized -or by projecting on P ∁ h 1 , which works with probability 1 but may use more distance and so pay more penalty. We can now compute the difference between both scores.

R Ωnorm adv (h 1 , ϕ) -R Ωnorm adv (m q h , ϕ ′ ) (B.16) = ν 1 U 1 -λ∥x -π P ∁ h 1 (x)∥ -max 1 -α, 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) (B.17) + ν -1 U 1 -max(α, 1 -λ∥x -π U + (x)∥)dµ -1 (x) (B.18) -ν 1 U + \Pϵ 2 max(1 -α -λ∥x -π U (x)∥, 0)dµ 1 (x) (B.19) + ν 1 U + ∩Pϵ 2 1 -λ∥x -π P ∁ h 1 (x)∥ -max 1 -α -λ∥x -π U (x)∥, 1 -λ∥x -π P ∁ h 1 (x)∥ dµ 1 (x) (B.20) + ν -1 U - 1 -λ∥x -π U (x)∥ -max 0, 1 -λ∥x -π N ∁ h 1 \U (x)∥, α -λ∥x -π U (x)∥ dµ -1 (x). (B.21)
Let us simplify Equation (B.16) using using additional hypothesis:

• First, note that Equation (B.18)> 0. Then a sufficient condition for the difference to be strictly positive is to ensure that other lines are ≥ 0.

• In particular to have (B.17) ≥ 0 it is sufficient to have for all

x ∈ U max 1 -α, 1 -λ∥x -π P ∁ h 1 (x)∥ = 1 -λ∥x -π P ∁ h 1 (x)∥. This gives us α ≥ λ(ϵ 2 -δ) ≥ λ max x∈U ∥x -π P ∁ h 1 (x)∥.
• Similarly, to have (B. [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF]) ≥ 0, we should set for all x ∈ U + \ P ϵ 2 α ≥ 1 -λ∥x -π U (x)∥.

Since min

x∈U + \Pϵ 2 ∥x -π U (x)∥ = δ, we get the condition α ≥ 1 -λδ.

• Finally (B.21) ≥ 0, since by definition of U -, for any x ∈ U -we have

∥x -π N ∁ h 1 \U (x)∥ ≥ ∥x -π U (x)∥.
Finally, by summing all these simplifications, we have (B.16) > 0. Hence the result hold for any α > max(1 -λδ, λ(ϵ 2 -δ))

B.9 Experimental results

In the experimental section, we consider X = [0, 1] 3×32×32 to be the set of images, and Y = {1, ..., 10} or Y = {1, ..., 100} according to the dataset at hand.

Adversarial attacks

Let (x, y) ∼ D and h ∈ H. We consider the following attacks: (i) ℓ ∞ -PGD attack. In this scenario, the Adversary maximizes the loss objective function, under the constraint that the ℓ ∞ norm of the perturbation remains bounded by some value ϵ ∞ . To do so, it recursively computes:

x t+1 = Π B ∥.∥ (x,ϵ∞) x t + β sgn ∇ x L h x t , y (B. 22
)
where L is some differentiable loss (such as the cross-entropy), β is a gradient step size, and Π S is the projection operator on S. One can refer to [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF] for implementation details.

(ii) ℓ 2 -C&W attack. In this attack, the Adversary optimizes the following objective:

arg min τ ∈X ∥τ ∥ 2 + λ × cost(x + τ ) (B. 23
)
where cost(x + τ ) < 0 if and only if h(x + τ ) ̸ = y. The authors use a change of variable τ = 1 2 (tanh(w) -x + 1) to ensure that x + τ ∈ X , a binary search to optimize the constant λ, and Adam or SGD to compute an approximated solution. One should refer to [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] for implementation details.

Experimental setup

Datasets. To illustrate our theoretical results we did experiments on the CIFAR10 and CI-FAR100 datasets. See [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] for more details.

Classifiers. All the classifiers we use are WideResNets (see [108]) with 28 layers, a widen factor of 10, a dropout factor of 0.3 and LeakyRelu activations with a 0.1 slope.

Machine used. 6 Tesla V100-SXM2-32GB GPUs

Experimental details

Sanity checks for Adaptive attacks In [START_REF] Tramer | On adaptive attacks to adversarial example defenses[END_REF], the authors give a lot of sanity checks and good practices to design an Adaptive attacks. We follow them and here are the information for Adaptiveℓ ∞ -PGD :

• We compute the gradient of the loss by doing the expected logits over the mixture.

• The attack is repeated 3 times with random start and we take the best perturbation over all the iterations.

• When adding a constant to the logits, it doesn't change anything to the attack • The loss doesn't fluctuate at the end of the optimization process.

Selecting the first element of the mixture. Our algorithm creates classifiers in a boosting fashion, starting with an adversarially trained classifier. There are several ways of selecting this first element of the mixture: use the classifier with the best accuracy under attack (option 1, called bestAUA), or rather the one with the best natural accuracy (option 2). Table B.6 compares both options.

Beside the fact that any of the two mixtures outperforms the first classifier, we see that the fisrt option always outperforms the second. In fact, when taking option 1 (bestAUA = True) the accuracy under ℓ ∞ -PGD attack of the mixture is 3% better than with option 2 (bestAUA = False). One can also note that both mixtures have the same natural accuracy (0.80), which makes the choice of option 1 natural.

Extension to more than two classifiers

As we mention in the main part of the paper, a mixture of more than two classifiers can be constructed by adding at each step t a new classifier trained naturally on the dataset D that contains adversarial examples against the mixture at step t -1. Since D has to be constructed from a mixture, one would have to use an adaptive attack as Adaptive-ℓ ∞ -PGD. Here is the algorithm for the extented version :

C.1 Contexte & motivations

Cette thèse vise à résoudre des problèmes multi-agents centralisés qui impliquent des interactions par paire entre agents. La configuration d'antennes dans un réseau cellulaire sans fil [START_REF] Siomina | Automated optimization of service coverage and base station antenna configuration in UMTS networks[END_REF] est un exemple de ces problèmes : le choix d'un paramètre pour une antenne a un impact à la fois sur sa propre qualité de signal et sur celle de chacune de ses antennes voisines en raison de l'interférence du signal. De même, dans un parc éolien, le réglage d'une éolienne a un impact non seulement sur sa propre efficacité de collecte d'énergie mais aussi sur celle de ses voisines en raison des turbulences du vent [START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF][START_REF] Van Dijk | Yaw-misalignment and its impact on wind turbine loads and wind farm power output[END_REF]. En considérant chaque antenne ou éolienne comme un agent, ces problèmes peuvent être modélisés comme un problème de bandit multi-agents (MA-MAB) [START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF] avec la connaissance d'un graphe de coordination [START_REF] Guestrin | Coordinated Reinforcement Learning[END_REF] où chaque noeud représente un agent et chaque arête représente une interaction entre deux agents. Un problème de bandit à bras multiples (MAB) est un problème de décision séquentiel où un apprenant doit choisir une action (aussi appelée bras) à chaque itération et obtient une récompense associée (éventuellement perturbée) qui informe sur la qualité de l'action choisie. Naturellement, l'apprenant ne connaît pas la distribution de la récompense pour chaque action possible. L'apprenant peut avoir des objectifs très différents, tels que maximiser les récompenses accumulées au cours du processus, ou bien en un nombre minimum d'essais et indépendamment des récompenses accumulées, déduire quelle est la meilleure action à choisir -c'est-à-dire la plus gratifiante. Par conséquent, un problème de bandit multi-agents à plusieurs bras est le cadre dans lequel plusieurs agents sont confrontés à un problème de bandit à plusieurs bras. Dans la littérature sur les bandits, on peut distinguer les bandits non structurés et les bandits structurés. Alors que le bandit non structuré considère que le fait de jouer une action et d'obtenir la récompense associée ne permet pas de déduire quoi que ce soit sur la distribution des récompenses des autres actions, le bandit structuré inclut les modèle de bandit où les récompenses des différentes actions partagent un paramètre commun [START_REF] Lattimore | Bandit Algorithms[END_REF]. Par exemple, une configuration de bandit structuré deja très etudiée dans la litterature est le bandit linéaire [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF] où la récompense associée à toute action dépend linéairement d'un vecteur paramètre inconnu θ. Par conséquent, à un moment donné, le fait de choisir une action et de recevoir la récompense qui lui est associée donne des informations sur θ et, par définition, également sur les récompenses de toutes les autres actions. Nous nous intéressons ici à de tels environnements structurés et nous présentons à ce sujet un nouveau bandit structuré multi-agents appelé Bandits Bilinéaires Graphiques. La spécificité de cet environnement réside dans l'interdépendance des récompenses obtenues par les agents voisins dans le graphe et dans l'hypothèse que ces récompenses sont bilinéaires, ce qui nous apparaît comme l'extension naturelle des récompenses linéaires lorsque les agents sont dépendants par paire. En effet, si les problèmes MA-MAB ont été étudiés dans le cadre de bandits non structurés avec des agents indépendants et dépendants (voir e.g., [START_REF] Agarwal | Multi-agent multi-armed bandits with limited communication[END_REF][START_REF] Amin | Graphical Models for Bandit Problems[END_REF][START_REF] Bargiacchi | Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems[END_REF][START_REF] Besson | Multi-player bandits revisited[END_REF][START_REF] Bistritz | Distributed multi-player bandits-a game of thrones approach[END_REF][START_REF] Heliou | Learning with bandit feedback in potential games[END_REF][START_REF] Landgren | Distributed cooperative decision making in multi-agent multi-armed bandits[END_REF][START_REF] Sankararaman | Social learning in multi agent multi armed bandits[END_REF][START_REF] Shahrampour | Multi-armed bandits in multi-agent networks[END_REF]101]), seul le cadre des bandits structurés avec des agents indépendants a été exploré (voir e.g., [START_REF] Amani | Decentralized multi-agent linear bandits with safety constraints[END_REF][START_REF] Cesa-Bianchi | A gang of bandits[END_REF][START_REF] Chan | Parallelizing contextual linear bandits[END_REF]). A travers cette thèse et les articles auxquels elle fait référence, nous voulons poser une première pierre à l'édifice.

C.2 Définition du problème Bandits Bilinéaires Graphiques Stochastiques

Soit G = (V, E) le graphe dirigé défini par V l'ensemble fini de noeuds représentant les agents et E l'ensemble d'arêtes représentant les interactions entre les agents. Nous supposons que si (i, j) ∈ E alors (j, i) ∈ E. Le graphe pourrait être considéré comme non dirigé mais nous supposons que les interactions entre deux voisins ne sont pas nécessairement symétriques par rapport aux récompenses obtenues, nous choisissons donc de conserver le graphe dirigé pour mettre en évidence cette asymétrie potentielle. Pour tout agent i ∈ V , nous désignons N i l'ensemble de ses agents voisins. Soit n = |V | le nombre de noeuds, m = |E| le nombre d'arêtes et X ⊂ R d un ensemble de bras fini où K = |X | désigne le nombre de bras. Le bandit bilinéaire graphique avec un graphe G et un ensemble de bras X consiste en le problème de décision séquentiel suivant : Bandits Bilinéaires Graphiques Stochastiques Pour chaque tour t > 0,

1. Chaque agent i ∈ V choisit un bras x (i) t dans X .
2. Ensuite, chaque agent i ∈ V reçoit une récompense bilinéaire bruitée pour chacun de ses voisins j ∈ N i :

y (i,j) t = x (i)⊤ t M ⋆ x (j) t + η (i,j) t , (

Objectifs

Comme nous l'avons brièvement mentionné précédemment, il existe deux principaux objectifs différents qu'un apprenant (ici l'entité centrale) peut vouloir atteindre dans un problème de bandit.

Identifier la meilleure allocation. Le premier objectif que nous voulons traiter dans cette thèse est celui où l'apprenant est intéressé à trouver, en un minimum de tours, la meilleure allocations de bras (x

(1) ⋆ , . . . , x (n)
⋆ ) qui maximise la récompense globale moyenne obtenue sur le graphe :

(x (1) ⋆ , . . . , x (n) ⋆ ) = arg max (x (1) ,...,x (n) )∈X n (i,j)∈E x (i)⊤ M ⋆ x (j) .
Cet objectif implique que l'entité centrale ne se soucie pas de choisir une allocation sous-optimale (x (1) t , . . . , x (n) t ) à chaque instant t tant qu'elle donne suffisamment d'informations sur le paramètre inconnu M ⋆ afin de construire une estimation précise M. Cet objectif est connu sous le nom d'exploration pure ou d'identification du meilleur bras [START_REF] Audibert | Best arm identification in multi-armed bandits[END_REF][START_REF] Bubeck | Pure exploration in multi-armed bandits problems[END_REF].

Maximiser les récompenses cumulées. Le deuxième objectif que nous voulons traité est le plus souvent considéré dans la littérature sur les bandits où l'apprenant souhaite maximiser la somme des récompenses (en esperance) obtenues au cours des tours. Dans notre cas, l'entité centrale souhaite maximiser les récompenses globales cumulées, données par la formule suivante

T t=1 (i,j)∈E x (i)⊤ t M ⋆ x (j) t .
Alors que le premier objectif permet à l'apprenant d'être dans un cadre d'exploration pure, indépendamment des récompenses obtenues tout au long du processus, l'objectif de maximisation des récompenses cumulées nécessite un compromis entre l'exploration des différents bras possibles pour avoir une estimation précise M de M ⋆ et l'exploitation des bras qui semblent être les plus optimaux étant donné M afin d'obtenir les récompenses cumulées maximales.

Pour ces deux objectifs (identification du meilleur bras ou maximisation des récompenses cumulées) et étant donné une estimation M, l'apprenant devra résoudre à un moment donné le problème d'optimisation suivant

max (x (1) ,...,x (n) )∈X n (i,j)∈E x (i)⊤ Mx (j) . (C.2)
En effet, pour l'identification du meilleur bras, ce problème d'optimisation doit être résolu à la fin lorsque l'apprenant veut retourner la meillieure allocation étant donné l'estimation M construite pendant la procédure d'apprentissage. Pour la maximisation des récompenses cumulées, ce problème d'optimisation peut devoir être résolu pendant la procédure d'apprentissage lorsque l'apprenant veut exploiter et renvoyer le meilleur bras articulé estimé compte tenu de sa connaissance actuelle de l'environnement qui est l'estimation construite M.

La résolution de ce problème d'optimisation n'est pas triviale, aussi pour les deux objectifs nous considérons l'objectif sous-jacent commun de résolution de ce problème.

C.3 Trouver la meilleure allocation lorsque la matrice est connu C.3.1 Un problème NP-Dur

Nous abordons le problème de la recherche de la meilleure alloaction étant donné M ⋆ et nous le désignons comme suit :

(x (1) ⋆ , . . . , x (n) ⋆ ) = arg max (x (1) ,...,x (n) )∈X n (i,j)∈E x (i)⊤ M ⋆ x (j) . (C.3) Remarquez que si le couple (x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ est tel que x ⋆ = x ′
⋆ alors trouver la meilleure allocation est trivial et la solution est d'attribuer x ⋆ à tous les noeuds. À l'inverse, si x ⋆ ̸ = x ′ ⋆ , le problème peut être plus difficile : selon le graphe G, l'allocation optimal pourrait soit être composée exclusivement du couple (x ⋆ , x ′ ⋆ ), soit être composée d'autres bras dans X . On pourrait vouloir utiliser la programmation dynamique comme dans [START_REF] Amin | Graphical Models for Bandit Problems[END_REF] pour résoudre ce problème d'optimisation, cependant dans ce cadre particulier, cela conduirait à utiliser un algorithme en temps non-polynomial. En effet, le théorème suivant indique que, même en connaissant le vrai paramètre M ⋆ , l'identification de la meilleure allocation (x

(x (1) ,...,x (n) )∈X n (i,j)∈E x (i)⊤ M ⋆ x (j) .
Par conséquent, étant donné la vraie matrice M ⋆ , l'apprenant n'est pas assuré de trouver en temps polynomial l'alloaction x

C.3.2 Algorithmes d'approximation et guaranties théoriques

Étant donné la matrice M ⋆ , l'objectif est de concevoir un algorithme qui renvoie une allocation x (1) , . . . , x (n) telle que sa récompense globale associée y = (i,j)∈E x (i)⊤ M ⋆ x (j) ait la garantie d'être proche de la récompense globale optimale y ⋆ = (i,j)∈E x

(i)⊤ ⋆ M ⋆ x (j)
⋆ . En d'autres termes, nous voulons trouver un paramètre d'approximation 0 < α ≤ 1 tel que, y ≥ αy ⋆ . Bien que le problème d'optimisation que nous cherchons à résoudre se trouve dans la littérature sur les Champs aléatoires de Markov lorsqu'on traite un graphe multi-labeélisé (voir e.g., [START_REF] Ajanthan | Optimization of Markov random fields in computer vision[END_REF]), à notre connaissance, les algorithmes qui donnent un rapport d'approximation sur la solution optimale n'ont pas été explorés.

L'approche que nous présentons dans cette section consiste d'abord à considérer le problème localement, i.e., au niveau des arêtes. En effet, considérons deux noeuds voisins i et j dans V et seulement les récompenses liées à ces noeuds, qui sont

x (i)⊤ M ⋆ x (j) et x (j)⊤ M ⋆ x (i) . En additionnant ces deux quantités, 1 on obtient x (i)⊤ M ⋆ x (j) + x (j)⊤ M ⋆ x (i) = x (i)⊤ M ⋆ + M ⊤ ⋆ x (j)
qui représente la récompense entre les deux noeuds voisins (i) et (j). Une stratégie locale que l'entité centrale devrait mettre en oeuvre consiste donc à allouer (x

(i) , x (j) ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ = (x ⋆ , x ′ ⋆ )
. Naturellement, si cette stratégie locale est facile à appliquer pour un couple de voisins (i, j), elle ne peut pas etre simultanément appliquée à tous les autres couples du graphe puisque certains d'entre eux partagent les mêmes noeuds. Cepen-Algorithm 14: Algorithm d'approximation pour notre problème NP-Dur Entrée: Graphe G = (V, E), ensemble de bras X , matrice M

⋆ (V 1 , V 2 ) = Approx-MAX-CUT(G); Trouver (x ⋆ , x ′ ⋆ ) ∈ arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ ; for i in V 1 do x (i) t = x ⋆ ; // Peut etre fait en parallèle end for i in V 2 do x (i) t =
x ′ ⋆ ; // Peut etre fait en parallèle end retourner (x (1) , . . . , x (n) ) d'autres arêtes (i, j) ∈ E les bras alloués associés seront les couples sous-optimaux et non désirés

(x ⋆ , x ⋆ ) ou (x ′ ⋆ , x ′ ⋆ ).
Avant d'énoncer la garantie de cet algorithme par rapport à la récompense globale optimale, introduisons m 1 (respectivement m 2 ) le nombre d'arêtes qui vont des noeuds de

V 1 (respectivement V 2 ) aux noeuds de V 1 (respectivement V 2 ) et m 1→2 (respectivement m 2→1 ) le nombre d'arêtes qui vont des noeuds de V 1 (respectivement V 2 ) aux noeuds de V 2 (respectivement V 1 ). Remar- quez que le nombre total d'arêtes m = m 1→2 + m 2→1 + m 1 + m 2 et
(x,x ′ )∈X 2 m 1→2 • x ⊤ M ⋆ x ′ + m 2→1 • x ′⊤ M s tarx + m 1 • x ⊤ M s tarx + m 2 • x ′⊤ M ⋆ x ′ , (C.5)
nous optimisons la récompense globale totale que l'on obtiendrait en allouant seulement deux bras (x, x ′ ) ∈ X 2 dans le graphe. Cette stratégie est décrite dans l'Algorithme 15.

Algorithm 15: Algorithm d'approximation amélioré pour notre problème NP-Dur

Entrée :

Graphe G = (V, E), ensemble de bras X , matrice M ⋆ (V 1 , V 2 ) = Approx-MAX-CUT(G); m 1→2 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 2 }|; m 2→1 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 1 }|; m 1 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 1 }|; m 2 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 2 }|; Trouver (x ⋆ , x′ ⋆ ) solution of (C.5); for i in V 1 do x (i) t = x⋆ ; // Peut etre fait en parallèle end for i in V 2 do x (i) t = x′ ⋆ ; // Peut etre fait en parallèle end retourner (x (1) , . . . , x (n) ) + m 1 x⊤ ⋆ M ⋆ x⋆ -x ⊤ ⋆ M ⋆ x ⋆ + m 2 x′⊤ ⋆ M ⋆ x′ ⋆ -x ′⊤ ⋆ M ⋆ x ′ ⋆ .
Les nouvelles garanties que nous obtenons sur la récompense de l'allocation obtenue par l'Algorithme 15 sont énoncées dans le théorème suivant. un paramètre dépendant du problème qui mesure le gain relatif de l'optimisation sur les récompenses sous-optimales définies comme :

ϵ = ∆ M ⋆ x (j)
⋆ .

Corollary C.

= m 1→2 + m 2→1 m + m 1 + m 2 m ξ + ϵ .
Ce corollaire est utile pour comprendre en pratique le type de garanties que nous pouvons avoir en fonction de la structure du graphe et de l'algorithme d'approximation que nous utilisons pour résoudre le problème de Max-Cut.

C.4 Identification du meilleur bras pour les bandits bilineaires graphiques

Dans cette section, nous supposons que nous ne connaissons pas la matrice M ⋆ et qu'une entité centrale fait face à un problème de bandits bilinéaires graphiques où à chaque tour elle choisit un bras pour chaque noeud du graphe et observe une récompense bilinéaire pour chaque arête du graphe. Dans ce chapitre, nous nous concentrerons sur l'objectif d'identification du meilleur bras.

C.4.1 Préliminaires

Pour simplifier, nous considérons que la matrice inconnue M ⋆ est symétrique, ce qui simplifie grandement le raisonnement. Dans le chapitre C.3, nous avons conçu des algorithmes en temps polynomial qui nous permettent de calculer une solution d'approximation α au problème NP-Dur consistant à trouver la meilleure allocation étant donné M ⋆ . Remarquez que dans l'algorithme d'α-approximation 14, M ⋆ n'est utilisé que pour identifier la meilleure paire (x ⋆ , x ′ ⋆ ) comme suit :

x ⋆ , x ′ ⋆ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ x ′ . (C.6)
Ainsi, utiliser une estimation M de M ⋆ ayant la propriété suivante :

arg max (x,x ′ )∈X 2 x ⊤ Mx ′ = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ x ′ = x ⋆ , x ′ ⋆ , (

Condition d'arrêt

Remarquons que le problème d'optimisation (C.6) est équivalent au problème d'optimisation suivant

(x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X vec xx ′⊤ , vec (M ⋆ ) .
Simplifions les notations et désignons par θ ⋆ ≜ vec (M ⋆ ) la version vectorisée de la matrice inconnue M ⋆ . Utilisons également la notation

z xx ′ ≜ vec xx ′⊤ , et définissons Z = {z xx ′ |(x, x ′ ) ∈ X 2 } l'ensemble contenant de tels vecteurs. Alors, chercher le couple (x ⋆ , x ′ ⋆ ) revient à chercher le vecteur z ⋆ ∈ Z où z ⋆ = arg max z∈Z ⟨z, θ ⋆ ⟩ .
En d'autres termes, nous voulons trouver un bras z ∈ Z, tel que pour tout z ′ ∈ Z, (zz ′ ) ⊤ θ ⋆ ≥ 0. Cependant, nous n'avons pas accès à θ ⋆ , donc nous devons utiliser son estimation empirique.

Ainsi, à chaque tour t, l'entité centrale peut choisir pour chaque couple de voisins (i, j) un bras z ∈ Z et obtenir une récompense linéaire bruitée de la forme ⟨z, θ ⋆ ⟩ + η où η est une variable aléatoire σ-sous-gaussienne, qui peut être utilisée pour calculer une estimation θt .

Pour plus de clarté, nous appellerons tout x ∈ X un bras de noeud et tout z ∈ Z un bras d'arête. Si x (i) t ∈ X représente le bras de noeud alloué au noeud i ∈ V au temps t, pour chaque arête (i, j) ∈ E nous désignerons le bras d'arête associé par z

(i,j) t ≜ vec x (i) t x (j)⊤ t ∈ Z.
Le but ici est de définir la séquence optimale (z 1 , . . . , z mt ) ∈ Z mt qui devrait être tirée dans les t premiers tours de façon à ce que (C.7) soit atteint le plus tôt possible. Une approche naturelle consiste à s'appuyer sur les stratégies classiques développées pour l'identification du meilleur bras dans les bandits linéaires. Nous définissons (y 1 , . . . , y mt ) les récompenses bruitées correspondantes de la séquence (z 1 , . . . , z mt ). Nous supposons que les termes de bruit dans les récompenses sont i.i.d., suivant une distribution σ-sous-gaussienne.

Soit θt = A -1 t b t ∈ R d 2 la solution du problème des moindres carrés ordinaires avec A t = mt i=1 z i z ⊤ i ∈ R d 2 ×d 2 et b t = mt i=1 z i y i ∈ R d 2 .
En suivant les étapes de [START_REF] Soare | Best-arm identification in linear bandits[END_REF], nous pouvons montrer que s'il existe z ∈ Z tel que pour tout z ′ ∈ Z ce qui suit est vrai :

∥z -z ′ ∥ A -1 t 8σ 2 log 6m 2 t 2 K 4 δπ 2 ≤ ∆t z, z ′ , (C.8) où ∆t (z, z ′ ) = (z -z ′ ) ⊤ θt est l'écart empirique entre z et z ′ ,

Une strategie G-Allocation contrainte

Étant donné la condition d'arrêt (C.8) dérivée dans la section précédente, on veut trouver la séquence de bras d'arête z ⋆ mt = (z ⋆ 1 , . . . , z ⋆ mt ) telle que :

z ⋆ mt ∈ arg min (z 1 ,...,zmt)∈Z mt max z ′ ∈Z z ′⊤ mt i=1 z i z ⊤ i -1 z ′ . (G-opt-Z)
Ceci est connu sous le nom de G-allocation (voir e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]) et est NP-Dur à calculer ( [START_REF] Çivril | On selecting a maximum volume sub-matrix of a matrix and related problems[END_REF]104]). Une façon de trouver une solution approximative consiste à s'appuyer sur une relaxation convexe du problème d'optimisation (G-opt-Z) et à calculer d'abord une allocation à valeur réelle Γ ⋆ ∈ S Z telle que Néanmoins, (G-relaxed-Z) donne tout de même des informations précieuses sur le nombre de fois, en proportion, où chaque bras d'arête z ∈ Z doit être allouée au graphe. Dans la section suivante, nous présentons un algorithme satisfaisant à la fois les exigences de proportion et les contraintes graphiques.

Γ ⋆ ∈ arg min Γ ∈S Z max z ′ ∈Z z ′⊤ z∈Z Γ z z ⊤ -1 z ′ . (G-

C.4.2 Algorithme et garanties

Allocation aléatoire sur les noeuds

Notre algorithme est basé sur une méthode aléatoire d'allocation directe des bras de noeud aux noeuds, évitant ainsi la tâche difficile de choisir les bras d'arête et d'essayer de les allouer au graphe tout en s'assurant que chaque noeud a une affectation unique. La validité de cette allocation aléatoire est basée sur le théorème C.4 ci-dessous montrant que l'on peut tirer des bras de noeuds dans X et les allouer au graphe de telle sorte que les bras d'arête associés suivent la distribution de probabilité Γ ⋆ solution de (G-relaxed-Z).

Theorem C.4. Soit γ ⋆ une solution du problème d'optimisation suivant :

min γ s tar∈S X max x ′ ∈X x ′ ⊤ x∈X γ x xx ⊤ -1 x ′ . (G-relaxed-X ) Soit Γ ⋆ ∈ S Z défini pour tout z = vec xx prime⊤ ∈ Z par Γ ⋆ z = γ ⋆ x γ ⋆ x ′ . Alors, Γ ⋆ est une solution de (G-relaxed-Z).
Ce théorème implique que, à chaque tour t > 0 et pour chaque noeud i ∈ V , si x (i) t est tiré de γ ⋆ , alors pour toutes les paires de voisins (i, j) ∈ E, les bras d'arête associés z (i,j) t suivent la distribution de probabilité Γ ⋆ . De plus, comme γ ⋆ est une distribution sur l'ensemble des bras de noeud, X , Γ ⋆ peut etre considéré comme une distribution de probabilité jointe (produit) sur

Analyse de la convergence

Nous prouvons maintenant la validité de la procédure d'échantillonnage aléatoire détaillée dans l'Algorithme 16 en contrôlant la qualité de l'approximation max z∈Z z ⊤ A -1 t z par rapport à l'optimum du problème d'optimisation G-allocation max z ′ ∈Z z ′⊤ mt i=1 z ⋆ i z ⋆⊤ i -1 z ′ décrit dans (G-opt-Z).

Comme cela est généralement fait dans la littérature optimal design (voir e.g., [START_REF] Pukelsheim | Optimal Design of Experiments[END_REF][START_REF] Sagnol | Optimal design of experiments with application to the inference of traffic matrices in large networks: second order cone programming and submodularity[END_REF][START_REF] Soare | Best-arm identification in linear bandits[END_REF]), nous limitons l'erreur relative β t :

max z∈Z z ⊤ A -1 t z ≤ (1 + β t ) max z ′ ∈Z z ′⊤ mt i=1 z ⋆ i z ⋆⊤ i -1 z ′ .
Notre analyse s'appuie sur plusieurs résultats de la théorie de la concentration matricielle. On peut se référer par exemple à [START_REF] Tropp | An introduction to matrix concentration inequalities[END_REF] et à ses références pour une introduction approfondie sur ce sujet. Nous introduisons d'abord quelques notations supplémentaires.

Soit f Z la fonction telle que, pour toute matrice non singulière Q ∈ R d 2 ×d 2 , f Z (Q) = max z∈Z z ⊤ Q -1 z et pour toute distribution Γ ∈ S Z on rappelle que Σ Z (Γ ) ≜ z∈Z Γ z zz ⊤ est la matrice de covariance associée. Enfin, laissons A ⋆ t = mt i=1 z ⋆ i z ⋆⊤ i être la matrice de G-optimal design construite pendant t tours.

Theorem C.5. Soit Γ ⋆ une solution du problème d'optimisation (G-relaxed-Z). Soit 0 < δ ≤ 1 et soit t 0 tel que t 0 = 2Ld 2 log 2d 2 /δ /λ min , où λ min est la plus petite valeur propre de la matrice de covariance 1 K 2 z∈Z zz ⊤ . Alors, à chaque tour t ≥ t 0 avec une probabilité d'au moins 1 -δ, la stratégie randomisée G-allocation pour les bandits bilinéaires graphiques de l'Algorithme 16 produit une matrice A t telle que : De plus, soit τ le nombre de tours suffisant pour qu'un algorithme quelconque détermine le meilleur bras avec une probabilité d'au moins 1 -δ. Une borne inférieure sur l'espérance de τ peut être obtenue à partir de celle dérivée pour le problème de l'identification du meilleur bras dans les bandits linéaires (voir e.g., Théorème 1 dans [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF]) :

f Z (A t ) ≤ (1 + β t )f Z (A ⋆ t )
E[τ ] ≥ min Γ ∈S Z max z∈Z\{z⋆} log 1 2.4δ 2σ 2 ∥z ⋆ -z∥ 2 Σ Z (Γ ) -1 m (z ⋆ -z) ⊤ θ ⋆ 2 .
Comme observé dans [START_REF] Soare | Best-arm identification in linear bandits[END_REF], cette limite inférieure peut être bornée, dans le pire des cas, par 4σ 2 d 2 /(m∆ 2 min ), ce qui correspond à notre borne jusqu'aux termes logarithmiques et à l'erreur relative β t . C.4.3 Influence de la structure du graphe sur le taux de convergence Caractérisation de la variance associée à la stratégie aléatoire La limite de convergence du théorème C.5 dépend de v = ∥E (A 1 -EA 1 ) 2 ∥. Dans cette section, nous caractérisons l'impact de la structure du graphe sur cette quantité et, par extension, sur le taux de convergence. Les limites sont énoncées dans le tableau C.1.

Graphique

Limite supérieure sur ∥Var(A Ces quatre exemples mettent en évidence la forte dépendance de la variance à la structure du graphe. Plus les arêtes sont indépendantes (sans noeuds communs), plus la quantité ∥Var(A 1 )∥ est petite. Pour un nombre fixe d'arêtes m, le meilleur cas est le graphe couplage où aucune arête ne partage le même noeud et le pire cas est le graphe en étoile où toutes les arêtes partagent un noeud central.

Résultats expérimentaux validant la dépendance du graphe

Dans cette section, nous considérons la version modifiée d'une expérience standard introduite par [START_REF] Soare | Best-arm identification in linear bandits[END_REF] et utilisée dans la plupart des articles sur l'identification du meilleur bras dans les bandits linéaires [START_REF] Fiez | Sequential experimental design for transductive linear bandits[END_REF][START_REF] Tao | Best Arm Identification in Linear Bandits with Linear Dimension Dependency[END_REF]106,109] . La matrice M ⋆ a sa première coordonnée égale à 2 et les autres égales à 0, ce qui donne θ ⋆ = vec (M ⋆ ) = (2, 0, . . . , 0) ⊤ ∈ R d 2 . Le meilleur bras d'arête est donc z ⋆ = z (1,1) = e ′ 1 . On peut noter que lorsque ω tend vers 0, il est plus difficile de différencier z (1,1) et z (d+1,d+1) = vec x (d+1) x ⊤ (d+1) que z (1,1) et les autres bras. Nous fixons η (i,j) t ∼ N (0, 1), pour toute arête (i, j) et tour t. Nous considérons deux cas où ω = 0.1 qui rend les bras z (1,1) et z (d+1,d+1) difficiles à différencier, et ω = π/2 qui rend le bras z (1,1) facilement identifiable comme le bras optimal. Pour chacun de ces deux cas, nous évaluons l'influence de la structure du graphe, du nombre d'arêtes m et de la dimension de l'espace des bras d'arête d 2 sur la complexité d'échantillonnage. Les résultats sont présentés dans la figure C.2.

Lorsque ω = 0, 1, le type de graphe n'a pas d'impact sur le nombre de tours nécessaires pour vérifier la condition d'arrêt. Ceci est principalement dû au fait que l'ampleur de la variance associée est négligeable par rapport au nombre de tours. Par conséquent, même si nous faisons varier le nombre d'arêtes ou la dimension, nous obtenons les mêmes performances pour tout type de graphe, y compris le graphe matching. Cela implique que notre algorithme est aussi performant qu'un bandit linéaire qui tire m arêtes en parallèle à chaque tour. Lorsque ω = π/2, le nombre de tours nécessaires pour vérifier la condition d'arrêt est plus petit et l'amplitude de la variance n'est plus négligeable. En effet, lorsque le nombre d'arêtes ou la dimension augmente, on remarque que le graphe en étoile prend plus de temps pour satisfaire la condition d'arrêt. De plus, notons que les complexités d'échantillonnage obtenues pour le cercle et le graphe d'appariement sont similaires. Cette observation est en accord avec la dépendance à la variance montrée dans le Tableau C.1.

C.5 Algorithmes basés sur le regret pour les bandits bilinéaires graphiques

Dans cette section, nous supposons également que nous ne connaissons pas la matrice des paramètres M ⋆ et comme décrit dans la configuration du problème dans la section C.2, une entité centrale fait face au problème des bandits bilinéaires graphiques où à chaque tour elle choisit un bras pour chaque noeud du graphe et observe une récompense bilinéaire pour chaque arête du graphe. L'objectif de l'entité centrale est de concevoir un algorithme qui maximise l'espérance de la récompense globale cumulée obtenue pendant T tours T t=1 (i,j)∈E x pour toute instance de bandits bilinéaires graphiques décrits dans la section C.2, sauf si P = N P .

Par conséquent, l'objectif de concevoir un algorithme avec un regret sous-linéaire en T n'est pas réalisable en temps polynomial par rapport à n. Cependant, certains problèmes NP-Dur sont α-approximables (pour un certain α ∈ (0, 1]), ce qui signifie qu'il existe un algorithme en temps polynomial garanti pour produire des solutions dont les valeurs sont au moins α fois la valeur de la solution optimale. Nous renvoyons le lecteur à la section C.3 pour plus d'informations sur l'approximation de la solution optimale de notre problème. Pour ce type de problèmes, il est logique de considérer l'α-pseudo-regret comme dans [START_REF] Chen | Combinatorial multi-armed bandit: General framework and applications[END_REF][START_REF] Kakade | Playing games with approximation algorithms[END_REF] qui est défini pour tout α ∈ (0, 1] comme suit

R α (T ) ≜ T t=1   α (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ - (i,j)∈E x (i)⊤ t M ⋆ x (j) t   ,
et on se fixe comme objectif de concevoir un algorithme avec un α-regret sous-linéaire.

Enfin, comme nous l'avons fait dans le chapitre C.4, rappelons que la récompense obtenue pour chaque arête du graphe à chaque tour peut être vue comme une récompense linéaire bruitée en dimension supérieure [START_REF] Jun | Bilinear Bandits with Low-rank Structure[END_REF] avec

y (i,j) t = vec x (i) t x (j)⊤ t
, vec (M ⋆ ) + η (C.10)

Dans ce chapitre, nous choisissons de concevoir un algorithme basé sur le principe d'optimisme face à l'incertitude [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF], et dans le cas d'une récompense linéaire [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF][START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF], nous devons maintenir un estimateur du vrai paramètre θ ⋆ . Pour ce faire, définissons pour tous les tours t ∈ {1, . . . , T } l'estimateur MCO de θ ⋆ comme suit :

θt = A -

C.5.3 Algorithme amélioré et analyse du regret

Nous abordons le problème de concevoir une version améliorée de l'algorithme proposé en utilisant l'idée présentée dans l'Algorithme 15 qui améliore le taux d'approximation. Rappelons que dans la section précédente de l'Algorithme 17, l'entité centrale choisit le couple (x t , x ′ t ) tel que (x t , x ′ t ) = arg max (x,x ′ )∈X 2 max θ∈C t-1 (δ) ⟨z xx ′ + z x ′ x , θ⟩ , qui maximise la récompense optimiste obtenue entre deux noeuds si l'entité centrale était capable d'allouer x t à un noeud et x ′ t au second. Cette stratégie étant optimale localement mais compliquée par la prise en compte des dépendances entre les arêtes, l'entité centrale pourrait prendre en considération les bras d'arêtes de la forme z xx et z x ′ x ′ qui sont créés lors de l'allocation des noeuds du graphe en utilisant seulement deux bras de noeuds x et x ′ . Cette idée suit celle présentée dans l'Algorithme 15 où l'on rappelle avec des notations différentes que le couple (x ⋆ , x′ ⋆ ) choisi pour allouer les noeuds du graphe sont tels que Ici, au lieu de maximiser la récompense locale que l'on peut obtenir entre deux noeuds, l'entité centrale maximise la récompense optimiste globale que l'on obtiendrait en allouant seulement deux bras (x, x ′ ) ∈ X 2 dans le graphe. Cette stratégie est décrite dans l'Algorithme 18.

(x ⋆ , x′ ⋆ ) = arg max (x,x ′ )∈X 2 ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x + m 1 z xx + m 2 z x ′ x ′ ,

Algorithm 18: OFUL amélioré pour les bandits bilineqires graphiques

Input :

Graphe G = (V, E), ensemble X (V 1 , V 2 ) = Approx-MAX-CUT(G); m 1→2 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 2 }|; m 2→1 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 1 }|; m 1 = |{(i, j) ∈ E|i ∈ V 1 ∧ j ∈ V 1 }|; m 2 = |{(i, j) ∈ E|i ∈ V 2 ∧ j ∈ V 2 }|; for t = 1 to T do xt , x′ t , θt-1 = arg max (x,x ′ ,θ)∈X 2 ×C t-1 ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x + m 1 • z xx + m 2 • z x ′ x ′ , θ⟩; x (i) t = xt pour tout i dans V 1 ; x (i) t = x′
t pour tout i dans V 2 ; Obtenir pour tout (i, j) dans E les récompenses y

∆ = ⟨m 1→2 z x⋆ x′ ⋆ -z x⋆x ′ ⋆ + m 2→1 z x′ ⋆ x⋆ -z x ′ ⋆ x⋆ + m 1 (z x⋆ x⋆ -z x⋆x⋆ ) + m 2 z x′ ⋆ x′ ⋆ -z x ′ ⋆ x ′ ⋆ , θ ⋆ ⟩ .
Les nouvelles garanties que nous obtenons sur l'α-regret de l'Algorithme 18 sont énoncées dans le théorème suivant. On peut voir ici que l'amélioration se produit dans le α du α-regret. Dans la section suivante, nous confirmons ces résultats par des expériences.

C.5.4 Expériences numériques

Nous concevons une expérience qui compare en pratique les performances de l'Algorithme 17 et de l'Algorithme 18 avec l'algorithme Explore-Then-Commit (ETC) en utilisant la stratégie d'exploration conçue dans la section C.4 pendant la phase d'exploration, et en allouant les noeuds dans V 1 et V 2 avec le meilleur couple estimé (x, x ′ ) = arg max (x,x ′ ) ⟨z xx ′ + z x ′ x , θt ⟩ pendant la phase de d'exploitation. Cependant, puisque les algorithmes que nous avons présentés dans cette section ont des garanties sur les α-regrets avec différents α, nous traçons la fraction de la récompense globale optimale pour chaque itération. Comme dans le chapitre C.3, nous observons une nette amélioration en choisissant à chaque tour t le couple de bras (x t , x′ t ) au lieu de (x t , x ′ t ).

C.6 Conclusion & perspectives C.6.1 Résumé des résultats

Dans cette thèse, nous avons introduit un nouveau modèle que nous avons nommé Bandits Bilinéaires Graphiques qui modélise les problèmes multi-agents centralisés où des interactions par paires existent entre les agents.

• Dans la section C.3, nous avons mis en évidence le fait que l'apprenant était confronté à un problème d'optimisation sous-jacent qui est NP-Hard quel que soit le but que l'apprenant souhaite atteindre. Nous avons donc proposé un algorithme d'α-approximation avec α ≥ 1/2 qui ne nécessite que de trouver le couple de bras (x ⋆ , x ′ ⋆ ) pour retourner l'α-approximation. Nous avons ensuite affiné ce paramètre d'approximation par rapport aux paramètres dépendant du problème en nous basant sur la structure du graphe et sur une propriété de la matrice M ⋆ .

• Dans la section C.4, étant donné l'algorithme d'α-approximation conçu dans la section C.3, nous avons présenté un algorithme de pure exploration qui permettait à l'apprenant de construire une estimation M qui était statistiquement efficace en termes d'optimal design. En effet, le problème de trouver en un nombre minimum de tours le meilleur couple (x ⋆ , x ′ ⋆ ) utilisé dans l'algorithme d'approximation α revenait à trouver le G-optimal design, également appelé G-allocation dans la littérature bandit. Résoudre ce problème dans les bandits bilinéaires graphiques impliquait de traiter une contrainte supplémentaire. C'est pourquoi nous avons présenté un algorithme qui respectait cette contrainte et qui utilisait un échantillonnage aléatoire pour construire l'estimation M. Nos résultats théoriques ont révélé un terme qui dépendait de la structure du graphe, nous avons donc montré l'impact du graphe dans nos résultats.

• Enfin, dans la section C.

1. 1 R

 1 Illustration of the learner's decision process at a given round t for a simple graph of three nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 The δ-confidence level ellipsoid . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Variation of ϵ, ξ, α 1 and α 2 with respect to the parameter ζ. The closer ζ is to 0 the lower the reward of the unwanted couples (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ), the closer ζ is to 1 the higher the rewards of the unwanted couples. The dimension d of the arm-set is 10 (which gives linear reward with unknown parameter θ ⋆ of dimension 100). The plotted curve represents the average value of the parameters over 100 different matrices M ⋆ initiated randomly with positive values. . . . . 4.1 Collision when allocating directly edge-arms to the edges . . . . . . . . . . . . 4.2 Upper bound on the variance and convergence rate of Algorithm 8 for the star, complete, circle and matching graph with respect to the number of edges m and the number of rounds t. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Number of rounds t needed to verify the stopping condition (4.3) with respect to left: the number of edges m where the dimension of the edge-arm space Z is fixed and equal to 25 and right: the dimension of the edge-arm space Z where the number of edges is fixed and equal to 156. For both experiments we run 100 times and plot the average number of rounds needed to verify the stopping condition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Fraction of the optimal global reward obtained at each round by applying the Algorithm 9, Algorithm 10 and the Explore-Then-commit algorithm (here named GBB-BAI) using the exploration strategy in Chapter 4. We use a complete graph of 5 nodes, we run the experiment on 5 different matrices as in Figure 3.1 with ζ = 0 and run it 10 different times to plot the average fraction of the global reward. We set the confidence δ = 0.001. . . . . . . . . . . . . . . . . . . . . B.1 Representation of the µ -1 (blue dotted line) and µ 1 (red plain line) distributions, without attack (left) and with three different attacks: no penalty (second drawing), with mass penalty (third) and with norm penalty (fourth). On all figures blue area on the left of the axis is P h (ϵ 2 ) and red area on the right is N h (ϵ 2 ). . . xv B.2 Illustration of adversarial examples (only on class 1 for more readability) crossing the decision boundary (left), adversarially trained classifier for the class 1 (middle), and a randomized classifier that defends class 1. Stars are natural examples for class 1, and crosses are natural examples for class -1. The straight line is the optimal Bayes classifier, and dashed lines delimit the points close enough to the boundary to be attacked resp. for class 1 and -1. We focus the drawing on the star points. Crosses can be treated symmetrically. . . . . . . . . . . . . . . . . B.3 Illustration of the notations U , U + , and U -for proof of Theorem B.4. . . . . B.4 Illustration of the notations U , U + , U -and δ for proof of Theorem B.5. . . . B.5 Evolution of the accuracy under Adaptive-ℓ ∞ -PGD attack depending on the budget ϵ ∞ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.6 Comparison of the mixture that has as first classifier the best one in term of natural accuracy and the mixture that has as first classifier the best one in term of Accuracy under attack. The accuracy under attack is computed with the ℓ ∞ -PGD attack. NA means matural accuracy, and AUA means accuracy under attack. C.1 Borne supérieure de la variance et du taux de convergence de l'Algorithme 16 pour le graphe en étoile, le graphe complet, le cercle et le graphe couplage par rapport au nombre d'arêtes m et au nombre de tours t. . . . . . . . . . . . . . C.2 Nombre de tours t nécessaires pour vérifier la condition d'arrêt (C.8) par rapport à gauche: le nombre d'arêtes m où la dimension de l'espace de Z est fixée et égale à 25 et right: la dimension de l'espace de Z où le nombre d'arêtes est fixé et égal à 156. Pour les deux expériences, nous les exécutons 100 fois et nous traçons le nombre moyen de tours nécessaires pour vérifier la condition d'arrêt. . . . . . . C.3 Fraction de la récompense globale optimale obtenue à chaque tour en appliquant l'Algorithme 17, l'Algorithme 18 et l'algorithme Explore-Then-commit (appelé ici GBB-BAI) en utilisant la stratégie d'exploration de la Section C.4. Nous utilisons un graphe complet de 5 noeuds, nous exécutons l'expérience sur 5 matrices différentes avec ζ = 0 et l'exécutons 10 fois différentes pour tracer la fraction moyenne de la récompense globale . . . . . . . . . . . . . . . . . . . . . . . xvi Notations Set of real numbers R d

1

 1 An introduction to the stochastic multi-armed bandit problem . . .

7 2. 1 . 1

 711 Motivations and formalization . . . . . . . . . . . . . . . . .

7 2. 1 . 2

 712 Maximizing the cumulative rewards . . . . . . . . . . . . . . .

8 2. 2

 82 The stochastic linear bandit problem . . . . . . . . . . . . . . . . . 10 2.2.1 Formalization . . . . . . . . . . . . . . . . . . . . . . . . .

10 2. 2 . 2

 1022 Experimental designs serving the pure exploration setting . . . . .

11 2. 2 . 3

 1123 Optimism in the face of uncertainty for linear bandits (OFUL) . .

14 2. 2 . 4

 1424 Bilinear bandits are linear bandits in a higher dimensional space . .

15 2. 3

 153 Multi-agent bandits and combinatorial bandits . . . . . . . . . . .

17

 17

2. 3 . 1

 31 Parallelizing contextual linear bandits . . . . . . . . . . . . . .

17

 17

2. 3 . 2

 32 Bandit problems in graphs . . . . . . . . . . . . . . . . . . .

18 2. 3 . 3

 1833 Link with unstructured multi-agents bandits . . . . . . . . . .

19 2. 3 . 4

 1934 Combinatorial bandits . . . . . . . . . . . . . . . . . . . . .

19

 19

Theorem 3 . 2 .

 32 1 we have m 1→2 = m 2→1 ≥ m/4 and m 1 + m 2 ≤ m/2. Let us consider the graph G = (V, E), a finite arm set X ⊂ R d and the matrix M ⋆ ∈ R d×d given as input to Algorithm 5. Let (x

Figure 3 . 1 :

 31 Figure 3.1: Variation of ϵ, ξ, α 1 and α 2 with respect to the parameter ζ. The closer ζ is to 0 the lower the reward of the unwanted couples (e i ⋆ , e i ⋆ ) and (e j ⋆ , e j ⋆ ), the closer ζ is to 1 the higher the rewards of the unwanted couples. The dimension d of the arm-set is 10 (which gives linear reward with unknown parameter θ ⋆ of dimension 100). The plotted curve represents the average value of the parameters over 100 different matrices M ⋆ initiated randomly with positive values.

Figure 4 . 1 :

 41 Figure 4.1: Collision when allocating directly edge-arms to the edges

Figure 4 . 3 :

 43 Figure 4.3: Number of rounds t needed to verify the stopping condition (4.3) with respect to left: the number of edges m where the dimension of the edge-arm space Z is fixed and equal to 25 and right: the dimension of the edge-arm space Z where the number of edges is fixed and equal to 156. For both experiments we run 100 times and plot the average number of rounds needed to verify the stopping condition.

1

 1 Optimism in the face of uncertainty for graphical bilinear bandits . 55 5.1.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.1.2 Algorithm and analysis of the regret . . . . . . . . . . . . . . . 59 5.1.3 Improved algorithm and analysis of the regret . . . . . . . . . . 66 5.2 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3 Conclusion and perspectives . . . . . . . . . . . . . . . . . . . . . 73

Figure 5 . 1 :

 51 Figure 5.1: Fraction of the optimal global reward obtained at each round by applying the Algorithm 9, Algorithm 10 and the Explore-Then-commit algorithm (here named GBB-BAI) using the exploration strategy in Chapter 4. We use a complete graph of 5 nodes, we run the experiment on 5 different matrices as in Figure 3.1 with ζ = 0 and run it 10 different times to plot the average fraction of the global reward. We set the confidence δ = 0.001.

  (a) Evolution of score with n (b) Evolution of score with d (c) Evolution of score with d

  η>0 e -ηt tr exp n i=1 log E exp(ηX i ) , and similarly P λ min n i=1 X i ≤ t ≤ inf η<0 e -ηt tr exp n i=1 log E exp(ηX i ) .

de -tη+ e ηL - 1 Lλ

 1 3 yields:P(λ min (S) ≤ t) ≤ inf η<0 e -ηt tr exp e ηL -1 L ES .Reintegrating ∥ • ∥ into the RHS yields: min(ES) .

Figure B. 1 :

 1 Figure B.1: Representation of the µ -1 (blue dotted line) and µ 1 (red plain line) distributions, without attack(left) and with three different attacks: no penalty (second drawing), with mass penalty (third) and with norm penalty (fourth). On all figures blue area on the left of the axis is P h (ϵ 2 ) and red area on the right is N h (ϵ 2 ).

Figure B. 2 :

 2 Figure B.2: Illustration of adversarial examples (only on class 1 for more readability) crossing the decision boundary (left), adversarially trained classifier for the class 1 (middle), and a randomized classifier that defends class 1. Stars are natural examples for class 1, and crosses are natural examples for class -1.The straight line is the optimal Bayes classifier, and dashed lines delimit the points close enough to the boundary to be attacked resp. for class 1 and -1. We focus the drawing on the star points. Crosses can be treated symmetrically.

  4 we devise a new procedure called Boosted Adversarial Training (BAT) to construct a robust mixture of two classifiers. It is based on three core principles: Adversarial Training, Boosting and Randomization. B.6 Experiments: How to build the mixture Simple mixture procedure (BAT). Given a dataset D and a weight parameter α ∈ [0, 1], we construct h 1 the first classifier of the mixture using Adversarial Training 4 on D. Then, we train

Lemma B. 3 .

 3 Lemma B.1 holds. Let us consider ϕ ∈ F X |ϵ 2 2 . If we take h ∈ BR(ϕ), then for y = 1 (resp. y = -1), and for any B ⊂ P h (resp. B ⊂ N h ) one has P(Y = y|X ∈ B) ≥ P(Y = -y|X ∈ B) with Y ∼ ν and for all y ∈ Y, X|(Y = y) ∼ ϕ y #µ y .

  ) ̸ = y} d(ϕ y #µ y )(x). Moreover thanks to the transfer theorem, one gets the following:) ̸ = y} d(ϕ y #µ y )(x) ) ̸ = y}f y (x) dζ(x) = inf h∈H X y=±1ν y 1{h(x) ̸ = y}f y (x) dζ(x).

Figure B. 3 :

 3 Figure B.3: Illustration of the notations U , U + , and U -for proof of Theorem B.4.

Figure B. 4 :

 4 Figure B.4: Illustration of the notations U , U + , U -and δ for proof of Theorem B.5.

Theorem C. 1 .

 1 NP-Dur par rapport au nombre de noeuds n. Considérons une matrice donnée M ⋆ ∈ R d×d et un ensemble de bras finis X ⊂ R d . A moins que P=NP, il n'existe pas d'algorithme en temps polynomial pour trouver la solution optimale de max

( 1 )

 1 ⋆ , . . . , x (n) ⋆ maximisant la récompense globale attendue. Dans les sections suivantes, nous donnons des algorithmes d'approximation en temps polynomial qui ont des garanties sur la récompense globale attendue retournée par rapport à la récompense optimale.

Theorem C. 2 .C. 3 . 3

 233 que par définition de l'ensemble d'arêtes E et en utilisant la Proposition C.1 nous avons m 1→2 = m 2→1 ≥ m/4 et m 1 + m 2 ≤ m/2. Considérons le graphe G = (V, E), un ensemble fini de bras X ⊂ R d et la matrice M ⋆ ∈ R d×d donnée en entrée de l'Algorithme 14. Soit (x (1) ⋆ , . . . , x (n) ⋆ ) l'allocation optimale telle que défini dans (C.3) et soit 0 ≤ ξ ≤ 1 un paramètre dépendant du problème défini par ξ = min x∈X x ⊤ M ⋆ x Algorithme amélioré utilisant la structure du graphe

Theorem C. 3 .

 3 Considérons le graphe G = (V, E), un ensemble fini de bras X ⊂ R d et la matrice M ⋆ ∈ R d×d donnés en entrée de l'Algorithme 15. Soit (x (1) ⋆ , . . . , x (n) ⋆ ) l'allocation optimale telle que définie dans (C.3) et que 0 ≤ ξ ≤ 1 soit défini comme dans le Théorème C.2. Soit 0 ≤ ϵ ≤ 1 2

X 2 Algorithm 16 :

 216 avec pour marginale γ ⋆ . Étant donné la caractérisation dans le Théorème C.4 et notre objectif de vérifier la condition d'arrêt dans (C.8), nous présentons notre procédure d'échantillonnage dans l'Algorithme 16. Nous notons également qu'à chaque tour, l'échantillonnage des bras de noeuds peut être effectué en parallèle. Algorithme d'Exploration Pure : G-Allocation randomisé pour GBB Entrée : Graphe G = (V, E), ensemble de bras X Définir A 0 = I ; b 0 = 0 ; t = 1 ; Appliquer l'algorithme de Frank-Wolfe pour trouver la solution γ ⋆ de (G-relaxed-X ). while la condition d'arrêt (C.

  meilleur cas de graphe implique un v = O(m) correspondant au taux de convergence O 1/ √ mt d'un algorithme de bandit linéaire utilisant un échantillonnage aléatoire pour tirer mt edge-arms sans contraintes (graphiques). De plus, nous verrons que le graphique du pire cas implique que v = O m 2 . Puisque nous avons comblé l'écart entre notre objectif contraint et le problème de l'identification du meilleur bras dans les bandits linéaires, grâce au Théorème C.4 et C.5, nous sommes en mesure d'étendre les résultats connus pour l'identification du meilleur bras dans les bandits linéaires sur la complexité d'échantillonnage et sa borne inférieure associée. Corollary C.2 ([90], Théorème 1). Si la G-allocation est mise en oeuvre avec la stratégie aléatoire de l'Algorithme 16, résultant en une approximation de β t , alors avec une probabilité d'au moins 1 -δ, le meilleur bras obtenu avec θt est z ⋆ et t ≤ 128σ 2 d 2 (1 + β t ) log 6m 2 t 2 K 4 δπ m∆ 2 min , où ∆ min = min z∈Z\{z⋆} (z ⋆ -z) ⊤ θ ⋆ .

  pour évaluer la complexité d'échantillonnage de notre algorithme sur différents graphes. Nous considérons d + 1 bras de noeuds dans X ⊂ R d où d ≥ 2. Cet ensemble est constitué des d vecteurs (e 1 , . . . , e d ) formant la base canonique de R d et d'un bras supplémentaire x d+1 = (cos(ω), sin(ω), 0, . . . , 0) ⊤ avec ω ∈]0, π/2]. Notez que par construction, l'ensemble des bras d'arête Z contient la base canonique (e ′ 1 , . . . , e ′ d 2 ) de R d 2

Figure C. 2 :

 2 Figure C.2: Nombre de tours t nécessaires pour vérifier la condition d'arrêt (C.8) par rapport à gauche: le nombre d'arêtes m où la dimension de l'espace de Z est fixée et égale à 25 et right: la dimension de l'espace de Z où le nombre d'arêtes est fixé et égal à 156. Pour les deux expériences, nous les exécutons 100 fois et nous traçons le nombre moyen de tours nécessaires pour vérifier la condition d'arrêt.

C. 5 . 1 ( 1 )Corollary C. 3 .

 5113 nous appuierons naturellement sur certaines idées et résultats présentés dans la section C.3 où la matrice était supposée connue par l'entité centrale. Nous suivons les notations que nous avons établies dans la section C.2. Optimisme face à l'incertitude pour les bandits bilinéaires graphiquesPréliminairesRappelons que la maximisation des récompenses cumulées est équivalente à la minimisation du regret associé. Nous définissons donc le pseudo-regret global sur T tours comme suit : Nous rappelons que l'objectif de l'apprenant est d'avoir un pseudo-regret R(T ), tel que lim ,...,x (n) )∈X (i,j)∈Ex (i)⊤ M ⋆ x (j)est NP-Dur par rapport au nombre d'agents n. Nous étendons ce résultat dans le corollaire suivant. Il n'existe pas d'algorithme en temps polynomial en n tel que lim

.

  Pour simplifier la notation, désignons tout x ∈ X comme un node-arm. Utilisons la notationz xx ′ ≜ vec xx ′⊤ , et définissons l'ensemble de bras Z = {z xx ′ |(x, x ′ ) ∈ X 2 } où tout z ∈ Z sera appelé un bras d'arête. Si le bras x (i)t ∈ X représente le bras de noeud alloué au noeud i ∈ V au temps t, pour chaque arête (i, j) ∈ E nous désignerons le bra d'arête associé par z

  t ∈ Z et définissons θ ⋆ = vec (M ⋆ ) la version vectorisée de la matrice inconnue M ⋆ . Avec ces notations, la récompense (maintenant) linéaire peut être réécrite comme suit :

;

  Calculer θt comme dans (C.11) end retourner θt Avant d'énoncer les garanties sur l'α-regret, nous rappelons que nous avons défini dans la section C.3 la quantité ∆ ≥ 0 comme étant la différence entre la récompense de l'allocation (x ⋆ , x′ ⋆ ) et celle de l'allocation (x ⋆ , x ′ ⋆ ),

Theorem C. 7 .,

 7 Étant donné le problème de bandits bilinéaires graphiques défini dans la section C.2, on définit ξ comme dans le théorème C.6, laissez 0 ≤ ϵ ≤ 1 2 être un paramètre dépendant du prob-lème qui mesure le gain de l'optimisation sur les bras sous-optimaux et indésirables défini comme : et on fixe α = 1+ξ 2 + ϵ où α ≥ 1/2 par construction, alors le α-regret de l'Algorithme 18 satisfait R α (T ) ≤ Õ σd 2 + S √ λ T m max(2, (LS) 2 ) + LSm d 2 log 2 T mL 2 /λ δ , où Õ cache les facteurs logarithmiques.

Figure C. 3 :

 3 Figure C.3: Fraction de la récompense globale optimale obtenue à chaque tour en appliquant l'Algorithme 17, l'Algorithme 18 et l'algorithme Explore-Then-commit (appelé ici GBB-BAI) en utilisant la stratégie d'exploration de la Section C.4. Nous utilisons un graphe complet de 5 noeuds, nous exécutons l'expérience sur 5 matrices différentes avec ζ = 0 et l'exécutons 10 fois différentes pour tracer la fraction moyenne de la récompense globale .

  d 2 which is the concatenation of all the columns of A ∈ R d×d Context & motivations . . . . . . . . . . . . . . . . . . . . . . . . Problem setting . . . . . . . . . . . . . . . . . . . . . . . . . . . Stochastic Graphical Bilinear Bandits . . . . . . . . . . . . . . Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Abbreviations 1 Introduction
	i.e.,		id est
	e.g., Contents	exempli gratia
	cf. i.i.d.	confer 1.1 1 identically and independently distributed 1.2 2
	MAB		Multi-Armed Bandits 1.2.1 2
	MA-MAB Multi-Agent Multi-Armed bandits 1.2.2
	BAI		Best Arm Identification
	OFU		Optimism in the Face of Uncertainty
	UCB		Upper Confidence Bound
	GBB		Graphical Bilinear Bandits
	ETC		Explore-Then-Commit
			xvii xix

3 1.2.3 Outline of the thesis and contributions . . . . . . . . . . . . .

4

3 Computing the best allocation for known parameter matrices Contents 3.1 An NP-Hard problem

  . . . . . . . . . . . . . . . . . . . . . . . . 23 3.1.1 Reduction to the max-cut problem . . . . . . . . . . . . . . . 23 3.1.2 Approximation algorithm and guarantees . . . . . . . . . . . . 24 3.1.3 Improved algorithm using the graph structure . . . . . . . . . .

29 3.2 Numerical experiments: influence of the parameters on the solution 32 3.3 Conclusion and perspectives

  . . . . . . . . . . . . . . . . . . . . . 33

Table 3 .

 3 1: Values of several parameters with respect to the type of graph. Experiments were performed on graphs of n = 100 nodes, and results for the random graph are averaged over 100 draws.

			Graph types
		Complete	Random	Circle	Star Matching
	m1+m2 m	0.495	0.453	0.01	0	0
	α 1		0.5 + 0.5ξ
	α 2	0.505 + 0.495ξ + ϵ 0.547 + 0.453ξ + ϵ 0.99 + 0.01ξ + ϵ	1	1

4 Best-arm identification in graphical bilinear bandits Contents 4.1 Preliminaries

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

	4.1.1 A two-stage algorithm template . . . . . . . . . . . . . . . . . 36
	4.1.2 Stopping condition . . . . . . . . . . . . . . . . . . . . . . 36
	4.1.3 A Constrained G-Allocation . . . . . . . . . . . . . . . . . . 38
	4

.2 Algorithm and guarantees

  . . . . . . . . . . . . . . . . . . . . . . 39 4.2.1 Random Allocation over the Nodes . . . . . . . . . . . . . . . 39 4.2.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . 43 4.2.3 Case where M ⋆ is not symmetric . . . . . . . . . . . . . . . .

48 4.3 Influence of the graph structure on the convergence rate

  . . . . . .[START_REF] Ilyas | Adversarial Examples Are Not Bugs, They Are Features[END_REF]

	4.3.1 Characterization of the variance associated with the randomized strat-
	egy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
	4.3.2 Experimental results validating the dependence on the graph . . . 52
	4

.4 Conclusion & Perspectives

  . . . . . . . . . . . . . . . . . . . . . . 53

Table 4 .

 4

2: Upper bound on the variance and convergence rate of Algorithm 8 for the star, complete, circle and matching graph with respect to the number of edges m and the number of rounds t.

  Table B.1: Evaluation on CIFAR10 and CIFAR100 without data augmentation. Accuracy under attack of a single adversarially trained classifier (AT) and the mixture formed with our method (Ours). The evaluation is made with Adaptive-ℓ ∞ -PGD and Adaptive-ℓ 2 -C&W attacks both computed with 100 iterations. For Adaptive-ℓ ∞ -PGD we use an epsilon equal to 8/255 (≈ 0.031), a step size equal to 2/255 (≈ 0.008) and we allow random initialization. For Adaptive-ℓ 2 -C&W we use a learning rate equal to 0.01, 9 binary search steps, the initial constant to 0.001, we allow the abortion when it has already converged and we give the results for the different values of rejection threshold ϵ 2 ∈ {0.4, 0.6, 0.8}. As for EOT, we don't need to estimate the expected accuracy of the mixture through Monte Carlo sampling since we have the exact weight of each classifier of the mixture. Thus we give the exact expected accuracy. the second classifier h 2 on a data set D that contains adversarial examples against h 1 created from examples of D. At the end we return the mixture constructed with those two classifiers where the first one has a weight of 1 -α and the second one a weight of α. The parameter α is found by conducting a grid-search. In Table B.1 we present results for α = 0.2 under strong state-of-the-art attacks. The procedure is summarized in Algorithm 12 5 Boosted Adversarial Training Input : D the training data set and α the weight parameter. Create and adversarially train h 1 on D Generate the adversarial data set D against h 1 . Create and naturally train h 2 on D

			.88	0.00	0.00	0.00	0.00
		AT [66]	0.83	0.42	0.60	0.47	0.35
		Ours	0.80	0.55	0.60	0.57	0.53
		Natural	0.62	0.00	0.00	0.00	0.00
	CIFAR100	AT [66]	0.58	0.26	0.38	0.29	0.22
		Ours	0.56	0.40	0.45	0.41	0.38
	Algorithm 11:

Table B .

 B • When doing 200 iterations instead of 100 iterations, it doesn't change the performance of the attack• When increasing the budget ϵ ∞ , the accuracy goes to 0, which ensures that there is no gradient masking. Here are some values to back this statement: 5: Evolution of the accuracy under Adaptive-ℓ ∞ -PGD attack depending on the budget ϵ ∞

	Epsilon 0.015 0.031 0.125 0.250
	Accuracy 0.638 0.546 0.027 0.000

  C.1)où M ⋆ ∈ R d×d est une matrice inconnue, et η Le cadre bilinéaire apparaît comme une extension naturelle du cadre linéaire pour modéliser l'interaction entre deux agents.Notons que ce model peut être considéré soit dans un cadre décentralisé où les agents prennent des actions sans consulter les autres agents, soit dans un cadre centralisé où une entité centrale choisit les bras de tous les agents, agrège les récompenses obtenues et conçoit une stratégie globale pour les agents du graphe.Dans cette thèse, nous ne considérons que le cas centralisé où une entité centrale gère tous les agents, choisit à chaque instant t l'allocation (x

				(i,j) t	une variable aléatoire σ-sous-
	gaussienne de moyenne nulle.
	La récompense y	(i,j) t	reflète la qualité de l'interaction entre les noeuds voisins i et j lorsqu'ils
	tirent respectivement les bras x	(i) t et x	(j)
				(1) t , . . . , x	(n) t ) et reçoit ensuite les récompenses
	associées y	(i,j) t	pour tous les (i, j) ∈ E.

t à l'itération t.

  Dans cette section, nous voulons capitaliser sur l'Algorithme 14 et sa solution1+ξ 2 -optimale pour affiner l'allocation des bras x ⋆ et x ′ ⋆ de telle sorte que les récompenses sous-optimales obtenuesx ⊤ ⋆ M ⋆ x ⋆ et x ′⊤ ⋆ M ⋆ x ′⋆ pénalisent le moins possible la récompense globale. En effet, dans l'Algorithme 14, le choix du couple (x ⋆ , x ′ ⋆ ) est uniquement guidé par le gain potentiel que l'on pourrait obtenir au niveau des arêtes coupées (i.e., qui vont d'un noeud dans V 1 à un noeud dans V 2 ou vice versa). Elle ne prend pas en compte toutes les m 1 récompenses de la forme x ⊤ ⋆ M ⋆ x ⋆ et les m 2 récompenses de la formex ′⊤ ⋆ M ⋆ x ′ ⋆ que l'on obtient en attribuant x ⋆ aux noeuds de V 1 et x ′ ⋆ aux noeuds de V 2 . Ici,l'amélioration que nous pouvons apporter est de les inclure dans le problème d'optimisation et de pondérer les différentes récompenses obtenues par le graphe en utilisant les proportions m 1→2 , m 2→1 , m 1 et m 2 . En désignant par (x ⋆ , x′ ⋆ ) la solution du problème d'optimisation suivant,

	max

  C.7) nous permet d'identifier la paire (x ⋆ , x ′ ⋆ ), et nous donne donc les mêmes garanties que celles présentées dans le Théorème C.2. Nous abordons donc le problème de construire M tel que, en un nombre minimum de tours, avec grande probabilité, nous soyons capables d'identifier la paire (x ⋆ , x ′

⋆ ) et d'appliquer l'Algorithme 14.

  relaxed-Z)On pourrait soit utiliser un échantillonnage aléatoire pour tirer des bras d'arête comme i.i.d. échantillons de la distribution Γ ⋆ , soit des procédures d'arrondi pour convertir efficacement chaque Γ ⋆ z en un nombre entier. Cependant, ces méthodes ne prennent pas en compte la structure graphique du problème. En effet, à un tour donné, les arêtes choisies peuvent donner lieu à deux affectations différentes pour le même noeud, nous appelons ce phénomène une collision.Par conséquent, les procédures d'échantillonnage aléatoire ou d'arrondi ne peuvent pas être utilisées directement pour sélectionner les bras d'arête dans Z.

  [START_REF] Andersen | CVXOPT: A Python package for convex optimization, version 1.2.0[END_REF]) n'est pas vérifiée do // Échantillonnage des bras de noeud ⋆ et obtenir pour tout (i, j) dans E les récompenses y Estimation de θt avec les bras d'arête associésA t = A t-1 + (i,j)∈E zCette procédure d'échantillonnage implique que chaque bras d'arête suit la distribution optimale Γ ⋆ . Dans cette thèse, nous présentons une stratégie de G-allocation aléatoire simple et standard, mais d'autres méthodes plus élaborées pourraient être envisagées, tant qu'elles incluent le caractère aléatoire nécessaire.

	Tirer x	(1) t , . . . , x	(n) t	iid ∼ γ (i,j) t	;
	// (i,j) t	z	(i,j)⊤ t	;
	b t = b t-1 + (i,j)∈E z	(i,j) t	y	(i,j) t	;
	θt = A -1 t b t
	t ← t + 1 ;
	end
	retourner θt

  ∥E (A 1 -EA 1 ) 2 ∥. Nous venons de montrer que la valeur d'approximation max z∈Z z ⊤ A -1 t z converge vers la valeur optimale avec un taux de O √ v/(m √ t) . Dans la section C.4.3, nous montrons que le

							,
	où
	β t =	Ld 2 mλ 2 min	2v t	log	2d 2 δ	+ o	1 √ t	,
	et v ≜

Table C .

 C 1: Borne supérieure de la variance et du taux de convergence de l'Algorithme 16 pour le graphe en étoile, le graphe complet, le cercle et le graphe couplage par rapport au nombre d'arêtes m et au nombre de tours t.

		1 )∥	β
	Étoile Complète	mP + (M + N )O m 2 mP + (M + N )O(m √ m)	O 1/ O 1/ m √ 1 4 √ t	t
	Cercle Correspondance	mP + (M + N )O(m) mP + mN	O 1/ O 1/	√ √	mt mt

2 Algorithme et analyse du regret

  Nous définissons également l'ensemble de confiance C t (δ) = θ : ∥θ -θt ∥ A -1 où avec une probabilité de 1-δ, on a que θ ⋆ ∈ C t (δ) pour tout t ∈ {1, . . . , T }, et δ ∈ (0, 1].Dans la section C.3, nous avons présenté deux algorithmes (Algorithm 14 et 15) qui utilisent la vraie matrice de paramètres M ⋆ pour renvoyer une allocation de bras permettant d'obtenir une α-approximation du problème de maximisation de la récompense globale. Naturellement, puisque nous n'avons pas accès à la matrice M ⋆ , nous ne pouvons pas l'utiliser directement à chaque itération pour maximiser la récompense globale cumulative (et donc minimiser le regret associé). Néanmoins, on peut utiliser l'estimateur construit θt et le principe d'optimisme face à l'incertitude pour surmonter le fait que M ⋆ est inconnu.En effet, nous rappelons que dans l'Algorithme 14, le couple (x ⋆ , x ′ ⋆ ) est choisi comme suit,(x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X 2 x ⊤ M ⋆ + M ⊤ x,x ′ )∈X 2 ⟨z xx ′ + z x ′ x , θ ⋆ ⟩ , (C.14)et est utilisé pour créer autant que possible des bras d'arêtes de la forme z xx ′ et z x ′ x dans le graphe. Ici, à chaque tour t, l'entité centrale choisit de manière optimiste le couple (x t , x ′ ⟨z xx ′ + z x ′ x , θ⟩ , puis alloue les bras-noeuds pour maximiser le nombre de bras-bords localement optimaux z xtx ′ Étant donné le problème de bandits bilinéaires graphiques défini dans la section C.2, soit 0 ≤ ξ ≤ 1 un paramètre dépendant du problème défini par Adaptation de l'algorithme OFUL pour les bandits bilinéaires graphiquesInput : Graphe G = (V, E), ensemble X (V 1 , V 2 ) = Approx-MAX-CUT(G) for t = 1 à T do //Trouver le meilleur couple optimiste x t , x ′ t , θt-1 = arg max (x,x ′ ,θ)∈X 2 ×C t-1 ⟨z xx ′ + z x ′ x , θ⟩ ; // Allouer x t et x ′ t dans le graphe x = x t pour tout i dans V 1 ; x = x ′ t pour tout i dans V 2 ; Obtenir pour tout (i, j) dans E les récompenses y α (T ) ≤ Õ σd 2 + S √ λ T m max(2, (LS) 2 ) + LSm d 2 log 2 T mL 2 /λ δ ,

	Algorithm 17: (i)	t	≤ σ d 2 log (i)	1 + tmL 2 /λ δ	+	√ λS ,
	Calculez θt comme dans (C.11)		(i,j) t	;
	end
	return θt .
						⋆ x ′	(C.12)
							(C.13)
	= arg max
							t ) comme
	suit,
	(x t , x ′ t ) = arg max (x,x ′ )∈X 2	max θ∈C t-1 (δ)	1 t b t ,		(C.11)
	où,
				t
	A t = λI d 2 +	z (i,j) s	z (i,j)⊤ s	,
				s=1 (i,j)∈E
	avec λ > 0 un paramètre de régularisation et
	ξ = min x∈X	1 m	b t = ⟨z xx , θ ⋆ ⟩ t s=1 (i,j)∈E (i,j)∈E z ⋆ , θ ⋆ z (i,j) s (i,j)	y (i,j) ≥ 0 ,
	et on fixe α = 1+ξ 2 , alors l'α-regret de l'Algorithme 18 satisfait

s . C.5.(t et z x ′ t xt . La méthode est présentée dans l'Algorithme 17 Theorem C.6. t t R où Õ cache les facteurs logarithmiques.

  θ ⋆ ⟩ .Comme dans la section précédente, nous n'avons pas accès à θ ⋆ , nous utilisons le principe d'optimisme pour trouver à chaque tour le couple (x t , x′ t ) tel que(x t , x′ t ) = arg max (x,x ′ )∈X 2 max θ∈C t-1 (δ) ⟨m 1→2 • z xx ′ + m 2→1 • z x ′ x = m 1 z xx + m 2 z x ′ x ′ , θ⟩ .

	(C.15)

2 Perspective et travaux futurs

  5, nous avons capitalisé sur l'algorithme d'α-approximation donné dans le chapitre C.3 et appliqué le principe d'optimisme face à l'incertitude pour concevoir des algorithmes basés sur le regret qui ont atteint un α-regret sous-linéaire en T où α ≥ 1/2. De plus, nous avons présenté expérimentalement les performances des algorithmes proposés et utilisé en comparaison un algorithme Explore-Then-Commit s'appuyant sur l'algorithme d'exploration pure présenté dans le chapitre C.4. Cette thèse avait pour but d'introduire le nouveau cadre du bandit bilinéaire graphique et de fournir les premières solutions aux problèmes courants posés dans la littérature sur le bandit. De nombreuses autres approches et modifications peuvent être envisagées. Nous en présentons deux dans ce qui suit. Dans le cadre d'une décentralisation totale (sans communication), quel type d'algorithmes pouvons-nous concevoir pour tirer parti du cadre du bandit linéaire ? Si nous autorisons la communication, comment pouvons-nous adapter les algorithmes proposés et quel est le compromis entre la quantité de communication et la performance ?

	Questions ouvertes :

C.6.

If X = (e1, . . . , e d ), when the learner chooses xt = ei, the reward yt = ⟨ei, θ⋆⟩ + ηt = [θ⋆]i + ηt, which does not share any information with the rewards of the other arms.

We drop the β parameter, because in our case it is equal to 1.

(1) ⋆ , . . . , x(n) ⋆ ), we have x (i)⊤ ⋆ M ⋆ + M ⊤ ⋆ x (j) ⋆ ≤ x ⊤ ⋆ M ⋆ + M ⊤ ⋆ x ′ ⋆ . (3.2)

Those quantities are not equal since the matrix M⋆ is not necessarily symmetric.

m (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆, and set α = 1+ξ

. Then, the expected global reward y = (i,j)∈E x (i)⊤ M ⋆ x (j) associated with the allocation x[START_REF] Abbasi | Robustness to adversarial examples through an ensemble of specialists[END_REF] , . . . , x (n) ∈ X n returned by Algorithm 5 verifies:y ≥ αy ⋆ . where y ⋆ = (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ .Finally, the complexity of the algorithm is in O(K 2 + n 2 ).

(i,j)∈E x (i)⊤ ⋆

M ⋆ x 0 , = 0, then ξ = 0 as well and we are in the worst case scenario where we can only have a guarantee on an α-approximation with α =

1/2. In practice, having ξ = 0 is reached when given the couple (x ⋆ , x ′ ⋆ ) = arg max (x,x ′ )∈X2 x ⊤ M ⋆ + M ⊤ ⋆ x ′ , this x 0 is either x ⋆ or x ′ ⋆ .In other words, if the unwanted couples of arms (x ⋆ , x ⋆ ) and (x ′ ⋆ , x ′ ⋆ ) give low rewards, then the guarantee on the reward will be badly impacted. Hence we can wonder how to prevent this phenomenon. We answer this question in the next section by taking into account both the proportion of undesirable couple of arms and their potential rewards at the selection of the pair (x ⋆ , x ′ ⋆ ) in order to improve the performance of the algorithm both in practice and theoretically.3.1.3 Improved algorithm using the graph structureIn this section, we want to capitalize on Algorithm 5 and its 1+ξ 2 -optimal solution to refine the allocation of the arms x ⋆ and x ′ ⋆ such that the obtained suboptimal rewardsx ⊤ ⋆ M ⋆ x ⋆ and x ′⊤ ⋆ M ⋆ x ′ ⋆penalize as less as possible the global reward. Indeed, in the Algorithm 5, the choice of the couple (x ⋆ , x ′ ⋆ ) is only guided by the potential gain that could be obtained at the level of the cut edges (i.e., that goes from a node in V 1 to a node in V 2 or vice versa). It does not take into account all the m 1 rewards of the form x ⊤ ⋆ M ⋆ x ⋆ and 29

x⊤ ⋆ M ⋆ x′ ⋆ + m 2→1 • x′⊤ ⋆ M ⋆ x⋆ + m 1 • x⊤ ⋆ M ⋆ x⋆ + m 2 • x′⊤ ⋆ M ⋆ x′ ⋆ = m 1→2 x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 • x ′⊤ ⋆ M ⋆ x ⋆ + m 1 • x ⊤ ⋆ M ⋆ x ⋆ + m 2 • x ′⊤ ⋆ M ⋆ x ′ ⋆ + ∆ = m 1→2 x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 • x ′⊤ ⋆ M ⋆ x ⋆ + m 1 • x ⊤ ⋆ M ⋆ x ⋆ + m 2 • x ′⊤ ⋆ M ⋆ x ′ ⋆ + ϵ (i,j)∈E x (i)⊤ ⋆ M ⋆ x (i) ⋆ = m 1→2 x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 • x ′⊤ ⋆ M ⋆ x ⋆ + m 1 • x ⊤ ⋆ M ⋆ x ⋆ + m 2 • x ′⊤ ⋆ M ⋆ x ′ ⋆ + ϵy ⋆ = 1 + ξ 2 y ⋆ + ϵy ⋆ = 1 + ξ 2 + ϵ y ⋆ .What is ϵ and what value can we expect?This parameter measures the gain with respect to the optimal reward one could get by considering the undesirable couple of arms of the form (x, x).The value of ϵ is high when the rewards associated with the couples (x ⋆ , x′ ⋆ ) and (x ′ ⋆ , x⋆ ) are close to those of (x ⋆ , x ′ ⋆ ) and (x ′ ⋆ , x ⋆ ) respectively and when the rewards associated with the suboptimal couples (x ⋆ , x⋆ ) and (x ′ ⋆ , x′ ⋆ ) are much higher than those of (x ⋆ , x ⋆ ) and (x ′ ⋆ , x ′ ⋆ ) respectively. On the contrary, ϵ is low (and close to 0) if the suboptimal couples of arms (x ⋆ , x ⋆ ) and (x ′ ⋆ , x ′ ⋆ ) already give high rewards (or the highest among the other suboptimal couples of the form (x, x)), hence the central entity does not gain a lot by choosing another couple of arms than (x ⋆ , x ′ ⋆ ). Notice that α = 1+ξ 2 + ϵ ≤ 1 by construction. In fact when ξ = 1 it means that the couple of arms of the form (x, x) gives the highest reward, hence (x ⋆ , x ′ ⋆ ) = (x ⋆ , x′ ⋆ ) which gives ϵ = 0 and α = 1. Corollary 3.1. Let us consider the same setting as in Theorem 3.3, the approximation ratio can be defined with the parameters m 1→2 , m 2→1 , m 1 and m 2 that depend on the graph and the approximation algorithm of the max-cut problem such that

The design of Algorithm 6 and its associated guarantee are posterior to the best arm identification algorithm that we proposed in this chapter. This extension is an ongoing work and might be added to the final version of this thesis.

x (i)⊤ ⋆ M ⋆ x (j)⊤ ⋆ -T × f (T ) ,

(i)⊤ ⋆ M ⋆ x (j)⊤ ⋆ -c × (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j)⊤ ⋆ = (1 -c) (i,j)∈E x (i)⊤ ⋆

m ⟨m 1→2 • z x⋆ x′ ⋆ + m 2→1 • z x′ ⋆ x⋆ + m 1 • z x⋆ x⋆ + m 2 • z x′ ⋆ x′ ⋆ , θ ⋆ ⟩

In the remaining of this paper, we consider a finite set of experiments; some results could be easily transposed to a continuous setting.

We assume that the experiments span R d so that the matrix X ⊤ X is non singular. If this is not the case then we may project the data onto a lower dimensional space.

It suffices to consider exactly K(K -1)/2 directions as the result is the same for xx ′ and x ′ -x.

Ωnorm is not limited to ℓ2 norm. The results we present hold as long as the norm used to compare X and ϕY (X) comes from a scalar product on X .

We prove this result in the supplementary material.

The grey area should actually be bigger since the best response to the attack would also change the decision on the upper part between the OBC and the doted line. We focus on what happens on the star points for simplicity.

We use ℓ∞-PGD with 20 iterations and ϵ∞ = 0.031 to train the first classifier and to build D.

More algorithmic and implementation details can be found in the supplementary materials.

In order for the attack to succeed, it it more efficient to compute the expected transformation of the logits instead of taking the expectation over the loss. More details on this in the supplementary materials.

Ces quantités ne sont pas égales puisque la matrice M⋆ n'est pas nécessairement symétrique.

m (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ , et on fixe α = 1+ξ

. Alors, la récompense globale y = (i,j)∈E x (i)⊤ M ⋆ x (j) associée à l'allocation x[START_REF] Abbasi | Robustness to adversarial examples through an ensemble of specialists[END_REF] , . . . , x (n) ∈ X n retournée par l'Algorithme 14 vérifie :y ≥ αy ⋆ . où y ⋆ = (i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ .Enfin, la complexité de l'algorithme est de O(K 2 + n 2 ).

Pour comprendre et analyser ce nouvel algorithme, définissons ∆ ≥ 0 la différence de récompense globale de l'attribution de (x ⋆ , x′ ⋆ ) au lieu de (x ⋆ , x ′ ⋆ ) comme suit :∆ = m 1→2 x⊤ ⋆ M ⋆ x′ ⋆ -x ⊤ ⋆ M ⋆ x ′ ⋆ + m 2→1 x′⊤ ⋆ M ⋆ x⋆ -x ′⊤ ⋆ M ⋆ x ⋆

(i,j)∈E x (i)⊤ ⋆ M ⋆ x (j) ⋆ , et on fixe α = 1+ξ 2 + ϵ.Alors, la récompense globale y = (i,j)∈E x (i)⊤ M ⋆ x (j) associée à l'allocation x[START_REF] Abbasi | Robustness to adversarial examples through an ensemble of specialists[END_REF] , . . . , x (n) ∈ X n retournée par l'Algorithme 15 vérifie :y ≥ αy ⋆ . où y ⋆ = (i,j)∈E x (i)⊤ ⋆

Remerciements

A Refined bounds for randomized experimental design

This appendix contains the paper "Refined bounds for randomized experimental design", Neurips 2019 Workshop "ML with Guarantees", G. Rizk, I. Colin, A. Thomas and Moez Draief. Experimental design is an approach for selecting samples among a given set so as to obtain the best estimator for a given criterion. In the context of linear regression, several optimal designs have been derived, each associated with a different criterion: mean square error, robustness, etc. Computing such designs is generally an NP-hard problem and one can instead rely on a convex relaxation that considers probability distributions over the samples. Although greedy strategies and rounding procedures have received a lot of attention, straightforward sampling from the optimal distribution has hardly been investigated. In this paper, we propose theoretical guarantees for randomized strategies on E and G-optimal design. To this end, we develop a new concentration inequality for the eigenvalues of random matrices using a refined version of the intrinsic dimension that enables us to quantify the performance of such randomized strategies. Finally, we evidence the validity of our analysis through experiments, with particular attention on the G-optimal design applied to the best arm identification problem for linear bandits.

A.1 Introduction

Experimental designs consist in the selection of the best samples or experiments for the estimation of a given quantity. A well-known and extensively studied example is the one of the ordinary least squares (OLS) estimator in the linear regression setting. The OLS estimator being unbiased, Is there a classifier that ensures optimal robustness against all adversarial attacks? This paper tackles this question by adopting a game-theoretic point of view. We present the adversarial attacks and defenses problem as an infinite zero-sum game where classical results (e.g. Nash or Sion theorems) do not apply. We demonstrate the non-existence of a Nash equilibrium in our game when the classifier and the Adversary are both deterministic, hence giving a negative answer to the above question in the deterministic regime. Nonetheless, the question remains open in the randomized regime. We tackle this problem by showing that any deterministic classifier can be outperformed by a randomized one. This gives arguments for using randomization, and leads us to a simple method for building randomized classifiers that are robust to state-or-the-art adversarial attacks. Empirical results validate our theoretical analysis, and show that our defense method considerably outperforms Adversarial Training against strong adaptive attacks, by achieving 0.55 accuracy under adaptive PGD-attack on CIFAR10, compared to 0.42 for Adversarial training.

Evaluating against strong adversarial attacks. When evaluating a defense against adversarial examples, it is crucial to test the robustness of the method against the best possible attack. Accordingly, the defense method should be evaluated against attacks that were specifically tailored to it (a.k.a. adaptive attacks). In particular, when evaluating randomized algorithms, one should use Expectation over Transformation (EOT) to avoid gradient masking as pointed out by [START_REF] Athalye | Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples[END_REF] and [START_REF] Carlini | On Evaluating Adversarial Robustness[END_REF]. More recently, [START_REF] Tramer | On adaptive attacks to adversarial example defenses[END_REF] emphasized that one should also make sure that EOT is computed properly 6 . Previous works such as [START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] and [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF] estimate the EOT through a Monte Carlo sampling which can introduce a bias in the attack if the sample size is too small. Since we assume perfect information for the Adversary, it knows the exact distribution of the mixture. Hence it can directly compute the expectation without using a sampling method, which avoid any bias. Table B.1 evaluates our method against strong adaptive attacks namely Adaptive-ℓ ∞ -PGD and Adaptive-ℓ 2 -C&W.

Hard constraint parameter. The typical value of ϵ in the hard constraint depends on the norm we consider in the problem setting. In this paper, we use an ℓ 2 norm, however, the constraint parameter for ℓ ∞ -PGD attack was initially set to be an ℓ ∞ constraint. In order to compare attacks of similar strength, we choose different threshold (ϵ 2 or ϵ ∞ ) values which result in balls of equivalent volumes. For CIFAR10 an CIFAR100 datasets [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], which are 3 × 32 × 32 dimensional spaces, this gives ϵ ∞ = 0.03 and ϵ 2 = 0.8 (we also give results for ϵ 2 equal to 0.6 and 0.4 as this values are sometimes used in the literature). Since Adaptive-ℓ 2 -C&W attack creates an unbounded perturbation on the examples, we implemented the constraint from Equation B.6 by checking at test time whether the ℓ 2 -norm of the perturbation exceeds a certain threshold ϵ 2 ∈ {0.4, 0.6, 0.8}. If it does, the adversarial example is disregarded, and we keep the natural example instead.

Experimental results. In Table B.1 we compare the accuracy, on CIFAR10 and CIFAR100, of our method and classical Adversarial Training under attack with Adaptive-ℓ ∞ -PGD and Adaptive-ℓ 2 -C&W, both run for 100 iterations. We used 5 times more iterations for the evaluation as we used during training, and carefully check for convergence. the rational behind this is that, for a classifier to be fully robust, its loss of accuracy should be controlled when the attacks are stronger than the ones it was trained on. For both attacks, both datasets and all thresholds (i.e. the budget for a perturbation), the accuracy under attack of our mixture is higher than the single classifier with Adversarial Training. Our defense is especially more robust than Adversarial Training when the threshold is high.

Extension to more than two classifiers. In this paper we focus our experiments on a mixture of two classifiers to present a proof of concept of Theorem B.4. Nevertheless, a mixture of more than two classifiers can be constructed by adding at each step t a new classifier trained naturally on the dataset D that contains adversarial examples against the mixture at step t-1. Since D has to be constructed from a mixture, one would have to use an adaptive attack as Adaptive-ℓ ∞ -PGD. We refer the reader to the supplementary material for this extended version of the algorithm and for all the implementation details related to our experiments (architecture of models, optimization settings, hyper-parameters, etc.).

Hence, the best response h is such that for every x ∈ X , and y ∈ Y, one has h(x) = y if and only if f y (x) ≤ f -y (x). Thus, h is the optimal Bayes classifier for the distribution (ν, ϕ 1 #µ 1 , ϕ -1 #µ -1 ). Furthermore, for y = 1 (resp. y = -1), and for any B ⊂ P h (resp. B ⊂ N h ) one has:

with Y ∼ ν and for all y ∈ Y, X|(Y = y) ∼ ϕ y #µ y .

Theorem B.3 (Non-existence of a pure Nash equilibrium). In our zero-sum game with λ ∈ (0, 1) and penalty Ω ∈ {Ω mass , Ω norm }, there is no Pure Nash Equilibrium.

Proof. Let h be a classifier, ϕ ∈ BR Ω (h) an optimal attack against h. We will show that h / ∈ BR(ϕ), i.e. that h does not satisfy the condition from Lemma B.3. This suffices for Theorem B.3 to hold since it implies that there is no

According to Lemmas B.1 and B.2, whatever penalty we use, there exists δ > 0 such that ϕ 1 #µ 1 (P h (δ)) = 0 or ϕ -1 #µ -1 (N h (δ)) = 0. Both cases are symmetrical, so let us assume that P h (δ) is of null measure for the transported distribution conditioned by y = 1. Furthermore we have ϕ -1 #µ -1 (P h (δ)) = µ -1 (P h (δ)) > 0 since ϕ -1 is the identity function on P h (δ), and since µ -1 is of full support on X . Hence we get the following:

Since the right side of the inequality is null, we also get:

This inequality is incompatible with the characterization of best response for the Defender of Lemma B.3. Hence h / ∈ BR(ϕ).

Theorem B.4. (Randomization matters) Let us consider

Then for any α ∈ (max(λ, 1 -λ), 1) and for any andm q h is the mixture of h by q.

Natural Training. To train an undefended classifier we use the following hyperparameters.

• Number of Epochs: 200

• Batch size: 128

• Loss function: Cross Entropy Loss

• Optimizer : SGD algorithm with momentum 0.9, weight decay of 2 × 10 -4 and a learning rate that decreases during the training as follows:

Adversarial Training. To adversarially train a classifier we use the same hyperparameters as above, and generate adversarial examples using the ℓ ∞ -PGD attack with 20 iterations. When considering that the input space is [0, 255] 3×32×32 , on CIFAR10 and CIFAR100, a perturbation is considered to be imperceptible for ϵ ∞ = 8. Here, we consider X = [0, 1] 3×32×32 which is the normalization of the pixel space [0.255] 3×32×32 . Hence, we choose ϵ 2 = 0.031 (≈ 8/255) for each attack. Moreover, the step size we use for ℓ ∞ -PGD is 0.008 (≈ 2/255), we use a random initialization for the gradient descent and we repeat the procedure three times to take the best perturbation over all the iterations i.e the one that maximises the loss. For the ℓ ∞ -PGD attack against the mixture m q h , we use the same parameters as above, but compute the gradient over the loss of the expected logits (as explained in the main paper).

Evaluation Under Attack. At evaluation time, we use 100 iterations instead of 20 for Adaptiveℓ ∞ -PGD, and the same remaining hyperparameters as before. For the Adaptive-ℓ 2 -C&W attack, we use 100 iterations, a learning rate equal to 0.01, 9 binary search steps, and an initial constant of 0.001. We give results for several different values of the rejection threshold: ϵ 2 ∈ {0.4, 0.6, 0.8}.

Computing Adaptive-ℓ 2 -C&W on a mixture

To attack a randomized model, it is advised in the literature [START_REF] Tramer | On adaptive attacks to adversarial example defenses[END_REF] to compute the expected logits returned by this model. However this advice holds for randomized models that return logits in the same range for a same example (e.g. classifier with noise injection). Our randomized model is a mixture and returns logits that depend on selected classifier. Hence, for a same example, the logits can be very different. This phenomenon made us notice that for some example in the dataset, computing the expected loss over the classifier (instead of the expected logits) performs better to find a good perturbation (it can be seen as computing the expectation of the logits normalized thanks to the loss). To ensure a fair evaluation of our model, in addition of using EOT with the expected logits, we compute in parallel EOT with the expected loss and take the perturbation that maximizes the expected error of the mixture. See the submitted code for more details.

Library used. We used the Pytorch and Advertorch libraries for all implementations.

Training method

NA of the 1 st clf AUA of the

Here to find the parameter α, the grid search is more costly. In fact in the two-classifier version we only need to train the first and second classifier without taking care of α, and then test all the values of α using the same two classifier we trained. For the extended version, the third classifier (and all the other ones added after) depends on the first classifier, the second one and their weights 1 -α and α. Hence the third classifier for a certain value of α can't be use for another one and, to conduct the grid search, one have to retrain all the classifiers from the third one. Naturally the parameters α depends on the number of classifiers n in the mixtures. dant, on peut tirer un enseignement de cette stratégie, à savoir qu'étant donné le bras conjoint optimal (x

Ainsi, au lieu de chercher l'allocation optimal (ce qui est NP-Dur), on peut alternativement rechercher l'allocation qui, pour toute arête (i, j) ∈ E, construit autant de paires (x (i) , x (j) ) = (x ⋆ , x ′ ⋆ ) que possible. Affecter x ⋆ à un sous-ensemble de noeuds et x ′ ⋆ au complémentaire revient à couper le graphe en deux morceaux et à créer deux ensembles distincts de noeuds

Ainsi, la stratégie décrite se résume à trouver une coupe passant par le nombre maximal d'arêtes.

Ce problème est connu sous le nom de Max-Cut (voir e.g., [START_REF] Goemans | 879-approximation algorithms for max cut and max 2sat[END_REF][START_REF] Sahni | P-complete approximation problems[END_REF]), qui est également NP-Dur. Cependant, l'attention considérable portée à ce problème nous permet d'utiliser l'un des nombreux algorithmes d'approximation (voir, e.g., Algorithm 13) qui garantissent de produire une coupe passant par au moins une fraction donnée des arêtes du graphe. La plupart des garanties pour l'approximation du problème de Max-Cut sont données par rapport à la solution optimale de Max-Cut, ce qui n'est pas exactement la garantie que nous recherchons : nous avons besoin d'une garantie en proportion du nombre total d'arêtes. Nous devons donc faire attention à l'algorithme que nous choisissons.

Algorithm 13: Approx-MAX-CUT [START_REF] Sahni | P-complete approximation problems[END_REF] Entrée:

A partir de l'Algorithme 13, on peut avoir une garantie sur la proportion d'arêtes coupées par rapport au nombre total d'arêtes m. Nous énonçons cette garantie dans la proposition suivante.

Proposition C.1. Étant donné un graphe

Étant donné cette garantie par rapport au nombre total d'arêtes, il ne reste plus qu'à présenter la stratégie complète qui consiste à allouer aux noeuds de V 1 le bras x ⋆ et aux noeuds de V 2 le bras x ′ ⋆ . Nous donnons la stratégie dans l'Algorithme 14. Avec cet algorithme, étant donné l'allocation retournée (x (1) , . . . , x (n) ), pour certaines arêtes (i, j) ∈ E, les bras alloués associés seront les couples optimaux

Des matrices de paramètres différentes M

(i,j) ⋆ pour chaque arête (i, j) ∈ E. Alors que le traitement d'une matrice commune de paramètres M ⋆ pour toutes les arêtes (i, j) ∈ E était pratique pour agréger les récompenses et construire une estimation commune M pour tous les agents, lorsque les récompenses y (i,j) t sont définies avec des matrices différentes M (i,j) ⋆ , le problème devient plus compliqué. En effet, considérons le cas où la récompense y (i,j) t est définie comme suit pour chaque (i, j) ∈ E :

sont des matrices à paramètres inconnus et η (i,j) t des variables aléatoires σ-sousgaussiennes.

Ce paramètre est pertinent lorsque les agents n'ont pas les mêmes interactions entre chacun de leurs voisins, et donc pas la même fonction de récompense.

Question ouverte : Dans le contexte de l'exploration pure, comment la condition d'arrêt change-t-elle ? Existe-t-il une stratégie d'échantillonnage pour chaque agent telle que les estimations M(i,j) sont construites en satisfaisant un critère d'optimal design ? Cadre décentralisé . Lorsque les agents sont contrôlés par une entité centrale, il est possible d'agréger les différentes récompenses et de construire une estimation commune M de M ⋆ . De plus, nous avons vu que les différents objectifs qui apparaissent sont relatifs aux bras d'arêtes et non directement aux bras de noeuds sélectionnés par chaque agent. En effet, cela est dû au fait que nous pouvons exprimer le bandit bilinéaire graphique comme des bandits linéaires au niveau des arêtes. Cet aspect particulier rend le cadre décentralisé un peu délicat car la coordination de deux agents sans communication pour tirer respectivement les bras de noeuds qui construiront bras d'arêtes désirés devient encore plus compliqué.

Cependant, d'autres problèmes surgissent même si le problème de coordination est résolu. Par exemple, dans le problème d'identification du meilleur bras, nous avons déjà conçu une procédure d'échantillonnage qui peut être exécutée en parallèle pendant un tour, d'où un choix décentralisé pour chaque agent. Cependant, la condition d'arrêt dépend de l'estimation M construite avec les bras d'arêtes pendant la procédure d'apprentissage, mais lorsque les agents ne communiquent pas, cette estimation ne peut pas être construite. En effet, un agent ne connaît que le bras qu'il tire et observe la récompense. Or, la récompense est linéaire par rapport à au bras d'arête associée et l'agent n'a pas accès à ce bras d'arête puisqu'il est construit avec son bras de noeud mais aussi avec celui de ses voisins (auquel il n'a pas accès).

ABSTRACT

We introduce a new model called Graphical Bilinear Bandits where a learner (or a central entity) allocates arms to nodes of a graph and observes for each edge a noisy bilinear reward representing the interaction between the two end nodes. In this thesis, we study the best arm identification problem and the maximization of cumulative rewards. For the first problem, a learner wants to find the graph allocation maximizing the sum of the bilinear rewards obtained through the graph. For the second problem, during the learning process, the learner has to make a trade-off between exploring the arms to gain accurate knowledge of the environment and exploiting the arms that appear to be the bests to obtain the highest reward. Regardless of the learner's goal, the graphical bilinear bandit model reveals an underlying NP-Hard combinatorial problem that precludes the use of any existing best arm identification (BAI) or regret-based algorithms. For this reason, we first propose an 𝛼-approximation algorithm for the underlying NP-hard problem, and then tackle the two problems mentioned above. By efficiently exploiting the geometry of the bandit problem, we propose a random sampling strategy for the BAI problem with theoretical guarantees. In particular, we characterize the influence of the graph structure (e.g., star, complete or circle) on the convergence rate and propose empirical experiments that confirm this dependence. For the problem of maximizing the cumulative rewards, we present the first regret-based algorithm for graphical bilinear bandits using the principle of optimism in the face of uncertainty. Theoretical analysis of the presented method gives an upper bound of 𝑂 # $√T'on the 𝛼-regret and highlights the impact of the graph structure on the convergence rate. Finally, we demonstrate by various experiments the validity of our approaches.

MOTS CLÉS

Apprentissage séquentiel, Bandits Bilinéaires Graphiques, Multi-agents RÉSUMÉ Nous introduisons un nouveau modèle appelé Bandits Bilinéaires Graphiques où un apprenant (ou une entité centrale) alloue des bras aux noeuds d'un graphe et observe pour chaque arête une récompense bilinéaire bruitée représentant l'interaction entre les deux noeuds associés. Dans cette thèse, nous étudions le problème d'identification du meilleur bras et la maximisation des récompenses cumulées. Pour le premier, un apprenant veut trouver l'allocation du graphe maximisant la somme des récompenses bilinéaires obtenues à travers le graphe. Pour le second problème, au cours du processus d'apprentissage, l'apprenant doit faire un compromis entre l'exploration des bras pour acquérir une connaissance précise de l'environnement et l'exploitation des bras qui semblent être les meilleurs pour obtenir la récompense la plus élevée. Quel que soit l'objectif de l'apprenant, le modèle de bandits bilinéaires graphiques révèle un problème combinatoire sous-jacent qui est NP-Dur et qui empêche l'utilisation de tout algorithme existant pour l'identification du meilleur bras (BAI) ou pour la maximisation des récompenses cumulées. Pour cette raison, nous proposons tout d'abord un algorithme d'𝛼-approximation pour le problème NP-Dur sous-jacent, puis nous nous attaquons aux deux problèmes mentionnés ci-dessus. En exploitant efficacement la géométrie du problème du bandit, nous proposons une stratégie d'échantillonnage aléatoire pour le problème BAI avec des garanties théoriques. En particulier, nous caractérisons l'influence de la structure du graphe (par exemple, étoile, complet ou cercle) sur le taux de convergence et proposons des expériences empiriques qui confirment cette dépendance. Pour le problème de la maximisation des récompenses cumulées, nous présentons le premier algorithme basé sur le regret pour les bandits bilinéaires graphiques utilisant le principe d'optimisme face à l'incertitude. L'analyse théorique de la méthode présentée borne l'𝛼-regret par 𝑂 # $√T' et souligne l'impact de la structure du graphe sur le taux de convergence. Enfin, nous démontrons par diverses expériences la validité de nos approches.

KEYWORDS

Sequential learning, Graphical Bilinear Bandits, Multi-agents