Cindy Bokobza

Thomas Bour- Geois

David-Peter Cohen

Adrien Dufour

Jorge Gallego

Pierre Gressens

Christophe Le Priol

Amélia Madani

Boris Matrot

Amazigh Mokhtari

Baptiste Porte

Nélina Ramanantsoa

Éléonore Sizun

Juliette Van

Steenwinckel ) Ainsi

Omar Darwiche Domingues

Rémy Degenne

Andrea Tirinzoni

Keywords:

Given the ever increasing cost of designing de novo therapeutic molecules, and the huge amount of currently available biological data, the development of systematic explorative pipelines for drug development has become of paramount importance. In my thesis, I focused on drug repurposing, which is a paradigm that aims at identifying new therapeutic indications for known chemical compounds. I investigated how to leverage in a reproducible way for drug screening the huge collection of available transcriptomic data -relative to protein production through the transcription of gene DNA sequences. The current state of research in drug development indicates that such generic approaches might considerably fasten the discovery of promising therapies, especially for rare diseases research.

First, noting that transcriptomic measurements are the product of a complex dynamical system of regulatory interactions on genes, I worked on integrating diverse types of biological information in order to build a model of these regulations. That is where gene regulatory networks, and more specifically, Boolean networks, intervene.

Such models are useful for both explaining observed transcription levels, and for predicting the result of gene activity perturbations through molecules. Second, these models allow online in silico drug testing. While using the predictive features of Boolean networks can be costly, the core assumption of this thesis is that, combining them with sequential learning algorithms, such as multi-armed bandits, might mitigate that effect, and help control the error rate in recommended therapeutic candidates. This is the drug testing procedure suggested throughout my PhD. The question of the proper integration of known side information about the chemical compounds into multi-armed bandits is crucial, and has also been investigated.

Finally, I applied part of my work to ranking different treatment protocols for neurorepair in the case of encephalopathy in premature infants. On the theoretical side, I also contributed to the design of an algorithm which is able to extend the drug testing procedure to a collaborative setting, where personalized recommendation of drug candidates can be made to heterogeneous subpopulations of patients, while being learnt using observations made on the whole set of subpopulations.

Titre : Combinaison de réseaux de régulation génique et d'apprentissage statistique séquentiel pour le repositionnement de médicaments

Résumé :

À cause du coût toujours croissant de la conception de molécules thérapeutiques de novo, et la masse considérable de données biologiques disponibles actuellement, la création de méthodes d'exploration systématique pour le développement de thérapies est devenue un enjeu crucial. Lors de ma thèse, je me suis concentrée sur le paradigme du repositionnement de médicaments, qui cherche à identifier de nouvelles indications thérapeutiques pour des molécules chimiques connues. J'ai cherché des méthodes qui traitent de façon reproductible la quantité importante de données transcriptionnelles disponibles (relative à la production de protéines à travers la transcription des séquences ADN géniques) pour le criblage de molécules. Une revue de l'état de la recherche en développement de médicaments montre que de telles approches génériques peuvent permettre de considérablement accélérer la découverte de thérapies prometteuses, plus particulièrement contre les maladies rares. Premièrement, en remarquant que les mesures d'activité transcriptionnelle résultent d'un réseau dynamique complexe d'interactions régulatrices entre gènes, j'ai travaillé sur l'intégration d'information biologique de différent types afin de construire un modèle de ces régulations géniques. C'est là que les réseaux de régulation génique, et, plus spécifiquement, les réseaux booléens, interviennent.

Ces modèles permettent à la fois d'expliquer les mesures d'origine transcriptionnelle observées, et de prédire le résultat de perturbations de l'activité de certains gènes par des molécules. Ensuite, ces modèles permettent d'effectuer des essais cliniques in silico de médicaments. Tandis que l'utilisation des prédictions faites par des réseaux booléens peut s'avérer coûteuse, l'hypothèse centrale de ma thèse est que leur combinaison avec des algorithmes d'apprentissage séquentiel, comme les bandits à bras multiples, peuvent non seulement réduire ce coût, mais également contrôler le taux d'erreur dans les recommandations de candidats thérapeutiques.

Cette démarche est la procédure d'essai clinique in silico analysée tout au long de mon travail de thèse. Le problème de l'intégration des vecteurs caractéristiques connues des composants chimiques dans les bandits à bras multiples a également été étudié. Enfin, j'ai appliqué une partie de mon travail de thèse au classement de différents protocoles de traitement pour de la neuroréparation dans le cas d'encéphalopathies chez des enfants prématurés. J'ai également contribué à la conception d'un algorithme qui permet d'étendre la procédure d'essai médicamenteux in silico à un cadre collaboratif, où plusieurs sous-populations de patients sont considérées simultanément.

Mots-clefs : repositionnement de médicaments ; analyse de données transcriptionnelles ; inférence de réseaux de régulation génique ; réseaux booléens ; essai de médicaments ; apprentissage séquentiel ; bandits à bras multiples.

Résumé substantiel en français

Le développement de médicaments est un processus connu pour son coût en temps de recherche, pouvant prendre près de dix ans, et financier important, avec un coût chiffré en millions de dollars. De plus, il est sujet à d'assez forts taux d'échec de développement, 1 même tardivement dans le développement. Seulement 64 % des traitements entrés en phase III d'un essai clinique après 2007 ont été commercialisés en 2012. 2

Cependant, grâce aux améliorations technologiques dans le domaine de la bioinformatique, une large quantité de données biologiques est disponible publiquement ; par exemple, la base de données Gene Expression Omnibus (GEO) pour les données transcriptomiques. 3 Cela permet d'accélérer la phase de recherche de thérapie en explorant ces données. Afin de garantir la reproducibilité et la transparence des résultats obtenus (et donc, leur robustesse), il est nécessaire pour traiter ces données de faire appel à des méthodes in silico automatisées. 4 De plus, le développement de molécules thérapeutiques de novo est à la fois complexe (puisque requérant la conception de molécules synthétisables ayant la fonction désirée) et relativement risqué, car il n'y a d'informations ni sur leur capacité d'administration, ni sur leurs éventuels effets secondaires néfastes, ce qui explique une partie des échecs dans le développement. 5 Ce constat m'a incitée à m'intéresser à un paradigme nommé le « repositionnement de médicaments » (drug repurposing), qui propose de rechercher de nouvelles thérapies parmi des composants chimiques d'ores et déjà commercialisés, soit en tant que molécules-outils, soit en tant que médicaments approuvés. 6 Cela permet à la fois de limiter la phase préclinique et la phase I de développement, mais également de réduire la probabilité d'émergence d'effets secondaires non désirables inconnus. Ces observations sont basées sur une revue de l'état-de-l'art effectuée en début de thèse. 7En particulier, cette thèse s'intéresse à une approche spécifique, nommé l'inversion de signature (signature reversion). 8 Cette méthode a déjà démontré son efficacité, par exemple, pour le repositionnement de médicaments contre la grippe,9 où les auteurs ont pu vérifier in vitro que l'administration de nifurtimox ou de chrysine permettait de réduire la quantité de virions grippaux.

Cette approache exploite des données transcriptomiques d'expression de gène. Le niveau d'expression d'un gène donné, dans un contexte spatial et temporal fixé, correspond à une mesure du nombre de transcriptions de la séquence génique. Cette mesure est corrélée au nombre de copies effectuées du produit encodé par ce gène (par exemple, une protéine). Ce produit est ce par quoi le gène exécute sa fonction dans l'organisme, en participant à des réactions chimiques au sein de l'organisme. Plus le nombre de copies de ce produit est élevé, plus l'effet de la fonction du gène se fait ressentir. La mesure du niveau d'expression d'un gène consiste d'abord à dénombrer les transcrits (nombres d'ARN messager, ou ARNm associés à ce gène) présents dans un échantillon de cellules, ce qui peut être fait par les technologies de séquençage à très haut débit. Une normalisation de cette quantité donne le niveau d'expression, c'est-à-dire le nombre moyen de transcriptions du gène par échantillon.

L'approche d'inversion de signature utilise alors cette donnée dans le but de trouver un candidat thérapeutique contre une pathologie fixée :

(1). D'abord, le niveau d'expression de gène chez des échantillons traités par diverses molécules, avec leur population de référence (« contrôle » ), est mesuré. À partir de ces profils transcriptionnels, on en déduit une signature « traitement » , c'est-à-dire une quantification de la variation d'expression de chaque gène des échantillons traités par rapport aux contrôles due à la différence de traitement. Par exemple, si l'expression d'un gène A est globalement plus (respectivement, moins) importante chez les échantillons traités par rapport à son expression chez les contrôles, alors ce gène sera noté comme étant « up-régulé » (resp., « down-régulé » ). L'avantage de considérer des signatures, plutôt que les profils transcriptionnels eux-mêmes, est de permettre de comparer les évolutions au niveau de l'expression de gène corrélées au traitement, indépendamment du contexte spatial et temporel des mesures transcriptomiques ;

(2). De même, le niveau d'expression de gène est mesuré chez des individus sains pour la pathologie considérée et des individus malades. Pareillement, une signature « maladie » est construite, représentant la différence d'expression entre individus sains et malades due à la maladie ; (3). Pour chaque signature « traitement » , une comparaison avec la signature « maladie » est effectuée. Si une signature « traitement » est similaire à la signature « malade » , alors cela signifie que l'évolution au niveau de l'expression de gène chez un échantillon traité est corrélée à celle chez un échantillon malade ; autrement dit, le traitement semble « reproduire » un profil malade. Si au contraire, il existe une forte différence entre signatures « traitement » et « maladie » , alors, par un même raisonnement, on en déduit que le traitement semble « restaurer » des niveaux d'expression de gène similaires à ceux observés chez les individus sains ; (4). Traditionnellement, les candidats recommandés à la suite de cette procédure sont ceux dont la dissimilarité entre leur signature associée et la signature « maladie » est la plus grande.

Exemple d'application du principe de l'inversion de signature. J'ai appliqué10 cette procédure dans un projet de sélection d'un protocole de traitement par cellules souches mésenchymateuses humaines chez le modèle murin d'encéphalopathie de la prématurité.

Dans cette étude, deux voies d'administration, trois âges d'injection des cellules souches et trois grades de doses (dépendant du poids de l'animal) ont été testés sur des rats modèles dont les cellules microgliales ont été séquencées, ainsi que celles d'animaux modèles n'ayant pas reçu de cellules souches, et d'animaux sains injectés avec un traitement factice. En me basant sur les profils transcriptionnels obtenus et l'approche d'inversion de signature, j'ai calculé la signature « traitement » associée à chaque protocole de traitement (en comparant échantillons modèles traités selon le protocole par cellules souches, et échantillons modèles non traités) ainsi que la signature « contrôle » , en comparant échantillons sains et échantillons modèles non traités par cellules souches. Enfin, j'ai classé les différents protocoles de traitement par similarité décroissante entre chaque signature « traitement » et la signature « contrôle » .

Le classement obtenu ne semble pas dépendre de facteurs confondants liés à la mesure des données et non au traitement (« effet batch » ), et ne présente pratiquement que des scores positifs, dont certains sont proches de la valeur 1. Cela suggère que l'injection de cellules souches est globalement bénéfique pour le traitement de l'encéphalopathie de la prématurité chez le modèle murin. L'approche par inversion de signature semble donc être intéressante en pratique. Dans ce cas, elle permet de déterminer le mode d'administration (voie, âge, dose relative au poids) qui maximise l'effet des cellules souches mésenchymateuses humaines sur la restauration d'un phénotype sain.

Limites de l'approche d'inversion de signature. Cependant, l'utilisation de cette approche pour le repositionnement de médicaments est confrontée à plusieurs défis.

Premièrement, (A) il n'est pas clair que les signatures obtenues respectivement aux étapes (1) et (2) dans le paragraphe précédent soient le reflet des conséquences directes du traitement (ou de la pathologie) au niveau transcriptionnel. En particulier, l'existence bien connue de cascades transcriptionnelles de régulation,11 avec un effet « domino » de la perturbation sur l'expression des gènes, semble plutôt suggérer que ces signatures sont le fruit de profils transcriptionnels « stabilisés » . Cela pose donc le problème de distinguer la cause des symptômes de la pathologie étudiée, de même que les processus biologiques réellement ciblés par un traitement ; Deuxièmement, (B) d'un point de vue pratique, cette méthode requiert d'assez grandes quantités de données transcriptomiques. Il est alors capital d'exploiter les données de traitement et de profils patients de façon générique pour augmenter la reproductibilité (c.à.d., compatible avec n'importe quelle pathologie considérée) ; Troisièmement, (C) la procédure de recommandation de candidats, qui consiste à considérer les candidats les mieux classés (par exemple, les N premiers), ne permet pas ni de contrôler le taux d'erreur, ni de fournir une interprétation des résultats au niveau transcriptionnel ; Enfin, (D) la démarche d'inversion de signature ne prend en compte qu'une tendance globale au niveau de l'évolution de l'expression à travers la population de patients : définir un cadre personnalisé pour la recommandation de candidats thérapeutiques pourrait être intéressant pour l'étude de maladies avec de multiples sous-catégories, comme par exemple l'épilepsie, ou le cancer du sein.

Introduction des réseaux de régulation pour le repositionnement de médicaments. Tout d'abord, dans le but de résoudre les problèmes évoquées en (A) et en (B), j'ai considéré 12 l'intégration de réseaux de régulation génique dans l'approche d'inversion de signature.

En effet, un réseau de régulation génique (gene regulatory network) modélise l'effet d'interactions régulatrices de l'expression sur un ensemble de gènes. Ce réseau est un graphe dont les noeuds sont les gènes, et ses arêtes sont les interactions régulatrices entre les gènes.

Cependant, ce modèle reste statique, et dans le but de démêler les causes des conséquences (A), nous avons besoin d'un modèle prédictif de l'expression de gène, en particulier, en réponse à des perturbations externes. Plusieurs types de réseaux satisfont ce critère, et dans cette thèse, je considère le formalisme des réseaux booléens. 13 D'abord, parce que ce type de modèle permet d'obtenir des prédictions d'ordre qualitatif, ce qui en facilite l'interprétation ; ensuite, parce que le calcul des « états stables » dans le réseau a été bien étudié.

Un réseau booléen est défini par, premièrement, un graphe d'interactions B(G, E) qui connecte les noeuds dans l'ensemble G agissant les uns sur les autres à travers l'ensemble d'arêtes E ; et, deuxièmement, un système logique S(V, F, [[•]]) qui décrit la dynamique du modèle. Ce système comporte un ensemble de variables booléennes (c'est-à-dire, pouvant prendre la valeur 0 ou 1 uniquement) V , et de formules logiques F , en nombre égal à la taille de G. On confond en général l'ensemble des noeuds G et des variables associées V . Pour tout noeud g dans G (g ∈ G), la formule logique ϕ g ∈ F comporte des connecteurs logiques (« et » ∧, « ou » ∨, « non » ¬ . . . ) qui combinent les variables associées à des régulateurs directs du noeud g dans le graphe B. Si un régulateur r ∈ G est un activateur (resp., un inhibiteur) de l'expression du noeud g, alors il existe une arête de r vers g activatrice (resp., inhibitrice) dans l'ensemble des arêtes du graphe d'interactions E.

À un moment donné, l'état du réseau est décrit par l'ensemble des valeurs booléennes attribuéees à toutes les variables associées aux noeuds. Pour obtenir l'état du réseau après application de l'effet des interactions régulatrices sur les valeurs des variables, on utilise la fonction [[•]] (« sémantique » ). Elle définit la façon d'évaluer la proposition logique ϕ g =⇒ g pour chaque gène g, en fonction des valeurs actuelles des variables du réseau. Si pour une variable g donnée, cette proposition est vraie ([[ϕ g =⇒ g]] = 1) alors l'état du noeud associé g est égal à 1 (interprété biologiquement comme « le gène g est exprimé » ). Sinon, cet état est égal à 0 (« non exprimé » ). On ne détaillera pas dans ce résumé les sémantiques possibles. La sémantique définit alors une relation entre deux états s 1 , s 2 du réseau : s 1 → [[•]] s 2 si et seulement si évaluer ces propositions logiques avec [[•]] aboutit à l'état s 2 . On peut alors construire un diagramme de transition entre états, c'est-à-dire, un graphe dont les noeuds sont les états possibles du réseau et les arêtes les relations → [[•]] . En particulier, on s'intéresse aux états qui « bouclent sur eux-mêmes » dans ce diagramme : ce sont les états attracteurs stables. Autrement dit, une fois que le réseau se trouve dans cet état, l'application de la sémantique laisse cet état invariant. Ces états sont intéressants parce que, biologiquement, on les considère comme les phénotypes « stabilisés » des types de cellules modélisés. 14 On peut alors utiliser ce réseau pour prédire des profils transcriptionnels « stabilisés » , en particulier, après perturbation de certains gènes. Une perturbation du gène g est définie par le fait de rajouter dans la sémantique la règle [[g]] = 0 (knock-out) ou [[g]] = 1 (surexpression, ou knock-in) qui supplante l'évaluation de [[ϕ g ]]. En partant d'un état initial du réseau et en appliquant itérativement la sémantique choisie, on peut prédire l'état attracteur stable dans lequel on aboutit au bout d'un certain nombre d'itérations de la sémantique, s'il existe. Le nombre d'itérations est en général choisi suffisamment grand pour pouvoir détecter cet état attracteur s'il existe.

La construction d'un réseau booléen, en particulier sans la présence d'un modèle de régulation déjà existant ni de données pertinentes d'expression de gène, est souvent un processus manuel de collecte d'interactions entre paires de gènes (dans la littérature ou dans des bases de données), et de amélioration itérative du modèle en comparant les prédictions d'états obtenues avec des données transcriptomiques issues d'expériences de perturbation de gène. Cette procédure atteint rapidement ses limites à partir du moment où l'on considère quelques centaines de gènes dans le réseau, ou dans le cas où l'on étudie les régulations dans le contexte d'une maladie rare.

Construction automatisée de réseau booléen associé à une maladie donnée. Une première étape a porté sur le développement d'une méthode de construction automatique de réseau booléen associé à une maladie donnée, dans le but de résoudre le problème évoqué dans (B).

Cette méthode part d'un ensemble de gènes associés à une maladie, de données d'expression dans des expériences de perturbation de ces gènes (un gène à la fois est perturbé) issues de la base de données LINCS L1000, 15 et des interactions répertoriées entre protéines codées par ces gènes, via Actuellement, une des méthodes les plus courantes pour identifier ces régulateurs-clés est de calculer, pour chaque gène, une mesure relative à la centralité de la position de ce gène dans le réseau. Cependant, d'une part, cette méthode n'exploite que la topologie du réseau lorsqu'elle est appliquée à un réseau booléen. D'autre part, cette définition ne colle pas exactement à la définition d'un régulateur-clé. J'ai donc conçu une méthode qui retourne, pour chaque gène, un score (appelé « spread » ) proportionnel à la perturbation de l'état du réseau par la perturbation de ce seul gène. L'idée derrière ce score est de calculer la dissimilarité entre les états attracteurs stables atteints sans perturbation du gène, et ceux dans lesquels le système peut aboutir lors de la perturbation du gène dans la lignée cellulaire d'intérêt.

Le classement par « spread » décroissant est corrélé à la centralité du gène dans le réseau (Control Centrality, 20 degré total et sortant du noeud). Cependant, ce classement est davantage corrélé à des mesures de pathogénicité de gène, par exemple, à l'intolérance d'un gène à une mutation pertede-fonction (pLI). Dans l'application à l'épilepsie, des gènes associés aux encéphalopathies épileptiques et à de graves troubles du développement sont retrouvés de manière significative (« enrichis » ) parmi les 12 premiers gènes du classement par « spread » .

Repositionnement de médicaments par le réseau booléen. Ensuite, j'ai cherché à résoudre les points évoqués dans la partie (A), en définissant une procédure de repositionnement de médicaments, basée sur l'approche d'inversion de signature, qui exploite les propriétés prédictives du réseau booléen.

Cette procédure a été appliquée au réseau construit pour l'épilepsie, et les résultats obtenus ont été comparés à une méthode traditionnelle d'inversion de signature, appelée L1000 CDS 2 . Pour ce faire, j'ai récupéré des profils transcriptionnels d'hippocampes épileptiques et sains 21 ainsi qu'une signature construite sur des échantillons traités et contrôles pour chaque molécule à cribler, à partir de la base de données LINCS L1000. 22 Puis, pour chaque échantillon patient perturbé et chaque molécule, le réseau prédit un ensemble d'états attracteurs stables atteints à partir du profil patient perturbé selon la signature de la molécule. Les gènes notés comme up-(resp., down-) régulés dans la signature sont surexprimés (resp., knocked-out) tout au long de la simulation. Pour chaque état attracteur prédit, on calcule un score associé à la distance entre cet état et la frontière de classification qui sépare échantillons contrôles des échantillons patients dans un espace en 2D bien choisi. Ce score est positif si l'état attracteur est dans l'hyperplan des échantillons contrôles, négatif sinon. Un score final pour ce profil patient et cette molécule est obtenu en pondérant les scores par attracteur par la probabilité d'aboutir dans cet attracteur. Enfin, un seul score par traitement est obtenu en moyennant les scores sur l'ensemble des patients.

Cette approche a été testée sur un ensemble de 22 proconvulsivants et 12 antiépileptiques connus. On observe que cette méthode a des performances similaires à L1000 CDS 223 , avec une faible amélioration au niveau de la mesure d'aire sous la courbe (AUC) qui quantifie la relation entre le taux Introduction des algorithmes de bandit pour le repositionnement de médicaments. Les algorithmes de bandit appartiennent de la catégorie d'algorithmes d'apprentissage statistique par renforcement. L'algorithme (l'« agent » ) doit interagir avec un environnement dont il peut tirer des observations, à partir desquelles il prend des décisions ; il adapte alors sa prise de décision en fonction des retours reçus de l'environnement. Dans cette thèse, l'environnement est le réseau booléen qui retourne le score (« observation » ) du traitement sur un ensemble de gènes, et la décision à prendre est le choix du prochain traitement (« bras » ) à à évaluer. Le but est d'ultimement identifier les meilleurs candidats thérapeutiques à partir de leur effet moyen sur le réseau. Les bandits sont surtout populaires dans le domaine des systèmes de recommendation,24 même s'ils ont été introduits pour décrire les essais cliniques adaptatifs,25 c'est-à-dire avec un processus d'allocation des patients aux bras de traitement qui n'est pas uniforme.

Maintenant, introduisons le concept de bandit (ici, on ne considèrera que les « bandits stochastiques structurés à nombre de bras finis » ). On considère un nombre K de bras, associés à des vecteurs caractéristiques (X k ) k∈ [K] . Un agent sélectionne séquentiellement ces bras pendant un nombre de tours donnés ou lorsqu'un critère d'arrêt est satisfait. En tirant le bras I t ∈ [K] := {1, 2, . . . , K -1, K} au tour t, l'agent exécute une action sur l'environnement, et perçoit son impact sur l'environnement via une observation bruitée Y t . On fait l'hypothèse que cette observation est une réalisation d'une distribution de probabilité ν It (Y t ∼ ν It ), c'est-à-dire, un nombre tiré selon cette distribution. Celle-ci est considérée fixée et inconnue de l'agent ν It (hypothèse de stochasticité), et dépendante du vecteur caractéristique associé X It (hypothèse de structure). On note l'espérance de la distribution ν a de probabilité µ a := E {Y ∼νa} [Y ] pour tout bras a ∈ [K]. En particulier, ici, on suppose que le modèle de bruit pour tout bras a ∈ [K] est additif et suit une distribution sous-gaussienne de moyenne µ a de variance fixée σ 2

Y t = µ It + ψ t , où ψ t satisfait pour tout réel λ ∈ R + , E[exp(λψ t )] ≤ exp λ 2 σ 2 2 .
Pour la suite du résumé, on se restreint aux distributions gaussiennes de variance fixée σ 2 . Sans perte de généralité, on considère qu'une action positive a sur l'environnement émet une observation moyenne µ a plus grande que celle émise par une action négative sur l'environnement. Cependant, l'agent ne peut estimer l'observation moyenne associée à un bras qu'en tirant plusieurs fois certains bras. Un bras optimal a ⋆ est alors un bras dont l'observation moyenne est la plus grande de tous les bras :

µ ⋆ = µ a ⋆ := max a∈[K]
µ a (il peut y en avoir plusieurs) ; similairement, on définit le N ème meilleur bras

µ (N ) = µ a (N ) := N max a∈[K] µ a .
L'agent peut avoir différents objectifs. Par exemple, l'objectif le plus couramment défini est d'obtenir la plus grande somme cumulative d'observations en T tours, ce qui s'appelle la minimisation du regret cumulatif (cumulative regret minimization). 26 Dans ce cas, l'agent doit souvent jouer le bras que l'agent estime optimal à partir des observations précédentes, au risque d'être pénalisé si on tire un bras sous-optimal. Dans notre cadre de reposi-tionnement de médicaments (et en particulier, sachant que l'on utilise des simulations in silico), on n'a pas d'intérêt à pénaliser l'exploration, c'est-àdire l'observation de bras sous-optimaux pour acquérir de l'information sur l'environnement. L'objectif de l'agent en utilisant l'algorithme de bandit A est donc l'identification des N meilleurs bras (Top-N identification) avec un plus petit nombre d'observations τ A possible. Ce nombre est défini par le critère d'arrêt de l'algorithme. Pour répondre au problème en (C), l'algorithme A doit retourner un ensemble de bras ŜN (τ A ) de taille N , avec une probabilité d'erreur de recommandation inférieure à un seuil fixé δ

P ŜN (τ A ) ⊆ S ⋆ N ≤ δ, où S ⋆ N := a ∈ [K] : µ a ≥ µ (N ) , (1)
est l'ensemble des bras optimaux. On peut aussi définir l'ensemble des bras ε-optimaux (c.à.d., optimaux à ε près) S ⋆ N (ε) := a ∈ [K] : µ a ≥ µ (N ) -ε . Combinaison des réseaux géniques et des algorithmes de bandit pour le repositionnement de médicaments. La méthode de repositionnement de médicaments que je propose combine les scores obtenus par le réseau booléen et les algorithms de bandit. En réponse à la requête de l'agent, un score lié au traitement t est calculé par le réseau booléen, sur un profil patient tiré aléatoirement. Ce score est retourné à l'agent. Celui-ci doit alors choisir le prochain traitement à tester, de sorte à limiter le nombre de scores nécessaires avant de trouver les N meilleurs candidats. Contrairement aux travaux publiés avant le début de ma thèse,27 l'algorithme de bandit peut exploiter les signatures associées aux traitements. Premièrement, cette information supplémentaire permet de réduire le nombre d'évaluations nécessaires, en supposant que des traitements ayant des signatures similaires auront des scores de repositionnement similaires. Deuxièmement, cela permet d'apprendre un modèle mathématique des scores en fonction des signatures, et de pouvoir alors interpréter l'importance d'un gène sur la valeur du score.

Résolution du problème d'identification des N meilleurs bras avec vecteurs caractéristiques. D'abord, je me suis concentrée28 sur l'identification des N bras ε-optimaux sous taux d'erreur fixé δ se reposant sur les vecteurs caractéristiques.

N'importe quel tel algorithme A est défini par trois règles :

(a) la règle d'échantillonnage (sampling rule), qui choisit le bras à tirer en fonction des observations précédentes ; (b) le critère d'arrêt (stopping rule), qui choisit d'arrêter la phase d'apprentissage (et donc, choisit le temps τ A ) pour entrer dans la phase de décision ; (c) la règle de décision (decision rule), qui construit l'ensemble de candidats ŜN (τ A ).

L'analyse de cet algorithme se décompose en deux parties. D'une part, on prouve sa correction, c'est-à-dire si la Condition (1) est bien satisfaite. En pratique, on construit un « bon » évènement E de probabilité supérieure à 1-δ sur lequel l'évènement { ŜN (τ A ) ⊆ S ⋆ N } est toujours vrai (c.à.d., l'algorithme ne se trompe pas). D'autre part, on calcule une borne supérieure sur le nombre τ A d'observations suffisant pour déclencher le critère d'arrêt (sample complexity).

J'ai donc défini une famille d'algorithmes de bandit appelée GIFA (Gap-Index Focused Algorithms), qui suppose une relation linéaire entre les scores et les vecteurs caractéristiques des bras, avec un paramètre inconnu à estimer θ

∃θ ∈ R d ∀a ∈ [K], µ a := θ ⊤ X a .
(2)

En particulier, si le vecteur caractéristique X a est la signature du traitement a, alors le paramètre θ estimé à la fin de la phase d'apprentissage donne les coefficients de la combinaison linéaire θ ⊤ X a . J'ai proposé une analyse unifiée de sample complexity pour ces algorithmes, dont la correction peut être montrée même sur des algorithmes partiellement définis (c.à.d., l'une des trois règles ci-dessus n'est pas définie). Cette famille d'algorithmes se fonde sur les valeurs (« indices » ) (B a,b (t)) (a,b)∈[K] 2 ,t≥0 , tels que B a,b (t) surapproxime l'écart (gap) µ a -µ b entre les bras a et b à n'importe quel tour t, avec probabilité 1 -δ. J'ai montré que cette condition constitue un « bon » évènement pour tout algorithme de la famille GIFA, ce qui en garantit sa correction. Au tour t, un algorithme de la famille GIFA définit à l'aide de ces indices un ensemble de candidats J(t) ; un représentant de l'ensemble des candidats J(t) noté b(t) ; et un compétiteur c(t) qui n'est pas dans J(t). Ce dernier est défini comme le bras qui n'est pas dans J(t) qui maximise l'indice sur l'écart avec b(t) : c(t) ∈ arg max

a̸ ∈J(t)
B a,b(t) (t). Un algorithme GIFA A retourne à la fin de sa phase d'apprentissage l'ensemble J(τ A ).

Deux types d'algorithmes GIFA, nommés LUCB-GIFA et Gap-GIFA selon les définitions de J(t) and b(t), sont particulièrement intéressants car ils généralisent des travaux antérieurs 29 . Le premier traque la valeur de l'écart minimal entre un bras dans J(t) et un autre qui n'y est pas, via l'indice B c(t),b(t) (t), et s'arrête au tour τ LUCB := inf t ≥ 0 : B c(t),b(t) (t) < ε .

Le second suit la valeur de l'écart entre le bras b(t) et le N ème meilleur bras a (N ) (µ a (N ) := N max k∈ [K] µ k , qui est la N ème plus grande observation moyenne), en considérant Ces deux types d'algorithmes sont parfaitement définis une fois leur règle d'échantillonnage implémentée. J'ai montré que le critère d'arrêt τ UGapE était plus agressif que τ LUCB (c'est-à-dire, qu'avec les mêmes observations au cours de la phase d'apprentissage, les algorithmes Gap-GIFA tirent moins de bras que les algorithmes LUCB-GIFA).

Enfin, j'ai testé les performances de ces algorithmes avec deux algorithmes qui ne tiennent pas compte des vecteurs caractéristiques des bras. J'ai considéré un petit problème de repositionnement pour N = 3 candidats et un taux d'erreur maximal de 10 % sur un ensemble de 21 médicaments pour l'épilepsie (avec 10 anti-épileptiques, et 11 pro-convulsivants), en transformant les signatures de médicaments pour obtenir un modèle linéaire. Les algorithmes n'utilisant pas les signatures échantillonnent toujours plus de 10 000 bras, alors que le nombre d'échantillons pour les deux algorithmes GIFA proposés est inférieur à 400, ce qui montre que tenir compte de ces vecteurs caractéristiques permet effectivement d'efficacement réduire le nombre d'échantillons pendant l'apprentissage.

Intégration des vecteurs caractéristiques dans le modèle de bandit. Cependant, on peut remettre en cause l'hypothèse de linéarité (Condition (2)) dans le modèle de scores, en particulier dans le contexte de données réelles. En effet, si le modèle sous-jacent n'est pas adapté aux données, alors la régression effectuée sur le paramètre θ à partir des observations peut donner une approximation θ trop mauvaise (i.e., la distance entre θ⊤ X a et µ a est importante). Alors, le « bon » évènement, défini précédemment avec l'hypothèse de linéarité, peut être faux avec une probabilité supérieure à δ. Donc l'algorithme peut retourner des résultats incorrects avec une probabilité plus grande que δ. Toutefois, le modèle linéaire est intéressant car la procédure de régression ainsi que l'interprétation du modèle y sont simples. J'ai donc travaillé 30 par la suite sur le problème des erreurs de spécifi- 30 [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF].

"Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification". Advances in Neural Information Processing xiii cation (de modèle) pour le problème d'identification des N meilleurs bras. Pour ce faire, j'ai étudié un type de modèle, appelé « déviés de la linéarité » (linear misspecified models). Un tel modèle comprend une partie linéaire (avec un paramètre inconnu θ comme dans la Condition (2)) et une partie sans structure (avec un paramètre inconnu η) qui n'utilise pas le vecteur caractéristique associé au bras :

∃θ ∈ R d ∃η ∈ R K ∀a ∈ [K], µ a = θ ⊤ X a + η a .
(

) 3
La valeur absolue des coefficients θ est comme précédemment associée à l'influence de chaque gène sur le score moyen de repositionnement, en introduisant une valeur de biais η t propre au traitement t. On suppose comme seule hypothèse supplémentaire que l'on connaît une borne supérieure Ψ nommée « déviation maximale » sur la valeur absolue de la partie non structurée η a pour tout bras a : η ∞ := max a∈ [K] η a ≤ Ψ. Un tel modèle est dit « Ψ-dévié » . Le but est de construire un algorithme qui suppose que les scores résultent d'un modèle dévié de la linéarité (Condition ( 3)), et qui identifie les N meilleurs bras avec une probabilité supérieure à 1 -δ, avec le moins d'observations possible. Dans le travail fait dans cette partie de la thèse, une borne inférieure sur le nombre d'observations nécessaire pour qu'un algorithme A retourne un ensemble de candidats correct avec probabilité 1 -δ (Condition (1)) a été trouvée, et peut être calculée numériquement

E[τ A ] ≥ (C ⋆ ) -1 log 1 2.4δ où C ⋆ := sup ω∈△ K min i∈S ⋆ N j̸ ∈S ⋆ N inf λ alternatif 1 2σ 2 a∈[K]
ω a (µ a -λ a ) 2 , (4) est appelé « temps caractéristique » . C ⋆ est proportionnel à une combinaison convexe de différences (bras à bras) entre l'observation moyenne µ a dans l'environnement que l'on considère, et celle λ a dans un environnement « alternatif » . Cet environment alternatif satisfait la Condition [START_REF] Cheng | Systematic quality control analysis of LINCS data[END_REF], et est tel que l'ensemble des N meilleurs candidats dans cet environment soit différent de l'ensemble des meilleurs bras S ⋆ N . K := {p ∈ [0, 1] K : k∈ [K] p k = 1} est l'ensemble des allocations sur un ensemble fini à K éléments. L'allocation ω ∈ d donne alors la probabilité avec laquelle tirer chaque bras a ∈ [K] pour se rapprocher de la borne inférieure. Étant donné la Condition (4), si de plus un algorithme A satisfait la Condition (1) et qu'il atteint cette borne asymptotiquement pour de petites valeurs de δ, alors l'algorithme A est dit « asymptotiquement optimal » . L'avantage d'avoir une borne inférieure calculable numériquement, c'est qu'elle peut être utilisée dans le corps de l'algorithme afin que celui-ci puisse être asymptotiquement optimal. J'ai donc travaillé sur la conception et à l'analyse d'un tel algorithme nommé Systems, 34. xiv MisLid. Si N a (t) est le nombre de fois où l'agent sélectionne le bras a jusqu'au tour t inclus, le principe de l'algorithme est de tirer des bras tant que le critère d'arrêt suivant, lié à l'Équation (4) inf

λ alternatif 1 2σ 2 a∈[K]
N a (t)(λ a -μa (t)) 2 < T t-1 (δ) , n'est pas vérifié au tour t ≥ 1. (μ a (t)) a∈ [K] est la projetée, sur l'ensemble des modèles Ψ-déviés, des moyennes empiriques ( µ a (t)) a∈ [K] pour chaque bras a au tour t. T • (δ) est un seuil qui quantifie la distance maximale tolérée entre le modèle empirique et un modèle Ψ-dévié alternatif. La définition de T • (δ) garantit que MisLid satisfait la Condition (1). À la fin de sa phase d'apprentissage, MisLid retourne les N bras optimaux selon

( µ a (τ MisLid )) a∈[K] ŜN (τ MisLid ) := N arg max k∈[K] µ k (τ MisLid ) .
J'ai comparé cet algorithme avec un algorithme GIFA (N -LinGapE) et un autre algorithme (LUCB) n'utilisant pas la structure des scores sur le même problème de repositionnement sur 21 médicaments, en transformant les signatures pour satisfaire la Condition (3) et connaître la valeur de la déviation maximale Ψ. La connaissance de cette déviation est malheureusement indispensable pour que l'algorithme soit plus performant qu'un algorithme n'utilisant pas la structure des scores. La Extension à différentes populations de patients. Dans la dernière partie de la thèse, je m'intéresse 31 au problème en (D) relatif à la définition d'une recommandation personnalisée. En effet, toutes les approches de bandit précédentes supposent l'existence d'une population relativement uniforme dans ses réponses à un médicament (avec des observations bruitées avec une variance σ 2 = 1 dans mes expériences) pour laquelle on veut trouver un ensemble de candidats.

Maintenant, comment procéder si les populations sont disparates, par exemple au niveau génomique, ou au niveau du sous-type de maladie dont elles sont atteintes ? Dans ce cas-là, on aimerait pouvoir recommander des ensembles de candidats spécifiques à chaque population, tout en exploitant les observations obtenues sur d'autres populations, afin de limiter le nombre d'observations. Toutefois, partager chaque nouvelle information entre toutes les populations peut aussi être coûteux, et donc le coût de communication doit également être réduit, en plus du nombre d'observations.

Similairement au concept d'apprentissage de bandit fédéré introduit dans le cadre de la minimisation de regret cumulatif, 32 j'ai travaillé sur une version (plus générale) de l'identification centralisé de N meilleurs bras, qui n'utilise pas 33 les vecteurs caractéristiques. J'ai prouvé une borne inférieure sur l'espérance du nombre d'observations pour retourner un ensemble de candidats correct avec probabilité 1 -δ dans ce cadre centralisé. J'ai également travaillé sur une nouvelle approche pour obtenir un algorithme quasi optimal. Au lieu de résoudre de manière répétée le problème dans l'Équation (4), ou de le faire de façon incrémentale comme pour MisLid, l'idée est de procéder par phases, dans laquelle l'on résout un problème plus simple qui est proche, et l'on utilise l'allocation calculée sur ce problème pour tirer les bras. L'information sur les différentes populations n'est alors partagée qu'à la fin des échantillonnages de la phase courante pour tous les groupes.

J'ai prouvé que l'algorithme correspondant donnait des recommendations correctes avec probabilité 1 -δ, et l'ai ensuite appliqué cet algorithme au problème de repositionnement des 21 médicaments, où les patients sont clusterisés en M groupes en fonction de leur profil transcriptionnel sur d = 194 gènes de M30. J'ai comparé ces résultats avec une méthode de référence inspiré du cadre fédéré qui a motivé ce travail 34 . Les résultats sont présentés dans la Table 2. 32 C. Shi and C. Shen (2021). "Federated multi-armed bandits". Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI); C. [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF]. "Federated multi-armed bandits with personalization". International Conference on Artificial Intelligence and Statistics. PMLR, pp. 2917-2925. 33 Encore... 34 C. Shi and C. Shen (2021). "Federated multi-armed bandits".

= αI M + 1-α M 1 M ×M , avec α ∈ [0, 1], et 1 M ×M est la matrice remplie de 1.

Conclusion.

Ma thèse se situe à l'interface entre la bioinformatique et le traitement de données génomiques, à travers l'utilisation de réseaux booléens, et l'apprentissage statistique, et plus précisément les algorithmes de bandit, qui permettent d'exploiter le score retourné par le réseau booléen. Ce travail de thèse s'est concentré sur quatre objectifs :

(A) la construction d'un modèle simulant l'effet de médicaments sur un réseau de régulation transcriptionnel ; (B) l'automatisation quasi complète de cette procédure ; (C) la conception d'une méthode de recommandation de candidats thérapeutiques exploitant le modèle en (A) ; (D) l'extension de cette méthode à un contexte où l'apprentissage est centralisé.

Ce travail est une nouvelle étape vers l'automatisation du processus de recherche pour le développement de médicaments, qui permet de diminuer le temps d'identification de molécules d'intérêt, mais aussi, en exploitant les données disponibles de façon transparente et reproductible, de faire face au problème des maladies rares.

Publications

Articles de journaux scientifiques / Journal papers [START_REF] Delahaye-Duriez | Identification de cibles thérapeutiques et repositionnement de médicaments par analyses de réseaux géniques[END_REF]. "Identification de cibles thérapeutiques et repositionnement de médicaments par analyses de réseaux géniques". médecine / sciences, 35(6-7), pp. 515-518 [START_REF] Réda | Machine learning applications in drug development[END_REF]. "Machine learning applications in drug development". Computational and structural biotechnology journal, 18, pp. 241-252 Actes de conférences / Conference proceedings Réda, Kaufmann, and Delahaye-Duriez (2021). "Top-m identification for linear bandits". In: International Conference on Artificial Intelligence and Statistics. PMLR, pp. 1108-1116[START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF]. "Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification". Advances in Neural Information Processing Systems, 34 Réda and Delahaye-Duriez (2022). "Prioritization of Candidate Genes Through Boolean Networks". In: International Conference on Computational Methods in Systems Biology. Springer, Articles en revue de lecture / Papers under review : [START_REF] Réda | Near-Optimal Federated Learning in Bandits[END_REF]. "Near-Optimal Federated Learning in Bandits". In press

Articles en préparation / Papers in preparation :

Cet article est en cours de préparation pour soumission au Journal of Neuroinflammation cette année, et fait l'objet du Chapitre 7.

This paper is in preparation for submission to Journal of Neuroinflammation this year, and is the topic of Chapter 7. Bokobza, Réda, et al. (in prep.). "Therapeutic evaluation of Hu-MSCs in a rat model of perinatal inflammation: a systematic outcome scoring" xviii

List of Figures

II

Introduction

The contents of this chapter rely on some of my publications. a a [START_REF] Réda | Machine learning applications in drug development[END_REF]. "Machine learning applications in drug development". Computational and structural biotechnology journal, 18, pp. 241-252. Bottom plot : Illustration of the approach of "signature reversion" for drug repurposing.

DRUG RESEARCH

Development of new drugs is a time-consuming and costly process [START_REF] Réda | Machine learning applications in drug development[END_REF]. Indeed, in order to ensure both the patients' safety and drug effectiveness, prospective drugs must undergo a competitive and long procedure. This process, from the identification of a molecule of interest to its commercialization, is completed on average in 5 years, and can take up to 10 years, depending on the considered disease, and have millions of dollars spent -still next to $2 billion on average for major pharmaceutical labs in 2021 (Deloitte Centre for Health Solutions, 2022). Figure 1 illustrates the typical pipeline for drug development. Drug development is roughly split into four major stages, called phases. The preclinical phase comprises of basic research, drug discovery and preclinical tests, which aim at assessing the efficiency and body processing of the drug candidate. The last three stages are clinical trials : study of dose-toxicity, short-lived side effects, and kinetic relationships (Phase I) ; determination of drug performance (Phase II) ; and comparison of the molecule to the standard-of-care (Phase III). An optional Phase IV (post-drug marketing) can be set to monitor long-lasting side effects and drug combination with other therapies. In addition to being time-consuming and expensive, drug development is subject to high failure rates, event in the latest stages of the pipeline [START_REF] Burki | A new paradigm for drug development[END_REF]. Only 64% of the molecules which reached Phase III after 2007 -the third block on the timeline in Figure 1-were marketed by 2012 [START_REF] Lowe | The Latest on Drug Failure and Approval Rates[END_REF]. Main reasons for failure were the lack of efficacy (57% of the failing drug candidates), and safety concerns (17%). Among safety concerns were increased risk of death or of serious side effects, which were still the main reasons of failure in Phases II and III in 2012 [START_REF] Schuhmacher | Changing R&D models in research-based pharmaceutical companies[END_REF], and in 2019 [START_REF] Lowe | The Latest on Drug Failure and Approval Rates[END_REF].

Although political efforts have been made to promote orphan disease research -the Orphan Drug Act in 1983 in the United States, three national reasearch plans in France, with the latest one passed in 2018 (Ministère des solidarités et de la santé, 2018)-with some success (Institute for Clinical and Economic Review (ICER), 2022), this situation has led the pharmaceutical industry to focus on the most profitable diseases. Cancer still represents the most studied therapeutic area in 9 of the biggest 10 pharmaceutical companies ; although a shift has been observed towards coronavirus 2019 (Covid-19) treatments in all of these companies. At least 4 drug pipelines related to Covid-19 were launched per company for treatment, disease prevention, or complications. This observation raises the issue of finding therapies for rarer, complex diseases, where the limited number of patients might hinder meaningful studies to be carried on ; or for tropical "neglected" diseases, such as malaria, where the drug development cost might be too prohibitive with respect to the estimated selling profits [START_REF] Walker | Supporting Drug Development for Neglected Tropical Diseases Using Mathematical Modeling[END_REF]. As highlighted by [START_REF] Carbonell | Opportunities at the intersection of synthetic biology, machine learning, and automation[END_REF] [START_REF] Mak | Artificial intelligence in drug development: present status and future prospects[END_REF], one rather inexpensive way to improve these numbers might be to automate some important but repetitive data processing and analysis tasks. For instance, artifical intelligence is massively featured in drug development methods during the Covid-19 outbreak (Chen, See, et al., 2020; Jamshidi et al., 2020; Vaishya et al., 2020). Such collaborations between researchers in artificial intelligence, pharmaceutical labs, and hospitals slowly bridge the gap in bioinformatics between applied mathematics, computer sciences and biology. This would accelerate drug development pipelines as they might be computationally, thus automatically, performed and less prone to human-related technical mistakes.

Moreover, thanks to technological breakthroughs in bioinformatics, a large amount of biological information and experiments is publicly available ; for instance, the Gene Expression Omnibus (GEO) database [START_REF] Barrett | NCBI GEO: archive for functional genomics data setsupdate[END_REF], which collects various types of data. This allows a lesser reliance on the access to wet-lab experiments to study biological phenomena. Again, automated methods intervene in the transparent and reproducible processing of this data. This data availability, combined with the rise of machine learning techniques, has led researchers to consider screening known molecules for a specific therapeutic indication, and finding them a new purpose, instead of designing de novo molecules. This approach to drug development is called drug repurposing -or drug repositioning-and might allow to decrease the failure rates and the development time. Indeed, repurposed drugs have well-documented safety-profiles -that is, side effects are known, thus avoiding the discovery of negative side effects in the later stages of development [START_REF] Mouchlis | Advances in de novo drug design: from conventional to machine learning methods[END_REF]-and the preclinical phase of dose and administration route testing have already been performed. Different approaches have been used to tackle the drug repurposing problem. For example, some rely on automatic processing of Electronic Health Records (EHR), clinical trial data, and text mining methods to identify correlations between drug molecules and gene or protein targets in literature (Andronis et al., 2011; Bisgin et al., 2012; Tari and Patel, 2014). However, this approach might be sensitive, but not really specific, since text interpretation is still a hard problem, and the relationship between disease factors and drugs might not be clear. The current state-of-the-art methods seem to have turned to different paradigms of repurposing, as emphasized by the following review papers (Alaimo, Giugno, and Pulvirenti, 2016; Hodos et al., 2016; Sardana et al., 2011). However, most of these methods rely on a rather strong hypothesis, which is that similarity between elements -for instance, chemical composition of drug molecules-implies correlation at drug target level. Nonetheless, some counter-examples to this hypothesis have led to health disasters : for instance, thalidomide exists as two chiral forms -that is, with the same chemical composition but having mirrored structures. One of these forms can treat morning sickness ; the other form can have teratogen effects [START_REF] Vargesson | Thalidomide-induced teratogenesis: History and mechanisms[END_REF].

An attempt at quantifying more accurately drug effects is signature reversion, also called connectivity mapping, which focuses on transcriptomic measurements, that is, relative to the production of molecules as encoded by gene sequences. Indeed, the transcriptome is the set of the different types of ribonucleic acids (RNA), which contain a part of the genomic information encoded in the deoxyribonucleic acid (DNA). RNA is key to the production of proteins, and, as such, ensures that vital chemical reactions occur, as illustrated in Figure 2. While DNA is a double helix comprising of four different small unit called nucleobases : adenine, guanine, cytosine, thymine, which are usually denoted by the letters A, G, C, T and paired [START_REF] Watson | The structure of DNA[END_REF], RNA are single-strand molecules composed of four nucleobases (adenine, guanine, thymine, and uracil, which "replaces" cytosine). First, immature messenger RNAs are usually produced by "copying" a strand of DNA in the cell nucleus, where DNA resides, during a process called transcription. If a gene sequence is represented in the RNA, the sequence of RNA corresponding to that gene sequence is called gene transcript. After maturation, messenger RNA (mRNA) is transported to the cell cytoplasm, outside the nucleus, to be read by a small unit called ribosome during the process of translation. During translation, the RNA sequence is read by triplets of nucleobases (called codons). Once the ribosome encounters a start codon (usually in human, AUG), each following codon is successively matched to a single aminoacid, which is concatenated to previous aminoacids, until the ribosome reaches one of the stop codons. This procedure generates a single protein. Then RNAs are interesting (proxy) snapshots of the state of protein production at a given time. In particular, in order to quantify this, one needs to detect gene transcripts, and count them, assuming that the number of transcripts associated with a given gene will be proportional to the quantity of the product encoded by the gene that would have been produced post-translation. This count is called gene expression, and can be measured through RNA sequencing for instance. The basic idea of RNA sequencing is to align a genomic sequence, obtained from the mRNAs, to a reference genome, and then count the number of successful matches to a gene sequence of the reference genome. Signature reversion leverages gene expression data in order to screen molecules for a specific indication. Given a "signature", that is, a subset of genes which are perturbed in patients at expression level -for instance, with increased or decreased expression levels in patients with respect to healthy individuals-the objective is to identify which treatments are most able to revert this signature ; that is, increasing the expression levels of genes with decreased expression in patients, and vice-versa. This operation is performed via comparisons of the query signature with "drug signatures", that is, summaries of genewise expression changes, this time due to the drug treatment. This approach has recorded some successes in drug repurposing, as noticed in [START_REF] Musa | A review of connectivity map and computational approaches in pharmacogenomics[END_REF]. For example, [START_REF] Xin | Identification of Nifurtimox and Chrysin as Anti-Influenza Virus Agents by Clinical Transcriptome Signature Reversion[END_REF] recently applied this approach for drug repurposing against influenza. They identified among potential drug candidates nifurtimox, which is usually prescribed to treat Chagas' disease, and could successfully decrease the number of influenza virions in later experiments (World Health Organization (WHO), 2021). The bottom diagram in Figure 1 illustrates each step of signature reversion to find novel drugs against a fixed pathology of interest :

(1). First, gene expression levels are measured in each group of treated samples and their controls. 40 (for every molecule to screen) A drug signature is computed from these transcriptomic profiles, which reports genewise expression changes due to the effect of the treatment. A gene which expression is increased (resp., decreased) in treated samples with respect to controls is said up-regulated (resp., down-regulated). The advantage of drug signatures over directly considering transcriptomic profiles is that the changes in gene expression described in the signature only depends on the treatment, and not on other possibly confounding factors.

(2). A disease-specific signature -that is, the aforementioned query signature-is built in a similar fashion by comparing transcriptomic samples from patients to healthy individuals (or another appropriate control group).

(3). A series of comparisons are performed between the disease signature and each of the drug signatures. If a drug signature is deemed similar to the disease signature, then this means that the treatment incurs the same type of perturbations at gene expression level than the pathology itself, since the same genes are affected by both the treatment and the disease in a similar way. To the contrary, if a drug signature and the disease signature are opposite, then the drug seems to restore the gene expression levels conversely to the effects of the disease. Then the considered drug might be a good candidate for repurposing. Finally, drugs can be ranked according to their similarity to the disease signature.

In this thesis, I exhibit an application of signature reversion applied to the treatment of encephalopathy of prematurity, a devastating condition due to premature birth, in Chapter 7 (Bokobza, Réda, et al., in prep.). This project aimed at selecting the optimal treatment protocol for the injection of stem cells in a rat model of encephalopathy of prematurity : age of injection, administration route, and weight-dependent dose level. Disease and drug signatures were computed as previously described, from the transcriptomic profiles of rats exposed to the treatment, and from appropriate controls. The final ranking of the treatment protocols emphasized the global positive effect of the injection of stem cells in rat models, and further bioinformatical analyses showed that the first two candidates were indeed promising.

However, in that project, I was lucky to get access to appropriate control groups for each treatment, which allowed me to build drug and disease signatures of good quality. That was possible because that work was a collaboration which involved custom wet-lab experiments. However, most existing molecules have not been tested on every pathology and every cell line. This raises the question of designing proper drug signatures when several factors (outside the treatment) might mismatch between samples from patient, healthy individuals, treated and control individuals. Then, I have identified four main questions to further enable the use of automated signature reversion, which constitute the core work in my PhD :

(A). There are online databases which compile drug signatures, based on in vitro treatment of human cells in standardized pipelines for transcriptomics (Lamb et al., 2006; Subramanian et al., 2017). These signatures could be alternatives to drug signatures specifically run on diseased profiles. However, the existence of regulatory cascades [START_REF] Bolouri | Transcriptional regulatory cascades in development: initial rates, not steady state, determine network kinetics[END_REF] 41 rather suggests that appropriate drug signatures are the result of "stabilized" transcriptional profiles, after potential molecular interactions between the perturbed levels of gene expression in the initial patient sample, and the perturbations due to the treatment itself. The method developed in Chapter 3 aims at modelling these regulatory cascades, and stable in silico treated profiles, by combining the information about drug-induced gene expression and a cohort of transcriptional profiles from patient and healthy individuals. In order to achieve this goal, I have built a model of gene expression regulation, which is called a gene regulation network, that allowed the prediction of gene expression levels under gene perturbations. Drugs can be scored according to the proximity between the corresponding in silico treated profiles and profiles from healthy individuals.

(B).

In practice, the method mentioned in paragraph (A) requires a large amount of transcriptional data. A reproducible and transparent pipeline for collecting and processing these data, in a generic way that is diseaseagnostic, is crucial to obtain drug signatures of good quality, and to ensure the replicability of drug repurposing. Chapter 2 [START_REF] Réda | Prioritization of Candidate Genes Through Boolean Networks[END_REF] deals with building the aforementioned gene regulatory network used for predicting drug signatures in a fully automated, reproducible and transparent fashion, from publicly available data.

(C).

In the traditional pipeline for signature reversion, as shown at the bottom of Figure 1, drug candidates are ranked by increasing score, which might be the distance to a disease signature. Then, the top-N candidates are selected. However, this recommendation does not control for the error in recommendation. Moreover, a single scalar (score) might not be informative enough about the transcriptional interactions which accounts for its high (or low) value, which hinders the interpretability of the results. Chapters 5 (Réda, Kaufmann, and Delahaye-Duriez, 2021) and 6 [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF] describe the introduction of sequential learning algorithms to solve these two issues. More specifically, these algorithms adaptively select the drugs to be scored and aim at reducing the number of selections needed to make a recommendation, by leveraging information about drug-induced perturbations of gene expression. Chapter 5 assumes that a linear model connects the drug-associated perturbations to the scores, which increases interpretability, at the cost of a sometimes bad approximation ; whereas Chapter 6 relies on a type of structure which generalizes linear models. These algorithms sequentially interact with the gene regulatory network, by sequentially observing the noisy score obtained for a specific treatment until they can make a guess about the top candidates.

(D).

Finally, the pipeline shown at the bottom of Figure 1 only takes into account a global trend at gene expression level in the whole population of patients. However, for some diseases (for instance, epilepsy or brain cancer), there might be several subpopulations of patients with distinct disease subtypes. One might be interested in making personalized drug recommendations for each subpopulation, while leveraging the information about the response to treatment for all subpopulations. Chapter 8 [START_REF] Réda | Near-Optimal Federated Learning in Bandits[END_REF] describes the extension of the framework previously studied in (C) to a collaborative setting, where the information about subpopulations should be shared, but only at times. Indeed, running experiments per batches is more practical and cost-effective, instead of having to wait until all tests have been run on all subpopulations. This issue of delayed feedback, and the need for a finite number of interim analyses, is indeed one of the main issues in adaptive clinical trials (Pallmann et al., 2018; Villar, Bowden, and[START_REF] Villar | Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges[END_REF], which are the in vivo counterpart to drug repurposing.

The ultimate objective of this work is to combine sophisticated techniques for modelling gene expression and recommending candidates to enable automated and transparent in silico drug repurposing.

Part I Analyzing a disease-specific regulatory network

The contents of this chapter rely on some of my publications. a a [START_REF] Réda | Prioritization of Candidate Genes Through Boolean Networks[END_REF]. "Prioritization of Candidate Genes Through Boolean Networks". International Conference on Computational Methods in Systems Biology. Springer, pp. 89-121; Réda, Kaufmann, and Delahaye-Duriez (2020). "Machine learning applications in drug development". Computational and structural biotechnology journal, 18, pp. 241-252.

Boolean networks provide a qualitative summary of regulatory interactions at molecular level. Provided the current huge amount and diversity of biological data, being able to integrate all this data to automatize and control each step of the model inference procedure becomes crucial. On the one hand, it might help targeting true causal regulatory interactions, and replay probable regulatory cascades and mechanisms which had lead to a given experimental, observed, phenotype. On the other hand, this systematization of the inference allows higher replication, lower time and wet-lab costs.

We propose a fully automated Boolean network inference pipeline. In order to illustrate our method, we focus on a gene module named M30, which global gene expression has been shown to be anti-correlated to epileptic phenotypes [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]. Thus, targeting one of the M30 genes might potentially be of interest for therapeutic purposes. Advanced graph analysis of a graph of regulatory interactions in this module might help identifying master regulators related to epilepsy factors ; give indications about the potential of molecules -for instance, whether they have an antiepiletic or proconvulsant effect ; and predict their effect on patients.

First, we describe a reproducible pipeline to identify the network associated with M30. It relies on perturbation experiments of interest from several public databases, and on the integration of supplementary biological information to further constraint the network inference procedure. Second, we will suggest a method using influence maximization to find a set of maximally perturbating set of genes. In particular, we exploit this to identify master regulator genes, at the top of the regulatory hierarchy. These genes might be potential therapeutic candidates, and might shed a light on the mechanisms of the considered disease. Last, we propose a method which directly uses this inferred model, in order to predict and to compare in silico the effect of potential drug candidates on patient phenotypes.

Chapter 1

Introduction to Boolean networks

The power of systems biology and network-based approaches comes from the analysis of multiple genes in functionally enriched pathway, as opposed to traditional single gene and single target approaches. In an effort to encode the effects of a molecule on the transcriptional activity of genes, we consider gene regulatory networks. These graphs connect genes according to their regulatory interactions. An arrow is drawn from one gene to another if the former regulates the expression level of the latter, for instance by preventing or encouraging its transcription. Such a model describes a dynamical system, which can be used for predicting transcriptional profiles. Integration of system biology-related methods to drug development has been implemented for epilepsy in Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF], and has allowed the identification of a gene module which global expression is highly anti-correlated to epileptic phenotypes [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]. Then, a whole set of genes can be targeted for treatment, instead of screening drugs against a single relevant target. Prior literature emphasized on the importance of small-effect gene in a system biology model, as their belonging to highly interconnected gene regulatory networks implies that any slight perturbation on these genes might impact significantly "core" disease genes [START_REF] Dugger | Drug development in the era of precision medicine[END_REF]. A large and active literature [START_REF] Ahmed | Assessing the effectiveness of causality inference methods for gene regulatory networks[END_REF] has emerged about the formalization, the building and the validation of such gene regulatory networks, along with the identification of gene modules highly correlated with pathological phenotype. As gene regulatory networks are assumed to mirror gene activity with regard to other genes' expression, building them usually require (time-series) expression data. These data can be extracted from databases recording measurements of expression after genewise perturbations -for instance, in [START_REF] Young | A posterior probability approach for gene regulatory network inference in genetic perturbation data[END_REF] (Aiba et al., 1989; Hock et al., 2011). on knock down gene expression measurements collected in the LINCS L1000 database [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF].

In this thesis, we focus on a specific type of gene regulatory networks, called Boolean networks. This class of networks allows to model in a qualitative way the effects of the interconnected gene regulations. Moreover, what makes them particularly attractive is the ability to easily simulate transcriptional profiles that result from the perturbation of one or several genes in the network. This property is key to our solution to tackle drug repurposing while taking into account the effect of regulatory cascades [START_REF] Bolouri | Transcriptional regulatory cascades in development: initial rates, not steady state, determine network kinetics[END_REF].

Modelling gene expression with regulatory networks

As illustrated in Figure 2 in introduction, the main connection between our genomic information -encoded into our DNA and shared by almost all cells in our body-and the state of our organism at some point in time is described by the "central dogma" [START_REF] Crick | On protein synthesis[END_REF] : DNA material (that is a double helix comprising of nucleotides) is transcribed into immature messenger RNA (mRNA, which is a single strand of nucleotides) within the cell nucleus. Immature mRNA matures, and then exits the nucleus to enter the cytoplasm. At this point, if sequencing occurs, then the mRNAs are retrieved and aligned against a reference genome in order to identify their matching gene transcripts. The number of successfully aligned mRNAs corresponds to the expression level of the considered gene. Later in the cytoplasm, mature mRNA is translated into aminoacid chains (i.e., proteins) via ribosomes, which "read" mRNA by triplets of nucleotides. Each triplet of nucleotides uniquely encodes for an aminoacid (or for a signal that the current aminoacid chain should start or end). However, several combinations of nucleotides can lead to the same aminoacid. Although this dogma is now often challenged, for instance by the existence of reverse transcription [START_REF] Baltimore | Viral RNA-dependent DNA polymerase: RNA-dependent DNA polymerase in virions of RNA tumour viruses[END_REF], where RNA-like molecules can be turned into genomic material, this dogma interestingly highlights the link between our more-or-less static genetic information, and the adaptation of our organism to its environment.

If there were no regulatory interplay on gene expression, i.e., no regulatory interaction of a given gene on the expression level of another gene, then there would be no to little adaptability to the environment. Indeed, almost every cell possess the same genomic information, yet there are different cell behaviors, implying a change in expression, with hundreds of existing different cell types in humans. Table 1.1 gives examples of such interactions, and their implications in terms of cell behavior and structure. Such regulatory relationships between genes, proteins, or other molecules produced from DNA can be summed up into a graph, that is called regulatory network. This model allows the identification of upstream regulators of specific proteins, that have an impact on the cellular property or properties of interest, and their regulatory patterns, that is, the dynamics of expression levels in this network. In [START_REF] Zhao | Distinct and competitive regulatory patterns of tumor suppressor genes and oncogenes in ovarian cancer[END_REF], this model is used to report a competiting co-regulatory mechanism between tumor-suppressor genes and oncogenes -i.e., genes that promote cancer growth. Tumor-suppressor genes, which are primarily involved in DNA repair and apoptosis, target genes that are functionally linked to the response to hormone stimuli ; whereas the downstream targets of oncogenes, which regulate response to hormone stimuli, act upon apoptosis. This model might cast a light upon patient response to hormonotherapy. In other domains, such as evolutionary development (evo-devo), networks help understand the embryonic development, that is, the process that leads an embryo to a fully-fledged organism. For instance, [START_REF] Dunn | Defining an essential transcription factor program for naive pluripotency[END_REF] [START_REF] Dunn | A common molecular logic determines embryonic stem cell self-renewal and reprogramming[END_REF] successfully model the reprogramming of differentiated (i.e., "specialized") cells into pluripotent cells in mice by the Yamanaka transcription factors [START_REF] Takahashi | Induction of pluripotent stem cells from mouse embryonic and adult fibroblast cultures by defined factors[END_REF]. Transcription factors are genes that bound to the DNA to modulate the transcription of some sequences, and often play a key role in regulatory networks.

Overview of gene regulatory networks

A gene regulatory network is a model of the regulatory interactions which modulate the expression level of a set of genes. Those regulatory interactions may intervene at transcription, or by a subsequent mechanism. As a general rule, this network is represented as a diagram which can be decomposed into three main components, respectively called "Inputs", "Regulatory network" and "Outputs". The second component "Regulatory network" connects the first part of the network "Inputs" to the "Outputs" component. It is a graph which nodes are genes or proteins, or any actively regulating compound, and edges (if the graph is undirected) or arrows (otherwise) between the nodes are regulatory interactions of gene expression. This decomposition is illustrated in Figure 1.1 for one of the most well-known examples of gene regulation, that is, the adaptation of Escherichia Coli to an environment lacking in glucose [START_REF] Pardee | The genetic control and cytoplasmic expression of "inducibility" in the synthesis of β-galactosidase by E. coli[END_REF]. The inputs to a gene regulatory network usually refer to the "actable" points of the network, to which an external perturbation can be applied, whereas outputs represent interesting biological states (called "phenotypes") resulting from the gene regulatory interplay. Phenotypes might be cell behaviors (e.g., apoptosis, that is, cell death) or structures (e.g., differenciation of a pluripotent stem cell into a more specialized cell line).

Several methods have been investigated to implement these networks in practice, especially the "Regulatory network" component, which correspond to different venues of research and can achieve different goals. [START_REF] Karlebach | Modelling and analysis of gene regulatory networks[END_REF] classify these frameworks into three main groups, in increasing order of complexity. Discrete models. First, they consider discrete models of gene expression, for instance Boolean networks (Kauffman, 1969; Thomas, 1973) which will be discussed at length in the next section. The principle of discrete models is to consider discrete values of expression and a finite set of types of regulatory interactions. This simple description -to the price of some loss of information-has already been useful to show the relationship between the structure of the network, and the presence of steady stable attractor configurations, in which the network remain under no external perturbations (Thomas, 1978).

Deterministic continuous models. Second, continuous models of gene expression, where values which amount to expression levels are updated for each species (gene, protein, . . . ) of interest, are considered [START_REF] Goodwin | Temporal organization in cells. A dynamic theory of cellular control processes[END_REF]. The idea is to find, for each species, a mathematical function which connects the expression level of regulator genes to the decay and synthesis rate for the considered species. The resulting set of functions yield a system of coupled differential equations which can be solved, for instance in order to identify basins of attraction. This system represent chemical kinetics between species, and can be formally described and executed using the κ-language [START_REF] Danos | Formal molecular biology[END_REF] for instance. However, the predicting power of this framework greatly rely on accurate estimations of the kinetic constants in the chemical equations, which can be difficult to obtain in the first place.

Stochastic continuous models.

Last, a regulatory network can also be viewed as a global system, which comprises of several type of small molecules. Each interaction between two types of molecules occurs with an associated probability [START_REF] Mcadams | Stochastic mechanisms in gene expression[END_REF], which can also be seen as chemical reaction rates. Then, given the probabilities associated with the interactions, and the initial number of each molecular type, the Gillespie algorithm [START_REF] Gillespie | Exact stochastic simulation of coupled chemical reactions[END_REF] can compute the probability of a phenotype, along with the associated number of each species. This framework shows an accurate description of the regulatory dynamics, but can be hard to analyze [START_REF] Karlebach | Modelling and analysis of gene regulatory networks[END_REF].

Introduction of dynamics with Boolean networks

We now focus on Boolean networks to model gene regulation networks. In the graph associated with a Boolean network, called "interaction graph", only two types of interactions are considered, either activatory or inhibitory. 1The type of an interaction is called its sign. Interactions are also directed, meaning that in the two genes involved in the interaction, one is set as the regulator, which causes the regulation, and the other is the regulated gene. [START_REF] Pardee | The genetic control and cytoplasmic expression of "inducibility" in the synthesis of β-galactosidase by E. coli[END_REF], based on the regulatory network in [START_REF] Santillán | Quantitative approaches to the study of bistability in the lac operon of Escherichia coli[END_REF]. It shows the adaptation of the bacteria to the type of nutrients present in its environment. E. Coli preferably feeds on glucose, but when it is missing from its environment, the inhibition on the transcription starting site LacP through repressor R, that blocks the transcription of genes LacY and LacZ, do not hold anymore, which allows the bacteria to feed on lactose. Gene LacY is involved in the transport of external lactose to the bacteria, and LacZ is involved in the transformation of lactose (and allolactose) into galactose. In the presence of glucose, the production of protein cAMP and CAP is inhibited, which further blocks the transcription of LacY and LacZ due to the presence of repressor R. To allow a readable figure, I omitted an activatory link between glucose and the output node, and the reaction between allolactose and LacZ which results in the production of galactose. Going from a set of regulatory interactions between genes, to a Boolean network with both the interaction graph and the set of regulatory functions for each node in the network. Left plot : Regulatory interactions (of different types) are shown between three genes (nodes of rectangular shape). Center plot : The gene regulation network between those three genes. Right plot : One possible Boolean model associated with the gene regulatory network. There might be potentially several sets of regulatory functions which can match the regulatory network, in the absence of additional information about gene expression.

The expression level of a gene has two states : either it is equal to 0, which stands for absent to low expression, or it is equal to 1, which corresponds to a state of high expression. These states belong to the Boolean domain, that is, {0, 1}, hence the name of the framework. A so-called "regulatory function" is assigned to each node in the regulatory network. This function describes the behavior of the expression state of a given gene at a given time point, depending on the expression states of its direct regulators, that is, the predecessor nodes in the network. This function is encoded as a Boolean (logical) function of the variables associated with gene expression states, which can be evaluated to get the current gene expression state. Figure 1.2 illustrates how to go from a set of regulatory interactions to a Boolean network, on a dummy example with three genes. From now on, we will often abuse notation by indifferently using "genes" and "variables" to designate gene expression states or nodes in the network. In order to determine how the configuration of the network -i.e., the set of expression states for every node in the network-changes at a given time point, one has to define the dynamics of the system. That is, the type of update between network configurations, in addition to the interaction graph and the regulatory functions.2 Several update types have been proposed, listed in [START_REF] Chatain | Boolean networks: beyond generalized asynchronicity[END_REF] for example. The two most simple types of updates are the synchronous and the asynchronous updates. The synchronous update means that, at each update step, all the gene regulatory functions are simultaneously evaluated, based the gene expression states at the current time step ; whereas in the asynchronous update step, a single gene regulatory function is selected at time for evaluation. Different update types mean different dynamics. In particular, depending on the chosen update type, the network can reach different configurations. Once the update type is set, a state-transition graph (STG) can be built, connecting network configurations to each other with directed links, such that, if there exists an arrow going from a network configuration A to another one B, then the network can go from the initial configuration A to final configuration B in a single update step. A state-transition graph thus depends both on the set of gene regulatory functions and the definition of the update step. The STG is particularly interesting because steady attractors -that is, configurations that loop upon themselves in the STG-and unsteady attractor states -cycles in the STG-can be directly read from this diagram. The attractor states in a Boolean network -and in gene regulatory networks in general-usually correspond to interesting biological phenotypes (Bloomingdale et al., 2018; Wery et al., 2019). Examples of STGs, with different types of attractor states, are shown in Figure 1.3.

A Boolean network can be straightforwardly built from a known static gene regulatory network, or maps of molecular interactions, by automatically assigning gene regulatory functions based on the activators, resp. the inhibitors, of each gene, for instance using CaSQ [START_REF] Aghamiri | Automated inference of Boolean models from molecular interaction maps using CaSQ[END_REF]. In order to infer the Boolean network "from scratch", that is, from a set of genes, one must define both an interaction graph and an set of gene regulatory functions which match experimental observations under the selected update type. One way to find the appropriate Boolean network requires to know a set of possible genepairwise interactions -sometimes known as the "prior knowledge network" (Ostrowski et al., 2016; Vaginay, Boukhobza, and[START_REF] Vaginay | Automatic synthesis of boolean networks from biological knowledge and data[END_REF])-and a set of time-series (binarized) transcriptional profiles which (hopefully) exhibits enough of the true dynamics of the biological system. Then gene regulatory functions are tuned until all the experimental constraints are satisfied by the model. However, there are two main hurdles in this inference problem. First, the binarization step to go from transcriptional profiles of continuous gene expression to binary profiles -with potentially genes which have no known binary state, if their expression is not high or low enough-incurs an unavoidable loss of information, and should be performed carefully. Binary profiles might also been built using prior knowledge on the expected behavior of the system, for instance, the phenotypes that should be observed under specific perturbations, but this requires supplementary access to custom wet-lab experiments or to the literature, at the risk of propagating scientific bias. Second, the problem of infering a network, which satisfies both the topological (related to interactions) and the dynamical (related to observations) constraints, is usually underdetermined. As the number of genes in the network grows, so does the amount of experimental data needed in order to pinpoint a single model which can satisfy all the constraints. This fact makes unbiased model building difficult when done manually. Some tools allow the automation of the tuning step, such as Re:In [START_REF] Dunn | Defining an essential transcription factor program for naive pluripotency[END_REF] or BoNeSiS [START_REF] Chevalier | Synthesis of boolean networks from biological dynamical constraints using answer-set programming[END_REF]. These methods are based on enumerating putative network solutions, and testing whether the network satisfies all constraints using a solver of Boolean equations (SAT solver).

Boolean networks in drug repurposing

Now, using the same tools as in Boolean network inference, a Boolean network can be predict (binary) transcriptional profiles under the perturbation of one or several genes.

The perturbation of a given gene, either by knockout -i.e., the expression of the gene is totally inhibited-or by overexpression -that is, the gene is forcibly expressed-is defined by setting the corresponding gene variable to either 0 or 1. Then, under these new Boolean constraints, and starting from a fixed initial state, update steps are iteratively executed, until an attractor state is reached, or until a maximum number of update steps is performed (in order to get results in finite time). This maximum number of steps, that is, the length of the (configuration) trajectory, can be chosen large enough to end up in an attractor state, if any exists. However, the network might end up in an unsteady attractor state, that is, a cycle of configurations between which the network indefinitely oscillates in the absence of supplementary perturbations. This can be checked by considering the whole trajectory of configurations from the initial to the last reached configuration, and matching the corresponding cycle -if it exists-to one of those present in the STG. Nonetheless, note that computing the whole STG, especially as the number of considered nodes is large, becomes quickly non computationally tractable. Fortunately, model checking tools such as Z34Bio [START_REF] Yordanov | Z34Bio: An SMT-based framework for analyzing biological computation[END_REF] -which uses SAT solvers-or BioModelAnalyzer (BMA) [START_REF] Benque | BMA: Visual tool for modeling and analyzing biological networks[END_REF], can prove the stabilization of the system or that the final configurations belong to a cycle. This property of Boolean networks will be useful in the drug repurposing method proposed in this thesis. Indeed, it allows to explicitly model the transcriptomic cascades, and to compute the "stabilized" qualitative biological profile the regulatory network ends up in after a single gene or drug-induced perturbation.

However, as previously mentioned, the accurate building of a Boolean network requires a rather large amount of data, which is often manually retrieved from the literature [START_REF] Collombet | Logical modeling of lymphoid and myeloid cell specification and transdifferentiation[END_REF]. When modelling a network on a large number of genes, the process is often incremental and based on iterative improvements of the model [START_REF] Niarakis | A practical guide to mechanistic systems modeling in biology using a logic-based approach[END_REF]. At best, it requires at least collecting a set of putative regulatory interactions, and a set of time-series experiments of good quality, which reports the initial configurations, the perturbations (if needed), and the final observed configurations ; and then feeding these inputs to a Boolean network inference algorithm (Chevalier et al., 2019; Dunn and Yordanov, 2019; Réda and Wilczyński, 2020). This approach might show its limits when facing hundreds of genes in a relatively poorly studied cell line.

This led me to first design an automated, reproducible method for building Boolean networks, as described in the next chapter.

Chapter 2

Prioritization of candidate genes through Boolean networks

This chapter focuses on in silico detection of master regulator genes, which is a popular approach to speed up drug development. Master regulator genes might be directly related to the onset of the disease, or may act on one pathway which counteracts the associated symptoms. Then, one could perhaps screen drugs to select chemical compounds targeting these genes. In prior works, the detection of these candidates was performed through the identification of the regulatory interactions between genes of interest for the disease. Indeed, system biology approaches have proven a useful tool to integrate transcriptomic data and predict transcriptional profiles under gene perturbations. In particular, Boolean networks, where gene expression levels are reduced to two values, are a flexible framework for qualitatively modelling gene expression. However, for rare diseases, building such a regulatory model can become a tedious and time-consuming task.

In this work, we show how to build, in a reproducible fashion, a Boolean network related to a subset of interesting genes for the disease, using publicly available data. Then, we describe a method to identify master regulatory genes, that is, genes which have an impact on the dynamics of the gene regulation in a specific disease-related transcriptional context. As a proof-of-concept, we focus on a subset of genes related to various epileptic phenotypes, in order to find novel master regulator genes associated with epilepsy. These genes might help casting a light on the causes of refractory epilepsy, which are epilepsies which cannot be managed by conventional antiseizure medication. We show that our method for the identification of master regulatory genes is consistent with network controllability measures, while targeting genes which are significantly enriched for epilepsy-related terms. We show that in silico perturbation of these candidates can reproduce epileptic phenotypes. Our pipeline allows for systematic and automated synthesis of a Boolean network and identification of putative drug targets. This work was accepted at the 20 th conference on Computational Methods in Systems Biology (CMSB 2022) [START_REF] Réda | Prioritization of Candidate Genes Through Boolean Networks[END_REF]).1

Related work

We propose a novel generic method for the detection of master regulator genes, which can be applied to any disease, and relies on a dynamic interplay between a gene regulatory network and gene expression data. We focus here, as a proof-of-concept, on an application to epilepsy.

Epilepsy actually encompasses various neurological diseases and syndromes, which can originate from brain injury or genetic background, that have in common a propensity to trigger chronic epileptic crises. Epileptic crises are characterized by a transitory abnormal neuron electric discharge, which might lead to unconsciousness, seizures, and/or body stiffness. Epilepsy is one of the most common neurological diseases worldwide, with around 50 million people living with this disease (World Health Organization (WHO), 2022). Moreover, more than 25% of epileptic patients are afflicted with drug-resistant epilepsy [START_REF] González | Drug-resistant epilepsy: definition and treatment alternatives[END_REF], also called refractory epilepsy. Symptoms in those patients could not be managed by at least two different antiepileptic therapies. This shows the limits of conventional antiepileptic medication, which are often molecules with antiseizure effects, and emphasizes the need to look for novel therapeutic candidates. Epilepsy-related genes are shown to be usually mainly expressed in a specific brain region, called hippocampus [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF]. This region is also affected by morphological changes linked to neuronal discharges in some epileptic patients [START_REF] Ogren | Threedimensional surface maps link local atrophy and fast ripples in human epileptic hippocampus[END_REF]. The exact relationship between lesions in the hippocampus and epilepsy-associated symptoms is still unclear, but might be related to the fact that hippocampus is one of the most excitable parts of the brain [START_REF] Kuruba | Hippocampal neurogenesis and neural stem cells in temporal lobe epilepsy[END_REF]. Several animal models of epilepsy exist, including a mouse model where injection of pilocarpine induce symptoms similar to temporal lobe epilepsy [START_REF] Srivastava | Genome-wide analysis of differential RNA editing in epilepsy[END_REF], or another involving sodium channels, which play a role to convey electric potentials. For instance, a genetically modified animal with invalidation of gene Scn1a, which code for sodium channels, models a severe form of epilepsy called Dravet syndrome [START_REF] Kalume | Sudden unexpected death in a mouse model of Dravet syndrome[END_REF].

In prior works, the identification of master regulators in gene networks has been a powerful method to detect novel genes of interest for a given disease. Master regulator genes are genes which might have a large, global influence on the expression of a group of genes in a specific pathway. For instance, SESTRIN3 [START_REF] Johnson | Systems genetics identifies Sestrin 3 as a regulator of a proconvulsant gene network in human epileptic hippocampus[END_REF] and CSF1R [START_REF] Srivastava | A systems-level framework for drug discovery identifies Csf1R as an anti-epileptic drug target[END_REF] were prioritised as candidate antiepileptic drug targets using different systems-biology approaches dedicated to identifying master regulators of epilepsy-associated networks of gene expression. Such genes might be forcibly expressed or knocked out -i.e., no more expressedby molecules, which might be interesting antiepileptic drug target candidates. Other approaches exploit the location of a given gene inside a gene regulatory network. It is assumed that the more central it is, the most regulatory it should be. Other approaches use the concept of "network controllability" [START_REF] Liu | Controllability of complex networks[END_REF] which is loosely related to the centrality of the gene in the network, as it estimates the number of downstream targets possibly affected by a perturbation of this gene. Yet, most of the cited approaches for the detection of interesting regulatory genes only leverage topological knowledge about the network, without considering the actual dynamics of the regulatory system. In order to find master regulator genes related to rheumatoid arthritis based on expression data, [START_REF] Zerrouk | Identification of putative master regulators in rheumatoid arthritis synovial fibroblasts using gene expression data and network inference[END_REF] have considered an approach which combines a transcription factor (TF) co-regulatory network and gene expression in fibroblast-like synoviocytes in patients afflicted with rheumatoid arthritis.2 TF influence in these samples was assessed using the tool CoRegNet [START_REF] Nicolle | CoRegNet: reconstruction and integrated analysis of co-regulatory networks[END_REF], which computes a score of influence per TF on a set of transcriptional profiles. This score is defined for TF t, that activates a set A t of genes and inhibits a set I t of genes, and a matrix of transcriptional profiles M3 as follows Influence(t) :=

1 |At| a∈At M [a, :] -1 |It| i∈It M [i, :] (s At ) 2 /|A t | + (s It ) 2 /|I t | (2.1)
where s At (resp., s It ) is the standard deviation in gene expression of all genes in A t (resp., I t ) across all profiles in M . However, such a computation does not take into account downstream transcriptional cascades beyond genes directly regulated by the TF. Yet these regulatory cascades might allow to identify off-target genes -i.e., genes subject to non specific and involuntary changes-which might lead to serious side effects [START_REF] Huang | A framework for identification of on-and offtarget transcriptional responses to drug treatment[END_REF]. These regulatory cascades can be modelled through a dynamical gene regulatory network. In that case, setting up a reproducible pipeline for building this regulatory network -especially for the retrieval of raw biological data-is crucial to ensure replicability in drug development research. Finally, this gene regulatory network can be represented by a Boolean network. This type of model comprises of discrete, qualitative, gene regulatory interactions, which promote an increased interpretability of the predicted changes and less expensive computations compared to continuous models of gene expression.

To this aim, we developed a fully automated pipeline to infer a Boolean network which models the regulatory interactions in a subset of genes. Our method is based on perturbation experiments of genes that may participate in our network, and on the integration of supplementary biological information to further constraint our inference procedure. In our work, we exploit the database LINCS L1000, which provides a large number of gene expression profiles for various combinations of cell types, and genetic perturbations [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF].

In our application to epilepsy, we focus on a module of 320 genes, called M30, which global expression was shown to be anticorrelated with various epileptic profiles and with the severity of epilepsy (Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]. Using the Boolean network selected by this method, we rank genes in M30 according to their in silico estimated ability to permanently modify the expression of the whole network, and prioritized top genes. In favor of the important role in epilepsy-related biological processes of these genes, this prioritised set of candidate genes is significantly enriched in terms related to epilepsy and neurodevelopmental issues when compared to the whole M30 module.

Reproductible inference of a cell-line specific Boolean network

This part of our work aims at designing a method which, given a set of genes of interest, is able to retrieve a Boolean network that allows the prediction of transcriptomic profiles. We consider the formalism of Boolean networks, introduced in Kauffman (1969); [START_REF] Thomas | Boolean formalization of genetic control circuits[END_REF], which are popular models to describe gene-gene expression regulations as gene regulatory networks.

Boolean networks. A Boolean network is characterized: first, by a graph -i.e., the network-which connects genes by their regulatory interactions. Such connections are enriched with the direction of the interaction, which distinguishes between regulator and regulated genes, and with the sign of this interaction, that is, whether the regulator inhibits or activates the expression of its target. Second, by the dynamics of the regulatory system : a logical function, called "gene regulatory function", is assigned to each gene. Regulatory functions are logical Boolean formulas where variables correspond to the expression of genes. The expression of a given variable is set to 1 if the associated gene is expressed, otherwise 0. For a given gene, a logical formula contains in its premise variables associated with direct regulators of the considered gene -i.e., direct predecessors of this gene in the network-and, in its conclusion, the variable related to this gene. Then, given the expression states of the regulators, one can obtain the expression state of the considered gene by evaluating the corresponding formula. The vector collecting all the binary gene expression states at a given time point is called network state, or sometimes network configuration. From this model, one can build a state-transition diagram, which is a graph where an edge goes from a given network state A to another network state B if and only if one can reach state B from state A in a single update step. One can read from this diagram attractor states, that is, self-looping nodes, which are defined as steady stable network configurations. That is, the application of the update step to this configuration will lead to itself. Attractor states are interesting because they are commonly related to observable biological phenotypes (Bloomingdale et al., 2018; Wery et al., 2019). This diagram also displays cycles of configurations, which correspond to unsteady stable configurations ; the application of the update step makes the system oscillate between a set of configurations in a cyclic way. A state-transition diagram is associated with a given model and a type of update. Several types have been suggested in the literature [START_REF] Chatain | Boolean networks: beyond generalized asynchronicity[END_REF], the most well-known ones being the synchronous and the asynchronous updates. In the former, all regulatory functions are evaluated in a single step, whereas in the asynchronous update, only one regulatory function is evaluated at one update step. Recently, [START_REF] Paulevé | Reconciling qualitative, abstract, and scalable modeling of biological networks[END_REF] have introduced new dynamics for Boolean networks, which was shown to be flexible enough to represent (a)synchronous dynamics as well as multi-level formalisms, that is, beyond boolean values for gene expression.

Building a Boolean network from scratch. Our work focused on combining several trusted public data sources, and published methods for the design of an end-to-end pipeline for the synthesis of a Boolean network, represented in Figure 2.1. This network models the regulatory dynamics on a subset of genes, in the absence of external perturbation, from a well-chosen cell line ; for instance, the regulations between M30 genes in brain cell lines for our application to epilepsy. Contrary to the contemporaneous work of [START_REF] Montagud | Patientspecific Boolean models of signalling networks guide personalised treatments[END_REF] applied to cancer, here we do not have any access to a generic model which could model any type of epilepsy to start with. Moreover, relying too much on prior epilepsy-oriented knowledge might lead us to find already known gene targets. The big picture of this pipeline comprises of the following three main steps in chronological order, respectively denoted (A), (B) and (C) in Figure 2.1 :

(A) Data collection.
Step (A) represents the collection and filtering of information from public, large databases : measurements of transcriptomic data are retrieved from the LINCS L1000 database [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF] using careful filtering and quality control measures ; known unsigned, undirected protein pairwise regulatory interactions involving genes in M30 are obtained from the STRING database [START_REF] Szklarczyk | The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets[END_REF].

(B) Data processing.

Step (B) represents the processing of this information into appropriate inputs for the inference of Boolean networks. First, a set of binarized phenotypes is built, corresponding to profiles from single gene perturbations and their associated controls, from the LINCS L1000 transcriptomic data. Then, a signed network of admissible regulatory interactions is constructed from the STRING-extracted regulatory interactions, by filtering out and signing edges based on Spearman's ρ gene pairwise expression correlations computed on LINCS L1000 profiles.

(C) Network inference. Finally, step (C) is the inference of a set of Boolean networks which satisfy all the experimental and topological constraints given by the phenotypes and the signed network. The experimental constraints comprise of each knockout experiment, where the control phenotype is considered the initial condition, and the perturbed one the final configuration reached after stabilitization of the system perturbed by the gene knockout or overexpression. A Boolean network solution should satisfy all of these time-series constraints by only considering a subset of admissible regulatory interactions, represented in the signed network. The final step of the procedure is the selection of an optimal Boolean network among these solutions, according to its topology. This final inferred network is selected through a desirability function maximization [START_REF] Babichev | Technique of Gene Regulatory Networks Reconstruction Based on ARACNE Inference Algorithm[END_REF] which aims at maximizing a parameter called general topological parameter (GTP), which depends on several network topology measures. Details in the implementation are available in Appendix 9.1. Note that, although this issue did not have to be addressed for epilepsy, since the method is meant to be generic (i.e., usable for any disease), which is why, if no gene set is provided, the method automatically retrieves genes from DisGeNet [START_REF] Piñero | The DisGeNET knowledge platform for disease genomics: 2019 update[END_REF] from the disease Concept ID (CID) Step (C)

Step (A)

Step (B)

B E D C A B E D C Or A B E D C Figure 2
.1: Automated building of Boolean networks. Overview of the pipeline for the automated building of Boolean networks. Blocks are colored according to the step they belong to: step (A) for the lightest blocks, step (B) for blocks at the center of the picture, and step (C) for the darkest blocks. defined in PubMed [START_REF] Doğan | NCBI disease corpus: a resource for disease name recognition and concept normalization[END_REF]. Table 9.4 in Appendix reports the values used to filter out genes from the DisGeNet database. The single network obtained at the end of step (C) is a dynamical system which can predict the behavior of gene expression under one or several gene perturbations, by considering the stable states (attractors and cycles) reachable from a given initial state under these perturbations. We now describe on how we leveraged this network model to rank genes according to their regulatory influence on other genes.

Detection of master regulators in a specific disease-context

As mentioned in Section 2.1, when looking for therapeutic candidates, one might be interested in master regulators, that is, genes at the top of the gene regulation hierarchy. Change in expression in a master regulator gene generally induces a large change in downstream gene expression ; for instance, this master regulator gene encodes for a transcription factor which affects the transcription of other genes [START_REF] Mattick | A global view of genomic information-moving beyond the gene and the master regulator[END_REF]. In practice, it is frequently quantified using the node (outgoing) degree and by the detection of hub nodes in the network. Many measures defining this "centrality" property for nodes in a graph exist, and can be computed using for instance Cytoscape [START_REF] Shannon | Cytoscape: a software environment for integrated models of biomolecular interaction networks[END_REF], modules NetworkAnalyzer [START_REF] Assenov | Computing topological parameters of biological networks[END_REF] and CytoCtrlAnalyser [START_REF] Wu | CytoCtrlAnalyser: a Cytoscape app for biomolecular network controllability analysis[END_REF]. For example, control centrality [START_REF] Liu | Controllability of complex networks[END_REF] has been recently used to identify regulations between NFATC4 and Type 2 diabetes-associated genes [START_REF] Sharma | Controllability in an islet specific regulatory network identifies the transcriptional factor NFATC4, which regulates type 2 diabetes associated genes[END_REF]. However, these measures only use the topological information in the network, whereas our network inference pipeline allows, along with the identification of regulatory relationships, the inference of gene regulatory functions, which encode how regulators influence the expression of their target genes. This is why we designed a method which uses this supplementary information for the detection of master regulators.

Gibbs and Shmulevich (2017) used a machine learning technique called "influence maximization" in order to identify key genes in their continuous model of the yeast regulatory network. In our work, we adapted influence maximization to generic Boolean networks. For long, online recommendation and advertising researchers have been interested in influence maximization [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF], which aims at finding a subset of fixed size of nodes which influences most the remainder of the network. The most well-known use cases for influence maximization are in online sponsoring of influencers on social media [START_REF] Leskovec | The dynamics of viral marketing[END_REF]. We will describe below the general setting for the problem of influence maximization.

Influence maximization. Considering a (un)directed graph G on a set of nodes V connected through a set of edges E ⊆ V × V, the goal is to determine a seed set N of fixed size k -that is, a subset of k nodes which start propagating some quantity (influence) throughout the network from their direct neighbors-which maximizes the influence spread. This influence spread is the expected number of vertices ultimately (in)directly influenced by seed set N . k is selected by the user depending on the final application case -for instance, the number of influencers to sponsor on social media to increase the advertisement of a product. Then, we study the random variable I(N ) which is the random number of vertices influenced by seed set N by the cascades of propagating influence.

In order to make this technique applicable to Boolean networks, we need to explicitly define the concept of influence on gene expression in this type of regulatory networks, that is called "spread process". This quantity is proportional to the influence that propagates along the edges of the network. We define influence in an iterative way ; first we consider a single gene and initial network state, then we proceed to define multi-gene influence, and finally influence for a set of genes across a set of initial network states.

Genewise influence in a Boolean network. The most intuitive definition of influence (denoted in the remainder of the paper "spread value") SV B ({n}, {i}) of a given node n on the other nodes in a Boolean network, in a given initial state i, would be that any perturbation of this node would "greatly change" the attractor states reachable from state i, compared to the set of attractor states reachable from state i in the absence of any perturbation. We define this great change (i.e., a positive spread value) by the fact that those two sets of attractors have an empty intersection. Let us denote A(i, P ) the set of attractor states reachable from state i, under perturbations in set P . Set P contains pairs of gene names and their perturbation (either 0 for knockout, or 1 for overexpression). Let us also denote O the set of output genes, that we define here as the set of genes with a positive in-degree. Then, considering any similarity measure S between network states, this definition of spread value SV B({n},i) for node n and initial state i is defined as :

SV B ({n}, {i}) = 1 -max S a 1 |O , a 2 |O : a 1 , a 2 ∈ A(i, ∅)× ∈ A(i, {(n, ¬i[n])}) ,
where we restrict the attractor states a 1 and a 2 to the set of output genes O when computing their similarity.

The perturbation denoted by (n, ¬i[n]) means that gene n is perturbed in the opposite direction to its expression state i[n] in i : for instance, if n is expressed in state i, then we consider knockouts of gene n. The restriction a |O of any attractor state a to output genes in O is actually important in order to have consistent results when considering isolated nodes.

Note that, if n does not have a determined expression state in initial state i, we set the associated perturbation set to ∅. This implies that some genes with individual spread value equal to 0 can either have no true influence on the network, or have no determined expression state in initial state s, which means that they are not measured during the generation of transcriptomic data.

The value SV

B ({n}, {i}) is equal to 0 if and only if A(i, ∅) ∩ A(i, {(n, ¬i[n])}) = ∅ (that is, if there is any attractor state in common). But, note that, if A(i, ∅) ∩ A(i, {(n, ¬i[n])}) = ∅, then SV B ({n}, i)
is not necessarily equal to 1, as reachable attractors might still be close to those obtained without any external perturbation.

Geneset influence in a Boolean network. When considering several nodes in set N instead of a single one n, that is, to assess the influence of a subset of nodes all simultaneously perturbed, we consider :

SV B (N , {i}) = 1 -max S a 1 |O , a 2 |O : a 1 , a 2 ∈ A(i, ∅) × A(i, {(n, ¬i[n]) : n ∈ N }) .
Aggregation of values for several initial states. When considering several putative initial states in set I instead of a single one i, for a given gene set N , we consider the geometric mean of their spread values for each initial state i ∈ I:

SV B (N , I) = i∈I (SV B (N , {i}) + 1) 1/|I| -1 ,
where |I| is the number of considered initial states. Note that we need to correct for zeroes to order to avoid the collapse of this measure when one perturbation does not trigger a change in reachable attractors for one of the initial states, while keeping spread values between 0 and 1 for better interpretability.

Algorithm 1 Gene influence maximization. Greedy influence maximization algorithm for Boolean networks

Input: B = (V, E, F ) a Boolean network on node set V with edges in E and regulatory functions F ; K the number of simultaneous perturbations on the network ; I set of initial Boolean states

Initialize N = ∅, k = 0 repeat k ← k + 1 # Adding to set N nodes that maximize the spread value N ← N ∪ N k where N k ← arg max n∈V \N SV B (N ∪ {n}, I) # Ensuring submodularity until k = K or the following condition holds max n∈V \N SV B (N ∪ {n}, I) ≤ SV B (N , I)

Output: N and observed spread values

Matching to the general setting of influence maximization. The graph that describes the connections between nodes (genes) is the interaction graph of the considered Boolean network B. Random variable I(N ) is no longer an integer (the number of influenced nodes) but directly the minimum frequency of change in binary expression across all downstream genes in reachable attractor states due to the perturbation(s). Note that computing SV B (•, •) in practice runs several stochastic trajectories to compute possibly reachable attractor states.

Once the concept of influence is defined, we propose an algorithm which greedily builds the set of nodes which influence most the remainder of the network.

Influence maximization algorithm on Boolean networks. We describe how to use this spread value to identify a subset of master regulators in the network. Current literature on influence maximization [START_REF] Perrault | Budgeted online influence maximization[END_REF], which is NP-hard, relies on the fact that the spread value function is submodular : roughly, as the considered subset increases, the difference in the value of this function due to adding another single element to the subset decreases. However, no such property can be assessed for the definition of spread value defined in the previous paragraph.

Then, we slightly adapted the greedy algorithm designed by [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF], which determines the set of nodes of minimal size K which are the most influent, where K is a predefined fixed value. Algorithm 1 goes as follows: starting from an empty set of nodes N 0 , a fixed set of initial states I, at each step k ∈ {1, 2, . . . , K}, the algorithm selects the node n ∈ N k-1 which maximizes spread value SV B (N k-1 ∪{n}, I) and computes the set

N k = {n} ∪ N k-1 , until step k = K, or until the first step k when the spread value SV B (N k-1 , I) is no longer increasing, that is, max{SV B (N k-1 ∪ {n}, I) : n ∈ N k-1 } ≤ SV B (N k-1 , I) , (2.2)
which is necessarily to compensate for the fact that the function might not be submodular. If, at a given step k, several nodes maximize the spread value, they are all added to set N k . The iteratively built set N K is then the set of possible K-sized gene subsets to simultaneously perturb on the network, such that the set of attractors reachable from initial set I is greatly modified. In this work, we selected K = 1, that is, we looked at individual contributions of genes to the changes, and we ranked gene n according to its spread value SV B ({n}, I).

Set of initial network states (I).

We consider transcriptomic profiles from human hippocampi afflicted with temporal lobe epilepsy (TLE) in [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF] for the initial states, such that genes are ranked according to their influence in an epileptic context. Temporal lobe epilepsy is one of the most common forms of partial epilepsy, where seizures affect one part of the brain, and is often associated with cases of refractory epilepsy that cannot be surgically treated [START_REF] Han | Electrical stimulation of hippocampus for the treatment of refractory temporal lobe epilepsy[END_REF]. Details about the implementation and initial states are available in Appendix 9.4. 207 genes out of to expression levels in these states, which means that for these genes, we are sure that any spread value equal to 0 for any of these genes truly means that the gene has no influence over the remainder of the network.

Similarity between attractor states (S).

The definition of influence relies on a similarity function S between two network states, that we left to be defined in the concept of influence. In the implementation of the method, we used a distance function which is relevant for binary vectors. The main difference with other binary distances is that we wanted to compute the differences in the presence of 1's, but also of 0's in the vectors, which prevented us from directly using Jaccard's score. Based on previous surveys of the state-of-the-art on binary distances [START_REF] Choi | A survey of binary similarity and distance measures[END_REF], we decided to use a "normalized" ℓ 1 -norm distance. That is, if a 1 and a 2 are the two binary vectors to compare (of size d), then the resulting similarity S(a 1 , a 2 ) between a 1 and a 2 is :

S(a 1 , a 2 ) = 1 - 1 d d i=1 |a 1 [i] -a 2 [i]| . (2.3)
This expression is exactly the percentage of row-wise equal coefficients in a 1 and a 2 , and yields 1 when a 1 = a 2 , and 0 for a 2 = (a

1 + 1) ≡ [2] (modulo 2).
It penalizes in a symmetric way differences in 1's and 0's.

Results

Networks obtained from the inference procedure

We discuss the networks resulting from the inference procedure described in Section 2.2.

Inferred network. The final network obtained at the end of step (C) is shown in Figure 2.2. In this figure, nodes are colored by their degree ; the darker the color, the higher the degree. Edges in Figure 2.2 are colored according to their source of evidence as reported by the STRING database [START_REF] Szklarczyk | The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets[END_REF]. One can notice that there are a lot of undirect geneto-gene regulatory interactions in this network. This actually is not very surprising, since few gene pairwise interactions are experimentally tested compared to all possibly existing interactions. Moreover, our model does not aim at taking into account exclusively transcriptomic interactions, but possibly non-physical, post-transcriptomic effects. Comparison of the different solutions. Now, we consider all 254 solutions generated at the end of step (B), and estimate how far they are from each other, in terms of node degree distribution, edge numbers, redundancy in interactions, unicity of regulatory functions for each node across those solutions, and values of general topological parameter (GTP). GTP is a value comprised between 0 and 1 that is used to select the final network among the 25 ones (as further described in Section 9.1 in Appendix) and characterizes the proximity of a network topology to a scale-free topology.

Table 2.1 shows distribution statistics about the values of GTP and the unicity of gene regulatory functions across solutions. Note that all solutions present similar topologies, with similar GTP scores quite close to 1, which matches what can be expected from biochemical interaction networks in non-fungi systems [START_REF] Broido | The longer tail: The changing shape of Amazon's sales distribution curve[END_REF]. Moreover, except for less than 25% of the genes in the network, genes are assigned at most 3 different regulatory functions across all solutions, which shows that their function in the network is globally preserved.

Figure 2.3 displays the boxplots of edge number and node degree distributions across solutions. These two plots show that, as mentioned before, the typical scale-free topology, with a few "hub nodes" with large degree and a large number of genes with few regulatory interactions, is present in all solutions. There are 74 interactions (that is, around 30 -34% of edges) which are present in at least 75% of the solutions, among which 25 are present in all of them. They are shown in Table 9.5 located at the Appendix. These numbers are confirmed by plotting the network comprising of all genepairwise interactions which are present in at least one solution, shown in Figure 2.4.

All in all, the networks obtained just before the model selection step mostly seem similar, both functionally (at the level of regulatory functions) and topologically (considering the node degrees, the number of edges, and the GTP scores).

Figure 2.2: Inferred network for the application to epilepsy. Inferred network resulting from our pipeline applied to the M30 gene module. Tee-headed arrows represent inhibitory regulatory interactions, whereas triangle-headed arrows are activatory regulations. Edges are drawn according to their source of evidence (on the undirected interactions) as reported by the STRING database: contiguous arrows denote coexpression, solid line denote experimental proofs of interaction, and sinewave interactions are derived from text-mining procedures. Gene nodes are colored according to their out-degree: lightest color for genes with outdegree equal to 0, darkest for genes with an outdegree higher than 5. Isolated nodes are not shown. The darker and thicker an edge is, the more frequent it is across solutions. Sinewave edges are inhibitory interactions, whereas solid lines denote activatory interactions. Orange nodes correspond to the genes which are perturbed in the LINCS L1000 experimental profiles used for inference.

Recommended master regulator candidates

We now study the results from the method of detection of master regulators.

Comparison to known gene measures. We computed the correlation between spread values for our application to epilepsy, genewise Control Centrality [START_REF] Liu | Controllability of complex networks[END_REF] values computed with CytoC-trlAnalyser [START_REF] Wu | CytoCtrlAnalyser: a Cytoscape app for biomolecular network controllability analysis[END_REF], and genewise outgoing degrees. The outgoing degree ("outdegree") is the number of direct downstream targets, whereas Control Centrality is the number of nodes which are affected by a change in the considered node, based on the (directed) network topology. More specifically, in order to compute the Control Centrality for any gene g in the setting we consider, at some time step t, (continuous) expression levels x(t) ∈ R N are time-invariant and depend linearly on those x(t -1) ∈ R N at the previous time step t -1, where N is the number of genes in the network

∂x(t) ∂t = Ax(t) + u g (t) , (2.4)
where A is the adjacency matrix in R N ×N associated with the network, and u g (t) ∈ R is the external signal imposed on node g at time t (either overexpression if it is positive, or knockout otherwise). In such a system, computing the number of nodes which can be controlled by gene g boils down to getting the rank of the so-called controllability matrix related to A and g, which is a function of powers of matrix A. This rank can be computed by solving a combinatorial optimization problem described in Equation (3) in [START_REF] Liu | Controllability of complex networks[END_REF]. Moreover, since, as a general rule, the true nonzero values in A as well as u g (•) are unknown, Control Centrality aims at quantifying structural controllability, independently from the values of nonzero coefficients in A and u g (•). All in all, Control Centrality is a solid counterpart to our method. It does not take into account neither the set of regulatory functions nor the gene expression levels in patients, but models regulatory cascades through the differential equation in Equation (2.4).

We compared these measures, related to the influence of a node in the network, to scores associated with the pathogenicity of genes :

-probability of loss of function intolerance (pLI) [START_REF] Lek | Analysis of protein-coding genetic variation in 60,706 humans[END_REF], which quantifies the intolerance to a deleterious mutation of a given gene.

-enhancer-domain score (EDS) [START_REF] Wang | Enhancer domains predict gene pathogenicity and inform gene discovery in complex disease[END_REF], which studies the conservation of the regulatory domain around genes ; -residual variation intolerance score (RVIS) [START_REF] Petrovski | Genic intolerance to functional variation and the interpretation of personal genomes[END_REF], which quantifies functional genetic variation, and is anticorrelated to gene pathogenicity.

Finally, we computed influence scores [START_REF] Nicolle | CoRegNet: reconstruction and integrated analysis of co-regulatory networks[END_REF] as well, which expression is reported in Equation (2.1).

Figure 2.5 displays the Spearman's ρ correlation heatmap between these different measures. We observed that, contrary to influence values, spread values were consistent and strongly correlated with Control Centrality and the outgoing degree, that is, network-dependent measures. Moreover, spread values have a stronger correlation to the gene pathogenicity-related measures pLI and (opposite of) RVIS.

We tested whether the spread value was actually totally determined by the number of downstream (not necessarily direct) regulated genes. To do so, we performed a Spearman's ρ linear correlation test on the spread values and the number of downstream regulated genes. We confirmed that there is a strong, significative correlation between the two -which is expected, given the definition of the spread value-but that the spread value is not completely determined by this value, that is, the correlation value is not equal to 1 (ρ = 0.82, p = 3.10 -57 ).

Enrichment analysis in epilepsy-related terms of genes with high spread values. From Figure 2.6, it can be noticed that there is a lot of discrepancy between pLI scores and spread values on M30 genes. Nonetheless, it should be noted that [START_REF] Ziegler | A snapshot of some pLI score pitfalls[END_REF] warns against genes which are involved in recessive forms of diseases, while having a low pLI score. That is actually the case for gene GNB5, which has a central place in our network (see Figure 2.1) with spread value 0.024, and a pLI score close to 0, and is involved in a recessive form of epileptic encephalopathy [START_REF] Poke | The epileptology of GNB5 encephalopathy[END_REF].

In order to statistically test if the top genes for spread values are related to epilepsy-related symptoms and mechanisms, we performed a pathway enrichment analysis. This analysis allows the identification of the gene functions most represented among these top genes. An over-representation analysis (ORA) [START_REF] Yaari | Quantitative set analysis for gene expression: a method to quantify gene set differential expression including gene-gene correlations[END_REF] determines which sets of functionally similar genes are statistically surrepresented in the subset of genes, if they exist. Surrepresentation of a gene function set is quantified by comparing the gene function categories represented in the subset of genes (enrichments) to the categories present in a background set of genes, which contains the subset of interest. The statistical tests -one per gene function category-were run by the online tool WebGestalt [START_REF] Liao | WebGestalt 2019: gene set analysis toolkit with revamped UIs and APIs[END_REF].

In this case, the subset of considered genes is the set of 14 genes with spread value greater than 0.01 (displayed in Figure 2.6). The selected back- ground genes were the 233 genes present in the Boolean network. We considered the gene categories annotated in the DisGeNet database [START_REF] Piñero | The DisGeNET knowledge platform for disease genomics: 2019 update[END_REF].5

Indeed, the subset of genes is (weakly) significantly enriched in genes related to the term "Epileptic encephalopathy" (odds ratio OR = 7.5, Benjamini-Hochberg (BH) [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF]-adjusted p ≈ 0.038), and more strongly enriched with (neuro)developmental issues, for instance, "Loss of developmental milestones" (OR = 10.5, BH-adjusted p ≈ 0.012), as reported in Figure 2.5. Similar results can be observed on another family of gene annotations, GLAD4U [START_REF] Jourquin | GLAD4U: deriving and prioritizing gene lists from PubMed literature[END_REF], as shown in Figure 9.2 in Appendix. These enrichment results go beyond the fact that M30 is globally enriched in epilepsy-related de novo mutations compared to the whole measured genome, as shown in Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF] : what is shown is that, among genes in the M30 module, ranking by spread values pinpoints epilepsy-related genes. 2.2, we have selected a shorter list of candidate genes, comprised of rather large spread value (greater than 0.01) and pLI score greater than 0.9, and of genes with very large spread value (greater than 0.02), in order to avoid the shortcoming in the pLI score mentioned above. Candidate genes are CACNA1C, RBFOX1, STXBP1, DNM1, NRIP3, SCN8A, CHRM2, GNB5, TUBB2A, PAK7 and GRIN1, shown in Figure 2.6. Most of these candidates -except for NRIP3, which is notably mainly expressed in the hippocampus-have a relationship to epilepsy-related symptoms in humans as shown in prior works (Appenzeller et al., 2014; Butler et al., 2017; Cushion et al., 2014; Al-Eitan et al., 2019; Lal et al., 2013; Myers et al., 2016; Ohba et al., 2015; Poke et al., 2019; Stamberger et al., 2016), as expected due to their membership to the M30 module.

Selection of a list of candidates. Based on the results shown in

In silico validation of the candidates. In order to further confirm our list of candidate genes, we exploited the Boolean network inferred in Section 2.2 to predict whether a knockout of each of these genes would trigger a transcriptional profile close to epileptic states. Knockouts are easier 07 0.06 0.11 1.0 0.16 0.39 -0.06 0.04 -0.0 0.0 0.16 1.0 0.19 0.08 0.06 0.03 0.08 0.39 0.19 1.0 # Normalize the matrix of control and patient samples per feature (gene)

X ← ( 1 σg (X g -µ g )) g∈{1,...,|M 30|}
, where µ g = mean(X g ) and σ g = std(X g )

# Learn a mapping from the initial high-dimensional space to 2D Fit a Principal Component Analysis (PCA) to experimentally confirm than overexpression perturbations, and most of the selected genes have a reported pro-epileptic deletion mutation (Appenzeller et al., 2014; Griffin et al., 2021; Lal et al., 2013). Our procedure is summarized in Algorithm 2. The key idea behind this algorithm is that the repurposing score aims at determining in which part of the 2D plane an in silico treated profile located : either on the hyperplane globally assigned to control samples, or the one mainly associated with patient profiles. Then, the "signed" distance of this treated profile to the frontier which separates these two hyperplanes is computed. The sign of this distance is indicative of the hyperplane the treated profile belongs to (positive if in the "control" hyperplane, negative otherwise). We now describe Algorithm 2. We consider this time both sets of binarized epileptic and control (healthy for epilepsy) hippocampi samples in [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF], respectively denoted P (|P| = 24) and C (|C| = 23). These profiles are restricted to the set of M30 genes present in the network. The binarization step is performed similarly to what is described in Section 9.1 in Appendix. In order to score the perturbation related to a given gene, first, we fit a Principal Component Analysis (PCA) model M with the standard-normalized matrix of binary control and patient profiles (with values in {-1, 1}). The use of PCA allows to renormalize the distances on the 2D plane, and transform the profiles to make them more informative, by considering the two axes on which the most variance across samples is observed. Second, we consider the PCA-transformed Characteristic Direction (CD) signature [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF] M(S) computed on patient and control profiles.

M : {-1, 1} |M 30| → R 2 on matrix X # Run Characteristic
I refer the reader to the corresponding paper [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF] for the details about Characteristic Direction. The main principle that should be kept in mind is that the signature CD[G1 G2], computed through Characteristic Direction for two condition groups G1 and G2, is actually the vector normal to the decision frontier in a high-dimensional space, which classifies samples into either G1 or G2, and that is oriented in the direction from G2 to G1. That means in particular that the changes reported in the signature are changes in G1 compared to the reference group G2.

Then, for each candidate gene g and each control profile c available, we retrieve a set of attractor states denoted A (c,g) reachable from state c under knockout of gene g, as described in Section 9.4 in Appendix. To compute the score associated with an attractor state a (that is, how close to a control profile it is), first, we PCA-transform this attractor into a 2D vector M(a) ; second, we compute the associated projection π(M(a)) onto the frontier between control and patient samples (which normal vector is M(S)) ; finally, CACNA1A DNM1 GNB5 STXBP1 SCN8A PAK7 TUBB2A CHRM2 -0.17 -0.17 -0.17 -0.20 -0.20 -0.20 -0.22 -0.24 NRIP3 GRIN1 RBFOX1 -0.25 -0.29 -0.30 we compute the signed distance between M(a) and its projection π(M(a)) :

signed(M(a), π(M(a))) := n i=1 (π(M(a)) i -M(a) i ) .
Note that, contrary to the distance function used in Section 2.3, this quantity is signed. This value is positive if M(a) is in the zone delimited by the hyperplan which contains control samples, and negative otherwise. The large it is, the more the attractor state is considered in the control or patient zone. Such a score allows a quick interpretation of the results and a straightforward visualization of the improvement (or worsening) of the patient profile after treatment.

In the presence of several attractors, we use the probability p (a,c,g) of presence of attractor a reachable from the initial control state c under knockout of gene g, as provided by PyMaBoSS [START_REF] Stoll | MaBoSS 2.0: an environment for stochastic Boolean modeling[END_REF], using the same parameters as in Section 9.4 in Appendix. For an attractor which is not reachable from profile c, we have p (a,c,g) = 0. The final score for a given gene

g based on |C| control profiles is then : score(g) := 1 |C| c∈C a∈A (c,g) p (a,c,g) × signed(M(a), π(M(a))) .
If no attractor state is found, then we set this value to "not a number".6 Table 2.3 shows the scores obtained for each candidate gene. These scores show that any knockout of these genes will globally turn a control transcriptomic profile into one which is similar to an epileptic one, with some of these genes which may never been investigated in the research related to epilepsy, such as NRIPS3, and some of them, notably, STXBP1 and GRIN1, which orthologuous genes were associated with epileptic seizures in zebrafish [START_REF] Griffin | Phenotypic analysis of catastrophic childhood epilepsy genes[END_REF].

Discussion

We introduced in this work two main contributions to in silico disease research. First, we designed a method for the inference of gene regulatory network only from a subset of genes. This method carefully combines information from several public databases and methods, and infers a dynamical model of gene regulation adapted to the considered set of genes. This method yields to quite robust network solutions, and is easily reproducible. Second, we showed how to exploit the inferred network model, and more specifically its dynamics, to detect master regulatory genes, which may be interesting candidates to investigate a disease. We applied our methodology to tackle epilepsy, and to find novel gene candidates to investigate epilepsy. It allowed us to get a list of candidate genes which perturbations greatly impact the whole network in an epileptic transcriptomic context. The experimental validation of the knockout of some of these candidates in the zebrafish is an ongoing collaboration with Nadia Soussi-Yanicostas at Inserm UMR1141. This methodology allows reproducible and transparent research, while reducing the amount of data needed as input, which is one of the main caveats of researching on rare or tropical neglected diseases. Next, we will investigate, using the inferred network, the simulation of the effect of molecules -that is, which potentially targets several genes at a time.

Chapter 3

Drug efficacy scoring using a Boolean network I now introduce how to use the Boolean network inferred in Chapter 2 in order to simulate and score the effect of a given drug at transcriptomic level.

Had we had access to several clinical trials involving the considered disease and molecules -for instance, in the form of a binary matrix with molecules as rows and diseases as columns, where ones would indicate successful trialswe could have used traditional machine learning techniques for classification tasks (He, Yang, Gong, et al., 2020; Jarada, Rokne, and Alhajj, 2020; Xue et al., 2018; Yang, Luo, et al., 2019) to solve drug repurposing. However, when facing rare diseases, we actually lack of information about putative successful drugs, and even sometimes information about the disease itself (e.g., classification, biological mechanism, . . . ). This problem bears more than a passing resemblance to the "cold start" problems in recommender systems. Nonetheless, when no prior information can be collected before running the recommender system, the methods used to circumvent this hurdle mostly rely on the guilt-by-association principle, where diseases with similar transcriptomic profiles may have common successful treatments.

However, when it comes to medical purposes, simple similarity is not enough, as mentioned in Réda, Kaufmann, and Delahaye-Duriez (2020), as a change in chirality 1 is enough to turn a morning illness treatment into a teratogen molecule. In particular, in order to perform signature reversion on a fixed pair of disease and drug, simply comparing the signature built from transcriptomic profiles from healthy individuals and patients on the one hand, and the one obtained on control and in vitro treated cells on the other hand, might not be enough to estimate the actual reversing power of the considered molecule. Genes interact on the expression level of others through regulatory cascades [START_REF] Bolouri | Transcriptional regulatory cascades in development: initial rates, not steady state, determine network kinetics[END_REF], and they sometimes undirectly act upon their own expression. When screening thousands of molecules, the experimental measure of the actual posttreatment transcriptomic profiles on diseased patients or animal models is too time-consuming and expensive to be effectively implemented ; hence the proposed drug repurposing method, which exploits the Boolean network inferred in Chapter 2 to model regulatory cascades, and predict post-treatment transcriptomic profiles.

In this chapter, I will describe how to leverage the Boolean network to predict post-treatment transcriptomic profiles. These predictions will help scoring the therapeutic effect of the treatment, by comparing in silico treated profiles and appropriate control ones. A special attention is brought to the selection of the drug-induced perturbations at transcriptomic level. These scores allow ranking putative drug candidates by their performance, which effectively solves the problem of drug repurposing. As a proof-of-concept, this procedure has been applied to epilepsy, for which a Boolean network was inferred in Chapter 2. The resulting ranking on a subset of known epilepsyrelated molecules was compared to a baseline method, which does not model regulatory cascades.

Related work

In this section, I briefly recall important concepts related to this chapter.

Drug repurposing and signature reversion. Drug repurposing is a paradigm of drug development which aims at discovering new therapeutic indications for already commercially available molecules. The reuse of welldocumented and already approved drugs allows decreasing the allocated budget and time for drug discovery research, while limiting the possibility of undiscovered negative side effects (Hwang et al., 2016).

A promising method is the method of "signature reversion" (Lamb et al., 2006; Sirota et al., 2011) : drug candidates are ranked according to the similarity between disease-and drug-associated signatures. Signatures describe the changes in gene expression which were induced by the disease, resp. by the treatment. The key idea is that, if the same genes are affected in both signatures in a different direction -i.e., up-regulated in the drug signature, and down-regulated in the disease one, or vice-versa-then this treatment could be an interesting therapy. Many examples of applications of signature reversion are available in the following review papers (Hodos et al., 2016; Musa et al., 2018). This approach has further been facilitated by the creation of databases of treated transcriptomic profiles, which are systematically produced through a standard RNA-sequencing pipeline on immortalized human cells. One of the first databases collecting this data is the Connectivity map (CMap) database [START_REF] Lamb | The Connectivity Map: using gene-expression signatures to connect small molecules, genes, and disease[END_REF] in 2006, followed by the LINCS L1000 database in 2017 [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF] at a larger scale. Each drug signature is built by comparing significant genewise changes in expression between cells in vitro treated by a control molecule or by a specific active molecule. The main asset of LINCS L1000 is that the database is extremely large : up to 30, 000 drugs or genetic perturbations are tested across 98 cell lines, going from 1, 309 chemical compounds applied to 5 cell lines in CMap. This is due to the reduced cost in the generation of transcriptomic data. Only around a thousand genes are truly assayed, and the expression of the remainder of the genome is inferred through a linear regression model. This computational inference seems to give satisfying results compared to the fully-assayed genome [START_REF] Cheng | Systematic quality control analysis of LINCS data[END_REF]Li, 2016; Subramanian et al., 2017). However, these databases might provide discordant drug signatures, and thus rankings, probably due to differences in the technical tools used in their profile generation pipeline [START_REF] Lim | Evaluation of connectivity map shows limited reproducibility in drug repositioning[END_REF]. The latter work also highlights the importance of careful selection of drug signatures -in particular, in terms of cell lines-from these databases, as we will develop in the next sections.

The LINCS L1000 database has been exploited in a signature reversion method called "L1000 CDS2 " [START_REF] Duan | L1000CDS2: LINCS L1000 characteristic direction signatures search engine[END_REF]. First, this method computes the drug signatures on the whole LINCS L1000 genome (≈ 12, 000 genes) with Characteristic Direction (CD) [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF]. CD is run over a set of treated and control samples from the same plate. Control samples are treated with a "sham" treatment, for instance dimethyl sulfoxide (DMSO). Then, CD signatures are averaged across replicates. In our application, in order to have a single drug signature per drug, I have also averaged across the two cell lines NPC (neural progenitor cell) and SH-SY5Y (neublastoma line), doses and exposure times. 2 Second, a CD signature S is obtained from applying CD to healthy and patient profiles (respectively denoted C and P). 3 Last, L1000 CDS 2 ranks drugs by computing a cosine distance score between S and each drug signature. That is, for any drug signature s, the method computes cos (s, S)

:= 1 - gene g S[g] × s[g] gene g S[g] 2 gene g s[g] 2 .
The higher the cosine score is, the more epilepsy-affected genes are perturbed by the drug in the opposite direction with respect to the disease ; then the considered molecule is a putative drug candidate. This method is the baseline for the suggested drug repurposing procedure presented in this chapter.

Gene regulatory networks as Boolean networks.

A gene regulatory network (GRN) is a summary of gene regulatory interactions, which is depicted as a graph : nodes are genes or proteins, (directed) edges are regulatory interactions from a regulator to a regulated gene. We implement GRNs as Boolean networks. In this type of models, nodes can have two expression states : 0 for low expression, and 1 otherwise. A "regulatory function" is assigned to each node. This function computes the new expression state of the associated node, depending on the expression states of its direct regulators at the previous time step. A network state is the concatenation of all the expression states at a time point. The procedure to update a network state using the regulatory functions is defined by the dynamics of the model. Examples of dynamics are described in Chatain, Haar, and Paulevé (2018); [START_REF] Paulevé | Reconciling qualitative, abstract, and scalable modeling of biological networks[END_REF]. A Boolean model can predict the final binary network state(s) under the perturbation of one or several genes in the network, by iteratively applying the update step. These final states are attractor states. An attractor state is such that either (a) applying the update step leaves that state invariant (steady attractor) ; (b) the system oscillates in a cycle to which this state belong (unsteady attractor).

Application to epilepsy.

In Chapter 2, we focused on a module of 320 genes called M30, which has been shown to have a global gene expression that is anti-correlated to various epileptic transcriptomic profiles (Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]) ; and we inferred a Boolean network which models the regulatory interactions in this gene module.

At the end of the previous chapter, we actually described a method which allowed us to score the effect of a single gene perturbation on the network. The goal of the current chapter is to explain how we have extended Algorithm 2 to scoring chemical compounds ; and, more specifically, how we have modelled the effect of the treatment, based on a careful selection of drug signatures from the LINCS L1000 database [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF].

Selection of drug signatures

Extension of Algorithm 2 to drug perturbations. Algorithm 2 in Chapter 2 predicts (binary) transcriptomic profiles under the perturbation of a single gene. These predictions are (steady) attractor states reachable from a control profile4 under the knockout of a given candidate gene g. To each knocked-out gene, a score was computed as a weighted sum of "signed distances" from each attractor state to the classification frontier between control and patient samples. This frontier was defined by the Characteristic Direction signature [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF] obtained from healthy and epileptic patient profiles, which corresponds to the normal vector to this frontier. However, potentially several genes can simultaneously be perturbed.

Nonetheless, current lists of drug targets -i.e., genes which are directly affected by a drug at transcriptional level-are not completely satisfying for our purpose. This motivates the use of drug signatures, which are a proxy to estimate which genes are specifically affected by a treatment and in which direction, i.e., whether they would be overexpressed or underexpressed after the treatment.

Gene targets.

Several online databases list drug-associated gene targets : DrugBank [START_REF] Wishart | DrugBank 5.0: a major update to the DrugBank database for 2018[END_REF], MINERVA [START_REF] Hoksza | MINERVA API and plugins: opening molecular network analysis and visualization to the community[END_REF], Drug Central [START_REF] Ursu | DrugCentral: online drug compendium[END_REF], Therapeutic Targets Database (TTD) [START_REF] Zhou | Therapeutic target database update 2022: facilitating drug discovery with enriched comparative data of targeted agents[END_REF] or LINCS L1000 [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF]. Let us check the relevance of these lists with regards to our application to epilepsy.

We compiled a set of 71 known antiepileptics and proconvulsant drugs -i.e., which trigger seizures, which is shown in Table 10.1 (36 antiepileptics) and in Table 10.2 (35 proconvulsant drugs) in Appendix. The objective is to determine how well the information about the drug targets in the M30 module discriminates between antiepileptics and proconvulsant drugs, from the comparisons in Figure 3.1. Each list of gene targets (per database of origin, independently of the direction of change) is turned into a binary vector of size (71 × 320), where ones represent gene targets to a given molecule prior to the comparisons. In the right-hand plot in Figure 3.1, an "aggregated" target matrix in {0, 5} 71×320 was built, such that coefficient in position (i, j) is the number of times gene j is denoted as a target to drug Left plot : Spearman's ρ correlation matrix heatmap which represents the redundancy of targets across databases. Right plot : Heatmap associated with the aggregated target matrix described in the main text. This matrix is restricted to genes (39/320) and drugs (27/71) with at least one nonzero coefficient in the matrix. Antiepileptic, resp. proconvulsant, drugs are written using a green, resp. red, font. Rows and columns are ordered according to two hierarchical clusterings represented on the plot. GABRD, GABRB3, . . . , which are related to synaptic transmission and notably known for intervening in epilepsy-linked mechanisms [START_REF] Treiman | GABAergic mechanisms in epilepsy[END_REF]) ; and on the other hand, drugs targeting sodium channel genes : SCN1A, SCN8A, SCN4B, which are involved in the initiation and propagation of action potentials in the central nervous system [START_REF] Meisler | Sodium channel gene family: epilepsy mutations, gene interactions and modifier effects[END_REF] ; and calcium genes : CACNA1C, CACNB4, . . . , which might induce neuron hyperexcitability [START_REF] Steinlein | Calcium signaling and epilepsy[END_REF]. However, these lists are discordant across databases, as shown by the left-hand plot in Figure 3.1. Moreover, they do not usually provide the direction of expression changes (except for TTD). Last, these lists are too sparse, i.e., a large number of drugs have no reported gene targets, and some genes cannot be linked to any drug, as shown in the caption for the right-hand plot in Figure 3.1.

SCN1A GRIN1 SV2A CHRM2 SCN8A CHRNB2 PDE10A CYP2C8 CACNA1C CACNB2 CACNB4 KCNQ3 GABBR2 CACNG2 CALM3 KCNH5 SCN4B KCNA1 TUBB2A TUBA4A GABRA1 GABRA3 GABRB2 GABRG2 GABRB3 GABRD GABRA4 0 1 2 3 4 5
Nonetheless, these gene targets constitute an interesting ground truth to guide the feature selection to build drug signatures.

Types of drug signatures. We considered three different methods, denoted "BrainCell", "AllCell" and "BestCell", to build drug signatures from the transcriptional profiles in the LINCS L1000 database [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF] for the 71 selected antiepileptics and proconvulsant drugs. These three approaches are mentioned in both plots in Figure 3.2. They only differ on the considered type of cell lines. As mentioned in [START_REF] Lim | Evaluation of connectivity map shows limited reproducibility in drug repositioning[END_REF], the correct selection of the cell line(s) on which differential expression is considered is key in signature reversion.

Any of these methods proceeds as follows : for any molecule, transcriptional profiles from control and treated cells, with at least two technical replicates, are retrieved. Depending on the selected approach, these profiles are possibly filtered on the cellular type on which they were measured. Then, at most 30 of the most reproducible profiles are kept for each group of profiles (treated and control groups).5 Last, we run Characteristic Direction [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF]) on these two groups, which outputs a single real-valued vector. This vector reports the magnitude of change in expression for each gene. Based on the p-values (per gene) computed by CD,6 this vector is binarized by assigning ones to genes with positive magnitude and significant p-value at level 5%, resp., zeroes to genes with negative magnitude and significant p-value at level 5%. In particular, it means that ones correspond to genes which are significantly upregulated after treatment -with respect to the control group-whereas zeroes correspond to significantly downregulated genes. Genes which are not represented in this vector are non significantly differentially expressed, that is, almost unaltered by the treatment.

As previously claimed, we consider the three following filters on cell lines -In "BrainCell", we further restrict the set of transcriptional profiles to profiles obtained in one of the two brain cell lines in LINCS L1000 : neural progenitor cells (NPCs) and a neuroblastoma cell line (SHSY-5Y).

-"AllCell" does not use any filter on the cell line.

-"BestCell" only considers the cell line in which a treatment experiment with this drug has yielded the highest Transcriptomic Activity Score ("cell_tas"), as reported by the LINCS L1000 database. The highest this score is, the most reliable the profiles with this treatment in this cell line are. 7In all three methods, 194 genes (out of 320 genes present in M30) are significantly differentially expressed in at least one of the drug signatures computed for the 71 epilepsy-related molecules. Due to the fact that experiments involving drugs do not necessarily cover all 98 cell lines present in LINCS L1000, there are only 34 drug signatures in "BrainCell", whereas there are 68 drug signatures in "BestCell" and "AllCell". The list of 34 antiepileptic and proconvulsant drugs having a "BrainCell" signature is reported in Tables 10.1 (12 antiepileptics) and 10.2 (22 proconvulsants) in Appendix (column "BrainCell"). Note that not all 71 drugs are present, due to the condition on having at least two technical replicates.

Feature selection with respect to gene targets. We compare all three methods to the lists of gene targets mentioned earlier in Figure 3.2. We denote "Aggregated" the binary matrix in {0, 1} 71×320 , where the coefficient at position (i, j) is equal to 1 if and only if gene j is reported as a target of drug i in at least one of the following 5 databases : DrugBank, DrugCentral, LINCS L1000, TTD, and MINERVA. Similarly, for each type of drug signature, the associated drug signature matrix is converted into a binary matrix in {0, 1} 71×320 , where ones represent significantly differentially expressed genes (up-regulation or down-regulation) according to the drug signatures. Jaccard's Index Each pairwise comparison is restricted to common genes and drugs. Left plot : We compare the similarities of clusterings (obtained by running k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF]) with 3 clusters) by computing the Adjusted Rand Index (ARI) [START_REF] Hubert | Failure of investigational drugs in latestage clinical development and publication of trial results[END_REF]. a Right plot :

The Jaccard index [START_REF] Jaccard | The distribution of the flora in the alpine zone. 1[END_REF] is computed between approaches. In both heatmaps, the higher the score, the most similar two methods are.

Rows and columns of the heatmaps are ordered by a hierarchical clustering represented on the plots.

a The choice of 3 clusters was motivated by the minimum number of clusters which could be found using any of the three types of features.

Figure 3.2 shows that, out of the three methods for retrieving drug signatures, the one exclusively using profiles from brain cell lines ("Brain-Cell") is the most similar to the aggregated list of gene targets ; both in terms of regrouping drugs (left-hand plot), and in terms of intersection of genes (right-hand plot). However, there is still little correlation between the drug signatures from LINCS L1000 and the gene target lists. This might be due to the fact that a gene might be differentially expressed -up-or downregulated-even when it is not directly targeted by a given drug. Moreover, the number of currently reported targets might be a lot smaller than the actual number of targets. Based on these plots, from now on, we only consider only "BrainCell" drug signatures.

Scoring with the Boolean network

Now, we lay out the outlines of the extension of Algorithm 2 in Chapter 2 to scoring chemical compounds, instead of single gene perturbations. Algorithm 3 describes the associated pseudo-code, with differences with Algorithm 2 written in bold type. Notice that one of the main differences is that we consider as initial conditions perturbed patient profiles -instead of profiles of healthy individuals-and that perturbations on the initial states are defined by drug signatures, and no longer by single gene perturbations. That means in particular that, if a gene is denoted as upregulated in a drug signature, then the corresponding perturbation of this gene in the algorithm will be an overexpression. Conversely, a downregulated gene denotes a knockout perturbation of this gene.

In our application to epilepsy, we considered the sets of binarized epileptic and control hippocampi samples for temporal lobe epilepsy (TLE) in [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF], respectively denoted P (|P| = 24) and C (|C| = 23). The use of these profiles is justified by the link between TLE and refractory epilepsies (refer to Chapter 2). This procedure was run on the 34 drugs related to epilepsy with the "BrainCell" signatures, as defined in Section 3.2. This took several hours to complete.8 Indeed, the runtime is proportional to the number of nodes in the Boolean network B and the number of patient samples |P|. Solving the problem of finding a single attractor state reachable from an initial state, using the most permissive semantics [START_REF] Paulevé | Reconciling qualitative, abstract, and scalable modeling of biological networks[END_REF], is PSPACE-complete as a general rule.9 It is coNP-complete under some conditions on the behaviour of regulatory functions. Then, each call to the Boolean network to solve the reachable attractor problem in Algorithm 3 is polynomial (in memory space) in the number of nodes in the Boolean network, up to a multiplicative constant corresponding to the maximum total number of attractor states to enumerate. There are |P| × |L| such calls in total, where |L| is the number of considered drugs.

In [START_REF] Chevalier | Synthesis of boolean networks from biological dynamical constraints using answer-set programming[END_REF], using their implementation of the most permissive semantics, they have been able to solve a single call to the solver -with complex constraints, that is, a potentially even harder problem than the reachability of attractor states-in a couple hours of CPU time in networks with up to 200 nodes. In a nutshell, these problems are tractable in practice for a limited number of nodes in the network, but cannot be extended to a large-scale screening of hundreds of drugs. We will discuss the consequences of this high computational cost at the end of the chapter.

Results

We compare repurposing results on the set of 34 drugs mentioned in Section 3.2 from Algorithm 3 and L1000 CDS 2 [START_REF] Duan | L1000CDS2: LINCS L1000 characteristic direction signatures search engine[END_REF], described in Section 3.1. Recall that L1000 CDS 2 computes a cosine distance score on CD signatures derived from LINCS L1000 profiles. In both approaches, the higher the score is, the best the candidate. The rankings on the 34 drugs are shown in Figure 3.3. Their associated receiver operating characteristic and precision-recall curves are represented in Figure 3.4.

From Figure 3.4, we observe that Algorithm 3 slightly improves over L1000 CDS 2 in predictability. In order to additionally assess the relevance of the proposed drug scoring method, we compute the hit ratio at fixed rank k (Koren, 2008), denoted HR@k (or Recall@k), which is the accuracy restricted to the top-k elements. This quantity can be computed from the rankings in Figure 3.3. The hit ratio metric is useful for evaluating recommendations, as one might only be interested in the first top items. Hit ratios at rank 2, 3, 5 and 10 -which are reasonable ranks for drug repurposing-for both methods are reported in Table 3.1. This table confirms that, indeed, the proposed drug scoring fares better than the baseline. HR@2 HR@3 HR@5 HR@10 Scoring 0.5 0.7 0.4 0.3 L1000 CDS 2 0.0 0.3 0.2 0.4 Table 3.1: Hit ratios on the proposed drug scoring and the baseline. Hit ratios at ranks 2, 3, 5, 10 in the proposed scoring and L1000 CDS 2 [START_REF] Duan | L1000CDS2: LINCS L1000 characteristic direction signatures search engine[END_REF] , rounded up to the 1 st decimal place.

Discussion

In this chapter, we described how the algorithm in Chapter 2 for scoring single gene perturbations was extended to drug-induced perturbations. The proposed method relies on a careful selection of the drug signatures, which highlights the importance of the choice of the cell line, as previously remarked by [START_REF] Lim | Evaluation of connectivity map shows limited reproducibility in drug repositioning[END_REF]. This method slightly improves over a baseline approach on a dataset of 34 antiepileptics and proconvulsants. Yet it is unfortunately still not satisfying in terms of interpretability. Indeed, most of the drug candidates are assigned negative scores ; according to the way the score was designed in Chapter 2, it means that these drugs do not truly "treat" patient profiles. Indeed, a negative score means that, in average, in silico treated profiles remain in the part of the 2D plane that is globally associated with epileptic profiles.

Nonetheless, as discussed in the introduction to this thesis, except for the use case described in Chapter 7, there is no transcriptomic database on individual patient responses to treatment at a large scale for any pathology, 11 similar to what LINCS L1000 is to drug perturbations on in vitro human immortalized cells. This work is an attempt at bridging the gap between what is known about the drug-induced changes in expression and modelling in silico patient responses to treatment.

Yet the proposed method might be improved with further work on the feature selection ; e.g., by incorporating the influence of the treatment dose. Indeed, higher doses might be more effective and lead to more reliable transcriptional profiles -which was our primary criterion for feature selection. However, they might be toxic, and trigger unwanted side effects. Sometimes, high doses might even trigger paradoxical effects, that is, reinforcing the symptoms of the disease, as reported for some antiepileptics in in vivo experiments (Nakken et al., 2003; Osorio et al., 1989).

Moreover, as highlighted by the plots associated with Algorithm 3 (Figures 3.3 and 3.4), there exists a quite large variation, in terms of scores and predictability, across patient profiles. Indeed, on the right-hand plot in Figure 3.3, we can notice that some patients seem to respond well to the treatment (i.e., with positive scores). For instance, at least one patient outputs a score close to 0.18 when treated by Whitaferin-A, even if the related average score across patients is negative. This raises the question on how to properly aggregate the scores across patients.

Finally, as previously evoked when discussing the computational complexity of Algorithm 4, large-scale screening might not be tractable in practice. This fact led me to consider sequential learning methods (namely, multi-armed bandits) in the next chapters : combining an online sampler, which asks for the evaluation of a given drug at points in time, and Algorithm 4 (below) which performs the evaluation of a single drug on a single patient profile. Additionally, this paves the way for a personalized drug repurposing method, which suggests drug candidates which are the most adapted to a specific patient. Personalized drug recommendation in cancer was recently investigated in [START_REF] Montagud | Patientspecific Boolean models of signalling networks guide personalised treatments[END_REF], showing promising results.

In the next chapters, we will apply multi-armed bandit algorithms to a drug repurposing instance, on a subset of 21 antiepileptics and proconvulsant drugs. This instance comprises of "BrainCell" drug signatures and associated average repurposing scores derived from Algorithm 3. This instance is further described in Table 10.3 in Appendix. This restricted instance, which displays a larger gap in scores between the set of antiepileptic and proconvulsant drugs, allows us to illustrate our proof-of-concept on the application of multi-armed bandit algorithms to drug repurposing. In practice, the actual gap in score is of course unknown, and such a transformation only holds to support our proof-of-concept. In Chapter 3, we described how to build a model which simulates the transcriptional effect of a given drug, based on a Boolean network. In particular, this procedure can be seen as a black box, which takes as input a drug signature -as previously defined previously-and a patient transcriptomic profile, and returns a score proportional to its signature reversing power. However, in practice, we know that this model retrieves a set of reachable steady attractor states in order to output a single score. Moreover, since we know there can be a high variability across patients' scores, estimating robustly the average score (across patients) of a given drug would require iterating it for all patients. All in all, this might be a time-consuming and computationally expensive process, especially when performed for a large number of drugs. I wondered if there was some adaptive, "smart", procedure for exploring the space of drugs. This is why I resorted to multi-armed bandit algorithms, which are introduced in the first sections of this chapter. For a given disease, we are interested in identifying a subset of drugs that may have a therapeutic interest. Providing a group of 5 or 10 drugs, rather than a single one, can ease the decision of further investigation, as many leads are provided. The problem of determining the N best options out of K ones was introduced in [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], and is named EXPLORE-N [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], Top-N identification [START_REF] Chen | Nearly instance optimal sample complexity bounds for top-k arm selection[END_REF], or Best-N identification [START_REF] Jiang | Practical algorithms for best-k identification in multi-armed bandits[END_REF]. Moreover, drug signatures can be leveraged to discriminate between drug candidates, and make the algorithm more efficient. Hence, in my PhD, I worked on structured Top-N identification,13 which exploits the structure of the model, that is, the relationship between drug signatures and scores.

Chapter 4 Introduction to Top-N identification

We will use machine learning techniques to refine our drug repurposing method. Machine learning is a subfield of artificial intelligence in Computer Science. Here, a machine learning algorithm designates any computational method where results from past actions or decisions, or past observations, are used to improve predictions or future decision-making. Machine learning techniques are now extremely popular in drug development (Aliper et al., 2016; Ekins et al., 2019; Hodos et al., 2016) as they allow automation of highly-dimensional, noisy biological data analysis. Many different machine learning tasks have been studied, which fall broadly into three categories. The first one is supervised learning, in which the goal is to predict the label of new observations given a large database of labelled examples. Several supervised learning algorithms have been applied in a biological context, such as Support Vector Machines [START_REF] Noble | Support vector machine applications in computational biology[END_REF] or (Deep) Neural Networks [START_REF] Chen | The rise of deep learning in drug discovery[END_REF]. The second task is unsupervised learning, and it aims at detecting underlying relationships or patterns in unlabeled data. Dimension reduction methods, like Principal Component Analysis (PCA), fall in this class. But other unsupervised problems are also studied in the context of drug development, such as density estimation and clustering [START_REF] Zhang | Predicting drug-disease associations and their therapeutic function based on the drug-disease association bipartite network[END_REF]. 1 The third type of task is renforcement learning, where algorithms rely on trial-and-error, and iteratively use external observations in order to find the best decision with respect to the environment they interact with. A large literature has dwelled on the use of sequential learning algorithms, where an agent, i.e., a goal-oriented entity which interacts with its environment, must make one choice at a time according to previous observations of the environment. Offline -or batch-methods use batches of data in order to learn, whereas online -or sequential learning-algorithms process one data point at a time, and update their prediction or decision accordingly.

In this thesis, I am interested in multi-armed bandit algorithms. They constitute a popular and versatile family of sequential decision-making algorithms. In multi-armed bandit algorithms, a fixed set of actions, called arms, is available. An agent sequentially interacts with the environment by selecting arms, as illustrated in Figure 4.1 below. Each arm selection produces some noisy observation, often interpreted as a reward. However, the average reward associated with each arm is initially unknown to the agent, and has to be learnt in the process while achieving a certain objective. Typically, this objective could be to discover the most efficient arm(s), that is, the arm(s) with highest average reward [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], or to maximize the total reward accrued across iterated arm selections [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. See the following referenced paper [START_REF] Kaufmann | Learning the distribution with largest mean: two bandit frameworks[END_REF] for a comparison between these two bandit objectives.

In our application to drug repurposing, multi-armed bandit algorithms will help decreasing the number of calls to the Boolean network, introduced in Chapter 3, needed in order to find the best drug candidates with respect to their score. In terms of drug development, one might restrict their search for new therapies to N leads associated with the highest repurposing score. What is left to do is to design an appropriate recommender system to automatically test and recommend drug candidates. Recommender systems actually belong to different families of machine learning methods, since a recommender system broadly designates an algorithm which aims at predicting rating of a given user which tests a given object [START_REF] Adomavicius | Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions[END_REF]. A large part of the literature about recommender systems is motivated by commercial purposes [START_REF] Broido | The longer tail: The changing shape of Amazon's sales distribution curve[END_REF]Smith, 2010; Smith and[START_REF] Smith | Two decades of recommender systems at Amazon.com[END_REF], including works on multi-armed bandit algorithms [START_REF] Guillou | Large-scale bandit recommender system[END_REF]Preux, 2016; Mary, Gaudel, and[START_REF] Mary | Bandits and recommender systems[END_REF]. However, we will demonstrate that multi-armed bandit algorithms can actually be applied to solve drug repurposing.

As described in Chapter 3, one evaluation -that is, one call to the Boolean network-outputs a score, which is computed by comparing the attractor states reached after a drug-induced perturbation on a single patient profile, and the attractors states reached in the absence of perturbation. Computation of reachable steady states in Boolean networks, although easier than in continuous dynamical systems, 2 can still be expensive, especially as the network grows larger. Moreover, we only consider one patient profile at a time, and plots in Chapter 3 show that there is a high variability in scores across patients. In a nutshell, a single call to the Boolean network might be computationally expensive and noisy, hence the need to reducing the number of calls, especially when considering a large number of drugs to screen.

Observation

Agent

Set of arms Environment

Structured stochastic bandits

In this thesis, I consider a specific type of multi-armed bandits, 3 which are adapted to the use in real-life applications. We denote throughout the manuscript [L] := {1, 2, . . . , L -1, L} for any positive integer L ∈ N * . We consider a finite set of K arms, associated with feature vectors (X a ) a∈ [K] which belong to R d . Let us denote the feature matrix X ∈ R K×d such that the feature vectors are the rows of this matrix :

X := [X ⊤ 1 , X ⊤ 2 , . . . , X ⊤ K ].
An agent sequentially selects arms for a given number of rounds or when a specific stopping criterion is satisfied. Sampling arm

I t ∈ [K] at round t, based on past observations, 4 yields a noisy observation Y t ∈ R. Y t is a realization of ν It drawn independently from all past observations, denoted Y t ∼ ν It . Let µ a := E Y ∼νa [Y ]
be the expected reward associated with probability law ν a for any arm a ∈ [K]. The probability law ν a , for any arm a ∈ [K], is assumed fixed and unknown to the agent (assumption of stochasticity), and depends on feature vector X a (assumption of structure). In the cases studied during my PhD, I focused on zero-mean σ 2 -subgaussian additive noise, for a fixed value of σ that is known to the agent. 5 In order to reconcile all the works done on the structure of bandits, I introduce the following definition Definition 4.1.1. M-structured stochastic bandit with σ 2 -subgaussian noise. Let σ be a positive real number, and M ⊆ (R d → R) a class of functions mapping from R d to R. If µ is M-structured, then there is a deterministic function f ∈ M such that for any arm a, µ a = f (X a ). At round t, the agent then observes

Y t := f (X It ) + ψ t ,
where ψ t is a centered random variable independent from past observations, and is σ 2 -subgaussian,6 that is, satisfies for any

λ ∈ R E[exp(λψ t )] ≤ exp λ 2 σ 2 2 .
The idea of this generic definition is to put an emphasis on the structure of the expected rewards for all arms, that is, the relationship between the expected reward µ a and the feature vector X a for any arm a. In particular, the key ingredient in such a structure is that parameters of functions f ∈ M are shared across arms. This is what makes speeding up the bandit algorithm possible,7 since the set of putative models matching the observations is possibly reduced to M-structured models. What should be kept in mind is that, here, features are a fixed, known, input to the problem, and one of the main questions tackled in this PhD is about their integration in the underlying (drug repurposing) model. Three familiar structures of bandit models are presented below.

Unstructured bandits (M = (R d → R)). This means that any function mapping from R d to R can be considered, which implies that we do not actually restrict the set of possible models, since we do not exploit feature vectors. This is the most straightforward model for bandits -in regret minimization (Auer and Ortner, 2010; W. R. [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]-and was the first class of models considered for the identification of the N arms with highest expected rewards, see for instance Chen, Li, and Qiao (2017); Gabillon, Ghavamzadeh, and Lazaric (2012); Jiang, Li, and Qiao (2017); [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. Additional assumptions are often made, for instance, bounds on the value of any expected reward : for any arm a,

µ a ∈ [0, 1], or |µ a | ≤ 1.

Linear bandits (M

= {x → θ ⊤ x | θ ∈ R d }).
In this case, we know that the expected reward of an arm a is the product of a linear function of the associated feature vector. As mentioned in the previous paragraph, parameter θ is common to all arms in [K]. Such a setting has been studied in the context of regret minimization (Abeille andLazaric, 2017; Agrawal and[START_REF] Agrawal | Thompson sampling for contextual bandits with linear payoffs[END_REF], but also for best arm identification (Jedra and Proutiere, 2020; Soare, Lazaric, and Munos, 2014; Xu, Honda, and Sugiyama, 2018). We will mention this class of models in Chapter 5. Supplementary assumptions might be that feature vectors span R d , which is justified by numerical concerns related to matrix singularity.

Remark 4.1.2. Given all the expected rewards (µ a ) a∈ [K] , there is not necessarily a unique θ such that for any arm a, µ a = θ ⊤ X a , depending on the set of feature vectors (X a ) a∈ [K] . For instance, considering two arms with respective feature vectors [1, 1] ⊤ and [0.5, 0.5] ⊤ and expected rewards µ 1 = 1 and µ 2 = 0.5, both θ = [1, 0] ⊤ and θ ′ = [0, 1] ⊤ could be considered as valid parameters. Without supplementary assumptions, potentially several distinct functions in the class M can represent the same model µ (however, a function f ∈ M represent a single model). In particular, this is why I chose to parametrize expectations and probabilities with model functions in M instead of models µ themselves. This underdetermination raises questions, especially for the interpretability of the inferred parameter(s).

Generalized linear bandits (M

= {x → ℓ(θ ⊤ x) | θ ∈ R d , ℓ link function}).
Generalized linear models are known beyond the multi-armed bandit field, and comprise of a link function -denoted ℓ here-which belongs to the exponential family, i.e., ℓ is any probability distribution parametrized by some scalar γ ∈ R8 which density, with respect to a reference measure, is of the following form

P {X∼ℓ} [X = x] = h(x) exp(Γ(γ) × T (x) -A(γ)) ,
where h is a function with nonnegative values, and Γ, T , and A (log-partition function) are known functions. Such a family includes the normal, exponential, and Poisson distributions. They have already been studied extensively in the literature (Filippi et al., 2010; Li, Lu, and Zhou, 2017; Russo and Van Roy, 2013). In particular, when ℓ : y → 1 1+exp(-y) , the class of models becomes

M = x → 1 1 + exp(-θ ⊤ x) | θ ∈ R d ,
which correspond to logistic bandits [START_REF] Dumitrascu | Pg-ts: Improved thompson sampling for logistic contextual bandits[END_REF]Engelhardt, 2018; Faury et al., 2020).

The agent can estimate the expected rewards from a given arm by sampling it repeatedly, or, when exploiting feature vectors, by sampling other arms. We denote the optimal arm, that is, the arm yielding the highest expected reward, a ⋆ : µ ⋆ = µ a ⋆ := max a∈ [K] µ a . In a similar fashion, we define the N th best arm, that is, the arm which yields the N th highest reward a (N ) such that µ (N ) := N max a∈ [K] µ a . Note that there can be several of them, but we assume in the remaining of the manuscript that it is unique.9

Several types of objectives might be satisfied by the agent : for instance, the most commonly studied is the maximization of the cumulative sum of rewards in T rounds, where T is fixed, which is called regret minimization. [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multi-armed bandit problems[END_REF] In that case, the agent has to play a large number of times the estimated optimal arm, from the past observations. Sampling a suboptimal arm incurs a penalty in terms of payoff, while the agent needs to have a good idea of the performance of each arm in the set to correctly estimate the best arm : this is the well-known explorationexploitation dilemma. Inspired by the image of one-armed bandits in casinos -which gave their names to multi-armed bandits-let us try to illustrate this principle : a player enters a casino and considers a row of one-armed bandit machines, numbered from 1 to K. This player yields an average payoff µ a of winning on machine a, for any a ∈ [K]. That player then wants to win as much as possible with its fixed budget of T tokens. If they knew which was the most rewarding machine, playing it repeatedly would ensure to win the greatest average return. Of course, the owner of the casino would not disclose such an information, which drives the player to sample all machines enough so that they can rule out probably suboptimal arms, while trying to maximize their profits within their limited token budget.

Other types of objectives belong to in the field of pure exploration problems, that aim at answering a question about the set of arms, which have also received a lot of attention (Bubeck, Munos, and Stoltz, 2009; Bubeck, Wang, and Viswanathan, 2013; Degenne and Koolen, 2019; Garivier and Kaufmann, 2016; Jun and Nowak, 2016). In our setting of drug repurposing, since we are considering in silico simulations, the agent has to explore as much as possible in order to detect which molecules are the best candidates for repurposing. Consider once again the casino example. Let us assume that the player decides to sneak at night into the casino, having stolen a large number of tokens, and plays all night long the K one-armed bandit machines so that to estimate which one is the best. They cannot leave the casino with their payoff this time, but they might escape before sunset with the information about the best arm to sample the next day. This pure exploration setting is the closest to our drug repurposing use case.

Therefore, in my thesis, I studied the design of algorithms A which tackle the pure exploration problem of identifying the N best arms with as few observations as possible -called in this manuscript Top-N identification10 . At final round τ A , these algorithms return the set ŜN (τ A ), of size N , based on past observations. The number of samples τ A is a random variable which depends on a stopping rule, which ensures that each arm in ŜN (τ A ) belongs to the true set of N best arms with probability higher than 1 -δ. This is the fixed-confidence setting.

P {f } ŜN (τ A ) ⊆ S ⋆ N,f ≤ δ , where S ⋆ N,f := a ∈ [K] : f (X a ) ≥ N max i∈[K] f (X i ) .
S ⋆ N,f is the true set of N best arms (again, considered unique) in model µ associated with function f ∈ M. I drop the subscript f in the notation S ⋆ N,f when we consider a single M-structured model. One might also be interested in the set of (N, ε)-best arms, that is, the set of arms which are among the N -best arms up to some margin ε ≥ 0

S ⋆ N (ε) ≈ S ⋆ N,f (ε) := a ∈ [K] : f (X a ) ≥ N max i∈[K] f (X i ) -ε . ( 4
∀f ∈ M, P {f } [τ A ≤ C(f, δ, ε, N )] ≤ δ , (4.2) or in expectation (what is proven in Chapter 6) ∀f ∈ M, E {f } [τ A ] ≤ E(f, δ, ε, N ) , (4.3)
where C and E are some functions depending on the inputs of the problem.

In practice, as illustrated in the next sections of this chapter, in order to show that an algorithm is δ-PAC, we build a minimally "good" event which is defined as an event E such that

P {A} (E) ≥ 1 -δ and ŜN (τ A ) ⊆ S ⋆ N (ε) ⊆ E. Then it directly follows that P ŜN (τ A ) ⊆ S ⋆ N (ε) ≤ P[(E) c ] ≤ δ ,
where (E) c is the complementary event to E : given the universe Ω (i.e., the set of all possible events),

E ∩ (E) c = ∅ and E ∪ (E) c = Ω .
Typically, to prove a high-probability sample complexity bound as in Equation (4.2), we prove an upper bound on the total number of samples τ A when this event E holds, as done in Kalyanakrishnan et [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF]. The goal is to carefully craft a time-dependent event E t which probability exponentially increases as t increases (such that the complementary event seldom happens), combining events that allow the algorithm to stop and be δ-correct. Then, finding a suitable time T 0 > 0 which ensures that, for any t ≥ T 0 ,

E t ⊆ {τ A ≤ t}, yields E[τ A ] = +∞ t=0 P(τ A > t) ≤ T 0 -1 t=0 1 + +∞ t=T 0 P((E t ) c ) finite value E ≤ T 0 to upper bound +E .
Remark 4.1.5. Note that, in high probability, we do not actually end up with a true upper bound on the total number of samples needed, that is, an upper bound in the worst-case scenario. Indeed, when the "good event" does not hold anymore, it is likely that the algorithm is no longer δ-correct, and we do not control what happens in that case (i.e., the upper bound does not hold anymore). However, Kalyanakrishnan et al. (2012, Theorem 8) showed that the worst case in Top-N identification for unstructured bandit models yields an upper bound on the number of samples of order O(Kε -2 log(N /δ)) (for 0 < δ < 1 4 , 0 < ε ≤ 1/32, N ≥ 6 and K ≥ 2N ). That means that, for any bandit algorithm, we could add to the stopping rule the following condition : regardless of the satisfaction of the initial stopping criterion, if the number of samples is higher than Kε -2 log(N /δ), then the algorithm should stop and apply its decision rule.

Remark 4.1.6. In addition to the error probability and the sample complexity, we could also evaluate algorithm A on their simple regret [START_REF] Bubeck | Pure exploration in multi-armed bandits problems[END_REF]Stoltz, 2008; Gabillon, Ghavamzadeh, and[START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF]

Bandit algorithms for Top-N identification

Any algorithm for Top-N identification12 comprises of three distinct parts which fully define it. At each round t, algorithm A uses a sampling rule to select arm I t , based on past observations Y 1 , Y 2 , . . . , Y t-1 obtained from arms I 1 , I 2 , . . . , I t-1 conditioned on some internal randomization U 1 , U 2 , . . . , U t-1 . 13Then, considering the σ-algebra 14 F(t) := σ ({I s , Y s , U s | s ≤ t}) generated by all observations up to round t included, random variable I t is then F(t -1)-measurable. That is, intuitively, all the information needed to set the value of I t is contained in the subsets of random variables present in F(t -1). Algorithm A keeps sampling arms until the adaptive stopping rule is triggered at round t. As it is adaptive, this stopping rule is a stopping time with respect to the filtration F(t). This means that the time when the algorithm stops sampling is a random variable which depends on the variables in the filtration which contains all variables of interest collected at the previous time steps. At stopping time τ A , algorithm A outputs its answer ŜN (τ A ) which is computed from the decision rule. Set ŜN (τ A ) is of size N and is F(τ A )-measurable. The interactions between these three rules are illustrated in Algorithm 5.

Algorithm 5 Top-N identification. Skeleton of a bandit algorithm for Top-N identification 1: t ← 0 2: repeat 3:

# Selection of an arm to sample 4:

I t ← sampling_rule({I s , Y s , U s : s < t}) 5:
# Observation of sampled arm I t drawn from law ν It 6:

Y t ∼ ν It 7: t ← t + 1 8: until stopping_rule (t, {I s , Y s , U s : s ≤ t}) 9: # Making a decision for the Top-N set 10: ŜN (τ A ) ← decision_rule ({I s , Y s , U s : s ≤ τ A }) 11: return ŜN (τ A )

Sample complexity in bandits

As previously mentioned, the goal in the fixed-confidence setting is to design an algorithm A with an (expected) number of samples as small as possible, while keeping the error rate under the threshold δ on a given class of models M. Depending on what we want to achieve (Equation (4.3), resp. Equation (4.2)) E {•} [τ A ], resp. an upper bound on τ A , is called sample complexity, as observations are sometimes called samples. We would like to assess whether our algorithms perform well by checking out the behavior of the sample complexity depending on the expected rewards (when we know them). In particular, consider the extreme case where the difference between the expected rewards of the N th best arm and of any of the (K -N ) worst-performing arms is equal to some small value ε > 0, that is, the minimal gap in performance between the top N arms and the others. It is easy to guess that, as ε decreases, the harder it is to distinguish between these two arms. Then, a large number of observations will be needed to return the correct set of best arms (and vice-versa, had we considered one of the N best arms and the (N + 1) th best one). Hence, we already suspect that the sample complexity needed to return a δ-correct answer crucially depends on the vector of expected rewards µ. As suggested by the previous example, (expected reward) gaps to the N th or (N + 1) th will play a significant role in the analysis of structured bandit models in Chapter 5. We define the gap associated with any of the arms a ∈ [K] as follows Sample complexity bound in high probability

LUCB 292H LUCB (f ) ln H LUCB (f ) δ + 16 where H LUCB (f ) := a∈[K] max ∆ a , ε 2 -2 KLLUCB 2 exp(1)H KL (f ) log 6K δ + 2 log H KL (f ) δ where H KL (f ) := a∈[K] max ∆ 2 a , (ε/2) 2 -2 UGapE 2H UGapE (f ) log K δ + 12H UGapE (f ) log K + 2H UGapE (f ) log K δ + 6H UGapE (f ) where H UGapE (f ) := a∈[K] max ∆a+ε 2 , ε -2
Table 4.1: Prior results on the sample complexity for unstructured bandits. Sample complexity results for unstructured bandits, for any f :

∆ k := max µ k -µ (N +1) , µ (N ) -µ k .
In particular, it implies that

∆ k := µ (N ) -µ k if k ∈ S ⋆ N (ε) , µ k -µ (N +1) otherwise .
We further illustrate the importance of arm gaps by considering sample complexity results (in high probability, as in Equation (4.2)) from prior work on Top-N identification in unstructured bandits in Table 4.1.

Moreover, had we have a guess on the lower bound on the sample complexity needed to solve our pure exploration problem on any model function f ∈ M, and if we managed to upper bound the sample complexity of our algorithm on that model by some quantity not too far from that lower bound, then we could ensure that our algorithm was performant with regards to sample efficiency. There exists an instance-dependent lower bound on the minimum expected number of samples needed to make a δ-PAC decision in a pure exploration problem [START_REF] Kaufmann | On the complexity of bestarm identification in multi-armed bandit models[END_REF] pure exploration problem will be hard, that is, will need a large number of samples to output a probably correct answer. That distance between models is expressed in terms of Kullback-Leibler divergence [START_REF] Kullback | On information and sufficiency[END_REF], denoted KL, which is defined as follows Definition 4.3.2. Kullback-Leibler divergence. For a pair of probability densities ν a and α a (for a fixed arm a ∈ [K])

KL(ν a , α a ) = D KL (ν a α a ) := E {Y } ν a (Y ) log ν a (Y ) α a (Y ) .
In particular, for continuous probability densities ν a and α a with respect to Lebesgue measure,

KL(ν a , α a ) = +∞ -∞ ν a (x) log ν a (x) α a (x) dx .
It has a closed-form in the case of Gaussian distributions that is symmetric, contrary to the general definition of Kullback-Leibler divergence in Definition 4.3.2. In the remainder of the manuscript, since we consider a single class of models M at a time, we will systematically abuse notation by considering model functions

f, f ∈ M such that E {Y ∼νa} [Y ] = f (X a ) and E {Y ∼αa} [Y ] = f (X a )
E {f } [τ A ] ≥        sup ω∈△ K inf f ∈Alt(f ) a∈[K] ω a KL a (f, f ) = C ⋆ (f )        -1 log 1 2.4δ
, where Alt(f

) := f ∈ M | S ⋆ N,f (ε) ⊆ S ⋆ N, f (ε)
, and

K :=    p ∈ [0, 1] K | k∈[K] p k = 1    is the simplex on [K] .
The lower bound in Result (4.3.3) was shown to be rather tight for ε = 0 and Gaussian bandits : indeed, for Gaussian bandits with fixed variance σ 2 , the Kullback-Leibler divergence has the following closed-form

∀f, f ∈ M ∀a ∈ [K], KL a (f, f ) 2 = (f (X a ) -f (X a )) 2 2σ 2 . (4.4)
Then Garivier and Kaufmann (2016, Theorem 5) have shown that

H opt (f ) ≤ (C ⋆ (f )) -1 ≤ 2H opt (f ) where H opt (f ) := a∈[K] 2σ 2 ∆ 2 a , which is, up to a factor log(1/C ⋆ (f ))
, what is shown in Table 4.1. However, this lower bound is probably not reachable as a general rule. In order to compare algorithms to this lower bound, the concept of asymptotic optimality is defined as the ability of a δ-PAC algorithm A of recovering the optimal scaling (C ⋆ (f )) -1 in the asymptotic regime of small values of δ (i.e., δ tends to 0).

Known result 4.3.4. Asymptotic optimality for pure exploration bandit algorithms in Degenne and Koolen (2019, Theorem 11). δ-PAC

algorithm A is asymptotically optimal on M-structured models if and only if for any f ∈ M

lim inf δ→0 E {f } [τ A ] log(1/δ) = (C ⋆ (f )) -1 .
In order to prove such a result for a δ-PAC candidate algorithm A, we have to exhibit an upper bound on the expected sample complexity E {f } [τ A ] for any model function f ∈ M, which ratio to log(1/δ) tends to (C ⋆ (f )) -1 as δ tends to 0. For example, it has been shown in Degenne and Koolen (2019, Theorem 11) that Track-and-Stop (Garivier and Kaufmann, 2016, Section 3), for the best arm identification (N = 1) problem, is asymptotically optimal. 15In Chapter 6, an algorithm is proven asymptotically optimal for a specific structure which encompasses linear bandits.

Integration of bandits to drug repurposing

In my thesis, I considered the integration of multi-armed bandits in drug repurposing. First, in order to reduce the number of calls to the gene regulatory network in Chapter 3. Second, in order to control the error rate (through the formalism of δ-PAC in Definition 4.1.3). Last, in order to infer an explanable model, based on the drug signatures, which are the feature vectors associated with screened molecules. This last point was something rather novel at the beginning of my PhD, since the few papers about Top-N identification were dealing with unstructured bandits (Chen, Li, and Qiao, 2017; Gabillon, Ghavamzadeh, and Lazaric, 2012; Kalyanakrishnan et al., 2012; Kaufmann and Kalyanakrishnan, 2013). Figure 4.2 illustrates the connection between Boolean networks and multi-armed bandits for drug repurposing. Chapter 5

BANDIT ALGORITHM

Sequential identification in linear models

The most crucial question, with respect to the state-of-the-art at the start of my PhD, was about the integration of feature vectors into the bandit model. Inspired by prior works on structured best-arm identification and regret maximization (Auer, 2002; Fiez et al., 2019; Li, Chu, et al., 2010; Xu, Honda, and Sugiyama, 2018), I first considered the simpler setting of linear bandits : bandits where, for any arm a ∈ [K], the associated expected reward vector µ a linearly depends on feature vector X a ∈ R d , with a parameter θ shared by all arms

∃θ ∈ R d ∀a ∈ [K], µ a = X ⊤ a θ .
In the terminology introduced in Chapter 4, such models have structure

Related work

Roughly two types of fixed-confidence algorithms have been proposed for Top-N identification in an unstructured bandit :

-based on adaptive sampling such as LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], or UGapE [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF].

-based on uniform sampling and eliminations (Chen, Li, andQiao, 2017; Kaufmann and[START_REF] Kaufmann | Information complexity in bandit subset selection[END_REF].

At the time of publication, the only efficient algorithms for linear bandits were proposed for the best arm identification (BAI) problem, which corresponds to N = 1. This setting, first investigated by [START_REF] Soare | Best-arm identification in linear bandits[END_REF], recently received a lot of attention. An efficient adaptive sampling algorithm called LinGapE was proposed by [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF]. Subsequent elimination-based works such as Fiez et al. ( 2019) [START_REF] Hassidim | An optimal elimination algorithm for learning a best arm[END_REF] sought to achieve the minimal sample complexity. In particular, the LinGame algorithm of [START_REF] Degenne | Gamification of pure exploration for linear bandits[END_REF] was shown to exactly achieve the problem-dependent sample complexity lower bound for linear BAI, in the regime in which δ goes to zero (Result 4.3.4). We note that, in principle, LinGame can be used for any pure exploration problem in a linear bandit, which includes Top-N identification for N > 1. However, this algorithm relies on a game theoretic formalism which needs the computation of a best response for Nature in response to the player's selection ; a computable expression of this strategy was (at that time) not available to our knowledge for Top-N (N > 1). Besides, naively computing the information-theoretic lower bound for Top-N identification is computationally hard. 2 These remarks led us to investigate efficient adaptive sampling algorithms for Top-N identification (N ≥ 1) in linear bandits, which are still missing in the literature, instead of trying to propose asymptotically optimal algorithms, as done in linear BAI. First, by carefully looking at existing adaptive sampling bandits for unstructured Top-N identification, we propose a generic algorithm structure based on gap indices, called gap-index focused algorithms (GIFA), which encompasses existing adaptive algorithms for unstructured Top-N identification and linear BAI. This structure allows a higher order and modular understanding of the learning process, and correctness properties can readily be inferred from a partially specified bandit algorithm. It allows us to define two interesting new algorithms, called N -LinGapE and LinGIFA. Then, we present a unified sample complexity analysis of a subclass of GIFA which comprises existing methods. Finally, we empirically show that N -LinGapE and LinGIFA perform better than their counterparts for unstructured bandits, which showcases the importance of side information in sample efficiency.

General structure for efficient algorithms

As mentioned in Definition 4.1.1, we consider one-dimensional subgaussian distributions of fixed variance σ 2 > 0, which are uniquely defined by their vector of expected rewards per arm. At each round t, the agent observes Y t ∼ ν It from pulled arm I t . We solve Top-N identification with error rate δ ∈ (0, 1) and any margin ε ≥ 0, that is, for a model function

f ∈ M lin find S ⋆ N (ε) := a ∈ [K] | f (X a ) ≥ N max i∈[K] f (X i ) -ε .

Estimation of the linear structure

The learner needs to estimate the unknown linear parameter θ ∈ R d , which can be done via a (regularized) least-squares estimator, defined as follows.

For any arm a ∈ [K] and at round t, we define the number of times a is sampled up to round t included as

N a (t) := s≤t 1(I s = a) ,
and the κ-regularized design matrix and least-squares estimate of θ as

V κ (t) := κI d + a∈[K] N a (t)X a X ⊤ a and θ(t) := V κ (t) -1 s≤t Y s X It . Note that matrix σ 2 V κ (t) -1
can be interpreted as the posterior covariance in a Bayesian linear regression model, in which the covariance of the prior is σ 2 /κI d .

Introduction to gap indices

Any adaptive sampling-based algorithm mentioned in Section 5.1 -along with the GIFA family that we introduce-crucially relies on estimating the gaps (∆ i,j ) (i,j) between pairs of arms (i, j)

∈ [K] 2 for any model f ∈ M lin ∀i ∈ [K] ∀j ∈ [K], ∆ i,j := f (X i ) -f (X j ) .
One way to achieve this estimation is by building upper confidence bounds (UCBs) on these quantities ; i.e., indices (U i,j (t)) i,j,t such that the following inequalities all hold with high probability

∀i ∈ [K] ∀j ∈ [K] ∀t > 0, ∆ i,j ≤ U i,j (t) .
For this purpose, we respectively introduce the empirical arm-individual mean and the empirical arm-pairwise gap at round t

∀a ∈ [K], µ a (t) := θ(t) ⊤ X a and ∀(a, b) ∈ [K] 2 , ∆a,b (t) := µ a (t) -µ b (t) .
A first option to build UCBs on pairwise gaps (∆ i,j ) i,j consists in building individual confidence intervals -that is, lower and upper bounds for ∆ i,j for any i, j-on the mean of each arm a ∈ [K]. These (symmetrical) confidence intervals are of the form { µ a (t) ± W a (t)}, where W a (t) := σ 2 T δ (t) X a ( V κ (t)) -1 , for some threshold function T δ (•) to be specified later. • M is the Malahanobis norm for any positive definite matrix M , such that, for any vector x, x M := √

x ⊤ M x. For a well-chosen T δ (•), and for any pair of arms (i, j) and round t, the following quantity

B ind i,j (t) := ∆i,j (t) + W ind i,j (t) where W ind i,j (t) := W i (t) + W j (t)
is an upper bound on ∆ i,j ("individual" UCB). Yet, using the linear model, one can also directly build a "paired" UCB on the difference

B pair i,j (t) := ∆i,j (t) + W pair i,j (t) where W pair i,j (t) := σ 2 T δ (t) X i -X j ( V κ (t)) -1 .
Both constructions lead to symmetrical bounds, called gap indices, of the general form B i,j (t) := ∆i,j (t) + W i,j (t). To sum it up, Definition 5.2.1. (Symmetrical) gap index. Let us define B •,• (•) = (B i,j (t)) i,j,t so that B i,j (t) := ∆i,j (t)+W i,j (t) for any arms (i, j), where

W •,• (•) ∈ ([K]×[K]×N * → R + ). B •,• (•) is a gap index if, with probability 1 -δ ∀i ∈ [K] ∀j ∈ [K] ∀t > 0, ∆ i,j ≤ B i,j (t) .
Remark 5.2.2. As shown in Section 5.2, in order to ensure that an algorithm using gap indices is δ-PAC, we only need those upper confidence bounds to hold with probability 1 -δ for specific gaps, namely the gaps between the true good arms and arms which are not in the set of N best arms up to ε

∀j ∈ (S ⋆ N (ε)) c ∀i ∈ S ⋆ N ∀t > 0, ∆ i,j ≤ B i,j (t) .
However, to upper bound the sample complexity, we will need Definition 5.2.1.

This generic form of gap index allows us to study and compare in a modular way the gain (or loss) in performance incurred by different definitions. For instance, let us compare individual and paired indices. Paired indices may indeed increase sample-efficiency. First, it directly follows from the triangular inequality for the Mahalanobis norm

• ( V κ (t)) -1 that ∀(i, j) ∈ [K] 2 ∀t > 0, W pair i,j (t) = W i,j (t) ≤ W i (t) + W j (t) = W ind i,j (t) .
Therefore, if both types of gap indices use the same threshold function T δ (•), paired indices are smaller

B pair i,j (t) = ∆i,j (t) + W pair i,j (t) ≤ ∆i,j (t) + W ind i,j (t) = B ind i,j (t) .
Moreover, the following lemma implies that paired or individual indices using arm features can yield smaller bounds on the gaps than individual indices which do not exploit the structure. This is proven in Réda, Kaufmann, and Delahaye-Duriez (2021, Lemma 2) by induction, combining the Sherman-Morrison formula and the Cauchy-Schwartz inequality.

Remark 5.2.3. Note that, in the unstructured case, one can consider the vectors of the canonical basis of R K as feature vectors :

X a = (1(i = a)) i∈[K] .
As a consequence, for any round t, V κ (t) = diag(N 1 (t)+κ, N b (t)+κ, . . . , N K (t)+κ), and X a ( V κ (t)) -1 = 1/ N a (t) + κ.

Lemma 5.2.4. Individual versus paired gap indices. The following inequality holds

∀t > 0 ∀a ∈ [K] ∀y ∈ R d , y ( V κ (t)) -1 ≤ y 2 / N a (t) X a 2 2 + κ .
In particular, when κ = 0, for any t > 0, and arms a, b

∈ [K] X a -X b ( V κ (t)) -1 ≤ X a ( V κ (t)) -1 + X b ( V κ (t)) -1 ≤ 1/ N a (t) + 1/ N b (t) .
As we will see in the next sections, the gap index plays an important part in the stopping rule, and the tighter it is, the faster the stopping time will be reached. We will show in Section 5.4 that empirically, the choice of paired indices over individual ones considerably speeds up the sampling phase. We will describe later how to build these gap indices in practice.

Remark 5.2.5. For any symmetrical gap index B •,• (•), we actually get a full confidence interval for free. Indeed, for any pair of arms i, j and round t, the following inequalities hold

-B j,i (t) ≤ ∆ i,j ≤ B i,j (t) .

Gap-index focused algorithms (GIFA)

Algorithm 6 Structure of GIFA. GIFA for (ε, δ)-PAC Top-N identification, with four undefined rules stopping, compute_Jt, compute_bt and selection and gap indices. 1: t ← 1 2: # For unstructured bandits 3: initialization() 4: while stopping (B i,j (t)) i,j∈[K] 2 ; ε, t > ε do 5:

# J(t) is the set of estimated N best arms at t 6:

J(t) ← compute_Jt((B i,j (t)) i,j , ( µ i (t)) i ) 7: # b(t) is the estimated N th best arm a (N ) and c(t) its challenger at t 8: b(t) ← compute_bt J(t), (B i,j (t)) i,j and c t ← arg max a̸ ∈J(t) B a,b(t) (t) 9:
# Selection of the sampled arm I t 10:

I t ← selection b(t), c(t); V κ (t -1) 11: Y t ∼ ν It 12:
Update design matrix V κ (t), means ( µ i (t)) i (if needed), (B i,j (t + 1)) i,j 13: t ← t + 1 14: end while 15: ŜN (τ GIFA ) ← J(τ GIFA ) 16: return ŜN (τ GIFA )

The gap indices introduced in Subsection 5.2 allow us to introduce a generic family of algorithms that encompasses the state-of-the-art in adaptive sampling-based algorithms for BAI and Top-N identification. Using this generic structure, we can easily extend an efficient BAI algorithm, LinGapE, to Top-N identification : this extension is called N -LinGapE. We also define another algorithm, called LinGIFA, for Top-N identification which only needs to keep and track their (symmetrical) gap indices, regardless of their actual definition. We will discuss the consequences of this flexibility later in this section. In this section, unless specified, we consider any type of symmetrical gap index, as defined in Definition 5.2.1. [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF], LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF]. Greedy and optimized sampling rules are defined in the main text.

and to select the two most ambiguous arms : b(t) ∈ J(t), which can be viewed as a guess for the N th -best arm a (N ) , and a challenger c(t) ∈ J(t). c(t) is defined as a potentially misassessed µ (N ) . c(t) is defined as having the largest possible gap index to b(t) among the estimated (K -N ) worst arms

c(t) := arg max c∈[K] B c,b(t) (t) .
The idea of using two ambiguous arms goes back to the LUCB algorithm [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] for Top-N identification in unstructured bandits. 3 Selected arm at round t I t should help discriminate between b(t) and c(t). A naive idea is to either draw b(t) or c(t), but alternative selection rules will be discussed later. At the end of the learning phase, at stopping time τ GIFA , the final set ŜN (τ GIFA ) is equal to J(τ GIFA ). The part denoted initialization is an optional phase where all arms are sampled once. 4 We assume that ties are randomly broken. The yet unspecified parts of the bandit algorithm remain in the choice of the rules compute_Jt (definition of J(t)), compute_bt (definition of b(t)), selection rule (choice of I t ), and the definition of the stopping rule stopping. Some of these rules may also rely on the gap indices.

Rules compute_Jt, compute_bt. Table 5.1 presents some instanciations of the GIFA structure presented in Algorithm 6. Two types of algorithms can be distinguished :

-the LUCB-GIFA type, which most intuitively computes J(t) as the empirical set of µ (N ) arms and b(t) as the arm in J(t) closest to [K] \ J(t) in terms of (estimated) expected reward. It comprises of algorithms LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF], and N -LinGapE.

-the Gap-GIFA type. This subclass of GIFA includes the unstructured bandit algorithm UGapE [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF] and LinGIFA.

To better understand the rules associated with Gap-GIFA, we first prove the following result for any gap index Lemma 5.2.6. Upper bound on the gap to a

(N ) . If B •,• (•) is a gap index in the sense of Definition 5.2.1, then with probability 1 -δ ∀t > 0 ∀a ∈ [K], µ (N ) -µ a ≤ N max b∈[K] B b,a (t) .
Proof. Except for some degenerate cases where two arm features are equal and the past observations made from both arms are exactly the same, we can assume that, at fixed arm a ∈ [K] and round t > 0, values (B b,a (t)) b∈ [K] are distinct. Then, assume towards contradiction that there exists a round t > 0 and an arm i ∈

[K] such that N max j∈[K] B j,i (t) < µ (N ) -µ i = ∆ a (N ) ,i .
Then, using the definition of gap index in Definition 5.2.1, with probability

greater than 1 -δ, for any k ∈ S ⋆ N = N arg max a∈[K] µ a B k,i (t) ≥ ∆ k,i ≥ ∆ a (N ) ,i > N max j∈[K] B j,i (t) .
That means that at least N distinct values of (B j,i (t)) j∈ [K] are strictly greater .

Remark 5.2.7. In UGapE, which only uses individual indices, the following result [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF], Lemma 1) was shown

∀t > 0 ∀a ∈ [K], µ (N ) -µ a ≤ N max b̸ =a B b,a (t) .
However, it only holds for general gap indices when a ∈ S ⋆ N (see the previous proof). Nonetheless, for any Gap-GIFA A, at any time t

< τ A N max c̸ =b(t) B c,b(t) (t) > 0 , meaning that if b(t) ∈ S ⋆ N , then N max c̸ =b(t) B c,b(t) (t) > 0 ≥ µ (N ) -µ b(t)
which makes the lemma hold in that important case, which justifies the stopping rule, as shown below.

Then, in Gap-GIFA, at round t, the set J(t) is the set of arms which true expected rewards are more likely to minimize the (signed) distance to µ (N ) . In a similar fashion, arm b(t) is the arm which true expected reward µ b(t) is likely to maximize the distance to µ (N ) among arms in J(t). Then b(t) is a proxy for arm a (N ) . Indeed, using the definition of b(t) in LinGIFA, and Lemma 5.2.6 max

j∈J(t) N max i̸ =j B i,j (t) = N max i̸ =b(t) B i,b(t) (t) ≥ µ (N ) -µ b(t) .
Stopping times. We restrict our attention to two stopping times already proposed for the LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] and UGapE [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF] algorithms, respectively

τ LUCB := inf t ∈ N * | B c(t),b(t) (t) ≤ ε and τ UGapE := t ∈ N * | max j∈J(t) N max i̸ =j B i,j (t) ≤ ε .
Why do these stopping times work? Let us deconvolute their expression.

τ LUCB relies on quantity B c(t),b(t) (t), which is defined in all algorithms using stopping time τ LUCB as

B c(t),b(t) (t) = max j∈J(t) max i̸ ∈J(t) B i,j (t) ≥ max j∈J(t) max i̸ ∈J(t) ∆ i,j = max i̸ ∈J(t) µ i -min j∈J(t) µ j ,
by definition of b(t) and c(t), and using the fact that B

•,• (•) is a gap index. If the algorithm correctly identifies J(τ LUCB ) = S ⋆ N (ε) at stopping time, then ε ≥ B c(τ LUCB ),b(τ LUCB ) (τ LUCB ) ≥ µ (N +1) -µ (N ) .
So stopping time τ LUCB actually tracks the value µ (N +1) -µ (N ) , that is, the separation (i.e., the minimum gap) between arms in J(t) and in [K] \ J(t) at each round t.

What about τ UGapE ? Using Gabillon, Ghavamzadeh, and Lazaric (2012, Lemma 1) for individual indices at stopping time τ UGapE , if

J(τ UGapE ) = S ⋆ N (ε), then ε ≥ max j∈S ⋆ N (ε) N max i̸ =j B i,j (τ UGapE ) ≥ µ (N ) -min j∈S ⋆ N (ε) µ j = 0 ,
that is, stopping time τ UGapE tracks the distance of the worst-performing arm in J(t) to a (N ) at each round t. Now, intuitively, since this distance can reach 0 -contrary to what is tracked by τ LUCB -it seems that this stopping rule τ UGapE stops earlier than τ LUCB , given the same past observations. This claim can be proven for any gap index Lemma 5.2.8. τ UGapE stops earlier than τ LUCB . For any t > 0, for any subset J ⊆ [K] of size N , for any j ∈ J, and any values

(B i,j (t)) i,j N max i̸ =j B i,j (t) ≤ max i̸ ∈J B i,j (t) , which implies max j∈J N max i̸ =j B i,j (t) ≤ max j∈J max i̸ ∈J B i,j (t) .
Proof. Indeed, for any round t > 0 and arm j ∈ J

N max i̸ =j B i,j (t) = min S⊆[K] |S|=N -1 max i̸ ∈(S∪{j}) B i,j (t) ,
since the set S matching the outer bound is

[N -1] arg max i∈[K]
B i,j (t), meaning that we consider the maximum value over the set of (B i,j (t)) t>0,i∈ [K] from which the N -1 largest values (and B j,j (t), if j does not already belong to

[N -1] arg max i∈[K] B i,j (t)) are removed. Then consider S = J \ {j} ⊆ [K] of size N -1 (since j ∈ J). Then N max i∈[K] B i,j (t) ≤ max i̸ ∈((J\{j})∪{j}) B i,j (t) = max i̸ ∈J B i,j (t) .
We will empirically compare these two stopping rules in Section 5.4.

Selection rule.

Besides the gap indices, this is the part where all information about the structure is concentrated. Regarding the selection rules shown in Table 5.1, Gap-GIFA algorithms select the least sampled arm among b(t) and c(t), which coincides with the following rule that we propose for a general setting, that is, at round t

I t ← arg max a∈{b(t),c(t)} W a (t) , (largest variance) (5.1)
which is still defined even when considering paired indices. In the original version of LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], both arms b(t) and c(t) are sampled at time t, but the analysis that we propose in this paper obtains similar guarantees for LUCB using the largest variance rule, so we only consider this selection rule in the remainder of the chapter. In LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF], authors propose two different selection rules to possibly sample another arm that would reduce at round t the variance on the estimation of gap ∆ c(t),b(t)

I t ← arg min a∈[K] X c(t) -X b(t) ( V κ (t-1)+XaX ⊤ a ) -1 , (greedy) (5.2) and I t ← arg max a∈[K] ω ⋆ a (b(t),c(t))>0 N a (t) ω ⋆ (b(t), c(t)) 1 |ω ⋆ a (b(t), c(t))| , (optimized) (5.3)
where ω ⋆ (b(t), c(t)) ∈ R K is a minimizer to the following optimization problem

min ω∈R K ω 1 s.t. X b(t) -X c(t) = ωX .
(5.4)

As explained by [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF], these two rules are meant to bring the empirical proportions of arm selections close to an optimal design (ω 1 , ω 2 , . . . , ω K ) which asymptotically minimizes quantity X b(t) -

X c(t) ( a∈[K] ωaXaX ⊤ a ) -1
. This quantity is present in the lower bound for best arm identification in linear bandits (Fiez et al., 2019).

Novel algorithms.

As previously mentioned, we introduce two new algorithms : an extension of LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF] to Top-N identification named N -LinGapE, and a new algorithm of type Gap-GIFA named LinGIFA, with a completely new rule for the computation of arm b(t). The strength of LinGIFA is that it is completely defined in terms of gap indices, and, as such, may be improved by deriving tighter bounds on the gaps. Moreover, we emphasize that, provided that the regularizing constant κ in the design matrix V κ (t) is positive at any round t, both LinGapE and LinGIFA can be run without an initialization phase. This permits to avoid the initial sampling cost when the number of arms is large that was noticed by Fiez et al. ( 2019). However, both algorithms still need to compute maximizers or minimizers over the whole set of arms during the learning rounds (for instance, when computing set J(t)). This might be mitigated by considering only a subset of arms when computing these rules, as mentioned in the experimental part of [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF].

Theoretical guarantees in GIFA

In this section, we present theoretical guarantees on GIFA. The fact that these algorithms are (ε, δ)-PAC (Definition 4.1.4) is a consequence of generic correctness guarantees that can be obtained for partially specified GIFA algorithms, which rely on the definition of gap indices (Definition 5.2.1). We further analyze the sample complexity of LUCB-GIFA algorithms, for instance N -LinGapE, in a unified proof.

Correctness analysis

We justify that the two stopping rules τ LUCB and τ UGapE introduced in Section 5.2 lead to (ε, δ)-PAC algorithms, in the sense of Definition 4.1.4. This holds provided the following condition on the gap indices Definition 5.3.1. Good gap indices. Let us denote

E GIFA := t>0 i̸ ∈S ⋆ N (ε) j∈S ⋆ N (ε) {B j,i (t) ≥ ∆ j,i } .
Then a good choice of gap indices (B i,j (t)) i,j,t>0 satisfies P[E GIFA ] ≥ 1 -δ.

We will show that such a choice of good gap indices exist, and any symmetrical gap index as in Definition 5.2.1 will satisfy this constraint. First, we observe that on event E GIFA , for any GIFA A (Algorithm 6), both stopping rules τ LUCB and τ UGapE output a (ε, δ)-correct answer.

Theorem 5.3.2. On event E GIFA , if A is from the family GIFA, then

LUCB-GIFA. If b(t) := arg max j∈J(t) max i̸ ∈J(t) B i,j (t), then J(τ LUCB ) = ŜN (τ A ) ⊆ S ⋆ N (ε). Gap-GIFA. If b(t) ∈ J(t), then J(τ UGapE ) = ŜN (τ A ) ⊆ S ⋆ N (ε).
Proof. Assume towards contradiction that at stopping time τ A ∈ {τ LUCB , τ UGapE }, there exists an arm b ∈ J(τ A ) ∩ (S ⋆ N (ε)) c (meaning that there is a mistake in the recommendation). If there is another arm c ∈ S ⋆ N such that B c,b (τ A ) ≤ ε,5 then using event E GIFA and c ∈ S ⋆ N (ε)

µ (N ) -µ b ≤ ∆ c,b ≤ B c,b (τ A ) ≤ ε =⇒ µ (N ) -ε ≤ µ b ,
which is a contradiction to the fact that b ∈ S ⋆ N (ε). We will actually show that such an arm c always exists. Assuming towards contradiction that it does not exist, and introducing set

C N (b) := {a ∈ [K]\{b} | B a,b (τ A ) > ε}, the following two claims hold ∀c ∈ S ⋆ N , B c,b (τ A ) > ε =⇒ S ⋆ N ⊆ C N (b) , (5.5) ∀c ∈ S ⋆ N , B c,b (τ A ) > ε and |S ⋆ N | = N =⇒ N arg max a̸ =b B a,b (τ A ) ∈ C N (b) .
(5.6)

Let us split the proof in two parts for each stopping rule LUCB-GIFA (τ A = τ LUCB ). Using the definition of τ LUCB , c(t), b(t), and b ∈ J(τ LUCB ), it holds that for any c ∈ J(τ LUCB )

B c,b (τ LUCB ) ≤ max j∈J(τ LUCB ) max i̸ ∈J(τ LUCB ) B i,j (τ LUCB ) = B c(τ LUCB ),c(τ LUCB ) (τ LUCB ) ≤ ε . Then J(τ LUCB ) c ⊆ ([K]-{b})\C N (b). Then, by Equation (5.5), S ⋆ N ∩ J(τ LUCB ) c =
∅, which means S ⋆ N ∩ J(τ LUCB ) = ∅ since both sets are not empty, of size N , which implies that S ⋆ N = J(τ LUCB ). Then (i,j) and round t > 0, gap index B i,j (t) is of the form ∆i,j (t)+σ 2 T δ (t) Wi,j (t) , where Wi,j (t) :=

J(τ LUCB ) ∩ (S ⋆ N (ε)) c = S ⋆ N ∩ (S ⋆ N (ε)) c = ∅ , which is a contradiction to the fact that b ∈ J(τ LUCB ) ∩ (S ⋆ N (ε)) c . Gap-GIFA (τ A = τ UGapE ). Using the definition of τ UGapE , and b ∈ J(τ UGapE ) N max a̸ =b B a,b (τ UGapE ) ≤ max j∈J(τ UGapE ) N max i̸ =j B i,j (τ UGapE ) ≤ ε ,
X i -X j ( V κ (t)) -1 (paired) , X i ( V κ (t)) -1 + X j ( V κ (t)) -1 (individual) ,
and

T δ (t) := 2 ln 1 δ + d ln 1 + (t + 1)L 2 κd + √ κ σ S , with L := max a∈[K] X a 2 and S ∈ R + is such that θ 2 ≤ S, then B •,• (•) is a good
gap index in the sense of Definition 5.3.1.

Proof. The proof follows from the fact that

t∈N * θ(t) -θ V κ (t) ≤ σ 2 T δ (t) ⊆ E GIFA ,
together with Kaufmann (2014, Lemma 4.1) which yields

P ∀t ∈ N * , θ(t) -θ V κ (t) ≤ σ 2 T δ (t) ≥ 1 -δ .
For paired indices, the inclusion follows from the fact that, for any pair of arms (i, j) and round t > 0 ∆i,j (t)

-∆ i,j = ( θ(t) -θ) ⊤ (X i -X j ) ≤ (1) θ(t)-θ V κ (t) X i -X j ( V κ (t)) -1 ≤ (2) Wi,j (t) ,
where (2) is either an equality for paired indices, or the application of the triangle inequality for individual indices.

(1) is shown by proving that, as a general rule, for any pair of vectors x, y ∈ R d and positive definite matrix Σ

x ⊤ y ≤ x Σ y Σ -1 .
Indeed, using the Cauchy-Schwarz inequality

x ⊤ y = (Σ ⊤ x) ⊤ Σ -1 y ≤ Σ ⊤ x 2 Σ -1 y 2 = x Σ y Σ -1 .
Remark 5.3.4. Note that L can easily be computed, since the agent has access to all the feature vectors. In order to provide a good value for S, if we know an upper bound M on the maximum absolute value of any of the expected rewards (µ a ) a i n[K] (which is slightly easier to obtain depending on the application case), then using the fact that µ a = θ ⊤ X a for any arm a ∈

[K] ∀a ∈ [K], |µ a | = θ ⊤ X a ≤ L × S ≤ M ,
and then use S = L/M as a proxy.

Unified sample complexity analysis

We derive below a high-probability upper bound on the sample complexitythat is, on the maximum value of τ GIFA -of the subclass LUCB-GIFA, combined with different selection rules, and a conjecture on the sample complexity upper bound for Gap-GIFA algorithms. More precisely, we upper bound the sample complexity on event

E := t>0 i∈[K] j∈[K] (∆ i,j ∈ [-B j,i (t), B i,j (t)]) , ( 5
τ LUCB ≤ inf u ∈ R * + | u > 1 + H LUCB-GIFA (f )(T δ (u)) 2 ,
where, depending on the selection rule Largest variance rule (Eq. (5.1))/sampling both b(t) and c(t).

H LUCB-GIFA (f ) := 4σ 2 a∈[K] max ε, ε + ∆ a 3 -2
, Optimized rule (Eq. (5.3)).

H LUCB-GIFA (f ) := σ 2 a∈[K] max i,j∈[K] 2 |ω ⋆ a (i, j)| max ε, ε+∆ i 3 , ε+∆ j 3 2 ,
where ω ⋆ (i, j) is the minimizer of Problem (5.4) for any pair of arms (i, j).

We compare all these claims to known results from the literature in Table 5.2, in particular, with regard to their complexity constant H GIFA (f ) for

f ∈ M lin := x → θ ⊤ x | θ ∈ R d .
We note that all GIFA algorithms have a scaling which is roughly of order O( a∈[K] (ε + ∆ a ) -2 ) when using the largest variance selection rule, or, interchangeably, the selection of both arms b(t) and c(t) at each round t. This is on par with what is observed in unstructured bandits in prior works.

Moreover, more sophisticated rules -namely, greedy and optimized selection rules-allow a possibly smaller scaling which might account for their empirical performance compared to the more naive largest variance rule. However, at fixed arm a, no problem-dependent upper bound is known on quantity |ω ⋆ a (i, j)| for any arms (i, j), where ω ⋆ (i, j) is solution to Problem (5.4), which intervenes in the complexity constant for the optimized selection rule. If it exists, let us denote it Γ a (X). Then the associated constant would be upper-bounded by a quantity of order O σ 2 max b∈

Algorithm Upper bound

Unstructured inf {u

∈ R * + | u > 1 + H uns (f )(T δ (u)) 2 } LUCB H uns (f ) := 2 a max (ε/2, ∆ a ) -2 UGapE H uns (f ) := 2 a max (ε, (ε + ∆ a )/2) -2 GIFA ⋆ inf u ∈ R * + | u > 1 + H lin (f )(T δ (u)) 2 Largest var. H lin (f ) := 4σ 2 a max (ε, (ε + ∆ a )/3) -2 Optimized H lin (f ) := σ 2 a max i,j∈[K] 2 |ω ⋆ a (i, j)| (max (ε, (ε + ∆ i )/3, (ε + ∆ j )/3)) -2 Greedy ⋆ H lin (f ) := σ 2 a max i,j∈[K] 2 ∥X i -X j ∥ 2 2 ∥Xa∥ 2 2 (max (ε, (ε + ∆ i )/3, (ε + ∆ j )/3)) -2
[K] Γ b (X)K max a (ε + ∆ a ) -2 . If 1 σ 2 max b∈[K]
Γ b (X), then this quantity depends on something indeed smaller than the sample complexity of unstructured bandit algorithms. This bound for the optimized selection rule is similar to what was derived in the original paper of LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF] for best arm identification. 6For the conjectured bound for the greedy rule, consider the following ratio between the ℓ 2 norms of feature vectors at fixed arm a ∈

[K] 7 Γ ′ a (X) := max i,j∈[K] 2 X i -X j 2 X a 2 ≤ 2L( X a 2 ) -1 .
Th associated complexity constant is then upper bounded by a factor

O σ 2 max b∈[K] Γ ′ b (X)K max a (ε + ∆ a ) -2 .
Sketch of proof. The proof of Theorem 5.3.5 generalizes and extends the proofs for unstructured and linear Top-N identification, with both paired and individual gap indices. Only a big picture of the proof is provided. The full proof is contained in the supplementary part in Réda, Kaufmann, and Delahaye-Duriez (2021). Appendix 12 comprises of the computations that guided Conjectures 5.3.9 and 5.3.10. As mentioned in Table 5.1, the LUCB-GIFA type of algorithms comprises of the following two rules : at round t J(t) := arg max j∈ [K] µ a (t) (compute_Jt) and b(t) := arg max

j∈J(t) max i̸ ∈J(t) B i,j (t) (compute_bt).
This class of algorithms includes LinGapE [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF] for best arm identification, our proposal N -LinGapE for Top-N identification, but also LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. The key ingredient in the proof is a lemma that gives an upper bound for any round t on the stopping quantity B c(t),b(t) (t). This lemma holds for any gap index satisfying Definition 5.2.1

Lemma 5.3.6. Upper bound on the stopping quantity (LUCB-GIFA).

On event E, for any round t > 0

B c(t),b(t) (t) ≤ min -max(∆ b(t) , ∆ c(t) ) + 2W c(t),b(t) (t), 0 + W c(t),b(t) (t) .
This result is shown by case disjunction on the membership of b(t) and c(t) to S ⋆ N (ε). This is a counterpart to Lemma 4 in Xu, Honda, and Sugiyama (2018), but does not require |J(t)| = 1 at round t > 0, notably by noticing that, by definition of b(t) and c(t), for any N ≥ 1,

B c(t),b(t) (t) = max j∈J(t) max i̸ ∈J(t) B i,j (t) .
In order to get the upper bound in Theorem 5.3.5 using the optimized rule (Equation (5.3)), one can straightforwardly apply Xu, Honda, and Sugiyama (2018, Lemma 1) to the inequality in Lemma 5.3.6. For the selection rule which either selects both b(t) and c(t), or the largest variance rule (Equation (5.1)), by combining Lemma 5.3.6 with the definition of the stopping time τ LUCB , we obtain the following upper bound on the number of times N It (t) that arm I t has been sampled up to round t included, at any round t < τ LUCB Lemma 5.

Upper bound on the number of selections (LUCB-GIFA, largest variance rule). For any round

t > 0, t < τ LUCB N It (t) ≤ 4σ 2 (T δ (t)) 2 max ε, ε + ∆ It 3 .
Finally, we apply the following result Lemma 5.3.8. Inversion lemma. Let f : [K] × (0, 1) × N * → R * + be a nondecreasing function in its last argument, and I t the set of sampled arms at exactly round t. Let E be an event such that for any t < τ A , for any δ ∈ (0, 1), there is an arm δ, t). Then, on event E,

I t ∈ I t such that N It (t) ≤ f (I t ,
τ A ≤ inf    u > 0 | u > 1 + a∈[K] f (a, δ, t)    .
The proof for both lemmas is available in Réda, Kaufmann, and Delahaye-Duriez (2021). I also propose a few conjectures present in Table 5.2, as listed below Conjecture 5.3.9. Sample complexity of Gap-GIFA. On event E, stopping time τ UGapE satisfies on model function

f ∈ M lin τ UGapE ≤ inf u > 0 | u > 1 + H Gap-GIFA (f )(T δ (u)) 2 ,
where, depending on the selection rule Largest variance rule (Eq. (5.1))/sampling both b(t) and c(t).

H Gap-GIFA (f ) := 4σ 2 a∈[K] max ε, ε + ∆ a 3 -2 .
Conjecture 5.3.10. Greedy rule (Eq. (5.2)).

On event E and model function f ∈ M lin , using the same notations as in Theorem 5. [START_REF] Cheng | Systematic quality control analysis of LINCS data[END_REF].5

H LUCB-GIFA (f ) := σ 2 a∈[K] max i,j∈[K] 2 X i -X j 2 2 X a 2 2 max ε, ε + ∆ i 3 , ε + ∆ j 3 -2
.

Application to drug repurposing

We compared the empirical performances of both GIFA proposals -namely, N -LinGapE and LinGIFA-to the two adaptive sampling algorithms for unstructured models in GIFA, that is, LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] and UGapE [START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF]. These algorithms were applied to a drug repurposing instance created using the drug scoring in Chapter 3 on 21 drugs -comprised of 10 antiepileptics and 11 proconvulsants, that is, that trigger seizures. In practice, the scores for each drug and patient profile have been obtained prior to the experiments, and saved in a matrix.

When an agent asks for the evaluation of a given arm, a patient index is sampled at random, and the score related to the arm and this patient is returned. At first, drug signatures directly computed from LINCS L1000, as described in Chapter 3, were considered as feature vectors in R 194 . However, the underlying model is very far from being linear, and this fact incurred empirical error rates higher than δ. The next chapter (Chapter 6) is actually motivated by this issue. In the meanwhile, in order to test our theoretical results on GIFA, a feature transformation procedure is performed on the drug signatures, as done in Papini et al. (2021, Appendix F.4). This procedure only transforms the feature vectors such that the resulting model is -roughlylinear. This procedure goes as follows : a linear model is extracted from the data by first fitting a neural network that regresses a linear model from each feature vector X a to score µ a for any arm a. Then, the activations from the last layer are selected as feature vectors (of dimension d = 15), and the associated network parameters constitute the linear parameter θ. The ℓ ∞ norm of the difference between the predictions of the resulting linear model and the scores (µ a ) a∈ [K] is equal to 0.132, which is lower than the minimum gap ∆ a (N ) a (N +1) ≈ 0.14 for N = 3, which is enough to assume in practice that this transformed model is quasi-linear [START_REF] Ghosh | Misspecified linear bandits[END_REF]Gopalan, 2017; Réda, Tirinzoni, and[START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF]. To sum it up, the feature vectors of dimension d = 15 are considered, and observed rewards are computed true scores from a randomly selected patient for the sampled treatment.

Remark 5.4.1. Of course, in a real-life setting, one does not have access to the true labels (average scores) associated with candidate drugs. Such a setting is however useful to check if our theoretical results empirically hold. In practice, either a non-linear algorithm should be considered on the untransformed data (see next Chapter 6), or either the (quasi) linearity of the data should be assessed on some datapoints (drugs) such that the recommendations from linear algorithms can be trusted. For instance, if the number of features is a lot larger than the number of arms/candidate drugs, the problem is intrinsically linear -but remember that a large number of features will result with high computing cost, due to the matrix multiplications which are cubic in the number of features. Application to "real life" drug repurposing is performed in Appendix 14.

In this experiment, we consider σ 2 = 1, δ = 0.1, N = 3 and we set ε to 0.2. For each algorithm, we use the corresponding theoretically-supported gap index, and we run it 100 times on the drug repurposing problem, in order to estimate its average empirical error rate and sample complexity.

One boxplot per algorithm is displayed in Figure 5.1. The y-axis corresponds to the sample complexity while the x-axis shows the algorithmic variants, with their reported empirical error rate δ computed across the 100 iterations, and rounded up to the 5 th decimal place. Individual outcomes are shown as gray dots. Strikingly, both LUCB and UGapE, but also the version of LinGIFA which uses individual gap indices instead of paired ones, need more than 10, 000 samples at each iteration of the experiment -which is why they are not shown in the boxplot. We have tested several versions of our two proposals :

-LinGIFA with the largest variance selection rule (LinGIFALargevar).

-LinGIFA (that is the version mentioned in Table 5.1 with the greedy selection rule).

-LinGIFA with the stopping rule τ LUCB instead of τ UGapE (LinGIFATLUCB).

-N -LinGapE with the optimized selection rule (LinGapEOptimized).

-The default version of N -LinGapE shown in Table 5.1 with the greedy selection rule (LinGapE).

This plot hightlights several interesting points :

1. Taking into account feature vectors is crucial for sample-efficient algorithms, at the condition of selecting the appropriate structure. Especially in this drug repurposing instance, linear algorithms perform better by several orders of magnitude than unstructured ones.

The experiment with the version of LinGIFA using individual gap indices

shows that upper confidence bounds built with individual gap indices can be a lot larger than those associated with paired indices, which was predicted by our Lemma 5.2.4. Moreover, this suggests that the loss of performance between unstructured and linear bandit algorithms is mainly due to the use

B i,j (t).
of individual gap indices, in addition to less sophisticated selection rules.

3.

As expected from the conclusion of our Lemma 5.2.8, LinGIFA (the default version using the stopping rule τ UGapE ) is more sample-efficient than LinGIFATLUCB (the version of LinGIFA which only differs by the use of stopping rule τ LUCB ).

4.

On average, LinGapE (N -LinGapE with the greedy selection rule) ends before LinGapEOptimized (N -LinGapE which uses the optimized selection rule), although the speedup is roughly of 7%, which seems to confirm our Conjecture 5.3.10.

Discussion

This chapter introduced a family of algorithms aimed at solving Top-N identification for linear models. According to Table 5.2, the sample complexityrelated constant is always at most of order O( k (ε + ∆ k ) -2 ), which is similar to what holds for unstructured bandits. However, can we exhibit an algorithm for Top-N identification which would be theoretically optimal (that is, asymptotically optimal, in the sense of Result 4.3.4)? Can we even get an easily computable version of the lower bound in Result 4.3.3? Moreover, how robust are these algorithms to non-linearity in the model? These points motivate the next chapter.

Chapter 6

Dealing with non-linear models

In Chapter 5, I studied the integration of feature vectors into a Top-N identification problem for bandits. As discussed in the conclusion of Chapter 5, although linear models are extremely attractive for their interpretability and the existence of associated efficient algorithms, real-life models might deviate from linearity. In particular, there might not be any valid value of parameter θ in the linear regression described in Subsection 5.2. As a consequence, the confidence intervals might not hold anymore, and the algorithm might return errors with rate higher than δ. In Table 6.1, I exhibit a synthetic experiment that shows that an efficient linear bandit algorithm can fail a lot more often than δ% of the time when facing non-linear models. Ψ 0 1 2 2.7 2.8 3 4 5 δ 0% 0% 0% 1% 6% 28% 100% 100% Table 6.1: Running a linear bandit algorithm on an increasingly non linear model. Empirical error δ from running N -LinGapE for Top-N identification (introduced in Chapter 5) on synthetic models of the form θ ⊤ X + Ψ 1(i = a (N +1) ) i∈ [K] , using δ = 5%, d = 5, K = 10, N = 3, σ = 1, and the theoretically-supported gap index in Lemma 5.3.3 across 100 iterations. Feature vectors were drawn at random. a The minimum gap in the associated linear model θ ⊤ X is ∆ a (N ) ,a (N +1) ≈ 0.2786. The larger Ψ is, the more the underlying model deviates from linearity, and the error δ increases.

a The exact value of matrix X is available at https://github.com/clreda/ misspecified-top-m/tree/main/data ("simulated misspecified instance" in [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF]).

However, we might still want to preserve the linear part of the model : first, for better interpretability ; second, to design a flexible bandit algorithm that can switch between perfectly linear and completely unstructured models, when feature vectors no longer bring supplementary information about the scores. In this chapter, I consider misspecified linear models, that can interpolate between linear and unstructured models. The corresponding class M Ψ of Ψ-misspecified models is

M Ψ := {x → θ ⊤ x + η(x) | θ ∈ R d , η ∈ (R d → R) s.t. max k∈[K] |η(x)| ≤ Ψ} .
As their name alludes to, the deviation of these models to a linear model -in terms of the maximum absolute coefficient of the unstructured term ηis always smaller than a fixed Ψ ≥ 0. When Ψ = 0, this class is the class of linear models, whereas, as Ψ grows, it matches the class of unstructured models -with bounded means. For large values of Ψ, the contribution η a to the expected reward µ a of arm a can be a lot larger than the linear part θ ⊤ X a : most of the information about arm a is present in η a . Remark 6.0.1. In practice, we consider the vector of expected rewards µ

µ = θ ⊤ X + η , where θ ∈ R d , η ∈ R K .
In particular, here, compared to Equation 6, possibly two arms a, b with the same feature vectors (X a = X b ) might have a different "unstructured" coefficient η a = η b . However, in our drug repurposing problem, the rewards result from a function of the signatures, which prevents that case from arising. Remark 6.0.2. Notice that the class of Ψ-misspecified models is different from the following class

M = {x → θ ⊤ x | θ ∈ R d+1 } ,
where • is the operator that turns a vector in R d into a vector in R d+1 by appending vector [1] to the initial vector. Indeed, in that case, the last coefficient θ d+1 is shared by all arms, whereas coefficients η 1 , η 2 , . . . , η K can possibly have different values.

In this chapter, we focus on fixed-confidence exact Top-N identification (ε = 0). First, a tractable lower bound on the sample complexity of any δ-correct algorithm (Definition 4.1.4) is derived. Then, an algorithm adapted to misspecified linear models is designed. Finally, the performance of this algorithm is compared to unstructured and linear bandit algorithms on a drug repurposing instance. This is a joint work with Andrea Tirinzoni and Rémy Degenne (SCOOL Inria team), and it led to a publication at the 35 th Neural Information Processing Systems conference (NeurIPS 2021) in Réda, Tirinzoni, and Degenne (2021).1

Related work

Learning a model under misspecification of its structure requires adapting to the scale of misspecification, typically under the assumption that some information about the latter is known -e.g., an upper bound Ψ to η . Due to its importance, this problem has recently gained increasing attention in the bandit community for regret minimization, but has not been addressed in the context of pure exploration. [START_REF] Ghosh | Misspecified linear bandits[END_REF] introduced the class of misspecified models for regret minimization. In that work, the authors show that if T is the learning horizon, for any bandit algorithm which enjoys an optimal regret scaling on linear models, there exists a misspecified instance where the regret is necessarily linear. As a workaround, they design a statistical test based on sampling a subset of arms prior to learning, to decide whether a linear or an unstructured bandit algorithm should be run on the data. Similar ideas are suggested in [START_REF] Chatterji | Osom: A simultaneously optimal algorithm for multi-armed and linear contextual bandits[END_REF], where the authors design a sequential test to sequentially switch between linear and unstructured algorithms. More recently, elimination-based algorithms (Lattimore, Szepesvari, andWeisz, 2020; Takemura et al., 2021) and model selection methods (Foster et al., 2020; Pacchiano et al., 2020) were popular approaches to tackle model misspecification for regret minimization. Notably, these regret minimization algorithms adapt to the amount of misspecification Ψ without knowing it beforehand, at the cost of an additive linear term that scales with Ψ.

Moreover, while best-arm identification has been the focus of many prior works in the realizable linear setting, some suggesting asymptotically-optimal algorithms (Degenne, Ménard, et al., 2020; Jedra and[START_REF] Jedra | Optimal best-arm identification in linear bandits[END_REF], Top-N identification has been seldom studied in terms of problem-dependent lower bounds. Lower bounds on the sample complexity for the unstructured2 Top-N problem have been derived previously, either focusing on explicit bounds (Kaufmann, Cappé, and Garivier, 2016, Theorem 4), or on worst-case lower bounds [START_REF] Chen | Nearly instance optimal sample complexity bounds for top-k arm selection[END_REF], Theorem 1.1), or on getting the correct dependence in the problem parameters for any error rate δ (Simchowitz, Jamieson, and Recht, 2017, Proposition 6). Because of the combinatorial nature of the Top-N identification problem, obtaining a tractable, tight, problem-dependent lower bound is not straightforward.

Assumptions

We denote L := max k∈ [K] X k 2 , and we assume that the feature vectors span R d -otherwise, those vectors could be rewritten in a subspace of smaller dimension. Furthermore, we assume that there exists some C ∈ R + such that, for any model function 3 Since f is a Ψ-misspecified model function, let us consider associated parameters θ and η, such that for any arm a ∈ [K], f (X a ) = θ ⊤ X a + η a . With regard to interpretability, we can observe that the absolute values of the coefficients in θ ∈ R d are still related to the influence of each of the d features on the average score f (X a ) ; with the addition of an arm-specific bias term η a .

f ∈ M Ψ , |f (X)| ∞ := max k∈[K] |f (X k )| ≤ C.
As previously mentioned, we focus on the case where ε = 0, that is, we only aim at returning arms in the true top-N set S ⋆ N,f . However, the set of N best arms of model function f might not be well defined since the set

S ⋆ N,f := k ∈ [K] | f (X k ) ≥ N max i∈[K]
f (X i ) might contain more than N elements if some arms have the same mean. Thus, let S N,f := S ⊆ S ⋆ N,f | |S| = N be the set containing all subsets of N elements of S ⋆ N,f . Moreover, suppose that the true model function f has exactly N arms that are among the top-N , i.e., that

S ⋆ N,f = N and S N,f = {S ⋆ N,f }. Remember that any δ-correct algorithm A satisfies Definition 4.1.4, that is, for any f ∈ M Ψ P {f } ŜN (τ A ) ⊆ S ⋆ N,f ≤ δ .

Tractable lower bound for general Top-N identification

In this section, we consider any class of models M, not necessarily M Ψ . As mentioned in Section 4.3, being able to asymptotically match -in the sense of Result 4.3.4-the lower bound on the sample complexity might lead to performant algorithms. In prior works that tackle pure exploration tasks, authors have focused on recovering (an estimation of) the oracle allocation arg min ω∈△ K in the general pure exploration lower bound (Result 4.3.3) to design asymptotically optimal algorithms. This oracle allocation can be approached either by using tracking (Du et al., 2021; Garivier and Kaufmann, 2016; Russac et al., 2021), or through online optimization algorithms (Degenne andKoolen, 2019; Degenne, Ménard, et al., 2020).

The computation of an oracle is tractable and well-studied for best arm identification, i.e., N = 1, since there are K -1 sets of alternative models when the optimal arm a ⋆ is unique. Each alternative corresponds to one of the K -1 alternative best arms. However, in Top-N identification, the naive computation of the lower bound-related optimization problem is done over

K N = K! (K-N )!N ! sets, where k! = 1 × 2 × 3 × • • • × k for any positive integer k.
In this problem, the set of alternative models to f ∈ M -which can be any structure-is Alt(f

) := f ∈ M | S N, f ∩ S N,f = ∅ .
This is the set of all M-structured model functions f , such that the set of top-N arms differs from S ⋆ N,f . Note that, while we assumed that the set of top-N arms under f is unique, this might not be the case for alternative f . Given any δ-correct algorithm A tackling Top-N identification in Mstructured models, for any δ ≤ 1/2, we can combine Result 4.3.3, which gives a lower bound on the sample complexity for any pure exploration problem, with the following lemma Lemma 6.

Expression of Alt(f ) in Top-N identification. For any model functions

f, f ∈ M such as |S N,f | = 1 f ∈ Alt(f ) ⇔ ∀S ∈ S N,f ∃i ∈ S ∃j ∈ S, f (X i ) > f (X j ) . Proof. Implication. If f ∈ Alt(f ), then it means that S N,f ∩ S N, f = ∅. Suppose that ∃S ∈ S N,f ∀i ∈ S ∀j ∈ S, f (X i ) ≤ f (X j ) . (6.1)
Then, this implies that for any arm i ∈ S, f (X i ) ≤ min j∈S f (X j ). Since, by definition, |S| = N , then S is a valid Top-N set under model function f , which means that S N,f ∩ S N, f = ∅, and then f ∈ Alt(f ). In conclusion, Equation (6.1) implies that f ∈ Alt(f ). Then the contraposition also holds.

Inverse implication.

Consider S ∈ S N,f (unique top-N set associated with f ), and arms i, j that satisfy the right-hand expression in Lemma 6.3.1. We now consider two cases :

-If there exists a top-N set S associated with f , i.e., S ∈ S N, f , such that j ∈ S, then by definition of i, i ∈ S ∩ (S) c .

-Otherwise, j is not among the best N arms under model function f . Then for any S ∈ S N, f , j ∈ S ∩ S c . This means that, in both cases, S ∈ S N, f , and then S N, f ∩ S N,f = ∅.

The combination of these two results yields the following lower bound for any structured Top-N identification problem Theorem 6.

Characteristic time for Top-N identification. Let KL be the Kullback-Leibler divergence on continuous probability densities (Definition 4.3.2). For any δ-correct algorithm A for Top-N identification where

δ ≤ 1/2, and any f ∈ M, the following lower bound holds on the sample complexity

E {f } [τ A ] ≥ (C ⋆ (f )) -1 log 1 2.4δ
, where

C ⋆ (f ) = sup ω∈△ K min i̸ ∈S ⋆ f min j∈S ⋆ f inf f ∈M f (X i )> f (X j ) a∈[K] ω a KL a (f, f ) ,
and KL a (f, f ) := KL(ν a , α a ) where ν a , α a are two Gaussian laws with fixed variance σ 2 that satisfy

E {Y ∼νa} [Y ] = f (X a ) and E {Y ∼αa} [Y ] = f (X a ), and
K :=    p ∈ [0, 1] K | k∈[K] p k = 1    is the simplex on [K].
Computing this quantity now requires to : (a) maximize once over the simplex -which can be still hard ; (b) minimize over the N (K -N ) half-spaces

{ f ∈ M | f (X i ) > f (X j )} for (i, j) ∈ S ⋆ f c × S ⋆
f . Those minimizations are convex optimization problems, and can be solved efficiently for M = M Ψ . Our algorithm inspired from that bound will need to perform only those minimizations, since it uses an online approach to estimate the oracle allocation ω ∈ K . For Gaussian bandits with fixed variance σ 2 -i.e., when rewards follow a Gaussian distribution with variance σ 2 -using the expression of the Kullback-Leibler divergence in that case (Equation (4.4)) yields the following optimization problem related to characteristic time

C ⋆ min i̸ ∈S ⋆ f min j∈S ⋆ f inf f ∈M Ψ f (X i )> f (X j ) 1 2σ 2 a∈[K] ω a (f (X a ) -f (X a )) 2 . (6.2)
This problem can be solved in Python by iterating over all pairs (i, j) ∈ S ⋆ f c × S ⋆ f , and independently solving at fixed (i, j) the quadratic program derived in Réda, Tirinzoni, and Degenne (2021, Appendix C.3) using module quadprog based on [START_REF] Goldfarb | A numerically stable dual method for solving strictly convex quadratic programs[END_REF]. Finally, the solution which yields the minimum value across all pairs of arms is selected.

Note that a lower bound for Top-N identification using Result 4.3.3 has been obtained in Kaufmann, Cappé, and Garivier (2016, Theorem 4). Aiming to be more explicit, they defined the set of alternative models as the set of models where one of the best arms is switched with the (N + 1) th best arm under f , or one of the K -N worst ones with the N th best one under f . These models are a strict subset of Alt(f ) for any f ∈ M. Hence this bound is not as tight as the one in Theorem 6.3.2, which is why the algorithm we introduce in the next section will rely on the latter instead. Note that for Ψ = 0 (linear models) and N = 1 (best arm identification), this lower bound is exactly the one derived by Fiez et al. (2019, Theorem 1).

Moreover, it was shown in Réda, Tirinzoni, and Degenne (2021, Section 3.1) using Theorem 6.3.2 that knowing that a problem is misspecified without knowing the upper bound Ψ on η ∞ is the same as not knowing anything about the structure of that problem. This means that there is no optimal δ-correct algorithm which is oblivious to Ψ ; if an algorithm A is δ-correct and optimal -in terms of sample complexity-on the set of model functions M Ψ , then it cannot be optimal as well on a strict subset of model functions M Ψ ′ , where Ψ ′ < Ψ. Hence, the algorithm described in the next section will crucially need a good estimate Ψ of the deviation to linearity.

In a related work on δ-correct best arm identification dealing with misspecified model structures [START_REF] Zhu | Pure Exploration in Kernel and Neural Bandits[END_REF], the proposed algorithm also relies on an estimated upper bound on the approximation error between the misspecified model (represented by a neural network) and the true expected rewards.

Misspecified Top-N identification

Now we introduce an algorithm for fixed-confidence Ψ-misspecified Top-N identification, named misspecified linear identification (MisLid). Its structure is outlined in Algorithm 7. This algorithm is only well-defined for Gaussian bandits with fixed variance σ 2 = 1. On the one hand, the design of MisLid builds on top of recent approaches for constructing pure exploration algorithms from lower bounds (Degenne, Koolen, and Ménard, 2019; Degenne, Ménard, et al., 2020; Jedra and Proutiere, 2020; Zaki, Mohan, and Gopalan, 2020). On the other hand, its main components and their analysis introduce several technical novelties to deal with misspecified Top-N identification, that might be of independent interest for other settings. We describe these components below. Let us define for any vector v ∈ R K

D v := diag(v 1 , v 2 , . . . , v K ) and V κ (t) := s≤t X It X ⊤ It (note that κ = 0 here) .
The core principle of the algorithm is to sample arms according to the oracle allocation ω(t) at round t. This allocation is estimated by an online learner L, which is iteratively updated based on the expression of characteristic time C ⋆ (f ) in Theorem 6.3.2. Sampling stops at round τ MisLid so that

τ MisLid := inf t>0 inf f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) > 2T δ (t) ,
where f t ∈ M Ψ is a model function which corresponds to the projection of the vector of empirical means ( µ a (t)) a∈ [K] onto the set of Ψ-misspecified model functions ; T δ (•) is a time-dependent threshold function to be defined. Remember that the empirical mean µ a (t) for arm a ∈ [K] at round t is defined as

µ a (t) := 1 N a (t) s≤t 1(I s = a)Y t ,
where N a (t) is the number of selections of arm a up to round t. For short, we denote f t (X) vector ( f t (X a )) a∈ [K] . Provided that function T δ (•) is well-chosen, the stopping rule ensures that MisLid is δ-correct, and asymptotically optimal, as defined in Result 4. 3.4 inf

f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) = inf f ′ ∈Alt( ft) ( f t (X) -f ′ (X)) ⊤ D N•(t) ( f t (X) -f ′ (X)) = inf f ′ ∈Alt( ft) a∈[K] N a (t)( f t (X a ) -f ′ (X a )) 2 .
If the (N a (t)) a∈ [K] were replaced by some allocation ω ∈ K and f t by the true model function f , the expression would have matched the definition of C ⋆ in the lower bound for Gaussian bandits with fixed variance σ 2 = 1, up to constant 1/2 (Equation (6.2)). This quantity is the statistic associated with a parallel generalized likelihood ratio test (GLRT), as described in Garivier and Kaufmann (2021). 4 Here, the GLRT is a sequential statistical test of the following two non-overlapping hypotheses

H 0 : S ⋆ N,f ⊆ S ⋆ N, ft (null hypothesis) , versus H 1 : S ⋆ N,f ⊆ S ⋆ N, ft (alternative) ,
and rejects the null hypothesis when the statistic is large enough. If H 0 is rejected, then we cannot rule out the possibility that the top-N arms under current empirical model f t includes the true set S ⋆ N,f . At the end of the sampling phase in round τ MisLid , MisLid returns the N -best arms of the projected estimated mean vector f τ MisLid (X). I will now explicit the different steps in Algorithm 7. Pull I t , receive Y t , and set ω(t) ← (1(i = I t ) i∈ [K] 7: end for 8: Compute empirical mean µ • (t 0 ) := ( µ a (t 0 )) a∈ [K] and the model function associated with its projection onto the set of Ψ-misspecified models

f t 0 ← arg min f ′ ∈M Ψ f ′ (X) -µ • (t 0 ) 2 D N•(t 0 ) .
9: for t = t 0 + 1, t 0 + 2, . . . , do 10:

# Stopping rule 11: Obtain ω(t) from learner L 15:

if inf f ′ ∈Alt( f t-1 ) f t-1 (X) -f ′ (X) 2 D N•(t-1) > 2T δ (t -
Compute closest alternative f (t) ← arg min Update L with gain Update µ • (t) and compute its projection f t ← arg min

f ′ ∈Alt( f t-1 ) f t-1 (X) -f ′ (X)
g(t) : ω → k∈[K] ω k f t-1 (X k ) -f (t)(X k ) + c(t -
f ′ ∈M Ψ f ′ (X) -µ • (t) 2 D N•(t)
22: end for Initialization phase. Algorithm 7 starts by sampling a deterministic sequence of t 0 arms that make the minimum eigenvalue of the resulting design matrix V κ (t 0 ) larger than 2L 2 . Since the rows of feature matrix X are assumed to span R d (Section 6.2), such sequence can be easily found by taking any subset of d arms that span the whole space -e.g., by computing a barycentric spanner. A barycentric spanner of R d of size d is a set of d arms if any element in R d can be expressed as a linear combination of these arms with coefficients in [-1, 1]. An approximation of this set can be computed in polynomial time [START_REF] Awerbuch | Adaptive routing with end-to-end feedback: Distributed learning and geometric approaches[END_REF]. Once this barycentric spanner is built, arms in the spanner are sampled in a round robin fashion until the desired condition is met. This is required to make design matrix V κ (•) invertible, and to ensure that concentration inequalities hold. While prior works typically avoid this step by regularizing [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF], in our misspecified setting it is crucial not to do so to obtain tight concentration results for the estimator of f (X). In order to get an upper bound on the length t 0 of the initialization phase, let us denote σ min (M ) the minimal singular value of matrix M . Let us consider

B = {b 1 , b 2 , . . . , b d } ⊆ [K], such that | B |= d,
the barycentric spanner of size d computed on feature matrix X. Then, if we stopped the round-robin sampling at round t 0 such that each arm in the barycentric spanner is sampled a number u 0 of times,

V κ (t 0 ) = u 0 b∈B X b X b .
The following condition is enough to ensure that

V κ (t 0 ) 2L 2 I d -that is, that matrix ( V κ (t 0 ) -2L 2 I d ) is positive definite u 0 σ min b∈B X b X ⊤ b ≥ 2L 2 . (6.3) Let us denote Γ ′ d (X) := min B d-sized spanner σ min ( b∈B X b X b ). Then u 0 = 2L 2 Γ ′ d (X) satis- fies Condition (6.3), and then t 0 ≤ d 2L 2 Γ ′ d (X) .
Remark 6.4.1. When the condition number of the matrix is very largee.g., feature vectors are collinear-this initialization phase can be timeconsuming. Specifically in that case, feature selection (or transformation) prior to the application of bandits is of paramount importance.

Estimation. At each round t ≥ t 0 , MisLid updates an estimator f t of the true bandit model f , by first computing the empirical mean for each arm a µ a (t

) := 1 N a (t) s≤t 1(I s = a)Y t .
Then this empirical model is projected onto the set of Ψ-misspecified models, according to the norm weighted by the individual number of draws N a (t) of each arm a f t := arg min

f ′ ∈M Ψ f ′ (X) -µ • (t) D N•(t) .
In [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF], this projection can be computed efficiently as two independent quadratic optimization problems, one for each of the two parameters θ and η.

Stopping rule. MisLid uses the standard stopping rule adopted in many existing algorithms for pure exploration (Degenne, Koolen, and Ménard, 2019; Garivier and Kaufmann, 2016; Shang et al., 2020), which is based on generalized likelihood ratio tests. But the difficulty lies in its calibration, that is, the choice of an appropriate threshold function T δ (•) for which the algorithm is δ-correct. MisLid requires a careful combination of concentration inequalities. First, for linear bandit models, to make the algorithm adapt well to (quasi) linear models with low misspecification Ψ ; and, second, for unstructured bandits, to guarantee asymptotic optimality in the sense of Result 4.

β uns δ (t) := 2KW 1 2K log 2 exp(1) δ + 1 2 log (8 exp(1)K log(t)) , β lin δ (t) := 4 √ 2 √ tΨ + 1 + log(δ -1 ) + 1 + 1 log(δ -1 ) d 2 log 1 + t 2d log(δ -1 ) 2 , and W (x) := -W -1 (-exp(-x)) ≈ x + log(x) ,
where

W -1 is defined such that, if z ∈ [-exp(-1), 0) , x ≤ -1, and x = W -1 (z) then x exp(x) = z (W -1 is the negative branch of the Lambert W function) .
Note that this stopping rule crucially relies on the knowledge of misspecification level Ψ.

Proof sketch. The objective is to show that MisLid is δ-correct on Ψmisspecified models, that is, according to Definition 4.1.4, for any f ∈ M Ψ ,

P f ŜN,f (τ MisLid ) ⊆ S ⋆ N,f ≤ δ.
If τ = τ MisLid , then

P f ŜN,f (τ ) ⊆ S ⋆ N,f = P f inf f ′ ∈Alt( fτ ) f τ (X) -f ′ (X) 2 D N•(τ ) > 2T δ (τ ) and f τ ∈ Alt(f )
And then

P f ŜN,f (τ ) ⊆ S ⋆ N,f ≤ P f ∃t > 0, inf f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) > 2T δ (t) and f t ∈ Alt(f ) ≤ P f ∃t > 0, inf f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) > 2T δ (t) ,
By the definition of threshold function T δ (•) and applying a union bound

P f ŜN,f (τ ) ⊆ S ⋆ N,f ≤ P f ∃t > 0, inf f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) > 2β uns δ (t) ≤ δ/2 + P f ∃t > 0, inf f ′ ∈Alt( ft) f t (X) -f ′ (X) 2 D N•(t) > 2β lin δ (t) ≤ δ/2 ≤ δ ,
where the last two inequalities respectively stem from the unstructured and the linear concentration inequalities derived in Réda, Tirinzoni, and Degenne (2021, Appendix F.1).

Sampling strategy and online learners. The sampling strategy of MisLid aims at achieving the optimal sample complexity from the lower bound from Theorem 6.3.2. Instead of using a tracking procedure as done in [START_REF] Garivier | Optimal best arm identification with fixed confidence[END_REF], MisLid uses an online regret minimization learner L, which incrementally estimates the optimal allocation ω ∈ K in the expression of the characteristic time in Theorem 6.3.2. 5 Arm I t is selected according to this allocation ω(t). Learner L iteratively updates its estimation ω(t) at round t through gains g(t). Gains translate the regret incurred by a bad approximation of the true allocation ω. If we knew the true model f , we would use the gradient at ω(t) of the following gain function to quantify the regret incurred by estimate ω(t)

g ⋆ : ω → inf f ′ ∈Alt(f ) f (X) -f ′ (X) 2
Dω .

Since f is not available, we settle for an upper bound on arg min

f ′ ∈Alt( ft) f t (X) - f ′ (X)
Dω for any ω ∈ K , using minimizer f (t), and optimistic bonuses c(t) ∈ R K such that, with high probability

∀ω ∈ K ∀t > 0, g t (ω) = k∈[K] ω k | f t (X k ) -f (t)(X k ) | + c(t) k 2 ≥ g ⋆ (ω) .
In pratice, c(t) k is defined for any arm k ∈ [K] as follows

c(t) k := min 8(LK + 1) 2 Ψ 2 + 4α lin t 2 X k 2 ( V κ (t)) -1 , 2α uns t 2 N k (t) , 4C 2 ,
where α uns t := β uns 1/(5t 3 ) (t) and α lin t := log(5t 2 ) + d log(1 + t/(2d)). As briefly mentioned in Section 6.3, using an online learner avoids having to rely on minmax oracles which might be computationally inefficient -due to the maximization over the simplex K -and might be prone to numerical approximations that prevent convergence. Given these parameters, Algorithm MisLid is provably asymptotically optimal (in the sense of Result 4.3.4). The full proof of the theorem is available in the supplementary section of [START_REF] Réda | Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification[END_REF].

Theorem 6.4.3. Sample complexity of MisLid. The following inequality about the expected sample complexity of MisLid holds

E {µ} [τ MisLid ] ≤ τ δ + 2 ,
where τ δ satisfies the following inequality in t (ℓ t := log(t) and O is the sum of O of each argument)

T δ (t) ≥ tC ⋆ + O min{tK 2 Ψ 2 +d √ tℓ t , √ Ktℓ t }; log K √ t; min{tK 2 Ψ 2 +dℓ t , Kℓ t } log(1/δ) .
In particular, for Ψ ≈ 0 (quasi linear models)

τ δ ≤ (C ⋆ ) -1 log(δ -1 ) + Ψ 2 τ δ + log(K) √ τ δ + d √ τ δ log(τ δ ) + d log(τ δ ) o(log(δ -1 )) ,
and for Ψ 0 (close to unstructured models)

τ δ ≤ (C ⋆ ) -1 log(δ -1 ) + log(K) √ τ δ + √ Kτ δ log(τ δ ) + K log(τ δ ) o(log(δ -1 )) .
Based on Theorem 6.4.3, for small values of δ,

τ δ ≈ (C ⋆ ) -1 log(δ -1 ) + C µ o(log(δ -1 )) ,
where C µ is a problem-dependent constant. Then

lim inf δ→0 E {µ} [τ MisLid ] log(δ -1 ) = lim inf δ→0 τ δ log(δ -1 ) = (C ⋆ ) -1 ,
which confirms that MisLid is indeed asymptotically optimal. Note that when Ψ ≈ 0 (i.e., , the model is quasi linear) there is only a logarithmic dependence on the number of arms K, which is on par with the state-of-the-art (Jedra and Proutiere, 2020; Kirschner et al., 2021; Tirinzoni et al., 2020). Moreover, the bound exhibits an adaptation to the value of Ψ. As Ψ grows, the upper bound transitions to terms matching the optimal unstructured bound without any dependence in the number of dimensions d and deviation Ψ.

Application to drug repurposing

Since our algorithm is the first to solve Top-N identification in Ψ-misspecified models, we compare it against an efficient linear algorithm, N -LinGapE, introduced in Chapter 5 -which is an extension to algorithm LinGapE described in [START_REF] Xu | A fully adaptive algorithm for pure exploration in linear bandits[END_REF]-and an unstructured one, LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. In all experiments, we consider δ = 10% and σ = 1. It has frequently been noted in the fixed-confidence literature that stopping thresholds which guarantee δ-correctness tend to be too conservative and to yield empirical error frequencies that are actually much lower than δ. Moreover, these thresholds are different between linear, misspecified and unstructured bandit models. In order to ensure a good trade-off between performance and computing speed, and fairness between the tested algorithms, as in prior works (Kaufmann and Kalyanakrishnan, 2013; Réda, Kaufmann, and Delahaye-Duriez, 2021), we use the following heuristic value ∀t > 0, T δ (t) := ln 1 + ln(t + 1) δ .

We consider the drug repurposing problem on 10 antiepileptics and 11 proconvulsants mentioned in Chapter 5. However, contrary to what was previously done, this time we truly stick to the theoretical setting, and make the environment return a realization of a Gaussian distribution of fixed variance σ 2 = 1 when the agent samples arm

I t ∈ [K] Y t ∼ N (µ It , 1) .
Moreover, knowing that our algorithm relies on a good knowledge of the maximum misspecification Ψ, we apply the same type of feature transformation procedure as in Chapter 5. We compute Ψ as the ℓ ∞ norm of the difference between the predictions of the regressed linear model, and the average drug repurposing scores. Selecting the most deviated model to linearity yields Ψ = 0.236 with d = 21 dimensions. Since the misspecification Ψ is a lot larger than the minimum gap min

k∈[K]
∆ k -contrary to the model built in

Chapter 5-this model is highly misspecified, which will allow us to observe the same behavior as in the experiment in Table 6.1.

Figure 6.1 shows one boxplot per algorithm, which reports the sample complexity on the y-axis, and the error frequency δ across 100 iterations rounding up to the 5 th decimal place. Individual outcomes are shown as gray dots. In order to speed up LUCB, we consider the PAC version of Top-N identification -that is, ε > 0-choosing as stopping threshold ε = 0.042 ≈ min k∈ [K] ∆ k (for N = 4), so that the algorithm stops earlier while returning the exact set of N best arms.

As previously mentioned, we expect N -LinGapE to dramatically fail since the misspecification scale Ψ is large. Moreover, from the sample complexity bound derived in Theorem 6.4.3, we expect MisLid to have an average sample complexity at most as large as the one incurred by a good unstructured bandit algorithm. As shown by the boxplot in Figure 6.1, linear algorithm N -LinGapE dramatically fails, whereas LUCB (for unstructured models) and MisLid remain δ-correct. Moreover, even with the early stopping rule using ε > 0, in average LUCB needs 40% more samples than MisLid, which does not use the information about ε.

Discussion

We have designed the first algorithm to tackle misspecification in fixedconfidence Top-N identification. The proposed algorithm can be applied to misspecified models which can deviate from linearity (i.e., Ψ ≥ 0), which encompass both unstructured (for large values of Ψ) and linear models (i.e., Ψ = 0).

The main limitation of MisLid is its computational complexity : at each round, O(KN ) convex optimization problems need to be solved for both the sampling and stopping rules, which can be expensive if the number of arms is large. Moreover, since the sampling of our algorithm is designed to aim at a lower bound, we can expect it to suffer from the same shortcomings as that bound. Indeed, it is known that the bound in question does not capture some lower order (in O(δ -1 )) effects, in particular those due to the multiplehypothesis nature of the test we perform, which can be very large for small times. Work to take these effects into account to design algorithms has started recently (Katz-Samuels, Jain, Jamieson, et al., 2020; Katz-Samuels and Jamieson, 2020; Wagenmaker, Katz-Samuels, and Jamieson, 2021) and we believe that it is an essential avenue for further improvements in pure exploration.

Finally, a good estimation of the maximum deviation to linearity, that we denoted Ψ, is crucial to obtain a sample-efficient algorithm. Unfortunately, this information is usually not available when dealing with real-life datasets. If the estimate Ψ is smaller than the true value Ψ, then there is a risk of losing the property of δ-correctness ; however, if this estimate is too large, then the performance of MisLid should be similar to a good bandit algorithm for unstructured models. Finding a good procedure to heuristically estimate the deviation level Ψ -without having access to the expected rewards-at the price of some loss in accuracy in the estimation of the true scores could be as well an interesting subsequent work. Indeed, determining the relative ranking of arms is more important in drug repurposing than estimating exactly the scores. 6 a Bokobza, Réda, et al. (in prep.). "Therapeutic evaluation of Hu-MSCs in a rat model of perinatal inflammation: a systematic outcome scoring"; [START_REF] Réda | Near-Optimal Federated Learning in Bandits[END_REF]. "Near-Optimal Federated Learning in Bandits". In press.

In this part of my thesis, I present two works related to the drug repurposing method I have proposed. Chapter 7 exhibits a instance of systematic treatment protocol ranking procedure through signature reversion when the appropriate conditions are reunited to build good quality signatures. Chapter 8 deals with the extension of any adaptive clinical trial to a collaborative setting.

In Chapter 7, I have collaborated with a team of biologists in order to select the optimal treatment protocol of stem cell injections in a rat model of infant encephalopathy. Transcriptional data obtained through sequencing comprised of samples subject to varying doses, modes and ages of injection. The goal was to rank the different treatment protocols depending on the estimated recovery of treated samples through signature reversion. This project is a proof-of-concept that present a systematic method to rank treatments based on transcriptomic data.

Chapter 8 deals with the theoretical setting where there are several heterogeneous subpopulations of patients which are recruited to join a collaborative adaptive clinical trial, in order to find the most interesting drug candidates for their own population of patients, while exploiting the results from other populations. This work is a step towards personalized clinical trials, which are a topical question in drug research, especially in cancer [START_REF] Maitland | Clinical trials in the era of personalized oncology[END_REF], where there is a plethora of subtypes and biomarkers which are sometimes unique to a patient.

Chapter 7

Application of systematic drug scoring

This chapter focuses on an application of signature reversion in practice, in a particularly favorable context for computing drug signatures : indeed, the drug signatures will exactly represent the transcriptional changes due to the drug treatment applied on patients. The simulating procedure using the Boolean network will not be needed in this case. This application highlights how signature reversion allows to discriminate between treatments.

The work presented in this chapter aims at validating umbilical cordderived human mesenchymal stem cells (MSCs) as a regenerative therapy for encephalopathy of prematurity (EoP), which is due to brain lesions in premature birth. Several treatment protocols -with varying weight-dependent doses, modes and ages of injection of MSCs-are performed on rat models of EoP, from which transcriptomic profiles are obtained, for control rats (which do not receive stem cells), rat models, and rat models treated with stem cells. The signature reversion method outputs a score for each treatment protocol, based on this transcriptomic data. These scores allow ranking treatment protocols according to their rescuing of a healthy transcriptomic profile. Further pathway analyses seem to confirm the impact of top-ranked treatment protocols on inflammation-related and developmental genes. A manuscript is in preparation for the Journal of Neuroinflammation (Bokobza, Réda, et al., in prep.).

Related work

Preterm birth represents an estimated 15 million births every year, and related complications are the leading cause of death in children before the age of 5 in 2018, according to the World Health Organization (World Health Organization (WHO), 2018). Deliveries before 37 on 40 weeks of gestation lead to a constellation of neurodevelopmental injuries, regrouped under the term of encephalopathy of prematurity (EoP) [START_REF] Bokobza | Neuroinflammation in preterm babies and autism spectrum disorders[END_REF]. White matter injury (WMI) is the most common brain injury associated to preterm birth. WMI is due to a myelination deficit during development. Myelination is an important step in nervous system development, as layers of myelin are produced and wrapped around the neuronal axons. This allows "insulating" the transmission of electric action potentials along the axon, and faster information processing and more complex brain processes. Myelin is produced by oligodendrocytes, that differentiate between 20 weeks of gestation to the first year of infant life. Preterm birth disturbs normal differentiation processes by blocking oligodendrocytes to a precursor state, which is unable to produce myelin. Moreover, microglial cells, which are a subtype of glial cells and the brain resident macrophages, produce cytotoxic molecules affecting precursor oligodendrocytes, which are highly vulnerable to such insults. This induces WMI through neuroinflammation in the white matter. In a good healthcare setting, more than half of babies who survive will suffer from lifelong disabilities, including cerebral palsy, severely impaired cognitive functions, and psychiatric disorders, such as attention-deficit and hyperactivity disorder, or autism spectrum disorder [START_REF] Crump | Preterm or early term birth and risk of autism[END_REF]. There is no existing treatment yet to repair brain damage incurred by EoP [START_REF] Chung | Neurodevelopmental outcomes of preterm infants: a recent literature review[END_REF].

In addition to their high regenerative capacity, mesenchymal stem cells (MSCs)1 are reported as anti-inflammatory, and with a low immunogenicity [START_REF] Passera | Therapeutic potential of stem cells for preterm infant brain damage: Can we move from the heterogeneity of preclinical and clinical studies to established therapeutics?[END_REF], meaning that they do not trigger a strong immune response. Moreover, several studies report that MSCs and moleculederived MSCs contribute to reduce microglial reactivity both in vitro and in vivo (Barati et al., 2019; Go et al., 2020; Liu, Zhang, et al., 2014). However, there is no current data which compiles effects of MSCs on microglial reactivity in a same animal model, at different time points. Indeed, for a common time of insult, MSCs will not correct the same microglial pathways depending on administration age -denoted by postnatal day (P) 5, P10, . . . , which are known to model prenatal humans.

This project introduces a new model of EoP, on rats, by neuroinflammation induced by the injection of interleukin 1 beta (IL1β) at postnatal day 5 (P5). The characterization of the microglial transcriptome has already been carried out in an identical mouse model by [START_REF] Krishnan | Integrative genomics of microglia implicates DLG4 (PSD95) in the white matter development of preterm infants[END_REF]. In that work, they have shown the existence of clusters of genes involved with inflammatory processes hyper-regulated at postnatal day 1 (P1) and P5, as well as clusters negatively regulated by IL1β and grouping together developmental genes.

The objective of this project is to assess the optimal administration settings for MSCs injections in the rat model of EoP, in terms of weight-dependent dose, route of administration and age of administration. In order to achieve this, we consider using the paradigm of signature reversion, reviewed in particular in [START_REF] Musa | A review of connectivity map and computational approaches in pharmacogenomics[END_REF], where post-treatment transcriptional levels are compared with control transcriptional profiles. The more similar these two groups of profiles are, the most interesting the associated treatment is. This method has already been successful, for instance, in drug repurposing against influenza, where candidates obtained through signature reversion have been validated in vitro [START_REF] Xin | Identification of Nifurtimox and Chrysin as Anti-Influenza Virus Agents by Clinical Transcriptome Signature Reversion[END_REF]. Combining this paradigm with the experimental transcriptional profiles generated for every selected treatment protocol allows us to rank the different treatment protocols depending on their similarity score to a healthy profile. This work was a collaboration with the Neurokines team (Inserm U1141), which performed the experimental part of this project, and the PREMSTEM consortium.

Setting and experimental data

All rat pups were intraperitoneally injected twice daily from post-natal day (P)1 to P4 and once in the morning of P5 with recombinant mouse IL1β, or the same volume of phosphate-buffered saline (PBS). Figure 7.1 shows the timeline of the experimental part. Rats were injected once with MSCs, according to their treatment protocol. Table 7.1 shows all parameters for the samples considered for treatment outcome scoring. In particular, note that there are 6 sequencing batches, with N = 3 replicates per condition. A condition represents the set of the following parameters : dose level, age and mode of administration, and treatment. IL1β samples are the rat models of EoP, whereas whereas PBS samples are the rat control group (without any neuroinflammation). The rats with a neuroinflammation induced by the injections of IL1β, and then treated with MSCs are considered the treated groups. Note that, since the doses of MSCs are weight-dependent, since we compare different ages of administration, the actual doses can be very dissimilar across groups. The numbering of doses 1, 2, 3 respectively stand for low, moderate and high doses of MSCs. In a nutshell, in addition to the injection with PBS, IL1β and IL1β followed by a treatment with MSCs, we consider three different dose levels (low, moderate and high doses), two administration routes (intranasal and intravenous), and three ages of administration (postnatal days 5, 10 and 20).

Age * Dose (#cells) Age * * Mode * † Treatment #repl. Batch P5 P7 INAS PBS, 3, 1 IL1β, 3, Dose 1 : 200, 000
IL1β+Dose 1, 3, Dose 2 : 500, 000

IL1β+Dose 2, 3, Dose 3 : 1, 000, 000

IL1β+Dose

3 3 IV PBS, IL1β, 3, 2 IL1β, 3, IL1β+Dose 1, 3, IL1β+Dose 2, 3, IL1β+Dose 3 3 P10 P12 INAS PBS, 3, 3 IL1β, 3, Dose 1 : 350, 000
IL1β+Dose 1, 3, Dose 2 : 900, 000

IL1β+Dose 2, 3, Dose 3 : 2, 000, 000

IL1β+Dose 3 3 IV PBS, IL1β, 3, 4 IL1β, 3, IL1β+Dose 1, 3, IL1β+Dose 2, 3, IL1β+Dose 3 3 P20 P22 INAS PBS, 3, 5 IL1β, 3, Dose 1 : 750, 000
IL1β+Dose 1, 3, Dose 2 : 2, 000, 000

IL1β+Dose 2, 3, Dose 3 : 4, 500, 000

IL1β+Dose 3 3 IV PBS, 3, 6 IL1β, 3, IL1β+Dose 1, 3, IL1β+Dose 2, 3, IL1β+Dose 3

Systematic outcome scoring

Once the transcriptional profiles from all condition groups in Table 7.1 were generated, I applied the principle of signature reversion in order to rank the different condition groups depending on their ability to rescue the IL1βinduced inflammatory profiles. In signature reversion, the in silico identification of the most appropriate treatment protocols relies on a scoring of the changes at gene expression level. These changes are computed between the samples treated associated with a single condition, and a carefully determined reference group. As mentioned in previous chapters (Chapter 3), changes in expression for a group of genes between treated and reference groups can be quantified by a vector of numbers called signature, which is as large as the number of genes in the group. A signature accounts for the magnitude and the direction of the genewise expression change that is only due to this treatment. It can be used to characterize the treatment effect at transcriptional level. In particular, the magnitude of the change in expression of gene g is inferred from the absolute value of the coefficient at position g in the signature. Likewise, the direction of the change of expression of gene g -up-regulation versus down-regulation-is encoded in the sign of this very same coefficient. Genes which are not subject to expression changes are either absent from the signature, or present with a coefficient equal to 0. Actually, signatures can be built from the results of a (univariate) differential expression analysis. In that case, the absolute values of coefficients are log 2 fold-changes : i.e., the binary logarithm of the ratio of gene expression levels in the treated and the reference groups. The set of genes associated with nonzero coefficients is restricted to significantly differentially expressed genes -i.e., associated with an adjusted2 p-value lower than some threshold, for instance 5%. However, we will consider here another method for differential expression analysis, which is not univariate. Indeed, we suppose that the changes in expression are mainly driven by groups of genes rather than individual genes. Leaving out the group interaction might incur important information loss about the perturbed pathways, as we will see in Section 7.4.

For each treatment protocol, the goal is to compute the similarity of the associated treatment signature to a reference signature. This reference signature is defined as the signature computed between corresponding PBScontrol rats and IL1β-EoP model samples ; it represents the transcriptional changes expected from a good treatment protocol that is able to rescue the disease phenotype.

Figure 7.2 illustrates the ranking procedure. In order to build a signature for a given treatment protocol T of MSCs injection (Step 1.a in Figure 7.2), we rely on a computational method called Characteristic Direction (CD) [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF]. 3 To build the treatment signature, this method takes as input RNA-sequencing (DESeq2 (Love, Huber, and Anders, 2014) normalized) profiles from two condition groups in the same sequencing batch. The first group is the IL1β-EoP model samples treated with T , and the second group comprises of IL1β-EoP model samples without any injection of MSCs. We denote the resulting vector CD[T IL1β]. The reference signature, denoted CD[PBS IL1β], to which treatment signature CD[T IL1β] should be compared, is built in a similar way (Step 1.b). CD is run on the condition groups of PBS-control samples and IL1β-EoP model samples without any injection of MSCs from the same sequencing batch as the previous two groups. Ensuring to use samples from the same sequencing aims at limiting the effect of the sequencing batch, which could otherwise be a confounding factor for the effect of the treatment. Moreover, the correction of batch effects from raw data is a tricky procedure, which might lead to serious signal loss. This is why computations were run on normalized expression matrices (per batch), instead of batch-corrected raw expression data. Finally, in order to assign a score to treatment protocol T (Step 2), we compute the cosine similarity between the signature S T = CD[T IL1β], corresponding to T , and the associated reference signature R = CD[PBS IL1β] on the same set of genes G -even if it means adding zero coefficients to the signatures. This cosine score is defined as

cos (S T , R) := g∈G S T [g] × R[g] g∈G S T [g] g∈G R[g] . (7.1)
This score is comprised between -1 (strong dissimilarity) and 1 (strong similarity). As shown in Step 3 of the figure, the underlying idea is that, the higher the similarity score is, the most the injection of MSCs is able to reproduce transcriptional genewise changes, in the right direction, from the diseased to a healthy phenotype.

Results

The final ranking of treatment protocols is shown in Table 7.2. First, this ranking shows that the method described in Section 7.3 seems robust to batch effect -meaning that groups treated with different doses at the same age and through the same mode of administration may be associated with dissimilar scores ; for instance, samples treated at P20 by intranasal injection yield score 0.80 when treated with a moderate weight-dependent dose, 0.51 when treated with a high dose, and 0.08 when treated with a low dose. This brings us to a second claim : dose does not have a monotonic effect on the reversion of the inflammation-induced gene expression change. We can notice, as a sanity check, that most of the scores are indeed positive, some being even rather close to 1 (up to 0.80). This supports the fact that most human MSC injections have a positive, therapeutic effect at transcriptomic level. One can notice that treatments with the intranasal route seem to rank relatively higher than treatments with intravenous injections. We checked our assumption on the impact on gene expression made in Section 7.3 : that is, the fact that changes are rather driven by groups of genes than by individual genes. In order to do so, instead of building signatures based on Characteristic Direction [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF], we considered signatures which coefficients were log 2 fold-change values associ- ated with Benjamini-Hochberg-adjusted p-values [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF] lower than 5%, as derived with the classic (univariate) differential analysis approach DESeq2 [START_REF] Love | Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2[END_REF]. The top plot in Figure 7.3 shows that, for DESeq2-derived signatures, few genes were in common (that is, with a nonzero coefficient in the signatures) across condition groups. That means that cosine scores computed on these signatures have a value close to 0. As such, these cosine score are weakly informative of the relevance of a given treatment protocol. To the contrary, the bottom plot of Figure 7.3 -that considers signatures computed through Characteristic Direction-shows that there is a small core subset of genes which have a nonzero coefficient in all signatures. These genes are Sparc, Csf1r, Lgmn, Ctsb, Cst3, AABR07006310.1, Mt-co1, Hexb, Ttr, and Apoe. It means that every cosine score at least relies on changes on this core set ; such, the score is indeed reflective of some transcriptional changes due to the treatment. These two plots highlight the fact that reversal in expression is essentially driven by groups of genes, and confirm our choice of using Characteristic Direction to build signatures. Moreover, since only nonzero coefficients are involved in the computation of the cosine score (Equation (7.1)), one might think that there might be a bias of higher cosine scores for treatment protocols which signature has a large number of common differentially expressed genes with the control signature CD[PBS IL1β]. We have tested this assumption by running a Spearman's ρ correlation test between the scores and the number of common differentially expressed genes with the control group. The test showed no significative linear correlation between the scores and the number of genes on which this score is computed (Spearman's ρ : 0.19, p = 0.44). Moreover, a linear regression from the number of common differentially expressed genes to the score has shown a bad fit (quantified by the residual R 2 = 0.04), as displayed by Figure 7.4. It is still interesting to note that this plot most likely gives away the reason why groups from the batch treated with intravenous injections at P5 are associated with low, sometimes even negative, scores in Table 7.2 ; the signatures associated with these groups have only a few genes in common with the control signature CD[PBS IL1β]. As such, the support of genes on which the associated scores are computed might not be informative enough.

Finally, in order to assess the relevance of the Top-2 candidates, that is, P20-INAS-Dose 2 and P10-INAS-Dose 3, I performed a pathway analysis on the Top-2 treatment protocols. Both candidates are associated with a score around 0.80 in Table 7.2. A pathway analysis allows the identification of the biological pathways which are most perturbed by these treatment protocols. I considered the over-representation analysis (ORA) [START_REF] Yaari | Quantitative set analysis for gene expression: a method to quantify gene set differential expression including gene-gene correlations[END_REF], which determines which sets of functionally similar genes -if there are anyare statistically surrepresented among a given subset of genes. Surrepresentation of a gene subset is computed by comparing the gene function categories represented in the subset (enrichments) to those present among a background set of genes, which contains the subset of interest. The gene function categories are annotated in Gene Onthology (GO) [START_REF] Ashburner | Gene ontology: tool for the unification of biology[END_REF]. In these annotations, genes are regrouped by biological processes in which they are involved (category GO Biological Process NoRedundant). The statistical tests -one per gene function category-were run by the online tool WebGestalt [START_REF] Liao | WebGestalt 2019: gene set analysis toolkit with revamped UIs and APIs[END_REF].

The gene subset associated with a treatment protocol T was built from the list of one-to-one orthologuous human genes with nonzero coefficients in the associated signature CD[T IL1β]. 4 We consider the human genes instead of rat genes because the gene categories for Homo sapiens are more robust. We also built two other gene lists, which respectively contains genes with positive (up-regulated), resp. negative (down-regulated), coefficients in the signature. This procedure output three lists of genes per treatment protocol : 5, 263 genes for P20-INAS-Dose 2, with 2, 631 up-regulated genes and 2, 632 down-regulated genes ; 6, 233 genes for P10-INAS-Dose 3, with 3, 534 up-regulated genes, and 2, 699 down-regulated genes. The background set of genes comprises of the 16, 699 human genes which are orthologuous to rat genes measured in all sequencing batches shown in Table 7.1. p-values associated with the enrichments should be adjusted for multiple hypothesis testing, for each gene function category -e.g., using Benjamini-Hochberg's correction [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF]. According to Figure 7.5, intranasal injection of a moderate dose of MSCs at a later stage (P20-INAS-Dose 2), as well as the intranasal injection of a high dose at postnatal day 10 (P10-INAS-Dose 3) perturb the expression of genes involved in the activation of granulocytes and neutrophils, which are cellular subtypes which play an important role in the immune system, and in the response to inflammation. Moreover, both treatments significantly down-regulate the expression of genes involved in phagocytosis and autophagia. These processes are activated in microglia exposed to the IL1β-induced inflammation, and are partly responsible for the deficit in myelin. Bottom plot : considering the subset of down-regulated genes.

Moreover, a similar enrichment analysis was performed on the core set of 9 genes (excluding AABR07006310.1, having no human orthologuous gene) which are significantly differentially expressed in all signatures. The following terms are significantly enriched at level 5% with an adjusted p-value of 0.021 : "regulation of cell morphogenesis", "neutrophil mediated immunity", "granulocyte activation" and "extracellular structure organization". The four genes that contribute to the enrichments in neutrophil-mediated immunity and granulocyte activation -which are important terms with regards to EoP, as discussed above-are CST3, CTSB, HEXB and TTR. The fact that this core set of genes comprises of genes involved in the response to inflammation is quite conforting with respect to the relevance of our ranking.

Discussion

Other experimental analyses, through the detection of the presence of myelin basic protein (MBP) via Western blots, showed a restoration of the levels of the MBP in stem cell-treated samples. It means that what we observe at transcriptomic level -positive scores of signature reversion-seems to match what is measured at the level of proteomics. This project showcases signature reversion, which is the key to ranking treatments. Signature reversion crucially relies on the definition of appropriate reference groups, in order to accurately quantify the transcriptional changes due to the treatment. This approach is agnostic to the disease, which enables its use in a generic fashion for other drug research investigations. Furthermore, this application highlights the potential of transcriptomics for drug repurposing. Even if transcriptomics might not be as informative as proteomics -i.e., the direct study of protein production-analysis of transcriptomic data still provides important clues about groups of genes affected by a disease. This confirms our choice of focusing on transcriptomic data throughout the thesis.

Lastly, this project allows us to take a step back to contemplate the big picture of this thesis. Contrary to the drug repurposing method proposed in my PhD, the main focus of this project was the design of a score associated with drug signatures, which already matched the appropriate cell lines and diseased tissues. The main difficulty of the drug repurposing method, as mentioned in introduction, lies in the design of appropriate drug signatures when the only available data is a set of in vitro treated profiles in immortalized cell lines -most likely cancerous, in the case of the LINCS L1000 database [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF]. This issue was tackled in Chapter 3, which combines the Boolean network in Chapter 2 with carefully built drug-induced perturbation signatures.

One might wonder why we did not apply the cosine score directly on the "stabilized" binary profiles predicted by the Boolean network and signature CD[Healthy Patients] built on patient and control profiles, as done in the project on EoP. This scoring method was actually shown to be less predictive than the current score function on a smaller subset of 10 molecules (4 antiepileptic and 6 proconvulsant drugs), where the proconvulsant or antiepileptic effect is provably assessed, except for Withaferin-A. The resulting rankings and receiver operating characteristic (ROC) curves for both methods are respectively shown in Figures 7.7 and 7.8. 5 This can actually be explained by the fact that the profiles predicted by the Boolean network are not exactly drug signatures per se : they are in silico treated patient profiles. As such, they do not represent changes in expression, but predicted expression levels in treated profiles. This is why the repurposing score described in Algorithm 3 aims at determining in which part of the 2D plane the in silico treated profile is -either on the hyperplane globally assigned to control samples, or the one mainly associated with patient profiles-and then computes the distance of that point to the frontier separating the two sides of the plane. For the sake of completeness, we also report the ranking and associated ROC curve when the cosine score method is applied to the whole set of 34 drugs mentioned in Chapter 3 in Figure 11.1 in Appendix. and the cosine score applied on "stabilized" profiles (right) on the smaller subset of 10 drugs. The baseline is L1000 CDS 2 [START_REF] Duan | L1000CDS2: LINCS L1000 characteristic direction signatures search engine[END_REF].

Chapter 8

Extension to a collaborative setting

In the last part of my PhD, I was interested in extending to a multi-agent setting the principles laid out for a single agent facing a problem of identification. That setting might be of interest in personalized drug repurposing, in which potentially different sets of drug candidates are recommended to different subpopulations of patients. Heterogeneous patient subpopulations are a common sight for versatile diseases which actually regroup a large number of pathologies : e.g., in breast cancer [START_REF] Sims | Origins of breast cancer subtypes and therapeutic implications[END_REF] or in epilepsy (Mirza, Stevelink, et al., 2021; Walker, Mirza, et al., 2015). In these diseases, a few number of patients might be afflicted with a specific subtype of disease, hence the idea of exploiting drug responses from other related patient subpopulations, while looking for the subpopulation-specific best drug candidates.

In the single-agent setting, the observations from a given action on the environment were assumed to be relatively uniform, as observations are "true" average scores with some additive noise of fixed variance equal to σ2 . Now, in order to model several subpopulations, a set of M agents is considered, across which expected rewards from the same arm might differ. Taking into account the observations made by other agents might decrease the total number of samples needed to determine the best drug candidates for each agent -especially as the reliance on others subpopulations growscontrary to the case where each agent works separately. However, in practice, broadcasting every new information about the observations to every agent might also be costly : indeed, launching a batch of trials is easier than having to wait for all other subpopulations to determine the next arm allocation. Moreover, with respect to patient data privacy -particularly in the context of adaptive clinical trials-sharing the raw patient drug response might also be harmful. Therefore, inspired by the "federated"1 bandit learning setting which was proposed for regret minimization [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF], a generalized framework for collaborative bandit Top-N identification is introduced. An associated novel information theoretic lower bound on the sample complexity was derived. Furthermore, a phased elimination-based algorithm, which is nearly optimal with respect to the proposed lower bound, 2 was designed. This algorithm features two novel key ideas -First, the tracking of an oracle allocation through a relaxed optimization problem, which is related to the characteristic time involved in the lower bound.

-Second, a data-dependent sampling scheme that selects the arms to sample in a more adaptive way than in prior works.

The combination of these two elements defines a new computationally cheaper approach to design near-optimal algorithms. This work has been accepted at the 36 th Conference on Neural Information Processing Systems (NeurIPS 2022) [START_REF] Réda | Near-Optimal Federated Learning in Bandits[END_REF], and was done in collaboration with Sattar Vakili (MediaTek Research).3

Collaborative Top-N (identification)

In this chapter, we do not consider specific model structures, and we consider exact Top-N identification (ε = 0). This work introduces a general multi-agent bandit model in which each agent is facing a finite set of K arms, and may communicate with the other M -1 agents through a central controller. The objective for an agent is to identify its own N best arms, in a context where optimality is defined with respect to mixed rewards. The mixed reward of an arm is a weighted sum of the rewards of this arm across agents, where the weight assigned by one agent to rewards observed by another is known. If positive (nonzero) weights are set on other agents' rewards, then communication between agents becomes necessary. This general setting recovers and extends several recent models for (centralized) collaborative bandit learning.

Let us now discuss the differences between this framework and the single-agent one discussed in Chapters 4 and 5. At each round, an agent can either sample an arm, or remain idle. In our case -dealing with a pure exploration problem-remaining idle means not sampling any arm, or, equivalently, sampling a "ghost" arm denoted 0, which returns observations equal to the smallest possible reward, so that it is quickly discarded from the set of top arms. that maximizes her local expected reward ; but rather, the set that maximizes a mixed reward, which definition relies on a known weight matrix

When

W ∶ = (w n,m ) n,m∈[M ] ∈ [0, 1] M ×M .
The m th column of matrix W quantifies the normalized contributions of each agent to the mixed reward of agent m, such that for any agent m ∈ [M ] n∈ [M ] w n,m = 1 .

In practice, this weight matrix could be obtained by considering a columnstochastic similarity matrix between patient subpopulations, based on their biomarkers, or transcriptional profiles. Then, as previously alluded to, the mixed reward at round t > 0 for agent m ∈ [M ] and arm I m t ∈ [K] is defined as a weighted sum of the local rewards across agents, had they sampled the same arm

I m t at round t Y ′ m t := n∈[M ] w n,m Y ′ n t .
The expectation of this (unobserved) mixed reward, called expected mixed reward, is

µ ′ (k,m) := n∈[M ] w n,m µ (k,n) .
We denote by S m N,µ :=

N arg max k∈[K]
µ ′ (k,m) the set of N arms with largest expected mixed rewards for agent m, which is assumed unique. When it is obvious, we refer to this set as S m N . Besides the degenerated case in which w n,m = 1(n = m), that is, each agent can solve their own bandit problem in isolation, agents need to communicate to fulfill their own identification targets ; i.e., to share information about their local rewards to other agents.

The communication model for an agent m ∈ [M ] is defined as follows : at each round t, this agent either remains idle ; or samples arm I m t and observes Y ′ m t ; or communicates information to a central server -e.g., the empirical means of its past local observations. This central server broadcasts this information to all other agents, at the price of some fixed communication cost. Similarly to the communication model described in [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF] [START_REF] Tao | Collaborative learning with limited interaction: Tight bounds for distributed exploration in multi-armed bandits[END_REF], communication between agents, i.e., broadcasting from the server, only happens at the end of local sampling phases for all agents ; that is, when all agents are idle at the same time. Like in single-agent settings, the analysis of the strategy adopted by the M agents will rely on the definition of mixed gaps related to pairs (agent, arm) Definition 8.1.1. Characteristic gap for collaborative bandits. For agent m ∈ [M ] and arm k ∈ [K], the gap is defined as follows

∆ ′ k,m := max µ ′ (k,m) - N +1 max a∈[K] µ ′ (a,m) , N max a∈[K] µ ′ (a,m) -µ ′ (k,m)
.

In particular, it implies that

∆ ′ k,m :=      N max a∈[K] µ ′ (a,m) -µ k if k ∈ S m N , µ ′ (k,m) - N +1 max a∈[K] µ ′ (a,m)
otherwise .

The analysis will also depend on arm-pairwise gaps related to an agent, that is, for any pair of arms (k, ℓ) and agent m

∆ ′ m k,ℓ := µ ′ (k,m) -µ ′ (ℓ,m) .
We denote the matrix in

R K 2 ×M , ∆ ′ := (∆ ′ m k,ℓ ) k,ℓ,m .
Our goal is to construct a δ-correct algorithm A, i.e., a set of sampling, stopping and decision rules, so that, for any model of local expected rewards µ ∈ R K×M and weight matrix W , the algorithm returns at stopping time τ A M sets of N arms ( S 1 N (τ A ), . . . , S M N (τ A )). These sets satisfy the property of correctness where I t is the event that the central server broadcasts information to all agents at round t.

P {µ,W } ∀m ∈ [M ], S m N (τ A ) ⊆ S m N ≥ 1 -δ ,
In the algorithm we propose in this chapter, Com {µ,W } (A) will be equal to the number of sampling phases, where agents observe local rewards from selected arms before sending information to the server. Similarly to the single-agent setting, a bound on the exploration cost can be expressed either in high probability (Equation (4.2)) or in expectation (Equation (4.3)).

Related work

The weighted collaborative model proposed above in the context of Top-N identification -and of pure exploration as a general rule-is novel. However, it encompasses different recent frameworks in collaborative learning, which formalized the case where several multi-armed bandits collaborate to efficiently perform sequential decision making (Zhu, Zhu, et al., 2021).

In particular, Shi, Shen, and Yang (2021) study a special case in which, given some personalization coefficient α ∈ [0, 1], the mixed reward for arm k ∈

[K] and agent m ∈ [M ] is an interpolation between the local expected reward µ (k,m) and the average reward across agents 1 M n∈[M ] µ (k,n) . This amounts to choosing W = (1 -α)I M + 1-α M 1 M ×M , where 1 M ×M is the matrix of size M × M with all coefficients equal to 1. In that work, authors consider the objective of minimizing regret in as few communication rounds as possible. Another work aimed at regret minimization considers a similar weighted model [START_REF] Wu | Contextual bandits in a collaborative environment[END_REF] in a different setting in which a central controller chooses in each round an arm for the unique agent (corresponding to a sub-population) that arrives. 4The counterpart pure exploration problem of fixed-confidence Top-N identification in a collaborative context is the focus of this chapter. The weighted collaborative framework described in Section 8.1 extends the well-studied fixed-confidence Top-N identification problem (Chen, Li, and Qiao, 2017; Even-Dar et al., 2006; Gabillon, Ghavamzadeh, and Lazaric, 2012; Kalyanakrishnan et al., 2012).

Another related setting is the "single-model" collaborative pure exploration (Hillel et al., 2013; Tao, Zhang, and Zhou, 2019; Yang, Chen, et al., 2021; Zhu, Mulle, et al., 2021), where M agents face the same best arm identification problem. Variants of this problem have been investigated. For instance, [START_REF] Yang | Cooperative stochastic bandits with asynchronous agents and constrained feedback[END_REF] consider asynchronous agents, which can only sample at some times ; whereas in [START_REF] Zhu | Decentralized Multi-Armed Bandit Can Outperform Classic Upper Confidence Bound[END_REF] ; Zhu, Zhu, et al. (2021), agents can only communicate information to some of the other agents. The goal in these papers is to reduce the sample complexity needed to solve a single pure exploration problem, at the cost of some communication rounds. Our model recovers the synchronous setting when considering µ (k,m) = µ (k,n) for any arm k ∈ [K] and agents m, n ∈ [M ], and by setting the weight matrix to W = I M .

An interesting kernelized collaborative pure exploration problem was recently studied by [START_REF] Du | Collaborative Pure Exploration in Kernel Bandit[END_REF]. In that work, both agents and arms are described by feature vectors, and there is a known kernel encoding the similarity between the mean reward of each (agent,arm) pair. This independent work follows a similar approach as ours and also propose a near-optimal phased elimination algorithm inspired by a lower bound, although the models and related lower bounds are significantly different. Finally, [START_REF] Russac | A/B/n Testing with Control in the Presence of Subpopulations[END_REF] considers a pure exploration task in which the expected reward of an arm is the weighted sum of its rewards across M distinct subpopulations. A parallel can be drawn between their setting and the weighted collaborative framework by setting each weight w n,m to some α n independent from m, so that the best arm is common to all subpopulations. However, the proposed algorithms do not aim at a low communication cost across subpopulations. Moreover, in the setting described in Section 8.1, an agent can potentially consider any linear combination of the others' observations in its mixed rewards, that is, any degree of personalization across agents.

In bandits working in collaboration, the need for a small communication cost makes algorithms based on phased eliminations appealing. In this category of algorithms, agents maintain a set of a active arms that are candidates for being optimal. At the end of each sampling phase, arms are possibly eliminated from this set, until the stopping criterion is met. Adaptivity to the observed rewards -and, in our case, communication-is only needed between sampling phases, which are typically long. This type of structure has been used in various bandit settings, both for regret minimization [START_REF] Auer | Using confidence bounds for exploitation-exploration tradeoffs[END_REF]Ortner, 2010; Shi, Shen, andYang, 2021) and pure exploration (Fiez et al., 2019; Hassidim, Kupfer, and Singer, 2020; Hillel et al., 2013). In some of these algorithms, including [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF], the number of samples gathered from an arm which is active in some phase r is fixed in advance. Going beyond such a deterministic sampling scheme might be crucial to achieve optimal performance with phased algorithms. In order to achieve (near)-optimality, others phased algorithms rely on the computation of an oracle allocation from the optimization problem related to a lower bound on sample complexity (Du et al., 2021; Fiez et al., 2019; Garivier and Kaufmann, 2016; Russac et al., 2021). In [START_REF] Du | Collaborative Pure Exploration in Kernel Bandit[END_REF]); Fiez et al. (2019), based on this allocation, a total number of samples to get in the current round is computed, depending on the identity of the surviving arms, that is, arms that remain in the candidate set. The distribution of samples across arms for the current round is proportional to the oracle allocation, and is obtained through a rounding procedure. Compared to these works, the algorithm proposed in this chapter is built on an allocation inspired by a relaxation of a lower bound, which does not only depend on the identity of surviving arms, and an alternative to the rounding procedure.

In our setting, we do not put constraints on the type of information that is exchanged in each communication round -which can be interesting when we consider privacy issues [START_REF] Dubey | Differentially-private federated linear bandits[END_REF]Pentland, 2020; Zhu, Zhu, et al., 2021)nor on the lengths of the messages. Each communication round has a unit cost. In a communication round, all agents send messages to the central server -e.g., estimates of their local means-and the server can send back arbitrary quantities or instructions -e.g., how many times each arm should be sampled in the next exploration phase, and when to communicate next.

Moreover, contrary to the works of [START_REF] Hillel | Distributed exploration in multi-armed bandits[END_REF] [START_REF] Tao | Collaborative learning with limited interaction: Tight bounds for distributed exploration in multi-armed bandits[END_REF] on collaborative learning, we do not look at strategies that explicitly minimize for the number of communication rounds. Instead, our approach consists in proving a lower bound on the smallest possible exploration cost of a δ-correct algorithm which would communicate at every round ; and then, finding an algorithm which exploration cost matches this lower bound, while suffering a reasonable communication cost.

Lower bound for collaborative Top-N

We first present a lower bound on the exploration cost, i.e., the total number of samples across arms and agents, needed for any δ-correct algorithm A to make a decision ( S m N (τ A )) m∈ [M ] . This lower bound holds on the exploration cost of a collaborative exact Top-N identification algorithm, in which all agents communicate to the central server their latest observation as soon as they received it. Moreover, it assumes that we consider unstructured Gaussian bandits with fixed variance σ 2 ; meaning that the reward Y ′ m t from arm I m t ∈

[K] observed at round t > 0 by agent m ∈ [M ] will be drawn from N (µ

(I m t ,m) , σ 2 ) Y ′ m t ∼ µ (I m t ,m) + ψ m t where ψ m t ∼ N (0, σ 2 ) .
Similarly to the lower bound in single-agent settings (Result 4.3.3), this lower bound will be expressed as the value5 of an optimization problem involving the characteristic gaps, inflated by a factor of order O(δ -1 ).

Definition 8.3.1. Oracle problem. Optimization problem P ⋆ on any vector ∆ ∈ R K×K×M is defined as follows

P ⋆ (∆) := arg min T ∈(R + ) K×M m,k T k,m (1) s.t. ∀m ∈ [M ] ∀k ∈ S m N ∀ℓ ∈ S m N , n∈[M ] w 2 n,m 1 T k,n + 1 T ℓ,n ≤ (∆[k,ℓ,m]) 2 2 . (2)
If allocation T ∈ R K×M satisfies T ∈ P ⋆ (∆), it means that T is a minimizer of Problem P ⋆ , which minimizes the objective (1) while satisfying the constraints (2).

Before stating the lower bound, we further assume that the weight matrix

W satisfies w n,m = 0 for any agent m ∈ [M ].
This assumption means that the mixed reward of any agent -at least partially-depends on her own local reward.

Theorem 8.3.2. Lower bound on the exploration cost for collaborative Top-N identification. Let µ be a fixed matrix of expected rewards in R K×M . For any δ ∈ (0, 1/2], let A be a δ-correct algorithm under which each agent communicates each reward to the central server after it is observed. Then the expected exploration cost of algorithm A satisfies

E[Exp µ (A)] ≥ N ⋆ (µ) log 1 2.4δ
,

where N ⋆ (µ) := m∈[M ] k∈[K] τ ⋆ k,m and τ ⋆ ∈ P ⋆ (∆ ′ ), and
∆ ′ := (∆ ′ m k,ℓ ) k,ℓ,m .
The full proof (displayed below) uses standard change-of-distribution arguments, together with classical results from constrained optimization. Since the setting only deals with unstructured models, we drop all notations related to model functions, and directly consider bandit models instead.

Proof. Let us use Lemma 6.3.1 to define the set of alternative models to µ and λ is an alternative model to µ if and only if

(λ ′ (ℓ,m) -µ ′ (ℓ,m) ) -(λ ′ (k,m) -µ ′ (k,m) ) = n∈[M ] w n,m (γ ′ n + γ n ) ≥ ∆ ′ m k,ℓ := µ ′ (k,m) -µ ′ (ℓ,m) . (8.2)
Then, finding λ such that Equation (8.1) is as tight as possible -that is, building the closest alternative model to µ-yields the following constrained optimization problem in γ, γ

′ inf γ,γ ′ ∈(R + ) M Equation 8.2 holds   n∈[M ] τ k,n γ 2 n 2σ 2 + n∈[M ] τ ℓ,n γ ′2 n 2σ 2   .
The infimum can be computed in closed form using constraint optimization.

Introducing a Lagrange multiplier Γ ∈ R + and the minimizer (γ ⋆ , γ ′⋆ ), from the KKT conditions (Karush, 1939; Kuhn and[START_REF] Kuhn | Nonlinear programming[END_REF] and the fact that (τ q,n ) q,n are positive, we get that, for any agent n,

σ -2 τ k,n γ ⋆ n -Γw n,m = 0 =⇒ γ ⋆ n = Γw n,m σ 2 τ k,n σ -2 τ ℓ,n γ ⋆ n -Γw n,m = 0 =⇒ γ ′⋆ n = Γw n,m σ 2 τ ℓ,n Γ   ∆ ′ m k,ℓ - n∈[M ] w n,m (γ ⋆ n + γ ′⋆ n )   = 0 .
Solution Γ = 0 is not acceptable, because otherwise γ ⋆ n = γ ′⋆ n = 0 for any agent n ∈ [M ], and then λ ∈ Alt(µ). If Γ = 0, then

∆ ′ m k,ℓ = Γ σ 2 n w 2 n,m τ -1 k,n + τ -1 ℓ,n =⇒ Γ = σ 2 ∆ ′ m k,ℓ n w 2 n,m (τ -1 k,n + τ -1 ℓ,n ) ,
And finally, plugging this expression into γ ⋆ and γ ′⋆ yields the following minimizer

∀n ∈ [M ], γ ⋆ n = ∆ ′ m k,ℓ w n,m τ -1 k,n n ′ ∈[M ] w 2 n ′ ,m (τ -1 k,n ′ + τ -1 ℓ,n ′ ) and γ ′⋆ n = ∆ ′ m k,ℓ w n,m τ -1 ℓ,n n ′ ∈[M ] w 2 n ′ ,m (τ -1 k,n ′ + τ -1 ℓ,n ′ )
, and the conclusion follows by plugging these expressions to get the expression of the infimum.

Note that this theorem only focuses on the exploration cost, without attempting to minimize the number of communication rounds. Indeed, the number of communication rounds for an algorithm in which the central server shares information after each new observation is equal to the exploration cost, and is thus suboptimal in this regard.

Note that, for the single-agent case (M = 1), Theorem 8.3.2 recover the complexity of best arm identification in a Gaussian bandit model [START_REF] Garivier | Optimal best arm identification with fixed confidence[END_REF].

Structure for nearly-optimal Top-N

In single-agent pure exploration tasks, lower bounds can guide the design of optimal algorithms, as they allow to recover an oracle allocation -i.e., the arg min for τ ∈ (R + ) K×M in Problem (8.3.1). Algorithms may try to achieve this allocation by using some tracking (Du et al., 2021; Garivier and Kaufmann, 2016; Russac et al., 2021). Yet, these approaches may be computationally expensive, as they solve the optimization problem featured in the lower bound in every phase.

In this section, a third approach to design near-optimal algorithms in collaborative settings is introduced. This approach leverages the knowledge of the lower bound within a phased elimination algorithm. This is crucial to maintain a small communication cost Com µ (A), and also to reduce the computational complexity compared to a pure tracking approach. The algorithm derived from this approach relies on a relaxed complexity term N (µ). As we will see, this term is within constant factors of the true constant N ⋆ (µ). This implies that the upper bound on the exploration cost incurred by algorithm A will ultimately depend on N ⋆ (µ).

Relaxation of the lower bound optimization problem

The main issue with Problem (8.3.1) is its dependence on the knowledge of the true set S m N for any agent m ∈ [M ]. In other algorithms which aim at asymptotic optimality -for instance, MisLid for Top-N identification for misspecified bandits in Chapter 6-it requires using the empirical best arms as proxy, and controlling the gap between this empirical estimate and the true one.

A simpler approach is actually to consider the following problem P instead, for any matrix ∆ ∈ (R + ) K×M P(∆) := arg min

τ ∈(R + ) K×M m,k τ k,m s.t. ∀m ∈ [M ] ∀k ∈ [K], n∈[M ] w 2 n,m τ k,n ≤ (∆ k,m ) 2 2 . (8.3)
In Top-N identification, the value of Problem P on the characteristic gaps is where τ ∈ P (∆ ′ ) and τ ⋆ ∈ P ⋆ (∆ ′ ), then

(i). N (µ) ≤ N ⋆ (µ) .
Moreover, for N = 1

(ii). N (µ) ≤ N ⋆ (µ) ≤ 2 N (µ) .
Proof. Since we consider a single model µ here, we drop the indexing by µ in all notations in this proof.

(i).

The following set of constraints of the oracle problem

Const ⋆ N := t ∈ (R + ) K×M | ∀m ∈ [M ] ∀k ∈ S m N ∀l ∈ S m N , n w 2 n,m t -1 k,n + t -1 l,n ≤ ∆ ′ m k,l 2 2
is included in the set of constraints of the relaxed problem in Equation 8.3

Const

N := t ∈ (R + ) K×M | ∀m ∈ [M ] ∀k ∈ [K], n w 2 n,m t k,n ≤ ∆ ′ k,m 2 2 .
Indeed, if t ∈ Const ⋆ N , then for any m ∈ [M ], and any l ∈ S m N ,

∀k ∈ S m N , n 0 + w 2 n,m t l,n ≤ n w 2 n,m 1 t k,n + 1 t l,n ≤ ∆ ′ m k,l 2 2 =⇒ n w 2 n,m t l,n ≤ min k∈S m N (∆ ′ m k,l ) 2 2 = N max q∈[K] µ (q,m) -µ ′ (l,m) 2 2 = ∆ ′ k,m 2 2 ,
Symmetrically, for any agent m and k ∈ S m N , one can check that n w 2 n,m t -1 k,n ≤

= k m ⋆ n∈[M ] w 2 n,m 2τ k,n + w 2 n,m 2τ k m ⋆ ,n = 1 2   n∈[M ] w 2 n,m τk,n   ≤(∆ ′ k,m ) 2 /2 + 1 2   n∈[M ] w 2 n,m τk m ⋆ ,n   ( * ) ≤ ∆ ′ k,m 2 /2 .
where ( * ) uses the fact that by Definition 8.1.

1 for N = 1 ∆ ′ k m ⋆ ,m := µ ′ (k m ⋆ ,m) - 2 max q∈[K] µ ′ (q,m) = min q̸ =k m ⋆ ∆ ′ q,m ≤ ∆ ′ k,m .
Then 2τ ∈ Const ⋆ 1 , therefore once again by minimality, N ⋆ ≤ 2 N .

Compared to Problem P ⋆ , a nice feature of P is that its constraint set does not depend on the knowledge of (S m N ) m∈[M ] , which will allow us to design algorithms that do not suffer too much from bad empirical guesses on the N -best arms, especially in early phases.

Remark 8.4.2. We further note that solving P is slightly easier than solving P ⋆ . Indeed, this optimization problem can be decoupled across arms, and it is sufficient to compute, for any arm k ∈ [K] arg min

τ k ∈(R + ) M m∈[M ] τ k m s.t. ∀m ∈ [M ], n∈[M ] w 2 n,m τ k n ≤ ∆ ′ k,m 2 2 ,
and then consider as the oracle allocation for the problem on the whole set of arms (τ k,m ) k,m := (τ k m ) k,m . To solve this convex optimization problem, we may rely on solvers for disciplined convex optimization, such as those implemented in CVXPy [START_REF] Agrawal | A rewriting system for convex optimization problems[END_REF][START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF].

However, we cannot get access to the true gaps (∆ ′ k,m ) k,m . Thus, our algorithm will rely on gap proxies ( ∆ ′ k,m ) k,m which will depend on past samplings and observations. The following lemma will be crucial in the analysis of the algorithm, in order to compare values, under some condition, from P(∆ ′ ) and

1 β 2 k,m τ k,m ≤ k,m τ ′ k,m .

Collaborative Phased Elimination (CPE)

We now introduce an algorithm for collaborative Top-N identification, called CPE and stated as Algorithm 8. CPE proceeds in phases, indexed by r. In phase r, we let B m (r) be the set of active arms for agent m -that is, the set of candidate arms that agent m keeps sampling at phase r-and B(r) := m∈[M ] B m (r) be the set of arms that are active for at least one agent at phase r. The algorithm maintains proxy gaps ( ∆ ′ k,m (r)) k,m for the true gaps (∆ ′ k,m ) k,m that are halved at the end of each phase for arms that remain active, hence the dependence in past observations. At the beginning of each phase r, the oracle allocation t(r), with respect to the proxy gaps, is computed, as well as the number of new samples (d k,m (r)) k,m that player m should get from arm k in phase r. For any arm k and agent m, d k,m (r) is defined such that the total number of selections n k,m (r) of arm k by agent m up to round r included becomes close to -a quantity slightly larger thant k,m (r) log (1/δ).

We observe that any arm k ∈ B(r) will not get any new samples in phase r, as the associated proxy gaps

( ∆ ′ k,n ) n∈[M ] are identical to those in the previous phase r -1, hence t k,n (r) = t k,n (r -1) and d k,n (r) = 0 for any agent n ∈ [M ].
In contrast to prior works, where the allocation in each round only depends on the identity of the surviving arms and the round index (Du et al., 2021; Fiez et al., 2019), in CPE it also depends on when the arms have been eliminated (which condition which gaps are frozen). After each agent m samples arm k d k,m (r) times, they all send their empirical local expected rewards µ (k,m) (r) k to the central server, which computes the empirical mixed

expected rewards µ ′ (k,m) (r) k,m,r ∀m ∈ [M ] ∀k ∈ [K] ∀r ≥ 0, µ ′ (k,m) (r) := n∈[M ] w n,m µ (k,m) (r) .
The active sets (B m (r)) m∈[M ] of all agents are then updated by removing arms whose empirical mixed expected rewards are too small. The number of communication rounds in Algorithm 8 is then exactly equal to the number of phases needed until the stopping criterion in Line 29 is fulfilled.

As in several prior works [START_REF] Kaufmann | Analysis of bayesian and frequentist strategies for sequential resource allocation[END_REF]Kalyanakrishnan, 2013; Shi, Shen, andYang, 2021), we rely on confidence intervals to perform these eliminations. However, constructing confidence intervals on the mixed expected rewards under our adaptive sampling rule is more challenging than when the number of samples from an active arm in phase r is fixed in advance. We build an confidence interval of the form µ (k,m) (r) ± Ω (k,m) (r) Algorithm 8 CPE algorithm for the collaborative setting. Collaborative Phased Elimination (CPE)

1: Initialize r ← 0, ∀k, m, ∆ ′ k,m (0) ← 1, n k,m (0) ← 1, ∀m, B m (0) ← [K] 2:
B(r) ← m∈[M ] B m (r) 7: Compute t(r) ← P √ 2 ∆ ′ k,m (r) k,m 8: For all k ∈ [K], compute (d k,m (r)) m∈[M ] ← arg min d∈N M m d m s.t. ∀m ∈ [M ], n k,m (r -1) + d m T δ (n k,• (r -1) + d) ≥ t k,m (r)
B m (r+1) ← k ∈ B m (r) | µ ′ (k,m) (r) + Ω (k,m) (r) ≥ N max j∈Bm(r) µ ′ (j,m) (r) -Ω (j,m) (r
Ω (k,m) (r) := T δ (n k,• (r)) n∈[M ] w 2 n,m n k,n (r) , where T δ (•) : (R + ) M → R + , N → T δ (N )
is a threshold function to be defined.

A choice of threshold that gives valid confidence intervals can be derived by leveraging some time-uniform concentration inequalities from Kaufmann and Koolen (2021, Proposition 24). The proof is postponed to the Appendix (Lemma 13.1.2).

Lemma 8.4.5. Choice of threshold function T

δ (•). For any N ∈ (N + ) M T δ (N ) := 2   g M δ KM + 2 m∈[M ] ln (4 + ln(N m ))   ,
where g M is a function that satisfies g M (δ) ≈ log(1/δ) + M log log(1/δ). Then the good event

E CPE ∶ = ∀r ≥ 0 ∀m ∈ [M ] ∀k ∈ [K], µ ′ (k,m) -µ ′ (k,m) (r) ≤ Ω (k,m) (r)
holds with probability larger than 1 -δ.

From this lemma and the elimination criterion at Line 17 in Algorithm 8, it easily follows that CPE for Top-N identification is δ-correct for this choice of threshold function, as, for any agent m ∈ [M ], no good arm k ∈ S m N can ever be eliminated from B m (r) at any round r on event E CPE . Theorem 8.4.6. CPE is δ-correct. On event E CPE , CPE outputs the correct set of optimal arms S m N for each agent m.

The fact that the sample complexity of CPE scales with N (and N ⋆ as well, thanks to Lemma 8.4.1) stems from the interplay between the confidence interval width Ω (k,m) (t) -which, up to the threshold function T δ (•), is exactly one of the constraints featured in the lower bound (Theorem 8.3.2)-and the definition of the allocation t(r). This leads to the following crucial result Lemma 8.4.7. On event E CPE , for any arm k, agent m, and round r

Ω (k,m) (r) ≤ ∆ ′ k,m (r) .
Proof. For any round r and arm k, using first the definition of d(r) at Line 7 and then the definition of oracle allocation t(r) at Line 6 in Algorithm 8

Ω (k,m) (r) := T δ (n k,• (r)) n∈[M ] w 2 n,m n k,n (r) = n∈[M ] w 2 n,m T δ (n k,• (r -1) + d k,• ) n k,n (r -1) + d k,n (r) ≤    n∈[M ] w 2 n,m t k,n (r) ≤ ∆ ′ k,m (r) if k ∈ B(r) n∈[M ] w 2 n,m t k,n (r ′ k ) ≤ ∆ ′ k,m (r ′ k ) = ∆ ′ k,m (r) ( * ) otherwise . where r ′ k := sup {r ′ ≥ 0 | k ∈ B(r ′ )} when k ∈ B(r) and ( * ) uses the fact that d k,m (r) = 0 when k ∈ B(r).
All in all, the following upper bound in high probability on the exploration cost of CPE is proven Theorem 8.4.8. Upper bound on Exp µ (CPE). On any model µ, with probability 1 -δ, CPE outputs the set of N best arms for each agent with an exploration cost at most

32N ⋆ (µ) log 2 8 ∆ ′ min log (1/δ) + o (1/δ) where ∆ ′ min := min m∈[M ] min k∈[K] ∆ ′ k,m ,
and at most log 2 8 ∆ ′ min communication rounds.

A sketch of the proof will be given in this section. The full proofs of Theorems 8.4.6 and 8.4.8 are available in Appendix (Theorems 13.2.1 and 13.3.1). This theorem proves that CPE is matching the exploration lower bound of Theorem 8.3.2 in the asymptotic regime of small values of δ, up to a logarithmic term in O (1/∆ ′ min ). It achieves this using only log 2 (8/∆ ′ min ) communication rounds. We note that a similar extra multiplicative logarithmic factor is present in the analysis of near-optimal phased algorithms in other contexts (Du et al., 2021; Fiez et al., 2019). Such a quantity appears as an upper bound on the number of phases, and may be a price to pay for the phased structure.

We argue that the communication cost of CPE is actually of the same order of magnitude as that featured in some related work. In [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF], which is the closest setting to our framework, the equivalent number of communication rounds p needed to solve the regret minimization problem is upper bounded by O 2 log 2 algorithm can be reached by using at most log 2 (1/∆ min ) communication rounds, where ∆ min is the gap between the best and second best arms.

Sketch of proof.

We let R denote the random number of phases used by Algorithm 8 before stopping. On the good event E CPE , considering any agent m ∈ [M ], we have proven that the algorithm never eliminates any good arm k ∈ S m N (Theorem 8.4.6), hence R := max

m∈[M ] max k̸ ∈S m N R k,m
, where R k,m is the last phase r in which k ∈ B m (r). Using Lemma 8.4.7, we can easily establish that

R k,m ≤ r k,m := min r ≥ 0 | 4 × 2 -r < ∆ ′ k,m , which satisfies r k,m ≤ log 2 8/∆ ′ k,m . This yields R ≤ log 2 (8/∆ ′ min )
, and further permits to prove that the proxy gaps ∆ ′ k,m (r)

k,m,r can be lower bounded by the true gaps

∀r ≤ R ∀k ∈ [K] ∀m ∈ [M ], ∆ ′ k,m (r) ≥ 1/8∆ ′ k,m .
Using the monotonicity properties of the oracle that are stated in Lemma 8.4.3, we establish that the allocation t(r) computed from the proxy gaps at Line 7 in Algorithm 8 satisfies

∀r ≤ R, m∈[M ] k∈[K] t k,m (r) ≤ 32 N (µ) . (8.4)
To upper bound the exploration cost Exp µ (CPE), the next step is to relate the number of selections of arm k by agent m up to phase R n k,m (R) to the oracle allocations. To do so, we observe that if R k,m is the last phase r before R such that the number of selections of arm k by agent m in phase r

d k,m (r) is positive, then n k,m (R) = n k,m ( R k,m
), and by definition of the quantities

(d k,m (r)) k,m,r and T δ (•) n k,m ( R k,m ) ≤ t k,m ( R k,m )T δ (n k,• ( R k,m )) + 1 ≤ t k,m ( R k,m )T δ (n k,• (R)) + 1 . (8.5) Then τ CPE := m,k n k,m (R) is bounded from above by τ CPE ≤ m,k t k,m ( R ′ k,m )T δ (n k,• (R)) + KM ≤ m,k r≤R t k,m (r)β ⋆ (τ CPE ) + KM ≤ 32R N (µ)β ⋆ (τ CPE ) + KM ,
where we use Equation 8.4 and introduce

β ⋆ (τ CPE ) := 2 g M δ KM + 2M ln(4 + ln(τ CPE )) .
Then, using the upper bound on the total number of rounds R and Lemma 8.4.1

τ CPE ≤ 32 log (8/∆ ′ min ) N ⋆ (µ)β ⋆ (τ CPE ) + KM . (8.6)
The end of the proof consists in using the known upper bound on R, and finding an upper bound for the largest τ A satisfying Equality 8.6.

Application to drug repurposing

Our general framework for any weight matrix W and µ ∈ R K×M has not been studied in pure exploration prior to this work. However, we can compare it in known settings, namely, in the personalized collaborative best arm identification, which is a straightforward counterpart to personalized federated regret minimization introduced in Shi, Shen, and Yang (2021). As we did throughout this chapter, we always consider Gaussian bandits with fixed variance σ 2 = 1.

Baseline algorithm for collaborative best arm identification (BAI).

We state below as PF-UCB-BAI (Algorithm 9) a straightforward adaptation of the PF-UCB algorithm in [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF] to personalized collaborative best arm identification ; meaning that only weight matrices of the form w n,m = α1(n = m) + 1-α M for any pair of agents (n, m) ∈ [M ] 2 are considered.

The original regret algorithm uses phased eliminations designed for each agent to identify their best arm together with exploitation : when all best arms have been found, or when some agent is waiting for others to finish their own exploration rounds, agents keep playing their empirical best arm.

To turn this into a δ-correct algorithm, we remove the exploitation rounds ; keep the same sampling rule within each phase -in which the number of samples from each arm is proportional to some rate function f (r) ; and calibrate the size of the confidence intervals used to perform eliminations slightly differently, introducing for any δ ∈ (0, 1) function

∀r ≥ 0, B r (δ) := 2 log (KM ζ(β)r β /δ) M F (r) .
where F (r) = r p=1 f (p), for some β > 1, and where ζ is the Riemann zeta function. In practice, we use β = 2. This algorithm follows the same general structure as our algorithm, with the notable difference that the number of samples of an arm k ∈ B m (r) in phase r is fixed in advance. PF-UCB-BAI is indeed δ-correct. On the good event G introduced in the following Lemma 8.5.1, the optimal arm for any agent m can never be eliminated from the set B m (r) at phase r.

Algorithm 9 PF-UCB-BAI. Benckmark algorithm for Collaborative Top-N 1: f (r): sampling effort in phase r, B r (δ): size of the confidence intervals in phase r.

2: Initialize r ← 0 , ∀k, m, n k,m (0) ← 0 , ∀m, B m (0) ← [K] , km ← 0 . 3: repeat 4: 5: # * Central server side * 6: if |B m (r)| = 1 then 7: km ← the unique arm in B m (r) 8: B m (r) ← ∅ 9: end if 10: B(r) ← m∈[M ] B m (r) 11: for k ∈ [K], m ∈ [M ] do 12: if k ∈ B(r) then 13: d k,m (r) ← (1 -α)f (r) + αM f (r) 1 (k ∈ B m (r)) else d k,m (r) ← 0
B m (r + 1) ← k ∈ B m (r) | μ′ k,m (r) + B r (δ) ≥ max j∈Bm(r)
G := ∀r ∈ N * ∀m ∈ [M ] ∀k ∈ B m (r), µ ′ (k,m) (r) -µ ′ (k,m) ≤ B r (δ)
holds with probability 1 -δ.

This lemma is a simple adaptation of Shi, Shen, and Yang (2021, Lemma 1) combined with a union bound on r ∈ N * . Moreover, on event G, similarly to the analysis of PF-UCB [START_REF] Shi | Federated multi-armed bandits with personalization[END_REF], we can upper bound the number of rounds where arm k is sampled by agent m by

p k,m := inf{r : B r (δ) ≤ ∆ ′ k,m /4} . When f (p) = 2 p , one can prove that p k,m p=1 f (p) = O M -1 ∆ ′-2 k,m log(δ -1 ) .
Summing the (deterministic) global and local exploration cost over rounds, arms and agents, yields an exploration cost of order

O    k∈[K]       1 -α min n∈[M ] ∆ ′ k,n 2    +   m∈[M ] α ∆ ′ k,m 2      log 1 δ    .
This algorithm can therefore serve as a baseline to be compared to our proposal in this particular case.

Drug repurposing. As mentioned in Chapter 6, in fixed-confidence BAI as well, algorithms using symmetric confidence intervals of width

2T δ (n k,• (r))g 1 n k,• (r)
for some arm k and some linear function g -which ensures that the confidence interval holds with probability 1 -δ-are usually quite conservative. Indeed, their reported empirical error frequency is usually a lot lower than the expected bound δ. As often done in other fixed-confidence best arm identification papers [START_REF] Kaufmann | Analysis of bayesian and frequentist strategies for sequential resource allocation[END_REF]Kalyanakrishnan, 2013; Réda, Kaufmann, andDelahaye-Duriez, 2021) For the baseline PF-UCB-BAI introduced above, which relies on a function f of the length of the exploration phase at phase p, we consider f (p) := 2 p log(1/δ) for p ≥ 0. Our experiment compares the performance of Algorithm 8 with the baseline algorithm PF-UCB-BAI on the drug repurposing instance mentioned in Chapter 3 with K = 21 drugs -note that here, we do not take into account the feature vectors in R 194 , so we do not need to apply feature transformations as we did in Chapters 5 and 6. We consider the setting where N = 1 and the weight matrix is of the form αI M + (1 -α)/M 1 M ×M , since the correctness of PF-UCB-BAI only holds in that case. We set δ = 0.1, and change the degree of personalization α ∈ {0.4, 0.6, 0.8}.

In order to determine the M clusters of patients, we first applied a Principal Component Analysis (PCA) transformation (using the first two principal components) on the standard-normalized matrix of transcriptomic profiles from epileptic patients [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF]. Then, we ran k-means++ [START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF] with M clusters. The clusters and the patient datapoints on the PCA-transformed standard-normalized matrix are represented on the scatter plot on Figure 8.1. According to the scatter plot of the datapoints, the choice M = 3 seemed the most appropriate. Then, per cluster m ∈ [M ] and treatment k ∈ [K], we defined µ (k,m) as the average repurposing score (as defined in Chapter 3) obtained by treatment k ∈ [K] across all patients in cluster m.

Results are shown in Table 8.1. Indeed, using the "heuristic" confidence interval has no consequences on the empirical error frequency δ. In terms of exploration and communication cost, CPE (Algorithm 8) improves considerably over the baseline, while being -for this specific instance of drug repurposing-extremely robust to changes in α, and thus to changes in ∆ ′ min .

Discussion

This work introduced a general framework for collaborative, or centralized, learning in multi-armed bandits, along with a lower bound on the associated exploration cost. Furthermore, we proposed a phased elimination algorithm for fixed-confidence Top-N identification. This algorithm tracks the optimal allocation from the pure exploration lower bound by considering a relaxed optimization problem instead, which is in contrast with prior works. Its exploration cost is matching the lower bound up to logarithmic factors, within a reasonable number of communication rounds.

As mentioned in introduction, our collaborative setting was motivated by the design of collaborative adaptive clinical trials for personalized drug recommendations, where several patient subpopulations (for instance, representing several subtypes of cancer) are considered and sequentially treated. However, in practice, especially when dealing with patient data, disclosing the mean response values to the central servor should be handled with care to preserve the anonymity of the patients. As such, the current solution is useful only to trustful partners. A possible solution to overcome this problem would be to carefully combine our algorithm with a data privacy-preserving method, for instance by adding some noise to the data [START_REF] Dubey | Differentially-private federated linear bandits[END_REF].

Finally, one might wonder why a given subpopulation should be interested in outputs from other subpopulations (and, as such, be associated with a non-entirely diagonal weight matrix). In the context of rare diseases or subtypes, the number of patients might not be large to give a robust estimate of the success rate/score of the selected arm, hence the use of outcomes from other populations, weighted by their similarity to the original set of patients. A first approximation of this setting is the suggested "fixed weighted collaborative setting" introduced in this chapter. An interesting subsequent work could rely on the relaxation of these fixed weights to a more adaptive weighting scheme. Another possibility is considering a kernel function instead of the fixed weights, as done in [START_REF] Du | Collaborative Pure Exploration in Kernel Bandit[END_REF] ; however, it raises the issue of selecting an appropriate kernel, as opposed to computing the similarity coefficients in our use case.

Summary

My PhD aims at tackling the problem of drug repurposing. This is the product of an interdisciplinary work, combining bioinformatics, through the modelling and analysis of drug-induced transcriptomic changes, and Computer Science, via the use of machine learning techniques for recommendation of treatments. I have focused on the approach of signature reversion, which compares the treatment-induced genewise expression changes to the expected changes associated with a good treatment. The main objective of this PhD was to develop a method which is disease-agnostic and easily reproducible -that is, that can be used in a generic and systematic fashion.

Chapter 7 aimed at describing a real-life application of signature reversion, to identify the optimal treatment protocol for stem cells against encephalopathy of prematurity, based on in vivo rat models. However, the main issue in drug repurposing based on signature reversion is data availability, especially for specific cell lines. To overcome this hurdle, Chapter 3 describes a method that is able to score the effect of a treatment on specific cell lines, based on drug signatures carefully built on in vitro experiments, and a dynamical system (Boolean network) inferred from scratch in Chapter 2. This network plays an important role in scoring treatments, as this dynamical system is able to score the effect of a treatment on a given patient for the considered disease. This score was slightly more predictive than traditional methods for signature reversion. Nonetheless, the previous method, when run on a set of repurposable drugs -which can be as large as millions of drugs to be screened-can be time-consuming. This motivates the works described in Chapters 5 and 6, which investigate the use of a specific type of sequential learning algorithms, called multi-armed bandit algorithms. Multi-armed bandits might be helpful to reduce the number of calls to the simulations run by the Boolean network, in order to identify the N most promising drug candidates. This approach can be extended to a collaborative context, where drug candidates can be simultaneously identified for heterogeneous subpopulations of patients, as described in Chapter 8. Application in a more realistic setting is laid out in Appendix 14. It confirms the theoretical and empirical observations made throughout the thesis, and shows that the methods developed during the time course of the PhD can be robust in the face of real life data. This thesis is a new step towards the full automation of drug development research, which might allow the speed-up of early clinical phases for the identification of interesting molecules, and further increase reprodubility of clinical results. Tackling poor data availability is particularly of interest when dealing with rare diseases.

Next steps

The importance of feature selection is highlighted throughout the thesis ; more particularly in Chapter 3, where the choice of the appropriate cell line to compute drug signatures impacts the identification of gene targets ; and in Chapter 6, where the number of samples needed at the start of the algorithm is correlated with feature collinearity. Several other venues for research could be explored :

Adding new types of features in the bandit algorithm. In order to better estimate the similarity between two drugs with close rewards from the gene regulatory network, one might be interested in adding supplementary information in feature matrix X. For instance, information about the chemical structure of the molecules, or about their drugability, that is, their ability to be administered as a drug. Even if that supplementary information is not exploited in the simulation of regulatory cascades, it may have an impact both on the sample-efficiency of the algorithm, and on discriminating between two drug candidates with similar transcriptomic effects but different potentials for commercialization.

Improving the inference of the Boolean network by integrating non-coding elements. Non-coding elements in the DNA might have a regulatory impact by binding to specific transcription factors for instance, without actually coding for a product. Inspired by [START_REF] Réda | Automated inference of gene regulatory networks using explicit regulatory modules[END_REF], adding the regulatory interactions between genes and non-coding elements might add a biologically meaningful structure which additionally constraints the network. The main challenges for this integration are the retrieval of relevant non-coding elements across cell lines, and assessing their true impact on regulation.

Interpretation of the parameters learnt in the bandit algorithm.

Although frequently mentioned in order to justify the choice of a given structure for a bandit model, a proper framework to analyze and exploit the parameters learnt during the sampling phase is missing. In the algorithm for misspecified linear models (Chapter 6), a sequential statistical test is performed to assess whether the null hypothesis, which claims that the current empirical model is an alternative model to the true model, should be rejected. In that case, if the empirical model is out of the alternative set at some point in time, one might be interested in leveraging the values of the coefficients associated with feature vectors to determine the role of a given feature -in our case, the change in expression for a gene-on the observed scores.

Chapter 9

Chapter 2 : Building the cell line specific Boolean network

Building the Boolean network

This section of the Appendix describes in details the procedure used to infer a Boolean network in our application to epilepsy.

Step (A) Building a undirected unsigned graph

This step builds an undirected, unsigned network of putative gene-to-gene regulations. The M30 module, as defined by Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF], comprises of 320 genes which global expression anticorrelates with epileptic phenotypes. We retrieved all 320 genes of the M30 module from the additional file 1 in Delahaye-Duriez, [START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]. Undirected and unsigned protein-pairwise interactions are then retrieved from the STRING database [START_REF] Szklarczyk | The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets[END_REF] on this specific set of genes for the human (NCBI taxon ID 9606). In order to perform the inference, it is necessary for computational reasons to restrict the set of edges to consider ; however, (weak) connectivity in the graph of interactions should also be preserved to fully exploit the dynamical constraints provided later on.

Considering the full network retrieved from the STRING database, we trimmed out isolated genes (i.e., without any interactions with any gene, not even themselves). For the M30 module, 318 genes out of 320 were retained after this procedure, with a total of 14, 662 undirected regulatory interactions. The STRING database also provides scores associated with each undirected edge named "combined scores", comprised between 0 and 1, 000, which are an aggregation of various scores related to the type of evidence supporting these edges (Von [START_REF] Mering | STRING: known and predicted protein-protein associations, integrated and transferred across organisms[END_REF]. This higher this score is, the more strongly supported the associated edge is. Provided a user-provided threshold η, we built the resulting gene regulatory network (GRN) by first preserving all edges with a STRING "combined score" greater than η ; second, considering all edges which contain at least one gene that do not appear in the set of edges at Step (1), we sorted them in the order of decreasing STRING "combined scores", and added them sequentially (by batch of edges with the same STRING "combined score") to the network until the number of weakly connected components is 1. We tested (weak) connectivity by performing a Depth-First Search [START_REF] Cormen | Introduction to algorithms[END_REF], which is a well-known procedure that explores all the nodes in a graph by favoring the exploration of child nodes instead of sibling nodes, until all nodes have been visited.

In order to select the threshold η = 400, we performed a gridsearch on [[100; 1, 000]] with a step of 5, and selected the value which minimized the number of edges. This step is automatically performed the first time the inference procedure in the repository1 is run, such that the user can use the threshold value η recommended by the grisearch. Choosing η = 400 allowed reducing the number of undirected edges from 14, 662 to 1, 633.

Step (A) Gene perturbation experiments

After that step, we restricted the set of genes (and thus, of interactions) to genes present in the database of transcriptional profiles LINCS L1000 [START_REF] Subramanian | A next generation connectivity map: L1000 platform and the first 1,000,000 profiles[END_REF]. In order to filter out genes, we first converted all gene identifiers in M30 into EntrezGene IDs using BioDBnet [START_REF] Mudunuri | bioDBnet: the biological database network[END_REF]. Then, we filtered out genes for which EntrezGene ID was not present in LINCS L1000 (using the API in LINCS L1000). After this step, 236 genes, out of 318, were retained. We selected all experiments present in LINCS L1000 such that at least one gene from M30 has been perturbed in a genetic experiment (knockdown or overexpression, along with control samples). All experiments involved any brain cell line in LINCS L1000 in our application to epilepsy. Unfortunately, there are no hippocampal neuron human (HN-h) cell lines in LINCS L1000, which are able to differenciate into neurons and glial cells as shown in the rat [START_REF] Eves | Immortal rat hippocampal cell lines exhibit neuronal and glial lineages and neurotrophin gene expression[END_REF]) ; although we can assume the neural progenitor cell (NPC) line present in LINCS L1000 might be appropriate. Among the genetic perturbation experiments listed in the database, we selected those which satisfy all following conditions -which are the richest, by considering metric distil_ss provided by the LINCS L1000 API, which is correlated to the number of significantly differentially expressed transcripts found in the differential analysis between the matching genetically treated and the control groups. In practice, this measure is correlated to the reproducibility of a drug signature [START_REF] Lim | Evaluation of connectivity map shows limited reproducibility in drug repositioning[END_REF].

-where there is at least two replicates from the same plate for the perturbed (of type pert_sh) and control (of type ctl_vector) conditions.

-which interference scale, as described in [START_REF] Cheng | Systematic quality control analysis of LINCS data[END_REF], is positive. This ensures that the associated genetic perturbation experiment was successful, meaning that a gene which has been perturbed by a knockdown (resp., an overexpression) has an expression lower (resp., greater) in treated profiles than in controls, compared to an appropriate housekeeping gene. The expression of the housekeeping gene should not dramatically change in both groups of profiles. A list of housekeeping genes is provided by [START_REF] Cheng | Systematic quality control analysis of LINCS data[END_REF].

-where the associated experiment is either using shRNA (knockdown perturbation), cDNA, also known as knock-in (overexpression perturbation), or CRISPR (knockout perturbation).

-where the associated cell line is either SHSY-5Y (neuroblastoma) or NPC (neural progenitor cells), which are the only brain cell lines in LINCS L1000.

The result of this step is a matrix of M30 genes by experimental profiles, which contains Level 3 LINCS L1000 data (normalized expression data for the whole genome) for each perturbation experiment. See Table 9.2 in Appendix for the list of experimental profiles used in the application to epilepsy.

Step (B) Binarization of experiments into binary profiles

Although there are known methods for the binarization of (single) RNA-seq data (Béal et al., 2019; Finak et al., 2015), probably due to the fact Level 3 LINCS L1000 data is a combination of measured and inferred expression data, for different platforms, 2 there were issues with the model fitting. Only a few genes were assigned a binary value 0 or 1 -the alternative being that they are not considered expressed "enough", according to the thresholds computed by these methods, to be assigned a state equal to 1, nor too weakly expressed to be assigned a state equal to 0. A data-driven method to tune the granularity of the binarization, adaptive to the selected perturbation expression data, was necessary in order to explicitly enforce a trade-off between a full reliance on the undirected edges provided by the STRING database, and on the experimental profiles from LINCS L1000.

Binarization. We designed an ad hoc binarization method to satisfy these constraints. This binarization was performed independently on each cell line. Gene expression (in normalized RNA counts) data was first quantilenormalized and clipped to the interval [0, 1]. Control samples, for the same cell line, were aggregated by considering the genewise median expression value. Given the threshold ζ, all genes with expression greater than 1 -ζ were considered greatly expressed (with assigned state 1), whereas genes with expression lower than ζ were considered non-expressed (with assigned state 0). Genes which expression levels were in the interval [ζ, 1 -ζ] had an undetermined expression state. Note that the quantile normalization is necessary, even though the initial expression data was normalized, in order to apply a same threshold ζ on all profiles. The higher ζ is, the more constrained the experiments are, as more genes have a determined expression state 0 or 1. Lower thresholds mean less constrained experiments, and a higher preference for the regulatory interactions filtered from the STRING database over expression data from LINCS L1000.

Using a bisection method in interval [0; 0.5] with precision 0.005, we identified ζ = 0.265 as the maximum threshold such that the inference of Boolean networks described in the next sections admits at least one solution. We recommend using this bisection method to determine the threshold ζ when using the pipeline with another dataset. Background expression data. However, this method relies on having enough data to compute reliable statistics of expression (minimum, maximum, mean) for each gene, which is why, for each cell line, we automatically retrieved from LINCS L1000 a "background" expression matrix, which we concatenated to the set of profiles before binarization. After binarization, we removed samples associated with the background dataset. In order to collect the background expression matrix, we selected all experiments in the considered cell line, with type pert_sh (knockdown experiments), and we filtered out experiments with less than two replicates, with metric distil_cc_q75 greater or equal to 0.2, and with metric pct_self_rank_q25 lower than 0.05. Metrics distil_cc_q75 and pct_self_rank_q25 are two measures associated with experimental profiles which quantify the reproducibility based on the correlation between the same technical replicates (distil_cc_q75) and the diversity of profiles for a given experimental setting (pct_self_rank_q25).

These rules correspond to the requirements for reproducible and distinct ("gold") profiles according to LINCS L1000 documentation. Finally, we selected the same-plate replicates with the highest value of distil_ss.3

Step (B) Implementation of topological constraints

The inference of a Boolean network relies on a set of admissible regulatory interactions and a set of time-series expression constraints. Indeed, solution networks only comprise of admissible interactions, such that all constraints provided by the experimental transcriptional observations are valid.

First, to build the set of admissible interactions (topological constraints), we restricted the selection of interactions to the network extracted from the STRING database. Since STRING-extracted interactions are unsigned, we decided to reduce the number of possible interactions by using the gene perturbation expression matrix retrieved from LINCS L1000 (Table 9.2), in order to improve the computational cost. A Pearson's r [START_REF] Bravais | Analyse mathématique sur les probabilités des erreurs de situation d'un point[END_REF] gene correlation matrix was computed from these profiles, and raised to the power of β coefficient-wise, which allowed signing the interactions using pairwise correlation signs. To preserve connectivity, we built the filtered signed undirected network similarly to what we previously did with a threshold value equal to τ = 0.4.

β was chosen as it is known that raising an adjacency matrix A to the power of β yields coefficients A [i, j] in position (i,j) equal to the number of paths (with eventually repeated edges) between node i and node j of length β. τ was chosen as a compromise between richness of the network (number of edges) and computational cost, by a bisection search in interval [0.01; 1] with step 0.005, which would be the way to go to apply our method to other datasets.

After this procedure, we removed isolated genes in the network (that is, with both in-degree and out-degree equal to 0). After this step, 232 genes were left in the network, with 637 × 2 putative genepairwise interactions (one for each direction between two genes). We stress on the fact that preserving connectivity will be crucial for properly exploiting the experimental data, which is why we trim out isolated genes.

Step (B) Implementation of experimental constraints

Second, we turned to building dynamical constraints, that is, the time-series binary expression states of genes in the network, according to the gene perturbation experiments from LINCS L1000. The experiments shown in Table 9.2 comprise of control and perturbed profiles in single gene perturbation experiments (either by knockdown through shRNA, or by overexpression using cDNA).

First, these profiles were binarized using the binarization procedure described above. Then, for each single gene perturbation experiment, we considered as initial condition the profile obtained from control samples, and as final condition those from perturbed samples, which are set as a (steady) attractor states. In order to implement the new dynamics in [START_REF] Paulevé | Reconciling qualitative, abstract, and scalable modeling of biological networks[END_REF], we used the Python package BoNeSiS [START_REF] Chevalier | Synthesis of boolean networks from biological dynamical constraints using answer-set programming[END_REF], which infers by answer-set programming Boolean networks (that is, both the set of regulatory interactions and regulatory functions) which satisfy the experimental constraints by only using the provided set of possible interactions. BoNeSiS is fed the binarized experimental profiles, along with their annotated gene perturbations, and the set of valid interactions determined in the previous section. Thus, the inferred GRN should satisfy all these constraints by assigning logical functions to genes and selecting regulatory interactions.

We use the procedure in BoNeSiS which randomizes the search for network solutions. Moreover, in order to avoid trivial solutions without interactions, we also implemented the constraint that the state where all genes were not expressed (i.e., with expression state 0) cannot lead to any of the reported final attractor states. This constraint can be challenged, as one might assume that a network could end up in the state where all genes are turned off in a transient way, if there are some genes which are only regulated by inhibitors. However, in practice, the inference procedure without this constaint yields singularly trivial and poorly connected solutions (i.e., most genes ending up without any regulators). We conjecture that it is linked to the procedure of answer-set programming, as similar methods, for instance Re:In [START_REF] Dunn | Automated Reasoning for the Synthesis and Analysis of Biological Programs[END_REF], give the option of adding supplementary constraints about the presence of an activator for some genes.

Step (C) Inference solutions and model selection Inference of putative Boolean network solutions. In BoNeSiS, we asked for the enumeration of at most 1 solution to the set of topological and experimental constraints defined above. In the implementation of the most permissive semantics in [START_REF] Paulevé | Reconciling qualitative, abstract, and scalable modeling of biological networks[END_REF] by [START_REF] Chevalier | Synthesis of boolean networks from biological dynamical constraints using answer-set programming[END_REF], the size of the Boolean function specification can be upper-bounded by a prespecified value. In my application, I have used the maximum total (ingoing and outgoing) degree of the underlying network in order to avoid spurious gene regulatory functions. Due to the intrinsic randomness stemming from the solver clingo [START_REF] Gebser | Theory solving made easy with clingo 5[END_REF], and the randomized search procedure used in BoNeSiS, we iterated this enumeration, such that we obtained 25 Boolean network solutions (among which 25 are unique in terms of regulatory functions).

Selection of an optimal model. In order to select a "representative" network consistent with what is known about the topology of biological networks, [START_REF] Babichev | Technique of Gene Regulatory Networks Reconstruction Based on ARACNE Inference Algorithm[END_REF] compiled a list of network measures to maximize in biological networks. Then, they computed a single scalar criterion value comprised in the interval [0, 1] (called in their paper "general topological parameter") to maximize through the Harrington desirability index [START_REF] Harrington | The desirability function[END_REF]. In practice, using notations from [START_REF] Babichev | Technique of Gene Regulatory Networks Reconstruction Based on ARACNE Inference Algorithm[END_REF], we considered the following weights a DS = 3, a CL = 3, a Centr = 3, and a GT = 1 , where -DS corresponds to the network density, that is, the ratio of the number of edges to the maximum number of possible connections between the nodes in the network (that is, if the network was fully connected) ; for a network of n nodes, this maximum number is equal to (n -1)n/2.

-CL corresponds to the network clustering coefficient which is the average of node-wise clustering coefficients. The clustering coefficient of a node is the ratio of the degree of the considered node and the maximum possible number of connections such that this node and its current neighbors form a clique (i.e., form a fully connected graph).

-Centr corresponds to the network centralization, which is correlated to the similarity of the network to a graph with a star topology.

-GT corresponds to the network heterogeneity, which quantifies the nonuniformity of the node degrees across the network by computing the ratio between the standard deviation of the node degrees and the average degree across the network.

The higher the weights, the more importance is given to having a large associated coefficient. Finally, for every network solution N returned by BoNeSiS, we computed exp (mean {-exp (x × a -1) : (x, a) ∈ V(N )}) , where V(N ) is the set of pairs (value, weight) associated with each topological measure V(N ) := {(DS(N ), a DS ), (CL(N ), a CL ), (Centr(N ), a Centr ), (GT(N ), a GT )} .

The final network was the one which maximized this quantity.

Robustness on a larger set of 50 solutions

Since the enumeration of solutions is still computationally expensive and time-consuming, we focused our work on a collection of 25 solutions. However, in order to assess the robustness of our inference procedure, we enumerated an additional set of 25 solutions, and reproduced the two plots shown in the results. Note that these 25 solutions were different from the first 25 ones, yielding a set of 50 unique solutions (in terms of gene regulatory functions). The selection of the optimal model run on these 50 models still returned the same network as shown in the main paper.

Implementation of the influence maximization algorithm

This section deals with supplementary data about the implementation of the influence maximization procedure.

Iteration of attractor states

In order to enumerate attractors under perturbations, we used PyMaBoSS [START_REF] Stoll | MaBoSS 2.0: an environment for stochastic Boolean modeling[END_REF]. We ran PyMaBoSS with 1, 000 trajectories, for reachable attractors within 50 time steps, and parameters time_tick = 1, use_physrandgen = 0. Unfortunately, this method does not guarantee the similarity of attractors from one iteration to another, but our tests showed that, although there is some noticeable change in the resulting spread values, it does not affect the final ranking on genes. We never had to deal with the case where no attractor state is retrieved with these parameter values.

Choice of initial states

In our application, we considered the integration of a disease-specific context by considering 24 hippocampi normalized transcriptional profiles of humans affected with Temporal Lobe Epilepsy (TLE) [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF] (EMTAB 3123 on ArrayExpress). The main idea is that we specifically target genes which perturbative power is high in a transcriptional context for epilepsy. We restricted these epileptic profiles to genes present in the network, by retrieving their associated Entrez ID identifiers and by matching with identifiers in LINCS L1000. Then, we binarized the profiles according to the binarization procedure described in the first section, with corresponding threshold ζ equal to 0.5, so that all genes have a determined binary expression state.

Additional results

This section shows additional results related to the inference of the Boolean network and the spread values. Then, similarly to the proof for LUCB-GIFA, I show an upper bound on the number of times sampled arm I t has been sampled up to round t included Lemma 12.1. .

Then we apply Lemma 5.3.8, which gives us the final upper bound.

Note that, empirically, LinGIFA has the same performance than N -LinGapE in terms of sample complexity, which makes us more confident in this conjecture.

12.2 Conjecture 5.3.10: analysis of the greedy selection rule Note that, for some κ ≥ 0, ∀t > 0, V κ (t) := κI d + a∈[K] N a (t)x a x ⊤ a . Greedy selection rule is selecting at round t > 0 the arm a t which satisfies a t ∈ arg min a∈ [K] x bt -x ct ( V κ (t-1)+xax ⊤ a ) -1 .

The goal is to upper bound the quantity x bt -x ct ( V κ (t-1)+xa t x ⊤ a t ) -1 with something depending on N at (t), in order to apply Lemma 5.3.8. Lemma 12.2.1. For any t > 0, for any a ∈ [K],

||x bt -x ct || ( V κ (t-1)+xax ⊤ a ) -1 ≤ x bt -x ct 2 λ bt,ct,a ,
where (λ i ) i≤d are the eigenvalues of ( V κ (t -1) + x a x ⊤ a ) associated with respective eigenvectors (v i ) i≤d , and λ bt,ct,a := min

j≤d v ⊤ j (xb t -xc t )̸ =0 λ j .
Proof. We denote V a := V κ (t -1) + x a x ⊤ a . Matrix V a is real, symmetric, and positive definite, thus all its eigenvalues (λ i ) i≤d are positive. Then, due to the spectral theorem, these eigenvalues are all real-valued, and there exists a orthonormal basis of associated eigenvectors (v j ) j≤d . In particular,

∀j ≤ d, v j = V -1 a V a v j = λ j V -1 a v j ⇔ V -1 a v j = λ -1 j v j .
Moreover, Span({v j } j≤d ) = R d , which means that there exists at least one j ≤ d such that v ⊤ j (x bt -x ct ) = 0. Then

x bt -x ct 2 V -1 a = (x bt -x ct )V -1 a j̸ =d v ⊤ j (x bt -x ct )v j = (x bt -x ct ) ⊤ j≤d v ⊤ j (x bt -x ct )λ -1 j v j ≤ λ -1 bt,ct,a x bt -x ct 2 2 .
Now, we will find a lower bound for the quantity λ bt,ct,at depending on N at (t), for any t > 0, with a t as defined by the greedy selection rule.

Lemma 12.2.2.

λ bt,ct,at ≥ κ + (N at (t -1) + 1) x at The second equality sign is due to Lemma 12.2.3 and the third one is due to the analysis of the eigenspace of xx ⊤ , where x = 0.

We know that λ pa(j ′ a ) (x a x ⊤ a ) ∈ {0, x a 2 2 } for any a ∈ [K]. Now all that is left is to prove that λ pa(j ′ a t ) (x at x ⊤ at ) = x at 2 2 where a t is the arm selected by the greedy rule, which means that any eigenvector v associated with the largest eigenvalue λ max of matrix V at satisfies v ⊤ (x ct -x bt ) = 0 and that, for any strictly smaller eigenvalue, associated eigenvector v satisfies v ⊤ (x bt -x ct ) = 0 (such that, necessarily, λ bt,ct,at = λ max ). This is the trickiest part of the proof, and is left unproven.

Combining this last lemma with stopping rule τ LUCB := inf t>0 {B ct,bt (t) < ε}, and the upper bound on B ct,bt (t) for Algorithm N -LinGapE, on event E GIFA at t < τ LUCB : Remark 13.1.3. Although the expression of g M is not in closed-form, its value can easily be retrieved through any scalar minimization procedure.

ε ≤ B ct,

Correctness analysis

Theorem 13.2.1. CPE is δ-correct. On event E CPE , CPE outputs the correct set of optimal arms S m N for each agent m.

Proof. If CPE was not δ-correct on event E CPE , then, for some agent m, there would be an arm ℓ ∈ S m N which is eliminated at round r from B m (r +1). But, on event E CPE , Lemma 8.4.5 implies that, for any r ≥ 0, m ∈ [M ], and (i, j) ∈ [K] 2 , µ ′ (i,m) (r)-µ ′ (j,m) (r)+Ω (i,m) (r)+Ω (j,m) (r) ≥ ∆ ′ m i,j ≥ µ ′ (i,m) (r)-µ ′ (j,m) (r)-Ω (i,m) (r)-Ω (j,m) (r) .

Then, combining the right-hand inequality for j = ℓ with the elimination criterion at Line 17 in Algorithm 8

N max i∈[K] ∆ ′ m i,ℓ ≥ N max i∈[K]
µ ′ (i,m) (r) -µ ′ (ℓ,m) (r) -Ω (i,m) (r) -Ω (ℓ,m) (r)

≥ N max i∈Bm(r)⊆[K]
µ ′ (i,m) (r) -µ ′ (ℓ,m) (r) -Ω (i,m) (r) -Ω (ℓ,m) (r) > 0 , which is absurd because ℓ ∈ S m N . That means, necessarily the only constraint that is violated is the one on (k ′ , m ′ ). Using the nondecreasingness of T δ (•)

Sample complexity analysis

n k ′ ,m ′ (r -1) + d k ′ ,m ′ (r) -1 = n k ′ ,m ′ (r -1) + d ′ k ′ ,m ′ (r) < t k ′ ,m ′ (r)T δ (n k ′ ,• (r -1) + d ′ k,• (r)) ≤ t k ′ ,m ′ (r)T δ (n k ′ ,• (r -1) + d k,• (r)) .
Combining the two ends of the inequality proves the claim. The upper bound on the communication cost follows from the upper bound on the number of phases given in Corollary 13.3.4.1.

At

Chapter 14

Chapters 5 & 6: Real-life conditions for drug repurposing One shortcoming of the experiments in Chapters 5 and 6 is that the setting is actually made closer to the studied frameworks -respectively linear and misspecified linear structures, by transforming the drug features in a supervised fashion, Gaussian rewards in Chapter 6 instead of raw patientspecific scores. In this chapter, we actually run the algorithms developed in this thesis (N -LinGapE, LinGIFA, MisLid, in their default versions) and the benchmark (unstructured) algorithm LUCB [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] in a setting closer to the drug repurposing application.

We consider again the set of K = 21 drugs which associated average scores are displayed in Table 10.3. A boxplot of these reward values is reported in the left-hand plot in Figure 14.1. The gap between the lowest average reward among antiepileptics and the highest average reward across proconvulsant drugs is around 0.07, which means that the associated instance (for N = 10) is moderately hard to solve.

We want to determine the N = 3 best treatments (∆ a (N ) a (N +1) ≈ 0.14), based on d = 10 features. 1 In order to compute these features from the original 194 ones, we apply Principal Component Decomposition (PCA) and select the d largest components. The a posteriori heatmap built on these PCA-reduced features, according to the true reported class (antiepileptic or proconvulsant) is displayed in Figure 14.1 (right-hand plot). This heatmap shows that using these features allows to cluster very roughly the two class of drugs (ARI ≈ 0.080 > 0), which tends to show that the underlying model is far from linear (with a full dependence on the features to infer the scores). Note that this feature transformation is reward-agnostic. Similarly to the experimental setting in Chapter 5, each time a treatment is selected, a patient is selected at random, and the score associated with this patient and the selected treatment is observed.

W it h a fe ri n -

When using MisLid, we need to determine the value to assign to parameter Ψ (please refer to Chapter 6). In practice, the repurposing scores (see Table 10.3) are roughly comprised between -1 and 1. In order to ensure that the property of correctness is satisfied, we use the reasoning laid out in the discussion of Chapter 6 ; we start by considering Ψ = 1, and decreasing its value by a fixed step size (size = 0.1) until the sample complexity is equal or lower than the one incurred by an unstructured algorithm on the same instance (e.g., LUCB). Of course, such a procedure is (1) not theoretically supported, (2) not tractable in practice because it would imply running first an unstructured instance. As previously mentioned in the thesis, an interesting subsequent work would be the design of an adequate finetuning method for Ψ.

The parameter values are reported in * Although this parameter is not related to the actual observations, it is used to build the confidence intervals in the algorithms.

MisLid was also run with a few lower values of Ψ in order to illustrate the effect of Ψ on the performance of the algorithm. The trend in sample complexity depending on Ψ is illustrated in the boxplot in Figure 14.2.

The comments made in the experimental part of Chapter 6 are still valid in this "real life" setting : linear bandit algorithms (including MisLid with Ψ = 0) do not preserve correctness anymore, whereas unstructured algorithms are not sample-efficient. In practice, as illustrated by Figure 14.2, selecting values around a educated guess about the absolute value of the upper bound on rewards seems rather robust and sample-efficient, while ensuring that the correctness property is preserved.

Considering MisLid and Figure 14.2, while the error rate drastically increases as the misspecification level ψ in input decreases (as described in the conclusion of Chapter 6), the trend of the sample complexity depending on ψ displays a different, non-monotonic pattern. In Table 14.2, the average sample complexity globally increases in the range ψ ∈ [0, 0.4], globally decreases in the range ψ ∈ [0.5, 0.8], and then slowly increases again starting from ψ = 0.8. The last "increasing" section of the plot is consistent with the conclusion in Chapter 6, which predicts that, the greater ψ is, the largest the set of models M ψ which the algorithm focuses on is, and gets closer to the set of all models (i.e., unstructured models). Then, the expected sample complexity of MisLid should be closer and closer to the expected sample complexity of a good unstructured bandit algorithm (e.g., LUCB). However, the observed behaviour of sample complexity for smaller values of ψ does not follow a linear pattern, and its study would be an interesting subsequent theoretical work.

  b(t)(t). Le critère d'arrêt correspondant estτ UGapE := inf t ≥ 0 : max a∈J(t) N max b̸ =a B b,a (t) < ε .

7. 1

 1 Figures / Liste des figures xxi

III

  stochastic bandits . . . . . . . . . . . . . . . . . . . . 4.2 Bandit algorithms for Top-N identification . . . . . . . . . . . . . 4.3 Sample complexity in bandits . . . . . . . . . . . . . . . . . . . . 4.4 Integration of bandits to drug repurposing . . . . . . . . . . . . 5 Sequential identification in linear models 5.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 General structure for efficient algorithms . . . . . . . . . . . . . 5.3 Theoretical guarantees in GIFA . . . . . . . . . . . . . . . . . . . 5.4 Application to drug repurposing . . . . . . . . . . . . . . . . . . . 5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Dealing with non-linear models 6.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.3 Tractable lower bound for general Top-N identification . . . . . . 100 6.4 Misspecified Top-N identification . . . . . . . . . . . . . . . . . . 103 6.5 Application to drug repurposing . . . . . . . . . . . . . . . . . . . 110 6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.2 Setting and experimental data . . . . . . . . . . . . . . . . . . . 117 7.3 Systematic outcome scoring . . . . . . . . . . . . . . . . . . . . . 119 7.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8 Extension to a collaborative setting 8.1 Collaborative Top-N (identification) . . . . . . . . . . . . . . . . . 131 8.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 xxvii

Figure 1 :

 1 Figure 1: Drug development timeline and principle of drug repurposing. Top plot : Timeline for a typical drug development pipeline, comprising of four main research stages followed by post-commercialization monitoring.Bottom plot : Illustration of the approach of "signature reversion" for drug repurposing.

Figure 2 :

 2 Figure 2: Transcriptomics.Transcriptomics : relationship between deoxyribonucleic and ribonucleic acids (DNA and RNA), and the measure of gene expression.

  Figure 1.1: Lac operon regulation as a gene regulatory network. Decomposition of the well-known lac operon regulatory network in the Escherichia Coli (E. Coli) bacteria from[START_REF] Pardee | The genetic control and cytoplasmic expression of "inducibility" in the synthesis of β-galactosidase by E. coli[END_REF], based on the regulatory network in[START_REF] Santillán | Quantitative approaches to the study of bistability in the lac operon of Escherichia coli[END_REF]. It shows the adaptation of the bacteria to the type of nutrients present in its environment. E. Coli preferably feeds on glucose, but when it is missing from its environment, the inhibition on the transcription starting site LacP through repressor R, that blocks the transcription of genes LacY and LacZ, do not hold anymore, which allows the bacteria to feed on lactose. Gene LacY is involved in the transport of external lactose to the bacteria, and LacZ is involved in the transformation of lactose (and allolactose) into galactose. In the presence of glucose, the production of protein cAMP and CAP is inhibited, which further blocks the transcription of LacY and LacZ due to the presence of repressor R. To allow a readable figure, I omitted an activatory link between glucose and the output node, and the reaction between allolactose and LacZ which results in the production of galactose.

Figure 1 . 2 :

 12 Figure1.2: Gene regulatory networks and Boolean networks. Going from a set of regulatory interactions between genes, to a Boolean network with both the interaction graph and the set of regulatory functions for each node in the network. Left plot : Regulatory interactions (of different types) are shown between three genes (nodes of rectangular shape). Center plot : The gene regulation network between those three genes. Right plot : One possible Boolean model associated with the gene regulatory network. There might be potentially several sets of regulatory functions which can match the regulatory network, in the absence of additional information about gene expression.

Figure 2 . 3 :

 23 Figure 2.3: Comparison of the inferred networks. Left-hand plot: Boxplots of node total degrees (in-and out-degree) per solution. The green lines represent median values. Right-hand plot: Boxplot of the number of edges across solutions. Again, the green line represent the median value.

Figure 2 . 4 :

 24 Figure 2.4: Illustration of the collapsed network. Network comprising of all gene-to-gene interactions which are present in at least one solution.The darker and thicker an edge is, the more frequent it is across solutions. Sinewave edges are inhibitory interactions, whereas solid lines denote activatory interactions. Orange nodes correspond to the genes which are perturbed in the LINCS L1000 experimental profiles used for inference.

  Figure 2.5 and Table

  Figure 2.5: Comparison between spread values and other gene measures. Top plot: Spearman's ρ correlation heatmap between different gene measures either related to the influence of a node on a network, or to the genetic variations associated with pathogenicity. Bottom plot: Enrichment results from the ORA analysis. All reported enrichments have an adjusted p-value lower than 20%.

Figure 2 . 6 :

 26 Figure 2.6: Gene ranking by decreasing spread value. Genes ranked by decreasing spread value, restricted to spread value greater than 0.01 (center bar), with their associated Control Centrality (CC) (top bar), and pLI scores (bottom bar).

  Direction on the set of control and patient samples # to determine the classification frontier between the two groups Compute S ← CD[P C] ∈ R |M 30| for g ∈ L do for c ∈ C do Initialize scores_profile(g) ← 0 for any gene g ∈ L Enumerate a set of attractors A (c,g) which are reachable from binarized control profile c under the knockout of gene g Initialize list(a) ← 0 for a ∈ A (c,g) for a ∈ A (c,g) do π(M(a)) ←projection of M(a) onto hyperplane of normal vector M(S) # Compute the "signed distance" between the considered attractor # and the classification frontier defined by S in 2D list(a) ← signed(M(a), π(M(a))) end for # Compute a score based on the probabilities of presence of attractors scores_profile(c, g) ← a∈A (c,g) p (a,c,g) list(a) where p (a,c,g) is the probability associated with attractor a, initial profile c and knockout of g end for # Obtain a single score per gene scores(g) ← 1 |C| c∈C scores_profile(c, g) end for Output: scores

i

  across the listed five databases. Indeed, as shown in the right-hand heatmap in Figure 3.1, the heatmap regroups on one hand (right-most cluster at the top of the plot) antiepileptics which target gamma-aminobutyric acid (GABAergic) receptors : GABRA4, D r u g B a n k D r u g B a n k M I N E R V A D r u g C e n t r a l T T D L I N C S L 1

Figure 3 . 1 :

 31 Figure 3.1: Visualization of gene targets for antiepileptic and proconvulsant drugs.Left plot : Spearman's ρ correlation matrix heatmap which represents the redundancy of targets across databases. Right plot : Heatmap associated with the aggregated target matrix described in the main text. This matrix is restricted to genes (39/320) and drugs (27/71) with at least one nonzero coefficient in the matrix. Antiepileptic, resp. proconvulsant, drugs are written using a green, resp. red, font. Rows and columns are ordered according to two hierarchical clusterings represented on the plot.

  B e s t C e l lA l l C e l l s B r a i n C e l l A g g r e g a t e r e g a t e d B r a i n C e l l B e s t C e l l A l l C e l l

Figure 3 . 2 :

 32 Figure 3.2: Comparison between the different types of drug signatures. Similarity between the three types of drug signatures retrieved from LINCS L1000 and the lists of gene targets, on the set of 71 epilepsy-related drugs. Associated matrices are converted into binary vectors of size (71×320).Each pairwise comparison is restricted to common genes and drugs. Left plot : We compare the similarities of clusterings (obtained by running k-means++[START_REF] Arthur | k-means++: The advantages of careful seeding[END_REF]) with 3 clusters) by computing the Adjusted Rand Index (ARI)[START_REF] Hubert | Failure of investigational drugs in latestage clinical development and publication of trial results[END_REF]. a Right plot : The Jaccard index[START_REF] Jaccard | The distribution of the flora in the alpine zone. 1[END_REF] is computed between approaches. In both heatmaps, the higher the score, the most similar two methods are. Rows and columns of the heatmaps are ordered by a hierarchical clustering represented on the plots.

  10

Algorithm 3

 3 Drug perturbation scoring. Scoring the effect of a drug candidate on epileptic transcriptional profiles Input: C set of control transcriptional profiles and P set of epileptic profiles ; a list of drug candidates L along with their drug signatures ; B = (V, E, F ) a Boolean network on node set V with edges in E and regulatory functions F Initialize scores(d) ← 0 for any drug d ∈ L Binarize profiles in C and in P in matrix X ∈ {-1, 1} (|C|+|P|)×|M 30|# Normalize the set of control and patient profiles, per feature (gene)X ← ( 1 σg (X g -µ g )) g∈{1,...,|M 30|}, where µ g = mean(X g ) and σ g = std(X g ) # Learn a mapping from the initial high-dimensional space and the # 2D plane Fit a Principal Component Analysis (PCA)M : {-1, 1} |M 30| → R 2 on matrix X # Run CharacteristicDirection on the set of control and patient samples Compute S := CD[P C] ∈ R |M 30| for d ∈ L do # We now consider patient profiles in initial conditions for p ∈ P do Initialize scores_profile(d) ← 0 for any drug d ∈ L Enumerate a set of attractors A (p,d) which are reachable from binarized patient profile p under the perturbations by drug d, as defined by its drug signature Initialize list(a) ← 0 for a ∈ A (p,d) for a ∈ A (p,d) do π(M(a)) ← projection of M(a) onto hyperplane of normal vector M(S) # Compute the "signed distance" between the considered attractor # and the classification frontier defined by S in 2D list(a) ← signed(M(a), π(M(a))) end for # Compute a score based on the probabilities of presence of an # attractor scores_profile(c, g) ← a∈A (p,d) p (a,p,d) list(a) where p (a,p,d) is the probability associated with attractor a, initial profile p and perturbation by d end for # Get a single value per drug candidate scores(d) ← 1 |P| p∈P scores_profile(p, d) end for Output: scores

Figure 3 . 4 :

 34 Figure 3.4: Comparison between the drug repurposing method and the baseline. Receiver operating characteristic (ROC) and precision-recall (PR) curves, representative of the performance of a method (prediction : Algorithm 3, baseline : L1000 CDS 2 ) on the set of 34 drugs. Left plot : ROC curves. Right plot : PR curves.

Algorithm 4

 4 Sequential drug perturbation scoring. Socoring of the effect of a drug candidate in a sequential setting # In practice, a patient profile is sampled at random from the available # pool of patient profiles Input: a patient profile p ∈ {0, 1} |M 30| and a drug candidate d with drug signature s d ∈ {-1, 0, 1} |M 30| Parameters: B = (V, E, F ) a Boolean network on node set V with edges in E and regulatory functions F , fitted Principal Component Analysis model M, classification frontier defined by normal vector M(S) in the 2D plane Enumerate a set of attractors A (p,d) which are reachable from binarized patient profile p under the perturbations by drug d, as defined by its drug signature s d Initialize list(a) ← 0 for a ∈ A (p,d) for a ∈ A (p,d) do π(M(a)) ← projection of M(a) onto hyperplane of normal vector M(S) # Compute the "signed distance" between the considered attractor # and the classification frontier defined by S in 2D list(a) ← signed(M(a), π(M(a))) end for # Compute a score based on the probabilities of presence of an attractor score ← a∈A (p,d) p (a,p,d) list(a) where p (a,p,d) is the probability associated with attractor a, initial profile p and perturbation by d Output: score Part IIAdaptive drug testing using banditsThe contents of this chapter rely on some of my publications. a a[START_REF] Réda | Machine learning applications in drug development[END_REF]. "Machine learning applications in drug development".Computational and structural biotechnology journal, 18, pp. 241-252; Réda, Kaufmann, and Delahaye-Duriez (2021). "Top-m identification for linear bandits". International Conference on Artificial Intelligence and Statistics.PMLR, pp. 1108-1116; Réda, Tirinzoni, and Degenne (2021). "Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification". Advances in Neural Information Processing Systems, 34.

Figure 4 . 1 :

 41 Figure 4.1: Multi-armed bandits. Illustration of the principle of multiarmed bandit algorithms.

3 .

 3 Fixed confidence Top-N identification for bandits with structure M. At fixed class M, the goal of fixed-confidence Top-N identification is to design an algorithm A which is δ-correct for any f ∈ M

  .1) In that case, algorithms which satisfy the inequality constraint in Definition 4.1.3 with set S ⋆ N (ε) are said to be (ε, δ)-Probably Approximately Correct (PAC) (Valiant, 1984) Definition 4.1.4. (ε, δ, N )-Probably Approximately Correct (PAC) for M-structured models. Algorithm A is (ε, δ, N )-PAC 11 if and only if, for any f ∈ M, A satisfies P {f } [ ŜN (τ A ) ⊆ S ⋆ N,f (ε)] ≤ δ . Then, we aim at minimizing the number of samples τ A among all δ-(PA)Correct algorithms A, either in high probability (what is proven in Chap-ters 5 and 8)

  , defined as follows for Top-N identification r N := N max k∈[K] µ k -min a∈ ŜN (τ A ) µ a . Note that, by definition, that quantity is always non negative (| ŜN (τ A )| = N ), and positive if and only if ŜN (τ A ) ⊆ S ⋆ N . Then, studying the error probability is estimating the expectation E A,µ [1 (r N > ε)]. However, in the context of drug repurposing, dealing with recommendation errors is more straightforward, especially in the case where ε = 0.

  R d → R (unstructured case). References : LUCB(Kalyanakrishnan et al., 2012, Corollary 7), KL-LUCB and KL-Racing[START_REF] Kaufmann | Information complexity in bandit subset selection[END_REF] (α = 2,C 0 (α) = 2 exp(1),k 1 = 3), UGapE[START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF], Theorem 2) (b = 1). To find the upper bound with the correct constants for UGapE, I have usedKaufmann, Ménard, et al. (2021, Lemma 15). I have omitted factors in O(K). Definition 4.3.1. Characteristic individual arm gap for M-structured bandit. For any f ∈ M such that, for any arm k ∈ [K], µ a = f (X a ), we define the positive gap quantity for arm k as follows

  , Lemma 1), in the regime of small, finite values of error rate δ. This lower bound relies on the definition of M-structured alternative model functions to f , which are the set of model functions in M where the set of N best arms differs from S ⋆ N,f . Intuitively, if there exists an alternative, similarly structured, model function f which is close enough to f such as S ⋆ N,f ⊆ S ⋆ N, f , then the associated

  , instead of probability laws ν a , α a , and then denote KL a (f, f ) := KL(ν a , α a ) .The lower bound stated inKaufmann, Cappé, and Garivier (2016, Lemma 1) is as follows : for any small δ, any δ-PAC algorithm A on any model function f ∈ M will sample in expectation a number of observations at least as large as some constant C ⋆ (f ) inflated by a factor O(log (1/δ)). Constant C ⋆ (f ) is called "characteristic time". It depends on f and reflects the intrinsic hardness of the problem in terms of alternative model functions.Known result 4.3.3. Lower bound on the expected number of samples in a pure exploration problem inKaufmann, Cappé, and Garivier (2016, Lemma 1). For any function f ∈ M and any δ ≤ 1/2, δ-PAC algorithm A satisfies the following inequality

M

  lin , where M lin := x → θ ⊤ x | θ ∈ R d (Definition 4.1.1). Linear models are interesting because they are easily interpretable ; if feature vector X a ∈ R d corresponds to the gene expression signature of molecule a, where d is the number of considered genes, then parameter θ exactly encodes the genewise contributions to the repurposing score. In this chapter, I will describe my contributions in solving the problem of Top-N identification in linear bandits. In particular, I introduced a set of generic algorithms with adaptive sampling, called gap-index focused algorithms (GIFA), which are δ-PAC (Definition 4.1.3) under weak assumptions about their definition (Section 4.2), and are on par with the state-of-the-art for Top-N identification in terms of empirical performance. This work was published in Réda, Kaufmann, and Delahaye-Duriez (2021) at the 24 th Artificial Intelligence and Statistics conference (AISTATS 2021).1

B

  j,i (t), which contradicts the definition of operator N max a∈[K]

3 .

 3 b (τ UGapE ) > ε, which is a contradiction. Hence, there always is c ∈ S ⋆ N such that B c,b (τ A ) ≤ ε, which allows us to prove the theorem. It easily follows from Theorem 5.3.2 that if LinGapE or LinGIFA are based on good gap indices in the sense of Definition 5.3.1, both algorithms are δ-PAC. We exhibit below a threshold function T δ (•), which defines the width of the upper confidence bounds, for which the corresponding gap indices are good gap indices. Lemma 5.3.Example of good gap indices. If for any pair of arms

Figure 5 . 1 :

 51 Figure 5.1: Drug repurposing instance with linear bandits. Drug repurposing instance with linear bandit algorithms (number of dimensions d = 15, σ = 1, δ = 10 %, ε = 0.2 > ∆ a (N ) a (N +1) ≈ 0.14, N = 3, 100 iterations) with their reported empirical error rates δ.

  Figure 5.2 plots the two curves associated with stopping quantities B c(t),b(t) (t) (for τ LUCB ), max j∈[K] N max i̸ =j B i,j (t) (for τ UGapE ) on the same 100 trajectories, which empirically confirms Lemma 5.2.8.

Algorithm 7 3 :

 73 MisLid algorithm for misspecified models. Require: Online learner L, stopping thresholds {T δ (t)} t≥1 Compute a sequence of arms I 1 , I 2 , . . . , I t 0 such that

  3.4. The precise definition of T δ (•) is given in the following result. Theorem 6.4.2. Correctness of MisLid. For the following expression of T δ (•), where δ ∈ (0, 1), Algorithm 7 is δ-correct in the sense of Definition 4.1.4 ∀t ≥ 0, T δ (t) := min β uns δ (t), β lin δ (t) , where

Figure 6 . 1 :

 61 Figure 6.1: Drug repurposing with misspecified bandits. Drug repurposing instance with reduced dimension (#arms K = 21, #dimensions d = 21, σ = 1, δ = 10 %, ε = 0.042, Ψ = 0.236, N = 4, 100 iterations) with empirical error rates δ.

3 Figure 7 . 1 :

 371 Figure 7.1: Timeline of the experiments. Timeline of the experimental part.

Figure 7 . 2 :

 72 Figure 7.2: Application of signature reversion to encephalopathy of prematurity. The procedure of scoring treatments protocols using signature reversion.

Figure 7 . 3 :

 73 Figure 7.3: Comparison between Characteristic Direction signatures and univariate differential analysis. Comparison between the numbers of common differentially expressed (DE) genes across groups of conditions, when differential gene expression analysis is performed through DESeq2 (Love, Huber, and Anders, 2014) (top plot) or through Characteristic Direction (Clark et al., 2014) (bottom plot). DE genes are genes which are associated with nonzero coefficients in the signatures. The number of genes contained in the most populous bars are reported in both plots.

Figure 7 . 4 :

 74 Figure 7.4: Correlation between score and number of common differentially expressed genes. Plot of the cosine score depending on the number of common differentially expressed (DE) genes of the associated signature (CD[IL1β + treatment IL1β]) with the control signature (that is, CD[PBS IL1β]). R 2 is the residual error of the linear regression (red line).

Figure 7 .

 7 Figure 7.5 shows the ORA results for P20-INAS-Dose 2, and Figure 7.6 those for P10-INAS-Dose 3. Similarly to differential expression analysis, the

Figure 7 . 5 :Figure 7 . 6 :

 7576 Figure 7.5: Enrichment analysis of differentially expressed genes in the best recommended treatment. Top left plot : enrichments (Top-10 in terms of decreasing p-value) of the subset of genes differentially expressed (both up-and down-regulated genes) by treatment protocol P20-INAS-Dose 2. Top right plot : considering the subset of up-regulated genes. Bottom plot : considering the subset of down-regulated genes.

Figure 7 . 7 :Figure 7 . 8 :

 7778 Figure 7.7: Rankings from cosine scores and repurposing scores on a subset of drugs. Boxplots of scores (left : repurposing score described in Algorithm 3 ; right : cosine score on "stabilized" profiles) on the smaller subset of 10 drugs, sorted by decreasing average score.

  agent m ∈ [M ] samples an arm I m t ∈ [K] at round t based on prior observations, m observes local reward Y ′ m t , which only she can observe. This local reward is drawn from a σ 2 -subgaussian distribution of mean µ (k,m) , independently from past observations and from other agents' observations -just like what happens in the single-agent context described in Definition 4.1.1.However, this agent does not seek to identify the set of N arms

  while : first, achieving a small exploration costExp {µ,W } (A)∶ = m∈[M ] k∈[K] N (k,m) (τ A ) , where N (k,m) (t) := s≤t 1(Y ′ m s = k) isthe number of selections of arm k by agent m up to round t included ; and, second, aiming at a small communi-cation cost, defined as Com {µ,W } (A) := t≤τ A 1(I t ) ,

  at least upper bound by the quantity N ⋆ (µ) from the oracle problem P ⋆ in Theorem 8.3.2. This result gets even more exciting for N = 1, as the value N (µ) of Problem P(∆ ′ ) is equal up to a multiplicative absolute constant to the characteristic time N ⋆ (µ).

  t ∈ Const N . Then, by minimality -that is, since τ ∈ Const N , t = τ ⋆ ∈ Const N ∩ Const ⋆ N , and τ ∈ P(∆ ′ )-we conclude that N ≤ N ⋆ . (ii). The lower bound is provided by (i) applied to the case N = 1. Now let us consider the solutions τ ⋆ ∈ P ⋆ (∆ ′ ) and τ ∈ P(∆ ′ ). Let us denote k m ⋆ := arg max k∈[K] µ ′ (k,m) the (unique) optimal arm for mixed expected rewards for agent m. Then, for any agent m ∈ [M ] and any arm k

P

  ( ∆), where ∆ := ( ∆ ′ k,m ) k,m . Its proof is given in Appendix (Lemma 13.1.1).Lemma 8.4.3. Comparison of values of Problem P with different gaps.Consider ∆, ∆ ′ ∈ (R + ) K×M , such that τ ∈ P(∆) and τ ′ ∈ P(∆ ′ ). Moreover, assume that there is a positive constant β such that for any agent m and arm k, ∆ ′ k,m ≤ β∆ k,m . Then

  Draw each arm k by each agent m once 3

  agent m ∈ [M ] (d k,m (r)) k∈[K] and d max (r) := max n∈[M ] k∈[K] d k,n (r) k ∈ B(r) d k,m (r) times, so that n k,m (r) = n k,m (r -1) + d k,m (r) 13: Remain idle for d max (r) -k∈[K] d k,m (r)rounds14: Send to server empirical local mean µ (k,m) (r) based on the n k,m (r) of candidate best arms for each user 19: for m = 1 to M do 20:

  if k ∈ B m (r + 1) and |B m (r + 1)| > N then until ∀m ∈ [M ], |B m (r)| ≤ N 33: Output: for any agent m, S m N (τ CPE ) := B m (r) to control empirical mixed expected rewards. The width Ω (k,m) (r) of the confidence interval scales with the following quantity Definition 8.4.4. Confidence interval for CPE. For any arm k, agent m, and round r ≥ 0, we define

  k,m (r)) k∈[K] to agent m m ∈ [M ] side * 19: Sample arm k ∈ [K] by agent m d k,m (r) times, so that the total number of samples is n k,m (r) = n k,m (r -1) + d k,m (r) 20: Compute the empirical mixed means (μ ′ k,m (r)) k,m based on the (n k,m (r)) k,m samples and send them to the central server 21Update set of candidate best arms for each user 24: for m = 1 to M do 25:

  until |B(r)| = ∅ 30: Output: km : m ∈ [M ]

  , to decrease the exploration cost, instead of implementing the theoretical value of T δ (•), we consider for any N ∈ N M

Figure 8 . 1 :

 81 Figure 8.1: Clustering of epileptic profiles. PCA plot with 2 principal components of patient profiles (centers are denoted by a star).

  1: Distribution statistics on the number of unique regulatory functions (RFs) across solutions per gene, and on the value of the general topological parameter (GTP) used for network selection in step (C) of the inference procedure. All values are rounded up to the 3 rd decimal place. Applied on the set of 50 solutions.

Figure 9 . 1 :η

 91 Figure 9.1: Left-hand plot: Boxplots of node total (ingoing and outgoing) degrees per solution. The green lines represent median values. Right-hand plot: Boxplot of the number of edges across solutions (each solution comprises of 232 nodes). The green line represent the median value. Applied on the set of 50 solutions. Note that the first 25 boxplots match the plot in Figure 2.3.

Figure 11 . 1 :

 111 Figure 11.1: Cosine method applied to the whole set of 34 drugs. Top left plot : Ranking obtained by computing cosine scores on the "stabilized" profiles predicted by the Boolean network. Top right plot : the associated ROC curve, where the baseline is L1000 CDS 2 (Duan et al., 2016). Bottom left plot : the associated PR curve. Bottom right plot : the PR curve in Chapter 3.

3 .

 3 For any arm a ∈ [K], let us denote for any b ∈ [K], b = a, Ṽb := κ K I d + N b (t -1)x b x ⊤ b and Ṽa := κ K I d + (N a (t -1) + 1)x a x ⊤ a , such that b̸ =a Ṽb + Ṽa = V a . Ṽb , for any b = a, and Ṽa are all real, symmetric, positive definite square matrices (if κ > 0 or if there is an initialization phase with uniform samplings). We will use these simple lemma to analyze the eigenspace of xx ⊤ , for any x ∈ R d Lemma 12.2.For any matrix A and for any constant α ∈ R, if ζ is an eigenvalue of A, then αζ is an eigenvalue of αA, and η + α is an eigenvalue of A + αI d . Proof. For any eigenvector v associated with eigenvalue ζ of A,(αA)v = (αζ)v and (A + αI d )v = ζv + αv = (ζ + α)v .We will now analyze the eigenspace of xx ⊤ , where x = 0. For any eigenvalueζ of xx ⊤ associated with eigenvector v, ζv = (xx ⊤ )v = x(x ⊤ v) = (x ⊤ v)x, which means that -any vector in Span(x) ⊥ := {v : x ⊤ v = 0} (of dimension d -1) is an eigenvalue of xx ⊤ associated with eigenvalue 0 (which then has a multiplicity of d -1).-any nonzero eigenvalue ζ (necessarily of multiplicity 1)satisfies v = x ⊤ v ζ x ∈ Span(x) := {αx : αR}. Moreover, we can show that (xx ⊤ )x = x(x ⊤ x) = x 2 2 x, which means that this eigenvalue is x 2 2 .Finally, we recall Weyl's matrix inequality[START_REF] Weyl | Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung)[END_REF] Lemma 12.2.4. Weyl's matrix inequality. If M and P are hermitian matrices of dimension d×d, 1 and λ k (A) denotes the k th largest (with multiplicity) eigenvalue of any hermitian matrix A, then max i,j≤d i+j=k+dλ i (M ) + λ j (P ) ≤ λ k (M + P ) ≤ min i,j≤d i+j=1+k λ i (M ) + λ j (P ) .By induction, if we consider K hermitian matrices P a , a ∈ [K], then∀a ′ ∈ [K], λ k k (P a ′ ) + a̸ =a ′ min j≤d λ j (P a ) .Now, let us apply this to eigenvalue λ bt,ct,a . Consider the permutation p a ∈ S d such that the eigenvalues of V a ordered by p a are in the order of decreasing value:λ pa(1) ≥ λ pa(2) ≥ • • • ≥ λ pa(d) .Let us denote j ′ a := maxj≤d v ⊤ pa(j) (xb t -xc t )̸ =0 j such that λ bt,ct,a = λ pa(j ′ a ) (V a ). λ bt,ct,a ≥ λ pa(j ′ a ) (N a (t -1) + 1)x a x ⊤ a N b (t -1)x b x ⊤ b = κ K + (N a (t -1) + 1)λ pa(j ′ a ) (x a x ⊤ a ) + b̸ =a κ K + N b (t -1) min j≤d λ j (x b x ⊤ b ) = κ K + (N a (t -1) + 1)λ pa(j ′ a ) (x a x ⊤ a ) + κ K (K -1) + b̸ =a N b (t -1) × 0 = κ + (N a (t -1) + 1)λ pa(j ′ a ) (x a x ⊤ a ) .

  Comparison of values of ProblemP from different gaps. Consider ∆ , ∆ ′ ∈ (R + ) K×M , τ ∈ P(∆) and τ ′ ∈ P(∆ ′ ). Then (i). If there is a constant α > 0 so that ∀k ∈ [K] ∀m ∈ [M ], α∆ k,m ≤ ∆ ′ kIf there is a constant β > 0 so that ∀k ∈ [K] ∀m ∈ [M ], ∆ ′ k,m ≤ β∆ k,m thenThe proof follows from the fact that τ and τ ′ are minimal. In particular, to prove(ii), let τ ′′ k,m = β 2 τ ′ k,m for any k ∈ [K], m ∈ [M ].Then, for any agent m and arm k, Choice of threshold function T δ (•). For any N ∈ (N + ) M g M is a function that satisfies g M (δ) ≈ log(1/δ) + M log log(1/δ). Then the good eventE CPE ∶ = ∀r ≥ 0 ∀m ∈ [M ] ∀k ∈ [K], µ ′ (k,m)µ ′ (k,m) (r) ≤ Ω (k,m) (r)holds with probability larger than 1 -δ.Proof. We prove that, if we define the threshold function for N ∈ (N * ) M asT δ (N ) := 2 g M δ ln(N m )) ,where-∀δ ∈ (0, 1), g M (δ) := M C g G (log(1/δ)/M ) , -∀x > 0, C g G (x) := min λ∈(0.5,1) (g G (λ) + x)/λ , -∀λ ∈ (0.5, 1), g G (λ) := 2λ -2λ log(4λ) + log(ζ(2λ)) -0.5 log(1 -λ) ,and ζ is the Riemann zeta function, then, the good eventE CPE := ∀r ∈ N ∀m ∈ [M ] ∀k ∈ [K], µ ′ (k,m) (r) -∆ ′ k,m ≤ Ω (k,m) (r) .holds with probability larger than 1 -δ. Using Kaufmann and Koolen (2021, Proposition 24) on µ ′ (k,m) , for any arm k and agent m, directly yieldsP ∃r ≥ 0, µ ′ (k,m) r -∆ ′ k,m > T δ (n k,• (r)) notation of the paper, consider µ = µ (k,•) and c = W •,m). Then all that is needed to conclude is to apply a union bound on [K] × [M ]P (E c ) = P ∃m ∈ [M ] ∃k ∈ [K] ∃r ≥ 0, µ ′ (k,m) r -∆ ′ k,m > 2T δ (n k,• (r))

Theorem 13. 3 . 1 .

 31 Upper bound on Exp µ (CPE). On any model µ, with probability 1 -δ, CPE (Algorithm 8) outputs the set of N best arms for each agent with an exploration cost at most 32N ⋆ (µ) log 2 Thanks to Theorem 8.4.6, CPE is shown to be δ-correct on event E CPE , of probability greater than 1 -δ (Lemma 8.4.5). Then we upper bound the exploration cost when E CPE holds. We denote by R the (random) number of rounds used by the algorithm, and, for all m ∈ [M ] and k ∈ S m N , by R k,m the (random) last round in which k is still a candidate arm for player m R k,m := sup {r ≥ 0 | k ∈ B m (r)} . By definition of Algorithm 8, R = max m∈[M ] max k̸ ∈S ⋆ m N R k,m . We first provide upper bounds on R k,m and R. To achieve this, we introduce the following notationMoreover, it also permits to prove that, in the last round R, the gap proxies are lower bounded by the true characteristic gaps.Corollary 13.3.4.2. At final round R, and for any agent m and suboptimal armk ∈ S m N , if E CPE holds, If R < r k,m , by definition of r k,m , ∆ ′ k,m (R) ≥ (1/4)∆ ′ k,m ≥ (1/8)∆ ′ k,m . If R ≥ r k,m , we first observe that ∆ ′ k,m (R) = ∆ ′ k,m (R k,m ) = (1/2) ∆ ′ k,m (R k,m -1) by definition of the algorithm (the gaps remain frozen when an arm is eliminated, and they are halved otherwise). As R k,m -1 < r k,m by Lemma 13.3.4, by definition of r k,m , it follows that4 ∆ ′ k,m (R k,m -1) > ∆ ′ k,mand we conclude that∆ ′ k,m (R) ≥ (1/8)∆ ′ k,m .Now, for any m ∈ [M ] and k ∈ [K], using Corollary 13.3.4.2, and the fact that the gap proxies are nonincreasing between two consecutive phases, we get∀k ∈ [K] ∀m ∈ [M ] ∀r ≤ R, ∆ ′ k,m (r) ≥ ∆ ′ k,m (R) ≥ ∆ ′ k,m /8 .Using Lemma 8.4.3 and once again the fact that gap proxies are nonincreasing, for any round r ≤ R, the optimal allocation t(r) k ∈ [K] and agent m ∈ [M ], we now introducer ′ k,m := sup{r ≤ R : d k,m (r) = 0} , so that n k,m (R) = n k,m (r ′ k,m) . Using Lemma 13.3.5 stated below, and the fact that threshold function T δ (•) is nondecreasing in each coefficient of its argument (see its definition in Lemma 8.4.5),n k,m (R) = n k,m (r ′ k,m ) ≤ t k,m (r ′ k,m )T δ (n k,• (r ′ k,m )) + 1 ≤ t k,m (r ′ k,m )T δ (n k,• (R)) + 1 .Lemma 13.3.5. For any k, m, r ≥ 0, either d k,m (r) = 0, orn k,m (r) = n k,m (r -1) + d k,m (r) < t k,m (r)T δ (n k,• (r)) + 1 .Proof. At fixed r ≥ 0, for any set S ⊆ [K] × [M ], let us prove by induction on|S| ≥ 1 1 ∀k ∈ [K] ∀m ∈ [M ], d ′ k,m (r) := (d k,m (r) -1 S ((k, m))) + =⇒ ∀(k, m) ∈ S, n k,m (r -1) + d ′ k,m (r) T δ (n k,• (r -1) + d k,• (r)) < t k,m (r) or d k,m (r) = 0 . At |S| = 1: Let us denote S = {(k ′ , m ′ )}. If d k ′ ,m ′ (r) = 0, then it is trivial. Otherwise, k,m d ′ k,m (r) < k,m d k,m (r), and then, by minimality of solution d(r), at least one constraint from the optimization problem of value k,m d k,m (r) has to be violated. For any (k, m) ∈ S, by definition of d(r) and nondecreasingness of T δ (•) n k,m (r -1) + d ′ k,m (r) = n k,m (r -1) + d k,m (r) ≥ t k,m (r)T δ (n k,• (r -1) + d k,• (r)) ≥ t k,m (r)T δ (n k,• (r -1) + d ′ k,• (r)) .

,

  |S| > 1: At fixed (k ′ , m ′ ) ∈ S, we can apply the claim to S \ {(k ′ , m ′ )}. Moreover, if d k ′ ,m ′ (r) = 0, then the claim is proven. Otherwise, towards contradictionn k ′ ,m ′ (r -1) + d ′ k ′ ,m ′ (r) ≥ t k ′ ,m ′ (r)T δ (n k ′ ,• (r -1) + d k,• (r)) .Let us then consider the following allocation∀k ∈ [K] ∀m ∈ [M ], d ′′ k,m (r) := d k,m (r) -1 {(k ′ ,m ′ )} ((k, m)) .It can be checked straightforwardly -using the nondecreasingness of T δ () -1 For any x ∈ N M , (x) + := (max(0, x m )) m∈[M ], and 1 S is the indicator function of set S.that d ′′ satisfies all required constraints for any pair(k, m) ∈ [K] × [M ], and that k,m d ′′ k,m (r) = k,m d k,m (r) -1, which, by minimality of d, is absurd. Then the claim is proven for |S| > 1. Then Lemma 13.3.5 is proven by consideringS = [K] × [M ] .By summing the upper bound on(n k,m (R)) k,m over [K] × [M ],we can upper bound the exploration cost τ asτ := k,m n k,m (R) ≤ k,m t k,m (r ′ k,m )T δ (n k,• (R)) + KM ≤ k,m t k,m (r ′ k,m )β * (τ ) + KM ≤ k,m r≤R t k,m (r)β * (τ ) + KM ≤ R max r≤R k,m t k,m (r) β * (τ ) + KM ≤ log 2 (8/∆ ′ min ) max r≤R k,m t k,m (r) β * (τ ) + KM ,where we use Corollary 13.3.4.1 and introduce the quantityβ * (τ ) := T δ (τ 1 M ) = 2 g M δ KM + 2M ln (4 + ln (τ )) where ∀n ∈ [M ], 1 M (n) = 1 .Using Equation (13.1) and Lemma 8.4.1,τ ≤ 32 N log 2 (8/∆ ′ min ) β * (τ ) + KM ≤ 32N ⋆ log 2 (8/∆ ′ min )β * (µ) + KM .Therefore, τ is bounded from above bysup {n ∈ N * : n ≤ 32N ⋆ log 2 (8/∆ ′ min )β * (n) + KM } .ApplyingKaufmann, Ménard, et al. (2021, Lemma 15) with∆ = 32N ⋆ log 2 (8/∆ ′ min ) KM + 2g M δ KM , b = 4M , c = 4 , d = e -1 using ∀n, log(n) ≤ ne -1 ,yields an explicit upper bound on τ T (µ) := 32N ⋆ log 2 (8/∆ ′ min ) KM + 2g M which satisfies τ T (µ) ≤ a + b ln(c + dτ T (µ)). Using that g M (x) x + M log log(x) in the regime of small values of δ, we obtain that T (µ) = 32N ⋆ log 2 (8/∆ ′ min ) log(1/δ) + o δ→0 log(δ -1 ) .

Figure 14 . 1 :

 141 Figure 14.1: Visualization of the "real life" repurposing instance. Left plot : Boxplot of rewards, by increasing order of average scores. Right plot : Pearson's r correlation heatmap based on the d = 10 PCA-reducred features, ranked by the order obtained through hierarchical clustering. The first row of colors (red/green) corresponds to the class (resp., proconvulsant drug/antiepileptic), whereas the second row of colors (7 different colors) corresponds to the clusters found through hierarchical clustering.

Table 1 :

 1 table 1 présente les résultats obtenus. En comparant les nombres d'observations entre LUCB et MisLid, on observe, comme précédemment, qu'il est avantageux de prendre en compte les signatures des médicaments pour limiter le nombre d'observations. De plus, sur ce modèle des scores non linéaires, N -LinGapE se trompe systématiquement sur les recommandations de candidats, contrairement à MisLid. La table des résultats associée. δ est la fréquence d'erreur dans les 100 itérations, ŝ est le nombre moyen (± l'écart-type), arrondi à l'entier le plus proche, d'échantillons utilisés.

	Algo.	δ	ŝ
	MisLid	0 % 158 869 ± 126 209
	LinGapE 100 %	161 ± 159
	LUCB	0 % 222 969 ± 22 798

Table 2 :

 2 La table des expériences dans le cadre de l'apprentissage centralisé (δ = 10 %, N = 1, K = 21, M = 3, 100 itérations). La fréquence d'erreur δ est à 0 % dans toutes les expériences. ŝ est le nombre moyen (arrondi à l'entier le plus proche) d'échantillons utilisés (± l'écart-type), et r est le nombre moyen de phases de communication. Notre algorithme est celui dénommé CPE. La référence n'est valable que pour N = 1 et pour W

	xvi

Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI).

  Gene regulatory networks and Boolean networks . . . . . . . . . 1.3 Dynamics in a Boolean network . . . . . . . . . . . . . . . . . . . Correlation between score and number of common differentially expressed genes . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Enrichment analysis of differentially expressed genes in the best recommended treatment . . . . . . . . . . . . . . . . . . . . . . . 7.6 Enrichment analysis of differentially expressed genes in the second best recommended treatment . . . . . . . . . . . . . . . . . 7.7 Rankings from cosine scores and repurposing scores on a subset of drugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Performance of cosine scores and repurposing scores on a subset of drugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Clustering of epileptic profiles . . . . . . . . . . . . . . . . . . . . 9.1 Comparison of the 50 inferred networks . . . . . . . . . . . . . . 9.2 Enrichment analyses for spread-selected genes . . . . . . . . . . Distribution statistics for the inferred networks . . . . . . . . . . 35 2.2 Distribution statistics of the spread values . . . . . . . . . . . . . 40 2.3 Scores for genes prioritized by spread values . . . . . . . . . . . 44

	1 2 1.1 Lac operon regulation as a gene regulatory network . . . . . . . Drug development timeline and principle of drug repurposing . Transcriptomics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Examples of regulatory interactions . . . . . . . . . . . . . . . . 13 1.2 List of Tables 2.1

2.1 Automated building of Boolean networks . . . . . . . . . . . . . 2.2 Inferred network for the application to epilepsy . . . . . . . . . . 2.3 Comparison of the inferred networks . . . . . . . . . . . . . . . . 2.4 Illustration of the collapsed network . . . . . . . . . . . . . . . . 2.5 Comparison between spread values and other gene measures . 2.6 Gene ranking by decreasing spread value . . . . . . . . . . . . . 3.1 Visualization of gene targets for antiepileptic and proconvulsant drugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Comparison between the different types of drug signatures . . . 3.3 Drug repurposing on a set of antiepileptics and proconvulsant drugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Comparison between the drug repurposing method and the baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Multi-armed bandits . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Drug repurposing with bandits . . . . . . . . . . . . . . . . . . . 5.1 Drug repurposing instance with linear bandits . . . . . . . . . . . 5.2 Comparison between the two stopping rules . . . . . . . . . . . 6.1 Drug repurposing with misspecified bandits . . . . . . . . . . . . 112 7.1 Timeline of the experiments . . . . . . . . . . . . . . . . . . . . . 119 7.2 Application of signature reversion to encephalopathy of prematurity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 7.3 Comparison between Characteristic Direction signatures and univariate differential analysis . . . . . . . . . . . . . . . . . . . . . 123 xx 7.4 11.1 Cosine method applied to the whole set of 34 drugs . . . . . . 14.1Visualization of the "real life" repurposing instance . . . . . . . . 14.2Results for the drug repurposing in epilepsy . . . . . . . . . . . . 3.1 Hit ratios on the proposed drug scoring and the baseline . . . . 56 4.1 Prior results on the sample complexity for unstructured bandits 72 5.1 Adaptive sampling-based algorithms for linear Top-N identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.2 Sample complexity results for linear bandits . . . . . . . . . . . 91 6.1 Running a linear bandit algorithm on an increasingly non linear model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.2 Results for drug repurposing with misspecified bandits . . . . .

Table of contents / Table des matières xxviii Introduction 2 I Analyzing a disease-specific regulatory network 10 1 Introduction to Boolean networks 12

  Selection of drug signatures . . . . . . . . . . . . . . . . . . . . . 3.3 Scoring with the Boolean network . . . . . . . . . . . . . . . . . 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.1 Modelling gene expression with regulatory networks . . . . . . . 13 1.2 Overview of gene regulatory networks . . . . . . . . . . . . . . . 15 1.3 Introduction of dynamics with Boolean networks . . . . . . . . . 16 1.4 Boolean networks in drug repurposing . . . . . . . . . . . . . . . 21 2 Prioritization of candidate genes through Boolean networks 23 2.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.2 Reproductible inference of a cell-line specific Boolean network . 26 2.3 Detection of master regulators in a specific disease-context . . 29 2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 xxvi 3.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2

of the screened drugs by distance to the disease-related signature Computation of the distance between each treatment-associated signature and the disease-related one

			PRECLINICAL STAGE	CLINICAL DEVELOPMENT	MARKETING FOLLOW-UP
	Identification of therapeutic targets, chemical compounds, ...	Assessment of the protocol of treatment	Phase I, Phase II, Phase III -toxicity study -drug efficacy testing -comparison to the standard-of-care	Phase IV, monitoring of side effects
	≈ 5 years	≈ 1-3 years	≈ 2-10 years	during commercialization
	gene 1 gene 2				Treatment 1 Treatment 2
	gene 3				Control
					Healthy
	gene 1 gene 2 gene 3				Patient High expression Low expression
					Upregulated
					Downregulated
	gene 1
	gene 2
	gene 3			Ranking

Table 1 .1: Examples of regulatory interactions. Two examples

 1 of regulatory interactions of various types, and their implications

	, which relies

Dynamics in a Boolean network. Dynamics in a Boolean network. Left plot : Two

  instances of Boolean networks, with the same interaction graph, but different sets of regulatory functions. Right plot : Corresponding state-transition graphs (STG), assuming synchronous updates, that is, simultaneous update of all gene regulatory functions.

						Synchronous state transition
		A			011	State: expression state of genes A, B, C Unsteady attractor state (cycle)
	C	B	000	001	101	Steady attractor state
	Gene regulatory	B ∨C ⇒ A A ⇒ B	100	010	110	111
	functions	¬B ∧ ¬A ⇒ C
		A
			000	001	101
	C	B
			010		110	111
	Gene regulatory functions	C ⇒ A A ⇒ B ¬B ∧ ¬A ⇒ C	100	011
		Activatory regulation
		Inhibitory regulation
		Node
	Figure 1.3:

  Min. 25 th quantile Median Mean 75 th quantile Max.Table2.1: Distribution statistics for the inferred networks. Distribution statistics on the number of unique regulatory functions (RFs) across solutions per gene, and on the value of the general topological parameter (GTP) used for network selection in step (C) of the inference procedure. All values are rounded up to the 3 rd decimal place.

	# RFs	1	1	2	2.202	3	11
	GTP	0.796	0.798	0.800	0.800	0.800	0.802

Table 2

 2 Distribution statistics (rounded up to the 5 th decimal place) of the spread values obtained for M30 genes present in the inferred network.

	Minimum 25 th quantile Median	Mean	75 th quantile Maximum
	0.0	0.0	0.0	0.00254	0.0	0.0556

.2: Distribution statistics of the spread values.

Table 2 . 3 :

 23 Scores for genes prioritized by spread values. In silico perturbation scores, rounded up to the second decimal place, resulting from the comparison of attractor states, reachable from control samples, to the set of control samples, after the knockout of one of the candidate genes in the Boolean network in Figure2.2.

repurposing on a set of antiepileptics and procon- vulsant drugs

		0.015		0.2
		0.010		0.0
	Score	Pentylenetetrazol (0.01) Brucine (0.01) Methocarbamol (0.01) Amoxapine (0.01) Pravastatin (0.01) Caffeine (0.01) Benzylpenicillin (0.01) Lamotrigine (0.01) Hydroxyzine (0.01) Primidone (0.00) Bemegride (0.00) Pindolol (0.00) Carbamazepine (0.00) Dextromethorphan (0.00) Lidocaine (0.00) Pyrimethamine (-0.00) Drug Quinidine (-0.00) Thiocolchicoside (-0.00) Riluzole (-0.00) Domperidone (-0.00) Diclofenamide (-0.01) Capsaicin (-0.01) Chloroquine (-0.01) Felbamate (-0.01) Pimozide (-0.01) Securinine (-0.01) Valproic-Acid (-0.01) Ropinirole (-0.01) Ticlopidine (-0.01) Mianserin (-0.01) Withaferin-A (-0.02) Bezafibrate (-0.02) Chlorambucil (-0.02) Dmcm (-0.02) 0.020 0.015 0.010 0.005 0.000 0.005	Score	1.0 0.8 0.6 0.4 0.2	Drug
	Figure 3.3: Drug

. Boxplots of rewards across patient profiles on the set of 34 drugs, sorted by decreasing average score, as reported in the x-axis. a The dot (average score) and labels on the x-axis are colored in green (resp., in red) if they match an antiepileptic (resp., a proconvulsant) drug. The black solid line is the function of median score values. Left plot : ranking from L1000 CDS 2 (baseline). Right plot : ranking from Algorithm 3.

a In L1000 CDS 2 , there is a single value equal to the average value.

Table 5

 5

.1: Adaptive sampling-based algorithms for linear Top-N identification. Adaptive sampling-based algorithms for (ε, δ)-PAC Top-N identification (our proposals are in bold type ; except for LUCB and UGapE, all algorithms use paired indices instead of individual indices). References : LUCB (Kalyanakrishnan et al., 2012), UGapE

Table 5 .

 5 2: Sample complexity results for linear bandits. Sample complexity results for unstructured and linear bandits on any model function f ∈ M lin . ω ⋆ is defined as the minimizer of Problem (5.4). Our proposed algorithms are in bold type, and conjectures are denoted with a superscript star. I omit factors in O(K) in unstructured bandit algorithms.

  Comparison between the two stopping rules. Plotting τ LUCB against τ LUCB on a same set of 100 trajectories of the version of LinGIFA using the stopping rule τ LUCB (LinGIFATLUCB) on the drug repurposing instance. The blue curve ("τ LUCB ") plots B c(t),b(t) (t) at each round t : the solid line being the average value at round t across the 100 iterations, the blue lighter zone the interval around the average value ± the standard deviation. Similarly,

	Stopping quantity	0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6	0	Round 50 100 150 200 LUCB UGapE
	Figure 5.2: the red curve ("τ UGapE ") plots max j∈J(t)	N max i̸ =j

Table 6 .2: Results for drug repur- posing with misspecified bandits.

 6 Table of results associated with the boxplots. δ is the empirical error rate across 100 iterations, ŝ is the average number of samples (± the standard deviation) rounded up to the closest integer.

	Algorithm	δ	ŝ
	MisLid	0% 158, 869 ±126, 209
	LinGapE	100%	161 ±159
	LUCB	0% 222, 969 ±22, 798

Table 9 .

 9 Table 2.1 and Figure 2.3 allows us to conclude similarly to the main paper that the networks obtained just before the network selection step are mostly functionally and topologically similar. Min. 25 th quantile Median Mean 75 th quantile Max.

	# RFs	1	1	2	2.635	3	17
	GTP	0.794	0.796	0.797	0.797	0.799	0.802

Table 9 .

 9 5: Regulatory interactions present in all of the 25 solutions. strongest evidence source from the STRING database. "Association in databases" means associated in curated pathway databases.

	Regulator Regulated Sign	Evidence source *
	RBFOX1	PEG3	Inhibitory Coexpression
	SLITRK3	IL1RAP	Inhibitory Text-mining
	TSPAN7	AFF2	Activatory Coexpression
	UQCRQ	TIMM17A	Inhibitory Coexpression
	CENPJ	ANAPC1	Activatory Coexpression
	SYT13	MLLT11	Inhibitory Coexpression
	GUCY1B3 AHNAK2	Activatory Text-mining
	PLEKHG3 CCDC68	Inhibitory Text-mining
	MLLT11	TTC3	Activatory Text-mining
	SULT4A1 KIAA0232 Inhibitory Text-mining
	GRIN1	CIT	Activatory Interaction
	GPI	ANXA6	Inhibitory Text-mining
	RBFOX1	ZMAT4	Activatory Coexpression
	FAR2	FAM49A	Inhibitory Text-mining
	RAB3A	REEP2	Activatory Coexpression
	CAMK2B	AGGF1	Inhibitory Association in databases
	GAP43	ELAVL2	Inhibitory Coexpression
	ADAM22	EPB41L3	Inhibitory Coexpression
	CDC42	ARHGAP44 Activatory Interaction
	GNB5	PAK1	Inhibitory Association in databases
	STMN2	PSMG1	Inhibitory Coexpression
	AMPH	AAGAB	Activatory Interaction
	GNB5	PRKCE	Inhibitory Association in databases
	ATP1B3	FXYD7	Inhibitory Association in databases
	ATP1A3	PANK2	Activatory Text-mining

*

Drug repurposing scoring 10.1 List of antiepileptic and proconvulsant drugs

  The initial tables of 71 epilepsy-related drugs on the next page (Tables 10.1 and 10.2) have been compiled thanks to Baptiste Porte.

	Caffeine (0.10) Felbamate (0.09) Brucine (0.09) Dextromethorphan (0.08) Withaferin-A (0.08) Methocarbamol (0.06) Ropinirole (0.06) Diclofenamide (0.06) Thiocolchicoside (0.06) Pimozide (0.06) Chlorambucil (0.06) Capsaicin (0.06) Ticlopidine (0.05) Hydroxyzine (0.05) Dmcm (0.05) Lamotrigine (0.05) Benzylpenicillin (0.05) Mianserin (0.05) Valproic-Acid (0.05) Carbamazepine (0.05) Primidone (0.05) Securinine (0.04) Pindolol (0.04) Chloroquine (0.04) Amoxapine (0.04) Pyrimethamine (0.03) Quinidine (0.02) Bezafibrate (0.02) Lidocaine (0.01) Pentylenetetrazol (0.01) Bemegride (0.01) Domperidone (0.01) Pravastatin (0.01) Riluzole (-0.

3. Upper bound on the number of samples from I t (Gap- GIFA, largest variance rule).

  For any round t > 0, t < τ UGapE N It (t) ≤ 4σ 2 (T δ (t)) 2 max ε, ε + ∆ It 3 Using successively the definition of the stopping rule τ UGapE at time t < τ UGapE , if a t ∈ arg max a∈{b(t),c(t)}W a (t) and I t is sampled according to the largest variance rule (Equation (5.1)), thenε < Bb(t) (t) ≤ minmax(∆ b(t) , ∆ c(t) ) + 6W at (t), 0 + 2W at (t)Then using Lemma 5.2.4, provided that κ ≥ 0

						-2
							.
	Proof. max ε,	ε + ∆ b(t) 3	,	ε + ∆ c(t) 3	≤ 2W at (t) = 2σ 2 T δ (t) max a∈{b(t),c(t)}	X a ( V κ (t)) -1
		max ε, ≤ 2σ 2 T max ε, ε + ∆ It 3 ε + ∆ It 3 ≤ 2σ 2 T δ (t)	X It 2 N It (t) X It 2 2 + κ	≤	2σ 2 T

δ (t) X It ( V κ (t)) -1 δ (t) N It (t)

.

which yields N It (t) ≤ 4σ 2 (T δ (t)) 2 max ε, ε + ∆ It 3 -2

  bt (t) ≤ min(-max(∆ bt , ∆ ct ) + 3W t (b t , c t ), W t (b t , c t )) (b t , c t ) = σT δ (t)||x ct -x bt || ( V κ (t-1)+xa t x ⊤ a t ) ⇔ N at (t) ≤ σ 2 T δ (t) 2 x bt -x ctApplying Lemma 5.3.8 allows us to conclude on the conjecture similarly to what was previously done.

	⇔ max ϵ,	∆ bt + ϵ 3	,	∆ ct + ϵ 3	≤ W t -1
					≤ σT δ (t)	x bt -x ct 2 κ + N at (t) x at 2 2	(a t pulled at t)
						x at 2 2	2 2	max ε,	∆ bt + ε 3	,	∆ ct + ε 3	-2	.

  Table 14.1, whereas the final error frequencies and sample complexities are reported in Table 14.2. Boxplots of the sample complexity for MisLid for Ψ ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0} for the "real life" drug repurposing instance. Green (resp., red) boxplots correspond to (resp., non) δ-correct algorithms.

	175000 200000	means
	0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 MisLid (value of ) 0 25000 50000 75000 100000 125000 150000 Sample complexity Figure 14.2: Results for the drug repurposing in epilepsy. Parameter Value K 21 d 10 σ 1 * δ 10% ε 0.14 N 3 #iterations 100 Table 14.1: Parame-ters values for the drug repurposing in
		epilepsy.	Parameter
		values for the "real-life"
		drug repurposing set-
		ting.

Table 14

 14 .2: Results for the drug repurposing in epilepsy. Results (in terms of empirical error frequency and sample complexity) for the "reallife" drug repurposing example. δ is the empirical error frequency across 100 iterations, ŝ is the average sample complexity across iterations (± the standard deviation), rounded up to the closest integer.

	Alg.	δ	ŝ
	LUCB	0%	37, 425 ±271
	MisLid (Ψ = 1)	0%	16, 189 ±1, 492
	MisLid (Ψ = 0.9)	0%	16, 070 ±1, 287
	MisLid (Ψ = 0.8)	0%	15, 846 ±1, 371
	MisLid (Ψ = 0.7)	0%	15, 942 ±1, 311
	MisLid (Ψ = 0.6)	0%	17, 205 ±1, 610
	MisLid (Ψ = 0.5)	0%	60, 992 ±4, 433
	MisLid (Ψ = 0.4) 100% 167, 233 ±13, 922
	MisLid (Ψ = 0.3) 100%	84, 337 ±5, 966
	MisLid (Ψ = 0.2) 100% 29, 586 ±24, 571
	MisLid (Ψ = 0.1) 100% 31, 144 ±11, 606
	MisLid (Ψ = 0)	100%	1, 393 ±491
	N -LinGapE	100%	157 ±13
	LinGIFA	100%	91 ±14

D. W.[START_REF] Carley | Drug repurposing: identify, develop and commercialize new uses for existing or abandoned drugs. Part I[END_REF]. "Drug repurposing: identify, develop and commercialize new uses for existing or abandoned drugs. Part I". IDrugs: the investigational drugs journal, 8(4), pp. 306-309.

[START_REF] Réda | Machine learning applications in drug development[END_REF]. "Machine learning applications in drug development". Computational and structural biotechnology journal, 18, pp. 241-252.

J.[START_REF] Lamb | The Connectivity Map: using gene-expression signatures to connect small molecules, genes, and disease[END_REF]. "The Connectivity Map: using gene-expression signatures to connect small molecules, genes, and disease". science,313(5795), pp. 1929-1935.

Y.[START_REF] Xin | Identification of Nifurtimox and Chrysin as Anti-Influenza Virus Agents by Clinical Transcriptome Signature Reversion[END_REF]. "Identification of Nifurtimox and Chrysin as Anti-Influenza Virus Agents by Clinical Transcriptome Signature Reversion". International Journal of Molecular Sciences, 23(4), p. 2372.

Bokobza, Réda, et al. (in prep.). "Therapeutic evaluation of Hu-MSCs in a rat model of perinatal inflammation: a systematic outcome scoring".

H.[START_REF] Bolouri | Transcriptional regulatory cascades in development: initial rates, not steady state, determine network kinetics[END_REF]. "Transcriptional regulatory cascades in development: initial rates, not steady state, determine network kinetics". Proceedings of the National Academy of Sciences, 100(16), pp. 9371-9376.

D.[START_REF] Szklarczyk | The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets[END_REF]. "The STRING database in 2021: customizable proteinprotein networks, and functional characterization of user-uploaded gene/measurement sets". Nucleic acids research, 49(D1), pp. D605-D612.

17 S.[START_REF] Chevalier | Synthesis of boolean networks from biological dynamical constraints using answer-set programming[END_REF]. "Synthesis of boolean networks from biological dynamical constraints using answer-set programming". 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, pp. 34-

41. 18 S.[START_REF] Babichev | Technique of Gene Regulatory Networks Reconstruction Based on ARACNE Inference Algorithm[END_REF]. "Technique of Gene Regulatory Networks Reconstruction Based on ARACNE Inference Algorithm."

IDDM, pp. 195-207; E. C. Harrington (1965). "The desirability function". Industrial quality control, 21(10), pp. 494-498.19 Delahaye-Duriez,[START_REF] Delahaye-Duriez | Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery[END_REF]. "Rare and common epilepsies converge on a shared gene regulatory network providing opportunities for novel antiepileptic drug discovery". Genome biology, 17(1), pp. 1-18.

D.[START_REF] Bouneffouf | A contextualbandit algorithm for mobile context-aware recommender system[END_REF]. "A contextual-bandit algorithm for mobile context-aware recommender system". International conference on neural information processing.Springer,

W. R.[START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]. "On the likelihood that one unknown probability exceeds another in view of the evidence of two samples". Biometrika, 25(3-4), pp. 285-294.

P.[START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF]. "The nonstochastic multiarmed bandit problem". SIAM journal on computing, 32(1), pp. 48-77;[START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF]. "A contextual-bandit approach to personalized news article recommendation". Proceedings of the 19th international conference on World wide web, pp. 661-670.

V.[START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF]. "Best arm identification: A unified approach to fixed budget and fixed confidence". Advances in Neural Information Processing Systems, 25; S.[START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. "PAC subset selection in stochastic multiarmed bandits." ICML. vol. 12, pp. 655-662.

 28 Réda, Kaufmann, and Delahaye-Duriez (2021). "Top-m identification for linear bandits". International Conference on Artificial Intelligence and Statistics. PMLR, pp. 1108-1116.

V.[START_REF] Gabillon | Best arm identification: A unified approach to fixed budget and fixed confidence[END_REF]. "Best arm identification: A unified approach to fixed budget and fixed confidence". Advances in Neural Information Processing Systems, 25; S.[START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. "PAC subset selection in stochastic multiarmed bandits." ICML. vol. 12, pp. 655-662.

[START_REF] Réda | Near-Optimal Federated Learning in Bandits[END_REF]. "Near-Optimal Federated Learning in Bandits". In press.

All parameters are similar but for the treatment, which can be a sham or no treatment at all.

That is, gene expression changes that trickle downstream of the initial gene perturbations incurred by a treatment.

Because of its straightforward qualitative interpretation, and its relatively low computational cost, my PhD revolved around discrete models of gene regulation, and more specifically, around Boolean networks.

Note that, in this thesis, we will not consider non monotonous regulatory interactions, which can be either inhibiting or activatory given the local context. This assumption is based on the fact that, in most models, it has been shown that non-monotonous regulations could be successfully modelled via post-transcriptional mechanisms, such as the involvement of non-coding DNA sequences, as mentioned in[START_REF] Réda | Automated inference of gene regulatory networks using explicit regulatory modules[END_REF]. Dynamics encoded by multi-level formalisms (i.e., with more than two expression states) can also be encompassed by carefully selecting the type of dynamics in the network(Paulevé et al.,

2020).

When it is obvious, the term "state" can also be used to designate network configurations instead of gene expression states.

Related code is located at https://github.com/clreda/PrioritizationMasterRegulators.

Fibroblast-like synoviocytes are a cellular subtype which is present in conjunctive tissues.

Rows are genes, and columns are samples.

That number was chosen for reasons related to computational cost and time.

Remember that, in the application to epilepsy, we do not use genes from DisGeNet, but the preselected set of genes M30.

This case never happened in practice.

That is, the orientation of the 3D molecular structure. Two molecules which only difference lies in their chirality cannot overlap, as the conformation of one is the mirror inverted version of the other.

This choice was made in order to ensure fairness when comparing this method to ours, since our drug signatures are computed over brain cell lines, as described later in this chapter.

Characteristic Direction (CD) has been introduced in the previous chapter. In a nutshell, this procedure creates a vector CD[G ∞ G ∈ ] which reports the genewise magnitude and direction of change in expression from reference sample group G ∈ to treated group G ∞ .

That is, the transcriptional profile of a healthy individual with respect to epilepsy.

This is done in practice by considering the reported value distil_ss in LINCS L1000 for each profile. This quantity is correlated to the number of differentially expressed genes (genes altered by the treatment), and is predictive of the reproductibility of the profile[START_REF] Lim | Evaluation of connectivity map shows limited reproducibility in drug repositioning[END_REF].

We set the number of permutations performed by CD at 10, 000.

This metric (like others such as distil_ss) is further discussed at https://clue.io/ connectopedia/glossary.

On a personal computer (processor Intel Core i7-8750H, 12 cores @2.20GHz, RAM 16GB).

i.e., this problem can be solved using an amount of memory at most polynomial in the number of nodes in the network.

i.e., the proof that the considered state is not a reachable attractor state is verifiable in polynomial time (in the number of nodes) by a deterministic algorithm.

e.g., the Genomics of Drug Sensitivity in Cancer (GDSC) database[START_REF] Yang | Genomics of Drug Sensitivity in Cancer (GDSC): a resource for therapeutic biomarker discovery in cancer cells[END_REF] for cancer subtypes.

Which is the name we will retain for the remainder of the manuscript.

That is, grouping data.

Even easier with the dynamics described in Paulevé et al. (2020).

I denote "(multi-armed) bandit" the setting, and "bandit algorithm" the algorithm itself.

And also some internal randomization variable U t ∈ [0, 1] in practice.

In practice, such an assumption could be discussed, especially for real-life applications. Nonetheless, it still remains an interesting simplification for our drug repurposing problem. Some theoretical work on the case of unknown variance is present in the literature, see for instance[START_REF] Chowdhury | Bregman Deviations of Generic Exponential Families[END_REF].

Familiar examples of σ 2 -subgaussian distributions are Gaussian distributions with fixed variance set to σ 2 , or bounded distributions.

That is, reducing the number of calls to the Boolean network, in our case.

We will only consider scalar parametrization here.

Because it simplifies the proof of correctness of the algorithms (refer to the next sections). This assumes that the expected rewards of the N th and the (N + 1) th arms are distinct, which is a reasonable assumption in practice.

Sometimes called best arm identification in the case where N = 1.

I will often abuse notation by only mentioning "δ-PAC".

It still holds for any algorithm for pure exploration.

The generation of noisy observations, and selection of arms (for instance, when choosing between two arms maximizing the same criterion) rely on the seed of a pseudo-random generator.

The σ-algebra ("tribu" in French) σ(X) associated with a set X is a collection of subsets of X, such that :X ∈ σ(X) ; if subset A ∈ σ(X), so is (A)c (closure under complement) ; if a finite number n of subsets A 1 , A 2 , . . . , A n belong to σ(X), then so is i≤n A i (closure under countable unions) and i≤n A i (closure under countable intersections).

For best arm identification, Sticky Track-and-Stop (mentioned in Degenne and Koolen (2019, Algorithm 1)) and Track-and-Stop coincide.

Related code is available at https://github.com/clreda/linear-top-m.

Note that we tackled both issues in Chapter 6.

That is, when M is the entire set of functions mapping from R d to R, see Definition

4.1.1.4 Which is a mandatory step for unstructured bandits in order not to sample at random in the first rounds, but might be ignored for structured ones, provided κ > 0.

Be careful that here we consider c ∈ S ⋆ N (one of the N best arms), and not c ∈ S ⋆ N (ε) (one of the N best arms up to ε).

However, note that the gaps depend on N and thus are defined differently.

We suspect that, if X is well-conditioned (w.r.t. a well-chosen norm), Γ ′ a (X) ≈ O(1).

The code related to experiments is available at https://github.com/clreda/ misspecified-top-m.

i.e., M := (R d → R), the whole set of functions mapping from R d → R.

In practice, the algorithm proposed in this work will not need to use the actual value of C.

However, GLRTs date back at least to[START_REF] Wilks | The large-sample distribution of the likelihood ratio for testing composite hypotheses[END_REF].

The use of online learners is a standard approach in bandits ; however, their description is outside the scope of this thesis. The interested reader can refer to De Rooij et al. (2014).

Even though the latter can be useful for interpretability, as previously mentioned.

That is, adult stem cells, which can be sourced from different body tissues.

The adjustment is necessary to compare p-values resulting from a multiple hypothesis testing, as done in a univariate differential expression analysis, here, one hypothese per gene.

Characteristic Direction was introduced in Chapter 2. In a nutshell, signature CD[G1 G2], computed through Characteristic Direction for two condition groups G1 and G2, is the vector normal to the decision frontier in a high-dimensional space, which classifies samples into either G1 or G2, and that is oriented in the direction from G2 to G1. That means in particular that the changes reported in the signature are changes in G1 compared to the reference group G2. Associated empirical p-values, relying on data permutations, might be computed using this method to assess the significance of the change in expression, and we used threshold 5% and 100 permutations in the computations.

That is, we match rat genes with human genes which have a close common genetic ancestor in the phylogenetic tree. As a general rule, orthologuous genes have the same function across species, which is the basis for running the analysis on human genes.

Note that we replaced zeroes by -1's in the binary profiles, in order to appropriately reflect down-regulation when computing the cosine score.

Their setting does not satisfy some requirements in federated learning, e.g., dealing with communication interruption.

That is, up to logarithmic multiplicative factors.

Related code is located at https://github.com/clreda/near-optimal-federated.

This type of bandits is an instance of contextual bandits, which are outside the scope of this thesis.

The value of a minimization problem is the objective evaluated at a minimizer (a similar definition can be given for maximization problems).

√ M ∆ ′ min . In the setting of collaborative learning, where M agents face the same set of arm distributions and W = I M , Hillel et al. (2013, Theorem 4.1) prove that an improvement of multiplicative factor 1/M on the exploration cost for a traditional best arm identification

https://github.com/clreda/PrioritizationMasterRegulators

RNA-sequencing data for the most recent version, microarray for the first generated profiles.

All these measures are further described at https://clue.io/connectopedia/glossary.

For real matrices, it means symmetric.

The choice of the value of d is guided by computational considerations, whereas N = 3 is based on the choice made in Chapter 5.

Remerciements

Acknowledgements / Remerciements xix

Part III Application & extension of drug repurposing

in Top-N identification

.

where λ ′ (k,m) := n∈[M ] w n,m λ (k,n) for any arm k and agent m. If, for δ-correct algorithm A, we assume that stopping time τ A is almost surely finite under model µ, then let event E µ := {∃m ∈ [M ], S m N (τ ) ⊆ S m N }. Since A is δ-correct, this event holds with probability at most δ under model µ and at least 1 -δ under any alternative model to µ. We define τ k,m := E {µ} [N (k,m) (τ A )], where N (k,m) (t) is the number of samples arm k is sampled by agent m up to round t included, and kl : (x, y) → x log(x/y)+(1-x) log((1-x)/(1-y)) the binary relative entropy function. Since A is δ-correct with δ ≤ 1/2, by Kaufmann, Cappé, and Garivier (2016, Lemma 1) and the expression of Kullback-Leibler divergence for Gaussian distributions with fixed variance σ 2 in Equation (4.4), for any alternative model λ to µ

Since all diagonal coefficients of W are positive, for any k

τ k,m > 0. Indeed, if w m,m = 0, it is possible to pick λ that only differs from µ by the entry λ (k,m) , in such a way that arm k becomes optimal -or sub-optimalfor agent m

otherwise .

From Equation (8.1) and δ ∈ (0, 1), we get

Consider now a fixed agent m ∈ [M ], and two arms k ∈ S m N (µ) and ℓ ∈ S m N (µ). We will build an alternative model λ, similar enough to µ, where only arms k and ℓ are modified for all agents, so that k ∈ S m N (λ) and ℓ ∈ S m N (λ). Given two nonnegative sequences γ, γ ′ ∈ (R + ) M , we define λ such that for all n ∈

Conclusion

Controlling for the dose-related and exposure time effects. I have focused most of my PhD on carefully crafting the selection process of transcriptomic profiles to obtain the most relevant drug signatures. This process relied on the selection of the appropriate cell line, as shown in Chapter 3. However, as demonstrated by the case study in Chapter 7, other parameters might significantly impact the changes on transcription, such as the dose or time of administration. Again, new challenges arise as there is a need to simulate the dose and time-related effect for drugs which have never been tried on patients.

Towards adaptive clinical and drug repurposing trials. Finally, Chapter 8 offers a glimpse into a new framework of uncovering new drug candidates, by sharing information across patient subpopulations in an communication-efficient way. However, this raises the issue of data privacy, as potentially, the actual outputs shared from agents to the central controller might be enough to threaten the anonymity of patients. This is one of the main topics studied in distributed or federated learning, which would be another approach to investigate the use of the drug repurposing method proposed in this thesis. 10.2: Initial list of proconvulsants with their associated references, and annotated presence of existing "BrainCell" signatures as described in Chapter 3. Drugs with existing "BrainCell" signatures are included in the ranking in Figure 3.3. Similarly to Table 10.1, I rank drugs in three categories : "Exploratory" (observed proconvulsant effect in animal or human), "Proven" (proven proconvulsant effect), "Ground truth" (in the class of proconvulsant drugs).

Drug repurposing instances in bandits

Note that the scores shown in Table 10.3 do not match the average scores in the right-hand boxplot in Figure 3.3, although the resulting ranking is similar to the one shown in the figure. This model has been obtained on the very same set of signatures as the one shown in Chapter 3.3. The one difference between the two models is in the initial patient states. On the one shown in Chapter 3, gene identifiers for genes measured in the profiles from [START_REF] Mirza | Genetic regulation of gene expression in the epileptic human hippocampus[END_REF] were converted into gene symbols present in LINCS L1000, which guaranteed a greater number of matches (207 matched genes instead of 194) than using the gene identifiers provided in the initial dataset (which were the profiles used for computing the scores shown in Table 10. 10.3: Bandit instance selected from the set of 34 ranked drugs using their "BrainCell" drug signatures (refer to Chapter 3). This instance is considered as it is in Chapter 8, whereas feature transformation (to make the model linear, resp. misspecified linear) is applied to the drug signatures in Chapter 5, resp. Chapter 6.

Chapter 11

Chapter 7 : Results for the cosine method applied to the "stabilized" profiles We report on Figure 11.1 the ranking, the ROC and PR curves obtained by computing a cosine similarity score between "stabilized" binary profiles predicted by the Boolean network, and the "disease" signature for epilepsy CD[Healthy Patient] based on Characteristic Direction [START_REF] Clark | The characteristic direction: a geometrical approach to identify differentially expressed genes[END_REF]. The reported baseline method is L1000 CDS 2 [START_REF] Duan | L1000CDS2: LINCS L1000 characteristic direction signatures search engine[END_REF], performed as described in Chapter 3. Note that on this larger dataset, the AUC is equal to the one reported in Chapter 3 for the drug repurposing method we proposed. However, the precision-recall curve is significantly worse.

Chapter 12

Chapter 5: Theoretical guarantees for GIFA 12.1 Conjecture 5.3.9: sample complexity analysis for Gap-GIFA

We consider now Gap-GIFA, where the following rules are fixed (see Table 5.1)

and the stopping rule is

Similarly to the proof for LUCB-GIFA in Réda, Kaufmann, and Delahaye-Duriez (2021), the objective is first to upper bound the stopping quantity

Lemma 12.1.1. Gap-independent upper bound on Bb(t) (t). For any t > 0, for any gap index satisfying Definition 5.2.1,

Proof. Using successively Lemma 5.2.8, the triangle inequality (when considering paired gap indices), and Corollary 1 from Gabillon, Ghavamzadeh, and Lazaric ( 2012)

Lemma 12.1.2. Upper bound on ( Bb(t) (t)) t>0 (Gap-GIFA). For any round t > 0, on event

, where a t ∈ arg max

Proof. Lemma 12.1.1 implies that Bb(t) (t) ≤ 2W at (t). It means that we only have to prove that in all cases,

Also note that, by triangle inequality, for any t > 0, for any pair of arms

Then, let us consider four cases :

and Bb(t

Moreover, there exists c ∈ (J(t)) c ∩ S ⋆ N ; otherwise, for any c ∈ J(t), c ∈ S ⋆ N , and since

, hence there is a contradiction. Then, using successively event E twice, the definition of c(t), event E again and the fact that c

and then we conclude similarly to (ii)

N and c(t) ∈ S ⋆ N . This is the part I haven't been able to show so far. I assume from now on that the lemma is correct. My conjecture is that, in this specific case

From the definitions of r k,m and r max , the following upper bounds can be easily checked.

Lemma 13.3.2. For any arm k ∈

Using the fact that CPE only halves the gap proxies of arms that are not eliminated, we can write down the value of the gap proxies for these arms

Using the important relationship between gap proxies and the confidence width as established in Lemma 8.4.7, we can further show that