Louis Arbez
email: louis.arbez@u-bourgogne.fr

Aurélien Royer

Jean-François Therrien

Sophie Montuire

Another mention of Meadow Vole (Microtus pennsylvanicus) found in pellets of Snowy Owl (Bubo scandiacus) in northern Ungava Peninsula, Canada

Keywords: Ungava Peninsula, Meadow Vole, Microtus pennsylvanicus, Ungava Collared Lemming, Dicrostonyx hudsonius, Péninsule d'Ungava, Microtus pennsylvanicus, Dicrostonyx hudsonius

The examination of raptor pellets can be used to evaluate Arctic biodiversity. We found the remains of Meadow Vole (Microtus pennsylvanicus) and Ungava Collared Lemming (Dicrostonyx hudsonius) in pellets from Snowy Owl (Bubo scandiacus geometric morphometric analysis of molar shape. The results of our study combined with historical data suggest that most commonly used range maps of Meadow Vole should be revised.

The northern part of the Ungava Peninsula, Canada, is remarkable for its low rodent diversity, with some distribution maps [START_REF] Hall | The Mammals of North America[END_REF][START_REF] Tamarin | Biology of New World Microtus[END_REF][START_REF] Shenbrot | Atlas of the Geographic Distribution of the Arvicoline Rodents of the Bulgaria[END_REF][START_REF] Wilson | Handbook of the Mammals of the World[END_REF] indicating the presence of only the endemic Ungava Collared Lemming (Dicrostonyx hudsonius). This low diversity is peculiar at this latitude, where additional rodent species are generally documented (e.g., North America: [START_REF] Hall | The Mammals of North America[END_REF] and [START_REF] Feldhamer | Wild Mammals of North America: Biology, Management, and Conservation. Edition[END_REF]; Eurasia: [START_REF] Ognev | R: a language environment for statistical computing[END_REF]). However, a few records of another small rodent, Meadow Vole (Microtus pennsylvanicus), from the northern end of the Ungava Peninsula have not been included in general range maps by some (e.g., [START_REF] Hall | The Mammals of North America[END_REF][START_REF] Feldhamer | Wild Mammals of North America: Biology, Management, and Conservation. Edition[END_REF][START_REF] Shenbrot | Atlas of the Geographic Distribution of the Arvicoline Rodents of the Bulgaria[END_REF][START_REF] Cassola | IUCN Red List of Threatened Species 2016: e[END_REF][START_REF] Wilson | Handbook of the Mammals of the World[END_REF]. Three records are in the Canadian Museum of Nature (CMN): two juvenile males stored in alcohol collected before 1934 from Salliut (formerly Sugluk) on the southwest side of Hudson Strait (CMNMA 12572 and 12573), and a male study skin (CMNMA 31130) collected from map for the species covers the entire peninsula and the 1954 record is mentioned in Desrosier et al. (2002) and compiled in the CMN database, as well as in the Global Biodiversity Information Facility database [START_REF] Khidas | Canadian Museum of Nature mammal collection[END_REF]. Additional specimens were collected in 2013 by [START_REF] Robillard | Étude des mouvements des neiges au Rapport d'activités au Service Environnement Mine Xstrata Nickel[END_REF][START_REF] Robillard | Variability in stable isotopes of snowy owl feathers and contribution of marine resources to their winter diet[END_REF] from Deception Bay, 60 km east of Salliut Inlet.

Vole were snap-trapped by [START_REF] Fortin | Aire de répartition du campagnol-lemming boréal au Québec: mentions les plus nordiques[END_REF] Synaptomys borealis) in the northern portion of the Ungava Peninsula near the Raglan Mine. Despite these previous records, the northern part of the Ungava Peninsula has not generally been included in widely used distribution maps of Meadow Vole (e.g., [START_REF] Hall | The Mammals of North America[END_REF][START_REF] Feldhamer | Wild Mammals of North America: Biology, Management, and Conservation. Edition[END_REF][START_REF] Shenbrot | Atlas of the Geographic Distribution of the Arvicoline Rodents of the Bulgaria[END_REF][START_REF] Cassola | IUCN Red List of Threatened Species 2016: e[END_REF][START_REF] Wilson | Handbook of the Mammals of the World[END_REF] We visually examined the teeth of small mammals collected from regurgitated pellets from Snowy Owl (Bubo scandiacus) nests near Salluit, northern Quebec, Canada (Figure 1), in July 2013 (see [START_REF] Robillard | Étude des mouvements des neiges au Rapport d'activités au Service Environnement Mine Xstrata Nickel[END_REF][START_REF] Robillard | Variability in stable isotopes of snowy owl feathers and contribution of marine resources to their winter diet[END_REF] for details) and used geometric morspecies. Unlike traditional morphometrics, landmarkbased geometric morphometric analyses use generalized procrustes analysis (GPA), which removes the shape parameters remain, with a minimum loss of information [START_REF] Adams | Geomorph: software for geometric morphometric analyses. R package version[END_REF]).

The regurgitated pellets of undigested material from birds of prey are commonly used to assess the diversity of small to medium sized animals around the world (e.g., [START_REF] Chaline | Les Proies des Rapaces[END_REF][START_REF] Ba | Les micromammifères du Djoudj (Delta du Sénégal) par l'analyse du régime alimentaire de la chouette effraie, Tyto alba[END_REF][START_REF] Meek | The use of Barn Owl (Tyto alba) pellets to assess population change in small mammals[END_REF][START_REF] Mcdonald | Utility of owl pellets for monitoring threatened mammal communities: an Australian case study[END_REF][START_REF] Heisler | Owl ping for broad-scale studies of small mammal communities[END_REF][START_REF] Linchamps | Neotaphonomic study of two Tyto alba assemblages from Botswana: palaeoecological implications[END_REF]. Cranial and dental remains level, hence determining prey diversity in the environment of a predator [START_REF] Chaline | Les Proies des Rapaces[END_REF][START_REF] Andrews | Owls, Caves, and Fossils: Predation, Preservation and Accumulation of Small Mammal Bones in Caves, with an Analysis of the Pleistocene Cave Faunas from Westbury-Sub-Mendip[END_REF]). The circumpolar Snowy Owl is a key predator in the Arctic tundra ecosystem. Its summer diet includes a diversity of small to medium sized animals, such as birds and hares, but generally consists mainly of rodents (e.g., [START_REF] Watson | The behaviour, breeding, and food ecology of the snowy owl Nyctea scandiaca[END_REF][START_REF] Gilg | Functional and numerical responses of four lemming predators in high arctic Greenland[END_REF][START_REF] Therrien | Écologie de la reproduction du harfang des neiges dans l'Arctique canadien[END_REF][START_REF] Robillard | Variability in stable isotopes of snowy owl feathers and contribution of marine resources to their winter diet[END_REF][START_REF] Royer | A taphonomic investigation of small vertebrate accumulations produced by the snowy owl (Bubo scandi acus) and its implications for fossil studies. Palaeogeography[END_REF].

In July 2013, 12 Snowy Owls were captured near Salluit, Ungava Peninsula (Figure 1) and equipped with global positioning system tracking transmitters [START_REF] Robillard | Étude des mouvements des neiges au Rapport d'activités au Service Environnement Mine Xstrata Nickel[END_REF]. The transmitters were pro- according to [START_REF] Hall | The Mammals of North America[END_REF]. The black square (in a) indicates the Ragalan Mine sector from which all pellets were obtained. Bubo scandiacus) that captured Meadow Voles are shown as ovals in b. The areas where [START_REF] Robillard | Variability in stable isotopes of snowy owl feathers and contribution of marine resources to their winter diet[END_REF] collected blood samples are within the owl home ranges. The dotted line indicates the erroneous northern limit of Meadow Vole [START_REF] Hall | The Mammals of North America[END_REF]). Map created using the "rnaturalearth" package v.0.2.0 in R (R Core Team 2020). grammed to communicate position every 4.5 days. Summer tracking showed that owls' movements were limited to their nesting areas. Regurgitated pellets-Snowy Owl nests a few days after the owls were captured and released (Table 1). Each pellet was dried and analysed to assess prey items from undigested materials (bones and teeth) at the Biogéosciences laboratory in Dijon, France.

Based on current distribution maps (e.g., [START_REF] Hall | The Mammals of North America[END_REF][START_REF] Feldhamer | Wild Mammals of North America: Biology, Management, and Conservation. Edition[END_REF] [START_REF] Desrosiers | Atlas des Micromammifères du Québec[END_REF].

lets based on the morphology of the lower and upper molars following [START_REF] Guilday | Dental variation in Microtus xanthognathus, M. chrotorrhinus, and M. pennsylvanicus (Rodentia, Mammalia)[END_REF], [START_REF] Semken | Key to Arvicoline ("Microtine" rodents) and Arvicoline-like lower cene archaeological and palaeontological sites in eastern North America[END_REF], [START_REF] Lupien | Recueil Photographique des Caractéristiques Morphologiques Servant à du Québec[END_REF], and [START_REF] Fauteux | An illustrated key to the mandibles of small mammals of eastern Canada[END_REF]. Ungava Collared Lemming molars lack cementum in the re-entrant angles, unlike the four other genera (see Meadow Vole and Ungava Collared Lem-triangles on the occlusal surface, in contrast with Microtus ure 2), P. ungava angles, M. gapperi with four closed triangles on the occlusal surface, and Synaptomys spp. with only two closed triangles on the internal side. The most reliable criterion for separating the Microtus species is the presence of an small "button-shaped" element at the upper molar for Meadow Vole, which is always absent in M. chrotorrhinus (Tamarin Semken and Wallace 2002). The occlusal pattern of the third upper molar can also be a distinguishing feature, as M. chrotorrhinus always has a complex posterior loop, which is rare in Meadow Vole [START_REF] Guilday | Dental variation in Microtus xanthognathus, M. chrotorrhinus, and M. pennsylvanicus (Rodentia, Mammalia)[END_REF][START_REF] Semken | Key to Arvicoline ("Microtine" rodents) and Arvicoline-like lower cene archaeological and palaeontological sites in eastern North America[END_REF]. One last criterion is the number of closed triangles on the lower second molar: M. chrotorrhinus has two while Meadow Vole has four.

geometric morphometric analysis was used on the et al. (2018) and [START_REF] Royer | A taphonomic investigation of small vertebrate accumulations produced by the snowy owl (Bubo scandi acus) and its implications for fossil studies. Palaeogeography[END_REF]. All teeth (MP-E 65 mm f/2.8 on a Canon EOS6D mark II camera, Canon, Inc., Tokyo, Japan) and saved in jpg format at a resolution of 72 dots per inch. Outlines were then extracted from pictures using ImageJ v.1.52a (free software with a Public Domain License available on GitHub; [START_REF] Schneider | NIH Image to ImageJ: 25 years of image analysis[END_REF]. Two-dimensional *MNI = minimum no. individuals contained in the pellet, calculated for each pellet based on counts of the maxillary and left or right mandible for each species. ing the orientation of the molar by manually marking four landmarks (white circles in Figure 3a) enabling the automatic detection of 14 additional landmarks and loop tips (Navarro et al. 2018). GPA and principal component analyses (PCA) were performed in R version 4.0.3 (R Core Team 2020) using the morph" package, 3.2.0 [START_REF] Adams | Geomorph: software for geometric morphometric analyses. R package version[END_REF]. Tooth length was estimated the Euclidean distance between landmarks 1 and 13 (Figure 3a,b).

The minimum number of individual (MNI) rodents was calculated for each pellet based on the most numerous skeletal component (i.e., left and right mandible or teeth). For example, a pellet with MNI of 5.

Nine of the 10 sets of owl pellets contained small vertebrate that did not show a high degree of digestion and which teeth were found (Figure to the species level, including 65 Ungava Collared Lemmings (86.6% of the MNI) and 10 voles (Microtus spp.; 13.3% the MNI; Table 1). All vole second upper molars possessed a supplementary "button shaped" element at the anterior part, no third upper molar exhibited a complex posterior loop morphology, and all second lower molars had four closed triangles, indicating that they were all Meadow Voles. These two species were the only rodent remains in the owl pellets and no juvenile lemmings or voles were found.

PCA (Figure 3c) showed that the morphological distribution of the specimens from the owl pellets was not uniform (PC 1: 40.3%; PC 2: 12.6%), but fell into of two species. These two distinct groups be iden-Collared Lemming on the positive side and the other shown by tooth length: Ungava Collared Lemmings have longer teeth than Meadow Voles (Figure 3b). Variability of Vole tooth (range 2.4-3.2 is commonly found among adults in other populations (e.g., Wallace 2006).

The distribution range maps for North American rodents generated by [START_REF] Hall | The Mammals of North America[END_REF] are widely used in many studies (e.g., [START_REF] Barnosky | Evolution of dental traits since latest Pleistocene in meadow voles (Microtus pennsylvanicus) from Virginia[END_REF][START_REF] Feldhamer | Wild Mammals of North America: Biology, Management, and Conservation. Edition[END_REF][START_REF] Spaeth | Morphological convergence and coexistence in three sympatric North American species of Microtus (Rodentia: Arvicolinae)[END_REF][START_REF] Jackson | A precarious for distinctive peripheral populations of meadow voles (Microtus pennsylvanicus)[END_REF]. According to [START_REF] Hall | The Mammals of North America[END_REF], Ungava Collared Lemming is the only lemming species inhabiting the Ungava Peninsula. Meadow Vole is widespread across North America; it occupies humid grassland, marsh, and riparian areas [START_REF] Tamarin | Biology of New World Microtus[END_REF] and is one of the most common Microtus species in subarctic taiga and tundra areas in Alaska and northern Canada [START_REF] Tamarin | Biology of New World Microtus[END_REF]. Meadow Vole is also widespread in the south and centre of the Labrador Peninsula and its distribution is thought to be limited northward by the Ungava trough [START_REF] Tamarin | Biology of New World Microtus[END_REF]. However, few occurrences of this species have been reported beyond this limit: two specimens in the Canadian Museum of Nature collected before 1934 (CMNMA 12572 and 12573), another individual collected near Sal-mens snap-trapped by [START_REF] Fortin | Aire de répartition du campagnol-lemming boréal au Québec: mentions les plus nordiques[END_REF]. [START_REF] Robillard | Étude des mouvements des neiges au Rapport d'activités au Service Environnement Mine Xstrata Nickel[END_REF] also mentioned the presence of carcasses of Meadow Voles in Snowy Owl nests. We analysed the owl pellets collected in such nests and report once again the of Meadow Vole in the region. Several individuals (at 10 of various sizes) were detected in Snowy Owl pellets, suggesting that the species is part of the owl's food web, although nearly 90% of the rodent individuals in the owl pellets were Ungava Collared Lemmings.

Will the abundance of Meadow Voles in the Ungava Peninsula increase with climate change? The species' presence has been known for some time (two specimens were found in 1934) and more specimens have been found since 2013, although few animals have been collected. In both our study and one by Fortin and Carron (2015), Meadow Voles represented 10% of the iden-with southern part of Ungava Peninsula getting dryer because of a decrease in precipitation and an in summer temperatures [START_REF] Furgal | Climate change and health in Nunavik and Labrador: lessons from Inuit Knowledge[END_REF][START_REF] Barrette | Nunavik and Nunatsiavut regional climate information update[END_REF]. These changes will most likely impact marshes and meadows, habitats sought preferentially by Meadow Voles, and could disadvantage dynamics of the Vole-Ungava Collared Lemming community in the area.

In conclusion, our results suggest that current widely used range maps for rodents should always be interpreted with caution and the one for Meadow Vole should be revised to include the northern tip of the Labrador Peninsula, namely the Ungava Peninsula.

Figure 1 .

 1 Figure 1. Distribution of Ungava Collared Lemming (Dicrostonyx hudsonius) and Meadow Vole (Microtus pennsylvanicus) according to[START_REF] Hall | The Mammals of North America[END_REF]. The black square (in a) indicates the Ragalan Mine sector from which all pellets were obtained.Bubo scandiacus) that captured Meadow Voles are shown as ovals in b. The areas where[START_REF] Robillard | Variability in stable isotopes of snowy owl feathers and contribution of marine resources to their winter diet[END_REF] collected blood samples are within the owl home ranges. The dotted line indicates the erroneous northern limit of Meadow Vole[START_REF] Hall | The Mammals of North America[END_REF]). Map created using the "rnaturalearth" package v.0.2.0 in R (R Core Team 2020).

Figure 2 .

 2 Figure 2. Characteristics of rodent teeth collected from Snowy Owl (Bubo scandiacus) pellets from Ungava Peninsula, Quebec. a and b. First lower molar of Meadow Vole (Microtus pennsylvanicus). c. First lower molar of Ungava Collared Lemming (Dicrostonyx hudsonius). d and e. Second upper molar of Meadow Vole. f. Second upper molar of Ungava Collared Lemming. g and h. Third upper molar of Meadow Vole. i. Third upper molar of Ungava Collared Lemming. Black arrows indicate the diagnostic supplementary "button shape" element in second upper molar of Meadow Vole, and the simplicity of the loop of the third upper molar of Meadow Vole. Scale bar: 1 mm. Photos: Louis Arbez.

Figure 3 .

 3 Figure 3. a. Landmarks for lemming (Dicrostonyx spp.) and vole (Microtus indicate manually applied black circles automatically detected landmarks, line shows tooth length. b. Boxplot of m1 length of Meadow Vole (Microtus pennsylvanicus) and Ungava Collared Lemming (Dicrostonyx hudsonius). c. Principal component analysis (PC 1 and PC 2 planes) of Meadow Vole (Microtus pennsylvanicus) and Ungava Collared Lemming (Dicrostonyx hudsonius) m1, including percentage of shape variation explained by each PC.-0.05 0.00 0.05

  ; Shenbrot and Krasnov 2005; Wilson et al. Northern Bog Lemming, and Eastern Heather Vole (Phenacomys ungava), with Ungava Collared Lemming the only species in the northern part of the peninsula. Rock Vole (Microtus chrotorrhinus) is mentioned at the very southern limits of the Ungava Peninsula by

	entire Ungava Peninsula: Ungava Collared Lemming,
	Meadow Vole, Southern Red-backed Vole (Myo-
	des gapperi),

Table 1 .

 1 Counts of Ungava Collared Lemming (Dicrostonyx hudsonius) and Meadow Vole (Microtus pennsylvanicus) remains found in 10 sets of Snowy Owl (Bubo scandiacus) pellets from Ungava Peninsula, Quebec, Canada.

	Owl no.	Collection date	Rodent species	MNI*	No. left mandibles	No. right mandibles	No. skulls
	2	8 July 2013	Dicrostonyx hudsonius	3	3	2	3
	3	9 July 2013	D. hudsonius Microtus pennsylvanicus	17 4	14 4	17 0	6 2
	4	4 July 2013	D. hudsonius M. pennsylvanicus	12 2	12 2	12 1	9 0
	5	8 July 2013	D. hudsonius	1	1	1	0
	6	14 July 2013	D. hudsonius	14	10	14	5
	8	13 July 2013	D. hudsonius M. pennsylvanicus	8 2	8 1	7 2	6 1
	9	-	D. hudsonius M. pennsylvanicus	5 2	5 1	4 2	2 2
	10	13 July 2013	D. hudsonius	2	2	2	2
	11	July2013	D. hudsonius	3	3	1	2
	Total			75
	Dicrostonyx		65 (86.7%)
	Microtus		10 (13.3%)

Acknowledgements

We thank reviewer Dominique Fauteux, William Halliday (Associate Editor), and Dwayne Lepitzki (Editor-in-Chief) for comments that improved our manuscript.

Author Contributions

Conceptualization: L.A., J.-F.T., A.R., and S.M.; Methodology: L.A., J.-F.T., A.R., and S.M.; Formal Analysis: L.A., J.-F.T., A.R., and S.M.; Data Curation: L.A., J.-F.T., A.R., and S.M.; Writing -Original Draft: L.A., J.-F.T., A.R., and S.M.; Visualization: L.A., J.-F.T., A.R., and S.M.


Jean-Baptiste Gaya
email: jbgaya@fb.com

Laure Soulier

Ludovic Denoyer

LEARNING A SUBSPACE OF POLICIES FOR ONLINE ADAPTATION IN REINFORCEMENT LEARNING

Deep Reinforcement Learning (RL) is mainly studied in a setting where the training and the testing environments are similar. But in many practical applications, these environments may differ. For instance, in control systems, the robot(s) on which a policy is learned might differ from the robot(s) on which a policy will run. It can be caused by different internal factors (e.g., calibration issues, system attrition, defective modules) or also by external changes (e.g., weather conditions). There is a need to develop RL methods that generalize well to variations of the training conditions. In this article, we consider the simplest yet hard to tackle generalization setting where the test environment is unknown at train time, forcing the agent to adapt to the system's new dynamics. This online adaptation process can be computationally expensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there is just a single train environment. To do so, we propose an approach where we learn a subspace of policies within the parameter space. This subspace contains an infinite number of policies that are trained to solve the training environment while having different parameter values. As a consequence, two policies in that subspace process information differently and exhibit different behaviors when facing variations of the train environment. Our experiments 1 carried out over a large variety of benchmarks compare our approach with baselines, including diversity-based methods. In comparison, our approach is simple to tune, does not need any extra component (e.g., discriminator) and learns policies able to gather a high reward on unseen environments.

INTRODUCTION

In recent years, Deep Reinforcement Learning (RL) has succeeded at solving complex tasks, from defeating humans in board games [START_REF] Silver | Mastering chess and shogi by self-play with a general reinforcement learning algorithm[END_REF] to complex control problems [START_REF] Peng | Sim-to-Real Transfer of Robotic Control with Dynamics Randomization[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. It relies on different learning algorithms (e.g., A2C - [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF], PPO - [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]). These methods aim at discovering a policy that maximizes the expected (discounted) cumulative reward received by an agent given a particular environment. If existing techniques work quite well in the classical setting, considering that the environment at train time and the environment at test time are similar is unrealistic in many practical applications. As an example, when learning to drive a car, a student learns to drive using a particular car, and under specific weather conditions. But at test time, we expect the driver to be able to generalize to any new car, new roads, and new weather conditions. It is critical to consider the generalization issue where one of the challenges is to learn a policy that generalizes and adapts itself to unseen environments.

Different techniques have been proposed in the literature (Section 6) to automatically adapt the learned policy to the test environment. In the very large majority of works, the model has access to multiple training environments (meta-RL setting). Therefore, the training algorithm can identify which variations (or invariants) may occur at test time and how to adapt quickly to similar variations. But this setting may still be unrealistic for concrete applications: for instance, it supposes that the student will learn to drive on multiple cars before getting their driving license.

(a) The figure represents the parameter space. The red (resp. blue) region is the space of good policies over the training (resp. testing) environment. A single learned policy (red point) may be inefficient for the test environment and has to be adapted (e.g., fine-tuning) to become good at test-time (blue point). Instead of learning a single policy, we learn a convex sub-space (the pentagon) delimited by anchor policies (red stars) that aims at capturing a large set of good policies. Then the adaptation is just made by sampling policies in this subspace, keeping the best one (blue star). (b) Qualitative example of k-shot adaptation on a modified Ant environment (20% of observations masked). 5 policies (i.e 5 values of z) are tested on one episode. In this case, for z = 0., the Ant is able to adapt to this new environment. More example of LoP trajectories in Figures 5 and6. See https://sites.google. com/view/subspace-of-policies/home for videos of the learned behaviors. In this paper, we address a simpler yet harder to tackle generalization setting in which the learning algorithm is trained over one single environment and has to perform well on test environments; preventing us from using meta-RL approaches. A natural way to attack this setting is to start by learning a single policy using any RL algorithm, and to fine-tune this training policy at test time, over the test environment (See red/blue points in Figure 1a), but this process may be costly in terms of environment interactions.

Very recently, the idea of learning a set of diverse yet effective policies (Kumar et al., 2020b;[START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF] has emerged as a way to deal with this adaptation setting. The intuition is that, if instead of learning one single policy, one learns a set of 'diverse' policies, then there is a chance that at least one of these policies will perform well over a new dynamics. The adaptation in that case just consists in selecting the best policy in that set by evaluating each policy over few episodes (K-shot adaptation). But the way this set of policies is built and the notion of diversity proposed in these methods have a few drawbacks: these models increase diversity by using an additional intrinsic reward which encourages the different policies to generate different distributions of states. This objective potentially favors the learning of policies that are sub-optimal at train time. Moreover, these approaches make use of an additional component in the policy architecture (e.g., a discriminator) that may be difficult to tune, particularly considering that, at train time, we do not have access to any test environment and thus cannot rely on validation techniques to tune the extra architecture.

Inspired by recent research on mode connectivity [START_REF] Benton | Loss surface simplexes for mode connecting volumes and fast ensembling[END_REF][START_REF] Kuditipudi | Explaining landscape connectivity of low-cost solutions for multilayer nets[END_REF] and by [START_REF] Wortsman | Learning neural network subspaces[END_REF] which aims to learn a subspace of models in the supervised learning setting, we propose to learn a subspace of policies in the parameter space as a solution to the online adaptation in the RL setting (see Figure 1a). Each particular point in this subspace corresponds to specific parameter values, and thus to a particular policy. This subspace is learned by adapting a classical RL algorithm (PPO and A2C in our case, see Section 3.3) such that an infinite continuum of policies is learned, each policy having different parameters. The policies thus capture and process information differently, and react differently to variations of the training environment (see Figure 1b). We validate our approach (Section 5) over a large set of reinforcement learning environments and compare it with other existing approaches. These experiments show that our method is competitive, achieves good results and does not require the use of any additional component of hyper-parameters tuning contrarily to baselines.

Reinforcement Learning: Let us define a state space S and an action space A. In the RL setting, one has access to a training Markov Decision Process (MDP) denoted M defined by a transition distribution P (s ′ |s, a) : S × A × S → R+, an initial state distribution P (i) (s) : S → R+ and a reward function r(s, a) : S × A → R.

A policy is defined as π θ (a|s) : S × A → R+, where θ denotes the parameters of the policy. A trajectory sampled by a policy π θ given a MDP M is denoted τ ∼ π θ (M). The objective of an RL algorithm is to find a policy that maximizes the expected cumulative (discounted) reward:

θ * = arg max θ E τ ∼π θ (M) [R(τ )]
(1) where R(τ ) is the discounted cumulative reward over trajectory τ .

Online adaptation: We consider the setting where the policy trained over M will be used over another MDP (denoted M) that shares the same state and action space as M, but with a different dynamics and/or initial state distribution and/or reward function2 . Importantly, M is unknown at train time, and cannot be used for model selection, making the tuning of hyper-parameters difficult.

Given a trained model, we consider the K-shot adaptation setting where the test phase is decomposed into two stages: a first phase in which the model adapts itself to the new test environment over K episodes, and a second phase in which the adapted model is used to collect the reward. We thus expect the first phase to be as short as possible (few episodes), corresponding to a fast adaptation to the new environment. Let us consider that a model π θ generates a sequence of trajectories τ1 , τ2 , ...., τ+∞ over M, the performance of such a model, is defined as:

P erf (π θ , M, K) = lim T →∞ 1 T T t=1 R(τ K+t ) (2)
which corresponds to the average performance of the policy π θ over M after K episodes used for adapting the policy. Note that we are interested in methods that adapt quickly to new a test environment and we will consider small values of K in our experiments. In the following, for sake of simplicity, K will refer to the number of policies evaluated during adaptation since each policy may be evaluated over more than a single episode when facing stochastic environments.

LEARNING SUBSPACES OF POLICIES

Motivation and Idea: To illustrate our idea, let us consider a toy example where the train environment contains states with correlated and redundant features, in such a way that multiple subsets of state features can be used to compute good actions to execute. Traditional RL algorithms will discover one policy π θ * that is optimal w.r.t the environment. This policy will typically use the state features in a particular way to decide the optimal action at each step. If some features become noisy (at test time) while, unluckily, π θ * particularly relies on these noisy features, the performance of the policy will drastically drop. Now, let us consider that, instead of learning just one optimal policy, we also learn a second optimal policy π θ * ′ , but enforcing θ * ′ to be different than θ * . This second policy may tend to make use of various features to compute actions. We thus obtain two policies instead of one, and we have more chances that at least one of these policies is efficient at test time. Identifying which of these two policies is the best for the test environment (i.e., adaptation) can simply be done by evaluating each policy over few episodes, keeping the best one. Our model is built on top of this intuition, extending this example to an infinite set of policies and to variable environment dynamics.

Inspired by [START_REF] Wortsman | Learning neural network subspaces[END_REF] proposing to learn a subspace of models for supervised learning, we study the approach of learning a subspace of policies in the parameter space, and the use of such a model for online adaptation in reinforcement learning. Studying the structure of the parameter space has seen a recent surge of interest through the mode connectivity concept [START_REF] Benton | Loss surface simplexes for mode connecting volumes and fast ensembling[END_REF][START_REF] Kuditipudi | Explaining landscape connectivity of low-cost solutions for multilayer nets[END_REF][START_REF] Wortsman | Learning neural network subspaces[END_REF] and obtain good results in generalization, but it has never been involved in the RL setting. As motivated in the previous paragraph, we expect that, given a variation of the training environment, having access to a subspace of policies that process information differently instead of a single policy will facilitate the adaptation. As a result, our method is very simple, does not need any extra hyper-parameter tuning and achieves good performance.

SUBSPACES OF POLICIES

Given Θ the space of all possible parameters, a subspace of policies is a subset Θ ⊂ Θ that defines a set of corresponding policies Π = {π θ } θ∈ Θ.

Since our objective is to learn such a subspace, we have to rely on a parametric definition of such a subspace and consider Θ as a simplex in Θ. Let us define N anchor parameter values θ1 , .... θN ∈ Θ. We define the Z-space as the set of possible weighted sum of the anchor parameters:

Z = z = (z 1 , ...z N ) ∈ [0, 1] N | z i = 1
. The subspace we aim to learn is defined by:

Θ = { N k=1 z k θk , ∀z ∈ Z} (3)
In other words, we aim to learn a convex hull of N vertices in Θ. Note that policies in this subspace can be obtained by sampling z ∼ p(z) uniformly over Z.

The advantages of this approach are: a) the number of parameters of the model can be controlled by choosing the number N of anchor parameters, b) since policies are sharing parameters (instead of learning a set of independent policies), we can expect that the learning will be sample efficient. Such a subspace is illustrated in Figure 1a through the "pentagon" (i.e., N = 5) in which angles correspond to the anchor parameters and the surface corresponds to all the policies in the built subspace.

K-shot adaptation: Given a subspace of policies Θ, different methods can be achieved to find the best policy over the test environment. For instance, it could be done by optimizing the distribution p(z) at test time. In this article, we use the same yet effective K-shot adaptation technique than Kumar et al. (2020b) and [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF]: we sample K episodes using different policies defined by different values of z that are uniformly spread over Z. In our example, it means that we evaluate policies uniformly distributed within the pentagon to identify a good test policy (blue star). Note that, when the environment is deterministic, only one episode per value of z needs to be executed to find the best policy, which leads to a very fast adaptation.

LEARNING ALGORITHM

Learning a subspace of policies can be done by considering the RL learning problem as maximizing:

L( Θ) = θ∈ Θ E τ ∼π θ [R(τ )]dθ (4)
Considering that Θ is a convex hull as defined in Equation 3, and using the uniform distribution p(z) over Z, the loss function of Equation 4 can be rewritten as:

L( θ1 , .... θN ) = E z∼p(z) [E τ ∼π θ [R(τ )]] with θ = N k=1 z k θk (5)
Maximizing such an objective function over θ1 , .... θN outputs a (uniform) distribution of policies trained to maximize the reward, all these policies sharing common parameters.

Avoiding subspace collapse:

One possible effect when optimizing L( θ1 , .... θN ) is to reach a solution where all θ k values are similar. In that case, all the policies would have the same parameters value, and will thus all achieve the same performance at test-time. Since we want to encourage the policies to process information differently, and following [START_REF] Wortsman | Learning neural network subspaces[END_REF], we encourage the anchor policies to have different parameters. This is implemented through the use of a regularization term denoted C( θ1 , .... θN ) that measures how much anchor policies are similar in the parameter space. This auxiliary loss is defined as a pairwise loss between pairs of anchor parameters:

C( θ1 , .... θN ) = i̸ =j cosine 2 (θ i , θ j ) (6)
The final optimization loss is then:

L( θ1 , .... θN ) = E z∼p(z) [E τ ∼π θ [R(τ )]] -β i̸ =j cosine 2 ( θi , θj ) with θ = N k=1 z k θk
where β is an hyper-parameter (see Section 5 for a discussion abot the tuning of this term) that weights the auxiliary term.

Initialize: θ1 , θ2 , ϕ (Critic), n batch size for k = 0, 1, 2... do Sample z 1 , ..., z n ∼ U [0,1] Define θ zi ← z i θ1 + (1 -z i ) θ2
Sample trajectories {τ i } n 1 using {π θz i } Update θ1 and θ2 to maximize:

1 n n i=1 L P P O (θ zi ) -β cosine 2 θ1 , θ2
Up. ϕ to minimize:

1 n n i=1 L M SE (ϕ, z i ) end

LINE OF POLICIES (LOP)

In the case of N = 2, the subspace of policies corresponds to a simple segment in the parameter space defined by θ1 and θ2 as extremities. θ1 and θ2 are combined through a scalar value z ∈ [0; 1]:

θ = z θ1 + (1 -z) θ2
(7) Computationally, learning a line of policies3 is similar to learning a single policy for which the number of parameters is doubled, making this particular case a good trade-off between expressivity and training speed. It corresponds to the following objective function:

L( θ1 , θ2 ) = E z∼U [0;1] E τ ∼π z θ1 +(1-z) θ2 [R(τ )] -cosine 2 ( θ1 , θ2 ) (8)
We provide in Algorithm 7 the adapted version of the clipped PPO algorithm [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] for learning a subspace of policies. In comparison to the classical approach, the batch of trajectories is first acquired by multiple policies sampled following p(z) (line 2-3). Then the PPO objective is optimized taking into account the policies used when sampling trajectories (line 4). At last, the critic is updated (line 5), taking as an input the z value so that it can make robust estimations of the expected reward for all the policies in the subspace. Adapting off-policy algorithms would be similar. Additional details are provided in appendix. Note that, for environments with discrete actions, we have made the same adaptation based on the A2C algorithm since A2C has less hyperparameters than PPO and is easier to tune, with similar results.

EXPERIMENTS

We perform experiments in 6 different environments. Implementations based on the SaLinA (Denoyer et al., 2021) library together with train and test environments will be released upon acceptance.

For each environment, we consider one train environment on which we trained the different methods, and multiple variations of the training environment for evaluation resulting in 50 test environments in total. The details of all the environment configurations and detailed performance are given in Appendix B. Note that the complete experiments correspond to hundred of trained policies, and dozens of thousands of policy evaluations. For simple control environments (i.e., CartPole, Pendulum and AcroBot), we introduce few variations of the physics constant at test-time, for instance by varying the mass of the cart, the length of the pole. For complex control environments (i.e., HalfCheetah and Ant using the BRAX library [START_REF] Freeman | Brax -a differentiable physics engine for large scale rigid body simulation[END_REF], we both use variations of the physics (e.g., gravity), variations of the agent shape (e.g., changing the size of the leg, or of the foot) and sensor alterations. At last, in MiniGrid and ProcGen we perform experiments where the agent is trained in one particular levels, but is evaluated in other levels (single levels on MiniGrid, and set of 10 levels in ProcGen). Note that ProcGen is a pixel-based environment where the architecture of the policy is much more complex than in control environments. Toy experiments on a simple Maze 2d are given in Appendix B.8 to show the nature of the policies learned by the different methods.

We compare our approach LoP 4 with different state-of-the-art methods: a) The Single approach is just a single policy learned on the train environment, and evaluated on the test ones. b) The DI-AYN+R(reward) method is an extension of DIAYN [START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF] where a set of discrete policies is learned using a weighted sum between the DIAYN reward and the task reward: R DIAY N +R (s, a) = r(s, a) + β log p(z|s) (9) Critically, this model requires to choose a discriminator architecture to compute log p(z|s) and modifies the train reward by defining an intrinsic reward that may drastically change the behavior of the policies at train time. c) At last, we also compare with the model proposed in [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF] denoted Lc (Latent-conditioned) that works only for continuous actions. This model is also based on a continuous z variable sampled uniformly at train time, but only uses an auxiliary loss without changing the reward. This auxiliary loss is defined through the joint learning of a density estimation model log P (z|s, a) where back-propagation is made over the action a. As in DIAYN+R, this model needs to carefully define a good neural network architecture for density estimation. Since Lc cannot be used with environment that have discrete actions, we have adapted DIAYN+R (called DIAYN+R Cont.) using a continuous z variable (instead of a discrete one) and a density estimation model log P (z|s) as in [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF]. Note that we do not compare to (Kumar et al., 2020a) for the exact same reason as the one identified in [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF]: SMERL assumes that the reward is known over the complete trajectories which results in unnatural adaptation of on-policy RL algorithms like PPO. Moreover, preliminary experiments with SMERL does not demonstrate any advantage against DIAYN+R correctly tuned. We also provide (see Table 4 in Appendix B.1) results where K independent policies are learned, the best one being selected over each test environment. This approach obtains lower performance than the proposed baseline and needs K more training samples making it unrealistic in most of the environments.

As network architectures, we use multi-layer perceptrons (MLP) with ReLU units for both the policy and the critic (detailed neural network architectures are described in Appendix). For DIAYN+R log P (z|s, ...) is also modeled by a MLP with a soft-max output. For Lc and DIAYN+R Cont., log P (z|s, ...) is modeled by a MLP that computes the mean of a Gaussian distribution with a fixed variance. For these baselines, z is concatenated with the environment observation as an input for the policy and the critic models.

To choose the hyper-parameters of the different methods, let us remind that test environments cannot be used at train time for doing hyper-parameters search and/or model selection which makes this setting particularly difficult. Therefore, we rely on the following procedure: a grid-search over hyper-parameters is made, learning a single policy over the train environment. The best value of the hyper-parameters is then selected as the one that provides the best policy at train time. These hyper-parameters are then used for all the different baselines. Concerning the β value, for LoP, we report test results for β = 1.0 while, for Lc and DIAYN+R, we use the best value of β on test environments. This corresponds to an optimistic evaluation of the baseline performances; aiming at showing that our method is much more efficient since it does not need such a beta-tuning (β = 1.0 giving good performance in the majority of cases). Said otherwise, we compare our model in the less favorable case where baselines have been unrealistically tuned.

For the adaptation step, each policy is evaluated over 10 episodes for stochastic environments or 1 single episode for deterministic environments. We repeat this procedure over 10 different training seeds, and report the reward over the different test environments together with standard deviation. All detailed results are available in Appendix.

ANALYSIS

We report the test performance of the models on different environments in Table 1. In all the environments, the adaptive models perform better than learning a single policy over the train environment which is not surprising. In most of the cases, LoP is able to achieve a better performance than other methods. For instance, on HalfCheetah where we evaluate the different methods over 16 variations of the train environments, LoP achieves an average reward of 10589 while Lc and DIAYN+R obtain respectively 9547 and 9680 (standard deviations are reported in Appendix B). Some examples of the discovered that behaviors in Ant and HalfCheetah5 for the different methods, and for different values of z are illustrated in Figures 1b, 5 and6. This outlines that learning models that are optimal on the train task reward, but with different parameter values, allows us to discover policies react differently to variations of the training environment. It seems to be a better approach than encouraging policies to have a different behaviors (i.e., generating different state distributions) at train time. Same conclusions can be drawn in most of the environments, including MiniGrid where LoP is able to explore large mazes while being trained only on small ones. On the ProcGen environment, where the observation is an image processed through a complex ConvNet architecture (See Appendix B.7), enforcing functional diversity (DIAYN+R) does not allow to learn good policies while the LoP model is able to better generalize to unseen levels. Note that the performance at train time is the same for all the different approaches reported (see the Table 3 for instance) but quickly decreases in DIAYN for larger values of β while it stays stable for LoP where the best results are obtained for β = 1.0.

Interestingly, in CartPole, DIAYN+R performs quite well. Indeed, when analyzing the learned policies, it seems to be a specific case where it is possible to obtain optimal policies that are diverse w.r.t the states they are sampling (by moving the cart more or less on the right/left while maintaining the pole vertical).

We have also performed experiments where test environments have the same dynamics as the training environment, but with defective sensors (i.e., some features at test time have a null value -see Appendix B.2 Table 7 on the Ant environment). The fact that LoP behaves also well confirms the effectiveness of our approach to different types of variations, including noisy features on which baselines methods were not applied in previous publications.

Sensitivity to hyper-parameters: One important characteristic of LoP is that it can be used with β = 1.0 and does not need to define any classifier architecture as opposed to DIAYN+R and Lc. Indeed, as shown in Figure 3 (left), the training performance of DIAYN drastically depends on a good tuning of β. Lc, which is less sensible, needs to use a correct classifier architecture as in DIAYN. LoP is simple to tune since the cosine term is usually easy to satisfy and our approach, at convergence, always reaches a 0 value on this term when β > 0.0. As illustrated in Appendix B.1 and B2, it is also interesting to note than, on the BRAX environments, the number of environment interactions needed to train LoP is similar than the one needed to train a single policy and LoP comes with a very small overhead in comparison to classical methods.

Online adaptation: One interesting property is the number of policies (and thus of episodes) to test over a new environment to get a good performance. For LoP and Lc, given a trained model, one can evaluate as many policies (i.e., different values of z) as desired. For DIAYN+R, testing more policies also means training more policies which is expensive and less flexible. Table 3 (right) provides the reward of the different methods when testing K policies on different HalfCheetah settings: as expected, the performance of DIAYN+R tends to decrease when K is large since the model has difficulties to learn too many diverse policies. For LoP and Lc, spending more episodes to evaluate more policies naturally leads to a better performance: these two models provide a better way to deal with the exploration-exploitation trade-off at test time. Again, please consider that Lc also needs to define an additional neural network architecture to model log P (z|s, a) while LoP does not, making our approach simpler.

Beyond a Line of Policies: While LoP is based on the learning of N = 2 anchor parameters, it is possible to combine more than two anchor parameters. We study two approaches combining N = 3 anchor parameters (that can be extended to N = 3): a) the first approach is a convex combination of policies (CoP) where z is sampled following a Dirichlet distribution. (b) The second approach is a Bézier combination (BoP) as explained in Appendix A.2. The results are presented in Table 3 (right) over multiple HalfCheetah environments. It can be seen that these two strategies are not so efficient.

LoP is thus a good trade-off between the number of parameters to train and the performance (Note that BoP and CoP need more samples to converge), at least given the particular neural network architectures we have used in this paper. We also performed an in-depth analysis of the evolution of the reward when K is increasing for LoP and CoP in Halfcheetah test environment (Figure 9 in Annex). While we expected CoP to outperform LoP when K is high, the best reward becomes stable when K=20 for both methods, and in most test environments, CoP is not able to reach the same best reward as LoP.

Analysis of the learned policies:

To better understand the nature of the policies discovered by the different approaches, we have made a qualitative study in which we analyze i) the robustness of the methods to corrupted observations, ii) the functional diversity induced by the different models, and iii) the specificity of the different learned policies to particular test environments. First, LoP is more robust to input feature corruption (see Table B.2 for the results, and Table 5 for the setting in Appendix) and we conjecture that it is because the diversity in the parameter space allows this model to learn policies that does not take into account the same input features equally. We also measure the functional diversity induced by the different models by training a posteriori a classifier that aims at recovering which policy (i.e which value of z) has generated particular trajectories (Exact protocol in Figure 8 in Appendix, with the training curves). On LoP with K = 5, such a classifier obtains a 82% accuracy at validation time showing that the 5 policies are quite diverse, but less than the DIAYN+R policies where the classifier reaches a 100% accuracy which is logical knowing the auxiliary loss introduced by DIAYN which enforces this type of diversity. It is interesting to note that with the trajectories generated in the test environments with LoP policies, the accuracy of the classifier is reaching 87 %: when LoP is facing new environments, it tends to generate more diverse policies. We think that it is due to the fact that, since the policies have different parameter values, they react differently to states that have not been encountered at train time. At last, Figure 4 (right) (and Figure 7 in appendix for K=10,20) illustrates which upon K = 5 policies is used for different test environments. It shows that both LoP and DIAYN+R use different policies over different test environments, showing that these methods are able to solve new environments by learning various policies and not a single but robust one. Examples of policies on a simple maze2d are given in Figure 4 (left) and Appendix which illustrate the diversity of the discovered policies.

RELATED WORK

Our contribution shares connections with different families of approaches. First of all, it focuses on the problem of online adaptation in Reinforcement Learning which has been studied under different terminologies: Multi-task Reinforcement Learning [START_REF] Wilson | Multi-task reinforcement learning: A hierarchical bayesian approach[END_REF][START_REF] Teh | Distral: Robust multitask reinforcement learning[END_REF], Transfer Learning [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF][START_REF] Lazaric | Transfer in reinforcement learning: a framework and a survey[END_REF] and Meta-Reinforcement Learning [START_REF] Finn | Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks[END_REF][START_REF] Hausman | Learning an embedding space for transferable robot skills[END_REF][START_REF] Humplik | Meta reinforcement learning as task inference[END_REF]. Many different methods have been proposed, but the best majority considers that the agent is trained over multiple environments such that it can identify variations (or invariant) at train time. For instance, [START_REF] Duan | Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning[END_REF] assume that the agent can sample multiple episodes over the same environments and methods like [START_REF] Kamienny | Learning adaptive exploration strategies in dynamic environments through informed policy regularization[END_REF][START_REF] Liu | Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices[END_REF] consider that the agent has access to a task identifier at train time.

More recently, diversity-based approaches have been adapted to focus on the setting where only one training environment is available. They share with our model the idea of learning multiple policies instead of a single one. For instance, DIAYN [START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF] learns a discrete set of policies that can be reused and fine-tuned over new environments. It has been adapted to online adaptation in (Kumar et al., 2020a) where the authors propose to combine the intrinsic diversity reward together with the training task reward. This trade-off is obtained through a threshold-based method (instead of a simple weighted sum) with good results. But this method suffers from a major drawback identified in [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF]: it necessitates to sample complete episodes at each epoch which is painful and not adapted to all the RL learning algorithms. [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF] also proposed an alternative based on learning a continuous set of policies instead of a discrete one without using any intrinsic reward.

The method we propose is highly connected to recent researches on mode connectivity with neural networks. Mode connectivity is a set of approaches and analyses that focus on the shape of the parameter space. It has been used as a tool to study generalization in the supervised learning setting [START_REF] Garipov | Loss surfaces, mode connectivity, and fast ensembling of dnns[END_REF], but also as a way to propose new algorithms in different settings [START_REF] Mirzadeh | Linear mode connectivity in multitask and continual learning[END_REF]. Obviously, the work that is the most connected to our approach is the model proposed in [START_REF] Wortsman | Learning neural network subspaces[END_REF]) that provides a way to learn a subspace of models in the supervised learning setting. Our contribution adapts this approach to RL for learning policies in a completely different setting which is online adaptation. At last, our work is sharing similarities with robust RL which aims at discovering policies robust to variations of the training environment [START_REF] Oikarinen | Robust deep reinforcement learning through adversarial loss[END_REF][START_REF] Zhang | Robust reinforcement learning on state observations with learned optimal adversary[END_REF]. The main difference is that robust RL techniques learn policies efficient 'in the worst case' and are not focused on the online adaptation to test environments (the objective is usually to learn a single policy efficient on different variations while we are learning multiple policies, just selecting one at test time).

CONCLUSION AND PERSPECTIVES

We investigate the idea of learning a subspace of policies in the reinforcement learning setting, and describe how this approach can be used for online adaptation. While simple, our method allows to obtain policies that are robust to variations of the training environments. Contrarily to other techniques, LoP does not need any particular tuning or definition of additional architectures to handle diversity, which is a critical aspect in the online adaptation setting where hyper-parameters tuning is impossible or at least very difficult. Future work includes the extension of this family of approaches in the continual reinforcement learning setting, the deeper understanding of the the built subspace and the investigation of different auxiliary losses to better control the shape of such a subspace.

A IMPLEMENTATION DETAILS

In this section, we provide details about the implementations of the baselines and our models. Appendix B provides details and additional results for each environment we used.

A.1 LOP-PPO

We detail the losses used in algorithm 7. First, we recall the clipped-PPO surrogate objective described in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. For a given trajectory τ = {(s t , a t , r t )} T 0 collected by the policy π θ old , and denoting ρ t (θ) = π θ (at|st) π θ old , the goal is to maximize:

L P P O (θ) := 1 T T t=0 min [ρ t (θ) A π θ old (s t , a t ) , clip (ρ t (θ) , 1 + ϵ, 1 -ϵ) A π θold (s t , a t )] (10)
Where function A π θ old is computed thanks to a value function V ϕ by using Generalized Advantage Estimation [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF]). This function is simultaneously updated by regression on mean-squared error over the rewards-to-go R t . In our case, this function not only takes s t as an input, but also the value z:

L M SE (ϕ, z) := 1 T T t=0 V ϕ (s t , z) -R t 2 (11)
In HalfCheetah and Ant experiments, we sampled actions from a reparametrized Gaussian distribution using a squashing function [START_REF] Ward | Improving exploration in soft-actor-critic with normalizing flows policies[END_REF]), but we set the standard deviation fixed (so it is an hyper-parameter encouraging exploration, called action std in Tables 3 and6): the policy network only learns the mean of this distribution.

A.2 BOP AND COP

The only change between LoP and these models resides in the way we combine the N anchor policies. For CoP, it is just the generalization of LoP for N > 2 (see 3.1). BoP, makes use of a Bezier parametric curve that uses Bernstein polynomials (the anchor parameters being the control points). For N = 3, it is defined by:

Θ = (1 -z) 2 θ1 + 2 (1 -z) z θ2 + z 2 θ3 , ∀z ∈ [0, 1] (12)
Concerning the policies z evaluated at test time, BoP uses the same strategy as LoP by testing values that are uniformly distributed in [0; 1]. For CoP, we opted for sampling K policies using a Dirichlet distribution over [0, 1] 3 .

A.3 DIAYN+R AND LC

In order to find the best trade-off between maximizing environment rewards and intrinsic rewards in DIAYN+R algorithm, we add the hyper-parameter β :

R DIAY N +R (s, a) = r(s, a) + β • log p(z|s) (13)
As an alternative to DIAYN+R Osa et al. (2021) proposes an algorithm where the discriminator takes not only observations as an input but also the policy output, updating both discriminator q ϕ and policy π θ when back propagating the gradient. In this case, it is not necessary to add an intrinsic reward. While [START_REF] Osa | Discovering diverse solutions in deep reinforcement learning[END_REF] illustrate their methods with TD3 and SAC, we adapted it to PPO. The surrogate loss is given by:

L LC := L P P O + β • log q ϕ (z | s, π θ (.|s, z)) (14)
B EXPERIMENTS DETAILS AND ADDITIONAL RESULTS

B.1 HALFCHEETAH

Task originally coming from OpenAI Gym [START_REF] Brockman | Minimalistic gridworld environment for openai gym[END_REF]. Instead of using MuJoCo engine, we decided to use Brax [START_REF] Freeman | Brax -a differentiable physics engine for large scale rigid body simulation[END_REF] as it enables the possibility to acquire episodes on GPU. We use the vanilla environment for training.The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations. The base learning algorithm is PPO.

Test environments: we operated modifications similar as the ones proposed in [START_REF] Henderson | Benchmark environments for multitask learning in continuous domains[END_REF]. Morphological variations: we changed the radius and mass of specific body parts (torso, thig, shin, foot). Variations in physics: we changed the gravity and friction coefficients. Table 2: Modified HalfCheetah environments used for testing. Morphological modifications include a variation on the mass and the radius of a specific part of the body (torso, thighs, shins, or feet). We also modified the dynamics (gravity and friction). Environment names are exhaustive: Big refers to a increase of 25% of radius and mass, Small refers to a decrease of 25%. For example, "BigFoot" refers to an HalfCheetah agent where feet have been increased in mass and radius by 25%. For gravity and friction, we also tried an increase/decrease by 50% (respectively tagged "Huge" and "Tiny"). . . Figure 8: We trained small discriminators over a dataset (100,000 environment interactions) of trajectories obtained with the learned policies of LoP and DIAYN+R when K=5. For each environment, for each seed, we trained a single discriminator and averaged the results. While the discriminators trained on DIAYN+R reach 100% accuracy rapidly on both train and test environments, they learn slower for LoP, with a slight advantage for the test environment, validating the fact that the diversity induced by the cosine similarity on the weights is more visible in variations of the environment rather than the environment on which the model has been trained. We evaluated the discriminator on a validation dataset (also 100,000 environment interactions) resulting in 100% accuracy for DIAYN in both train and test environments. For LoP, we obtained 82% accuracy on the training environment, and 87% on the test environments. The discriminator architecture consists in a neural network of two hidden layers of size 16, taking the unprocessed states as an input and outputting the predicted policy used (like in DIAYN).

Hyper-parameter

Figure 9: Evolution of the best reward obtained with respect to K for LoP (N=2) and CoP (N=3) for each Halfcheetah test environment. We ran the K-shot evaluation for each K from K=1 to K=100 using the method described in Appendix A.2: we simply sample K random coefficients using the uniform distribution over [0, 1] for LoP and the Dirichlet distribution over [0, 1] 3 for CoP. Results are averaged over 10 run for each K, and over the 10 models we learned for each method.

B.2 ANT

Task originally coming from OpenAI Gym [START_REF] Brockman | Minimalistic gridworld environment for openai gym[END_REF]). Instead of using MuJoCo engine, we decided to use Brax [START_REF] Freeman | Brax -a differentiable physics engine for large scale rigid body simulation[END_REF]) as it enables the possibility to acquire episodes on GPU. We use the vanilla environment for training. The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations. The base learning algorithm is PPO.

Test environments: As for HalfCheetah, we operated variations in physics (gravity and friction coefficients). We also designed environments with a percentage of masked features to simulate defective sensors (They are sampled randomly and remain the same for each run).

B.5 PENDULUM

We use the openAI gym implementation of Pendulum as a training environment. The 3 test environments are provided by [START_REF] Packer | Assessing generalization in deep reinforcement learning[END_REF] where two different factor may vary: the mass and the length of the pendulum. We have considered 5 discrete actions between -1 and +1. We have used A2C as a learning algorithm. -28.2 ± 13.3 Average -52.7 -28.9 -47.1 -44.0

Table 16: Results over Pendulum, using 10 policies, and 10 episodes per policy at adaptation time.

Figure 1 :

 1 Figure 1: (Left) An illustration of the process of learning a subspace of policies. (Right) Comparison between PPO and our model in a test environment.

Figure 2 :

 2 Figure 2: The adaptation of the PPO Algorithm with the LoP model. The different with the standard PPO algorithm is that: a) trajectories are sampled using multiple policies θ zi b) The policy loss is augmented with the auxiliary loss, and c) The critic takes the values z i as an input.

Figure 4 :

 4 Figure 4: (left) Trajectories generated by K = 10 policies (one rows) on an unseen maze (objective is to go from left to right, see details in Appendix B.8) for DIAYN+R (left column with best β value) and LoP (right column with β = 1.0). It illustrates the diversity obtained with DIAYN+R and LoP. (right) Number of times (y-axis) each policy (x-axis) (over K = 5 tested policies) is chosen over the 16 test environments of the HalfCheetah setting for each of the 10 seeds. Blue is LoP and orange is DIAYN+R. Different policies are used for different test environments showing the interest of learning a subspace of policies. Note that in LoP, the anchor policies are rarely chosen. Results for K = 10 and K = 20 in Appendix (Figure 7).

Figure 5 :

 5 Figure 5: Qualitative example of LoP trajectories on HalfCheetah "BigShins" test environment (5shot setting). The best reward is obtained for z = 0.75.

Figure 6 :

 6 Figure 6: Extreme case: when torso radius and mass are increased by 50%. Only one policy is able to adapt without falling down (z = 0.5).

Figure 7 :

 7 Figure 7: Number of times (y-axis) each policy (x-axis) is chosen by k-shot evaluation over the 16 test environments of the HalfCheetah for each of the 10 seeds (one table per seed). In blue, LoP, in orange, DIAYN+R. Please note that the 10 same LoP models are used for K=5, K=10, K=20 which is not the case for DIAYN+R.

Figure 10 :

 10 Figure 10: Evolution of the cumulative reward during training on the generic Ant environment for LoP, DIAYN+R and Lc for different values of beta. On can see that DIAYN+R struggles to perform well on the train set for β = 1 and β = 10. Results are averaged over 10 seeds.

Figure 12 :

 12 Figure 12: Trajectories learned by LoP (β = 1.0). Rows are environments, columns are the K = 10 policies test during online adaptation. For each test environment, at least one policy is able to reach the goal

Table 1 :

 1 Average cumulated reward of the different models over multiple testing environments averaged over 10 training seeds (higher is better). For DIAYN and Lc, we report the results and tested 10 policies for LoP, Lc and DIAYN+R using 10 episodes per policy for stochastic environments, and 1 episode per policy on deterministic ones. Performance is evaluated using the deterministic policy. Standard deviation is reported for each single test environment in Appendix B. Performance of the models at train time that shows that for LoP β is not hurting train performance while it is DIAYN+R. Standard error deviation is reported in TableB.2 for each environment. We also report the performace at train time that shows that a too high value of β hurts DIAYN+R performance while is less critical in LoP. (right) Ablation study on the number of policies K used at test time on 3 HalfCheetah environment variations (see Appendix B.1 for further details and additional results) together with the performance of the BoP and CoP variants. Standard deviation is given in appendix, Table4.

			CartPole Acrobot Pendulum Minigrid Brax HalfCheetah Brax Ant ProcGen	toy maze
	Nb. Test Env.	6	6		3	6	16		15	3	4
	Type of actions	Discr.	Discr.	Discr.	Discr.	Cont.		Cont.	Discr.	Discr.
	Single Policy	143.4	-99.7	-52.7	0.169	7697		3338	11.09	-83.2
	LoP		149.9	-93.2	-28.9	0.447	10589		4031	16.38	-33.6
	DIAYN+R	168.1	-97.0	-47.1	0.248	9680		3759	11.45	-42.2
	DIAYN+R L2	156.1	-93.6	-44.0	0.443	-		-	-	-
	Lc		-	-		-	-	9547		4020	-	-
										SmallFeet TinyFriction BigGravity
		K =	5	10	20	Train Perf.		LoP	K=5	8283	10425	10464
	LoP	β = 0.1 3905 3991 4164 β = 1. 4035 4031 4174 β = 10 3998 4012 4145	7659 7630 7670		K=10 K=20 DIAYN+R K=5 K=10	8805 8794 7580 7580	10662 10734 9132 9132	10578 10807 8989 8989
	DIAYN	β = 0.1 3558 3833 3949 β = 1. 3451 3759 2878 β = 10 3356 3400 3109	7739 5388 4430		Lc	K=20 K=5 K=10 K=20	8255 8186 8186 8107	10003 9521 9661 9661	9766 9360 9488 9506
		β = 0.1 3909 4020 4150	7767		BoP (N=3)	K=5 K=10	6775 6660	7867 7840	7878 8026
	Lc	β = 1.	3820 3947 4126	7650			K=20	6996	7963	8015
		β = 10	3870 3945 4108	7710		CoP (N=3)	K=5 K=10	8996 9210	9468 9523	9287 9568
									K=20	9155	9979	9695
	Figure 3: (left)

  Table 2 precisely indicates the nature of the changes for each environment.

	Env name	Modifications
	BigFeet	Feet mass and radius ×1.25
	BigFriction	Friction coefficient ×1.25
	BigGravity	Gravity coefficient ×1.25
	BigShins	Shins mass and radius ×1.25
	BigThighs	Thighs mass and radius ×1.25
	BigTorso	Torso mass and radius ×1.25
	SmallFeet	Feet mass and radius ×0.75
	SmallFriction Friction coefficient ×0.75
	SmallGravity Gravity coefficient ×0.75
	SmallShins	Shins mass and radius ×0.75
	SmallThighs	Thighs mass and radius ×0.75
	SmallTorso	Torso mass and radius ×0.75
	HugeFriction Friction coefficient ×1.5
	HugeGravity	Gravity coefficient ×1.5
	TinyFriction	Friction coefficient ×0.5
	TinyGravity	Gravity coefficient ×0.5

Table 3 :

 3 Hyper-parameters for PPO over HalfCheetah

	Value

Table 4 :

 4 Mean and standard deviation of cumulative reward achieved on HalfCheetah test sets per

model (see Table

2

for environment details). Results are averaged over 10 training seeds (i.e., 10 models are trained with the same hyper-parameters and evaluated on the 16 test environments). K is the number of policies tested at adaptation time, using 1 episode per policy since this environment is deterministic. Ensembling with K = 5 models takes 5 times more iterations to converge and testing values of K > 5 is very costly in terms of GPU consumption.

Table 5 :

 5 Table 5 precisely indicates the nature of the changes for each environment DefectiveSensor 10% 10% of env obs set to 0 DefectiveSensor 15% 15% of env obs set to 0 DefectiveSensor 20% 20% of env obs set to 0 DefectiveSensor 25% 25% of env obs set to 0 DefectiveSensor 30% 30% of env obs set to 0 DefectiveSensor 35% 35% of env obs set to 0 Modified Ant environments used for testing.

	Env name	Modifications
	BigFriction	Friction coefficient ×1.25
	BigGravity	Gravity coefficient ×1.25
	SmallFriction	Friction coefficient ×0.75
	SmallGravity	Gravity coefficient ×0.75
	HugeFriction	Friction coefficient ×1.5
	HugeGravity	Gravity coefficient ×1.5
	TinyFriction	Friction coefficient ×0.5
	TinyGravity	Gravity coefficient ×0.5
	DefectiveSensor 5%	5% of env obs set to 0

Table 14 :

 14 Pendulum train and test environments

	Hyper-parameter			Value
	learning rate:			0.001
	n acquisition steps per epoch:		8
	n parallel environments:		32
	critic coefficient:			1.0
	entropy coefficient:			0.001
	discount factor:			0.99
	gae coefficient:			0.7
	gradient clipping:			2.0
	n neurons per layer:			16
	n layers:			2
	LoP
	β:			1.0
	DIAYN
	β:			1.0
	n neurons per layer discriminator:	16
	n layers discriminator:		2
	learning rate discriminator:		0.001
	Table 15: Hyper-parameters for A2C over Pendulum
	Single	LoP	DIAYN+R	DIAYN+R L 2
	Light

In the experimental study, one training environment is associated to multiple test environments to analyze the ability to adapt to different variations.

Other ways to control the shape of the subspace can be used and we investigate some of them in Section

 4 We consider the LoP-A2C and the LoP-PPO models for environments with respectively discrete and continuous actions. LoP-PPO could be also used in the discrete case but requires more hyper-parameter tuning.

Videos available at https://sites.google.com/view/subspace-of-policies/home

https://github.com/facebookresearch/salina/tree/main/salina_examples/ rl/subspace_of_policies

We have made several efforts to ensure that the results provided in the paper are fully reproducible. In Appendix, we provide a full list of all hyperparameters and extra information needed to reproduce our experiments. The source code is available on SaLinA repository such that everyone can reproduce the experiments 6 .

 [START_REF] Packer | Assessing generalization in deep reinforcement learning[END_REF]where two different factor may vary: the intertia factor and the lengh of the system. We have used A2C as a learning algorithm. Table 13: Results over Acrobot, using 10 policies, and 10 episodes per policy at adaptation time.

Environment

B.6 MINIGRID

We have use Gym Minigrid to perform experiments on mazes [START_REF] Brockman | Minimalistic gridworld environment for openai gym[END_REF]. We have used the MultiRoom-N2-S4 for training considering one single maze. At test time, we have tested on three different MultiRoom-N2-S4 environments composed of two rooms, but also on three MultiRoom-N4-S5 composed of four rooms. This allows us to evaluate the generalization power of the different methods to larger mazes. We have used A2C as a learning algorithm. We performed experiments on pixel-based environment with ProcGen. We used the FruitBot game for training considering 10 levels sampled uniformly at each episode. At test time, we selected 3 different environments, each of them composed of 10 uniformly sampled levels, sampled uniformly, and that were not seen at train time. We used the CNN architecture described in the ProcGen paper (IMPALA architecture [START_REF] Espeholt | Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures[END_REF])). For LoP, the two first blocks are fixed and only the parameters of the last block depend on the value of z. For DIAYN, the z value is provided as a one-hot vector stacked to the observation. As for Brax environment, we used PPO algorithm.

Hyper

Hyper

B.8 MAZE 2D WITH WALLS

To visualize the policies learned by the different methods, we just implemented a simple discrete maze (4 actions = up,down, left, right) where the objective is to go from the top-middle tile to the bottom-middle tile by moving through a corridor (size is 21 × 11). Reward is -1 at each step until goal is reached and the maximum number of steps is -100. The optimal policy in the training environment achieves -16. At test time, we generate walls in the corridor such that the agent has to avoid these walls to reach the goal. The observation space is a 5×5 square around the agent. Policies are learned by using PPO. Illustrations of the trajectories over the train and the 4 test environments are illustrated in Figures 11,12,13