Mathieu Chalvidal
email: mathieu_chalvid@brown.edu

Thomas Serre
email: thomas_serre@brown.edu

Rufin Vanrullen
email: rufin.vanrullen@cnrs.fr

Meta-Reinforcement Learning with Self-Modifying Networks

Deep Reinforcement Learning has demonstrated the potential of neural networks tuned with gradient descent for solving complex tasks in well-delimited environments. However, these neural systems are slow learners producing specialized agents with no mechanism to continue learning beyond their training curriculum. On the contrary, biological synaptic plasticity is persistent and manifold, and has been hypothesized to play a key role in executive functions such as working memory and cognitive flexibility, potentially supporting more efficient and generic learning abilities. Inspired by this, we propose to build networks with dynamic weights, able to continually perform self-reflexive modification as a function of their current synaptic state and action-reward feedback, rather than a fixed network configuration. The resulting model, MetODS (for Meta-Optimized Dynamical Synapses) is a broadly applicable meta-reinforcement learning system able to learn efficient and powerful control rules in the agent policy space. A single layer with dynamic synapses can perform one-shot learning, generalizes navigation principles to unseen environments and manifests a strong ability to learn adaptive motor policies.

Introduction

The algorithmic shift from hand-designed to learned features characterizing modern Deep Learning has been transformative for Reinforcement Learning (RL), allowing to solve complex problems ranging from video games [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] to multiplayer contests [START_REF] Jaderberg | Human-level performance in 3d multiplayer games with population-based reinforcement learning[END_REF] or motor control [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF][START_REF] Timothy P Lillicrap | Continuous control with deep reinforcement learning[END_REF]. Yet, "deep" RL has mostly produced specialized agents unable to cope with rapid contextual changes or tasks with novel or compositional structure [START_REF] Brenden | Building machines that learn and think like people[END_REF][START_REF] Lake | Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks[END_REF][START_REF] Cobbe | Leveraging procedural generation to benchmark reinforcement learning[END_REF]. The vast majority of models have relied on gradient-based optimization to learn static network parameters adjusted during a predefined curriculum, arguably preventing the emergence of online adaptivity. A potential solution to this challenge is to meta-learn [START_REF] Schmidhuber | Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement[END_REF][START_REF] Thrun | Lifelong Learning Algorithms[END_REF][START_REF] Vilalta | A perspective view and survey of meta-learning[END_REF] computational mechanisms able to rapidly capture a task structure and automatically operate complex feedback control: Meta-Reinforcement Learning constitutes a promising direction to build more adaptive artificial systems [START_REF] Clune | Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF] and identify key neuroscience mechanisms that endow humans with their versatile learning abilities [START_REF] Botvinick | Reinforcement learning, fast and slow[END_REF].

In this work, we draw inspiration from biological fast synaptic plasticity, hypothesized to orchestrate flexible cognitive functions according to context-dependent rules [START_REF] Martin | Synaptic plasticity and memory: An evaluation of the hypothesis[END_REF][START_REF] Abbott | Synaptic computation[END_REF][START_REF] Wade G Regehr | Short-term presynaptic plasticity[END_REF][START_REF] Caporale | Spike timing-dependent plasticity: A hebbian learning rule[END_REF]. By tuning neuronal selectivity at fast time scales -from fast neural signaling (milliseconds) to experience-based learning (seconds and beyond) -fast plasticity can support in principle many cognitive faculties including motor and executive control. From a dynamical system perspective, fast plasticity can serve as an efficient mechanism for information storage and manipulation and has led to modern theories of working memory [START_REF] Mongillo | Synaptic theory of working memory[END_REF][START_REF] Masse | Circuit mechanisms for the maintenance and manipulation of information in working memory[END_REF][START_REF] Barak | Working models of working memory[END_REF][START_REF] Mark | activity-silent' working memory in prefrontal cortex: a dynamic coding framework[END_REF][START_REF] Sanjay | Neural mechanisms of attending to items in working memory[END_REF]. Despite the fact that the magnitude of the synaptic gain variations may be small, such modifications are capable of profoundly altering the network transfer function [START_REF] Yger | Fast learning with weak synaptic plasticity[END_REF] and constitute a plausible mechanism for rapidly converting reward and choice history into tuned neural functions [START_REF] Rȃzvan | Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity[END_REF]. From a machine learning perspective, despite having a long history [START_REF] Hinton | Using fast weights to deblur old memories[END_REF][START_REF] Schmidhuber | Learning to control fast-weight memories: An alternative to dynamic recurrent networks[END_REF][START_REF] Schmidhuber | Learning to control fast-weight memories: An alternative to dynamic recurrent networks[END_REF][START_REF] Malsburg | The Correlation Theory of Brain Function[END_REF][START_REF] Ha | Hypernetworks[END_REF], fast weights have most often been investigated in conjunction with recurrent neural activations [START_REF] Ba | Using fast weights to attend to the recent past[END_REF][START_REF] Miconi | Differentiable plasticity: training plastic neural networks with backpropagation[END_REF][START_REF] Schlag | Learning to reason with third-order tensor products[END_REF][START_REF] Irie | Going beyond linear transformers with recurrent fast weight programmers[END_REF] and rarely as a function of an external reward signal or of the current synaptic state itself. Recently, new proposals have shown that models with dynamic modulation related to fast weights could yield powerful meta-reinforcement learners [START_REF] Schlag | Learning associative inference using fast weight memory[END_REF][START_REF] Sarafian | Recomposing the reinforcement learning building blocks with hypernetworks[END_REF][START_REF] Luckeciano | Transformers are meta-reinforcement learners[END_REF]. In this work, we explore an original self-referential update rule that allows the model to form synaptic updates conditionally on information present in its own synaptic memory. Additionally, environmental reward is injected continually in the model as a rich feedback signal to drive the weight dynamics. These features endow our model with a unique recursive control scheme, that support the emergence of a self-contained reinforcement learning program.

Contribution:

We demonstrate that a neural network trained to continually self-modify its weights as a function of sensory information and its own synaptic state can produce a powerful reinforcement learning program. The resulting general-purpose meta-RL agent called 'MetODS' (for Meta-Optimized Dynamical Synapses) is theoretically presented as a model-free approach performing stochastic feedback control in the policy space. In our experimental evaluation, we investigate the reinforcement learning strategies implemented by the model and demonstrate that a single layer with lightweight parametrization can implement a wide spectrum of cognitive functions, from oneshot learning to continuous motor-control. We hope that MetODS inspires more works around self-optimizing neural networks.

The remainder of the paper is organised as follows: In Section 2 we introduce our mathematical formulation of the meta-RL problem, which motivates MetODS computational principles presented in Section 3. In Section 4 we review previous approaches of meta-reinforcement learning and we discuss other models of artificial fast plasticity and their relation to associative memory. In Section 5 we report experimental results in multiple contexts. Finally, in Section 6 we summarise the main advantages of MetODS and outline future work directions.

Background

Notation

Throughout, we refer to "tasks" as Markov decision processes (MDP) defined by the following tuple τ = (S, A, P, r, ρ 0 ), where S and A are respectively the state and action sets, P : S ×A×S → [0, 1] refers to the state transition distribution measure associating a probability to each tuple (state, action, new state), r : A × S → R is a bounded reward function and ρ 0 is the initial state distribution. For simplicity, we consider finite-horizon MDP with T time-steps although our discussion can be extended to the infinite horizon case as well as partially observed MDP. We further specify notation when needed by subscripting symbols with the corresponding task τ or time-step t.

Meta-Reinforcement learning as an optimal transport problem:

Meta-Reinforcement learning considers the problem of building a program that generates a distribution µ π of policies π ∈ Π that are "adapted" for a distribution µ τ of task τ ∈ T with respect to a given metric R. For instance, R(τ, π) can be the expected cumulative reward for a task τ and policy π, i.e where state transitions are governed by s t+1 ∼ P τ (.|s t , a t ), actions are sampled according to the policy π: a t ∼ π and initial state s 0 follows the distribution ρ 0,τ . Provided the existence of an optimal policy π * for any task τ ∈ T, we can define the distribution measure µ π * of these policies over Π. Arguably, an ideal system aims at associating to any task τ its optimal policy π * , i.e, finding the transport plan γ in the space Γ(µ T , µ π * ) of coupling distributions with marginals µ T and µ π * that maximizes R. Associating any task τ in T to its optimal policy π * τ in Π can be regarded as finding an optimal transport plan from µ T to µ π * with respect to the cost -R. Finding this transport plan is generally an intractable problem. b) Meta-RL approximates a solution by defining a stochastic flow in the policy space Π conditioned by the current task τ and driving a prior distribution µπ 0 of policies π0 towards a distribution µ θ,τ,t π of policies with high score R. c) Density and mean trajectories of our model dynamic weights principal components over several episodes of the Harlow task (see section 5.1) reveal this policy specialization. Two modes colored with respect to whether the agent initial guess was good or bad emerge, corresponding to two different policies to solve the task.

max γ∈Γ(µ T ,µ π * ) E (τ,π)∼γ R(τ, π) where R(τ, π) = E π,Pτ T t=0 r τ (a t , s t ) (1)
Most generally, problem (1) is intractable, since µ π * is unknow or has no explicit form. Instead, previous approaches optimize a surrogate problem, by defining a parametric specialization procedure which builds for any task τ , a sequence (π t ) t of improving policies (see Fig. 1). Defining θ some meta-parameters governing the evolution of the sequences (π t ) t and µ θ,τ,t π the distribution measure of the policy π t after learning task τ during some period t, the optimization problem amounts to finding the meta-parameters θ that best adapt π t ∼ µ θ,τ,t π over the task distribution.

max θ E τ ∼µ T E π∼µ θ,τ,t π R(τ, π)] (2)
Equation ( 2) cast meta-Reinforcement learning as the problem of building automatic policy control in Π. We discuss below desirable properties of such control for which we show that our proposed meta-learnt synaptic rule has a good potential.

Efficiency: How "fast" the distribution µ θ,τ,t π is transported towards a distribution of highperforming policies. Arguably, an efficient learning mechanism should require few interaction steps t with the environment to identify the task rule and adapt its policy accordingly. This can be seen as minimizing the agent's cumulative regret during learning [START_REF] Wang | Learning to reinforcement learn[END_REF]. This notion is also connected to the policy exploration-exploitation trade-off [START_REF] Ishii | Control of exploitation-exploration meta-parameter in reinforcement learning[END_REF][START_REF] Zintgraf | Varibad: A very good method for bayes-adaptive deep rl via metalearning[END_REF], where the agent learning program should foster rapid discovery of policy improvements while retaining performance gains acquired through exploration. We show that our learnt synaptic update rule is able to change the agent transfer function drastically in a few updates, and settle to a policy when task structure has been identified, thus supporting one-shot learning of a task-contingent association rule in the Harlow task, adapting a motor policy in a few steps or exploring original environments quickly in the Maze experiment.

Capacity: That property defines the learner sensitivity to particular task structures and its ability to convert them into precise states in the policy space, which determines the achievable level of performance for a distribution of tasks µ τ . This notion is linked to the sensitivity of the learner, i.e how the agent captures and retains statistics and structures describing a task. Precedent work has shown that memory-based Meta-RL models operate a Bayesian task regression problem [START_REF] Ortega | Meta-learning of sequential strategies[END_REF], and further fostering task identification through diverse regularizations [START_REF] Humplik | Meta reinforcement learning as task inference[END_REF][START_REF] Hafner | Dream to control: Learning behaviors by latent imagination[END_REF][START_REF] Zintgraf | Varibad: A very good method for bayes-adaptive deep rl via metalearning[END_REF] benefited performance of the optimized learning strategy. Because our mechanism is continual, it allows for constant tracking of the environment information and policy update similar to memory-based models. We particularly test this property in the maze experiment in Section 5, showing that tuned online synaptic updates obtain the best capacity under systematic variation of the environment.

Generality:

We refer here to the overall ability of the meta-learnt policy flow to drive µ θ,τ,t π towards high-performing policy regions for a diverse set of tasks (generic trainability) but also to how general is the resulting reinforcement learning program and how well it transfers to tasks unseen during training (transferability). In the former case. since the proposed synaptic mechanism is model-free, it allows for tackling diverse types of policy learning, from navigation to motor control. Arguably, to build reinforcing agents that learn in open situations, we should strive for generic and efficient computational mechanisms rather than learnt heuristics. Transferability corresponds to the ability of the meta-learned policy flow to generally yield improving updates even in unseen policy regions of the space Π or conditioned by unseen task properties: new states, actions and transitions, new reward profile. etc. We show in a motor-control experiment using the Meta-World benchmark that meta-tuned synaptic updates are a potential candidate to produce a more systematic learner agnostic to environment setting and reward profile. The generality property remains the hardest for current meta-RL approaches, demonstrating the importance of building more stable and invariant control principles.

3 MetODS: Meta-Optimized Dynamical Synapses What if a neural agent could adjust the rule driving the evolution of µ θ,τ,t π based on its own knowledge? In this work, we test a neural computational model that learns to compress experience of a task τ into its dynamic weights W t .

∀t ≤ T, π(a|s, W t ) ∼ µ θ,τ,t π (3)
Contrary to gradient-based rules whose analytical expression is a static and predefined function of activations error (∆(W t ) = ∂a ∂W . ∂L ∂a ), we define an update rule that directly depends on the current weight state through a parameterized non-linear recursive scheme, making the expression of the update much more sensitive to weight content itself.

∀t ≤ T, ∆(W t ) = F θ (W t ) W =Wt (4)
This self-reflexive evolution of the synaptic configuration W t allows for online variation of the learning rule during adaptation for a given task τ and opens an original repertoire of dynamics for the policy evolution µ θ,τ,t π . Specifically, at every time-step t, network inference and learning consists in recursive applications of read and write operations that we define below.

Read-write operations:

The core mechanism consists in two simple operations that respectively linearly project neurons activation v through the dynamic weights W followed by an element-wise non-linearity, and build a local weight update with element-wise weighting α:

φ(W , v) = σ(W .v) read ψ(v) = α v ⊗ v write (5)
Here, α is a matrix of R N ×N , ⊗ denotes the outer-product, is the element-wise multiplication, σ is a non-linear activation function. The read and write operations are motivated by biological synaptic computation: Reading corresponds to the non-linear response of the neuron population to a specific activation pattern. Writing consists in an outer product that emulates a local Hebbian rule between neurons. The element-wise weighting α allows locally tuning synaptic plasticity at every connection consistent with biology [START_REF] Larry | Synaptic plasticity: taming the beast[END_REF][START_REF]Tuning into diversity of homeostatic synaptic plasticity[END_REF] which generates a matrix update with potentially more than rank one as in the classic hebbian rule.

Recursive update: While a single iteration of these two operations can only retrieve a similar activation pattern (reading) or add unfiltered external information into weights (writing), recursively applying these operations offers a much more potent computational mechanism that mix information between current neural activation and previous iterates. Starting from an initial activation pattern v (0) and previous weight state W (0) = W t-1 , the model recursively applies equations (5) for s ∈ [1, S] on v (s) and W (s) such that:

for s ∈ [1, S] :        v (s) = s-1 l=0 κ (l) s v (l) + κ (s) s φ(W (s-1) , v (s-1) ) W (s) = s l=0 β (l) s W (l) + β (s) s ψ(v (s-1) ) (6) Parameters κ (l) s and β (l)
s are scalar values learnt along with plasticity parameters α, and correspond to delayed contributions of previous patterns and synaptic states to the current operations. This is motivated by biological evidence of complex cascades of temporal modulation mechanisms over synaptic efficacy [START_REF] Yun | Impact of second messenger modulation on activity-dependent and basal properties of excitatory synapses[END_REF][START_REF] Rylan | Synapse-specific control of experience-dependent plasticity by presynaptic nmda receptors[END_REF]. Finally, (v (S) , W (S) ) are respectively used as activation for the next layer, and as the new synaptic state W t .

Computational interpretation: We note that if S=1 in equation ( 6), the operation boils down to a simple hebbian update with a synapse-specific weighting α i,j . This perspective makes MetODS an original form of modern Hopfield network [START_REF] Ramsauer | Hopfield networks is all you need[END_REF] with hetero-associative memory that can dynamically access and edit stored representations driven by observations, rewards and actions. While pattern retrieval from Hopfield networks has a dense litterature, our recursive scheme is an original proposal to learn automatic updates able to articulate representations across timesteps. We believe that this mechanism is particularly benefitial for meta-RL to filter external information with respect to past experience. The promising results shown in our experimental section suggest that such learnt updates can generate useful self-modifications to sequentially adapt to incoming information at runtime.

Algorithm 1: MetODS synaptic learning 1: Require: θ = [f , g, α, κ, β] and W 0 2: for 1 ≤ t ≤ T do 3: v (0) ← f (s t , a t-1 , r t-1 ) 4: W (0) ← W t-1 5: for 1 ≤ s ≤ S do 6: v (s) ← s-1 l=0 κ (l) s v (l) + κ (s) s σ(W (s-1) .v (s-1) ) 7: W (s) ← s-1 l=0 β (l) s W (l) + β (s) s α v (s-1) ⊗ v (s-1)

8:

end for 9:

a t , v t ← g(v (s) ) 10: W t ← W (s) 11: end for
In this work, we test a single dynamic layer for MetODS and leave the extension of the synaptic plasticity to the full network for future work.

In order for the model to learn a credit assignment strategy, state transition and previous reward information [s t , a t-1 , r t-1 ] are embedded into a vector v t by a feedforward map f as in previous meta-RL approaches [START_REF] Duan | Rl2: Fast reinforcement learning via slow reinforcement learning[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF]. Action and advantage estimate are read-out by a feedforward policy map g. We meta-learn the plasticity and update coefficients, as well as the embedding and read-out function altogether: θ = [f , g, α, κ, β]. Additionally, the initial synaptic configuration W 0 can be learnt, fixed a priori or sampled from a specified distribution.

Related work

Meta-Reinforcement learning has recently flourished into several different approaches aiming at learning high-level strategies for capturing task rules and structures. A direct line of work consists in automatically meta-learning components or parameters of the RL arsenal to improve over heuristic settings [START_REF] Houthooft | Evolved policy gradients[END_REF][START_REF] Xu | Meta-gradient reinforcement learning[END_REF][START_REF] Gupta | Metareinforcement learning of structured exploration strategies[END_REF]. Orthogonally, work building on the Turing-completeness of recurrent neural networks has shown that simple recurrent neural networks can be trained to store past information in their persistent activity state to inform current decision, in such a way that the network implements a form of reinforcement learning over each episode [START_REF] Hochreiter | Learning to learn using gradient descent[END_REF][START_REF] Duan | Rl2: Fast reinforcement learning via slow reinforcement learning[END_REF][START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF]. It is believed that vanilla recurrent networks alone are not sufficient to meta-learn the efficient forms of episodic control found in biological agents [START_REF] Lengyel | Hippocampal contributions to control: The third way[END_REF][START_REF] Botvinick | Reinforcement learning, fast and slow[END_REF]. Hence additional work has tried to enhance the system with a better episodic memory model [START_REF] Santoro | Meta-learning with memory-augmented neural networks[END_REF][START_REF] Pritzel | Neural episodic control[END_REF][START_REF] Ritter | Episodic control as metareinforcement learning[END_REF] or by modeling a policy as an attention module over an explicitely stored set of past events [START_REF] Mishra | A simple neural attentive meta-learner[END_REF]. Optimization based approaches have tried to cast episodic adaptation as an explicit optimization procedure either by treating the optimizer as a black-box system [START_REF] Santoro | Meta-learning with memory-augmented neural networks[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] or by learning a synaptic configuration such that one or a few gradient steps are sufficient to adapt the input/output mapping to a specific task [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF].

Artificial fast plasticity: Networks with dynamic weights that can adapt as a function of neural activation have shown promising results over regular recurrent neural networks to handle sequential data [START_REF] Munkhdalai | Meta networks[END_REF][START_REF] Ha | Hypernetworks[END_REF][START_REF] Miconi | Differentiable plasticity: training plastic neural networks with backpropagation[END_REF][START_REF] Miconi | Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity[END_REF][START_REF] Schlag | Learning to reason with third-order tensor products[END_REF][START_REF] Schlag | Learning associative inference using fast weight memory[END_REF]. However, contrary to our work, these models postulate a persistent neural activity orchestrating weights evolution. On the contrary, we show that synaptic states are the sole persistent components needed to perform fast adaptation. Additionally, the possibility of optimizing synaptic dynamics with evolutionary strategies in randomly initialized networks [START_REF] Najarro | Meta-learning through hebbian plasticity in random networks[END_REF] or through gradient descent [START_REF] Miconi | Learning to learn with backpropagation of hebbian plasticity[END_REF] has been demonstrated, as well as in a time-continuous setting [START_REF] Choromanski | An ode to an ode[END_REF]. Recent results have shown that plasticity rules differentially tuned at the synapse level allow to dynamically edit and query networks memory [START_REF] Miconi | Differentiable plasticity: training plastic neural networks with backpropagation[END_REF][START_REF] Chalvidal | Go with the flow: Adaptive control for neural ODEs[END_REF]. However another specificity of this work is that our model synaptic rule is a function of reward and synaptic state, allowing to drive weight dynamics conditionally on both an external feedback signal and the current model belief.

Associative memory: As discussed above, efficient memory storage and manipulation is a crucial feature for building rapidly learning agents. To improve over vanilla recurrent neural network policies [START_REF] Duan | Rl2: Fast reinforcement learning via slow reinforcement learning[END_REF], some models have augmented recurrent agents with content-addressable dictionaries able to reinstate previously encoded patterns given the current state [START_REF] Weston | Memory networks[END_REF][START_REF] Zaremba | Reinforcement learning neural turing machines -revised[END_REF][START_REF] Pritzel | Neural episodic control[END_REF][START_REF] Botvinick | Reinforcement learning, fast and slow[END_REF]. However these slot-based memory systems are subject to interference with incoming inputs and their memory cost grows linearly with experience. Contrastingly, attractor networks can be learnt to produce fast compression of sensory information into a fixed size tensorial representation [START_REF] Bartunov | Meta-learning deep energy-based memory models[END_REF][START_REF] Zhang | Learning to update auto-associative memory in recurrent neural networks for improving sequence memorization[END_REF]. One class of such network are Hopfield networks [START_REF] Hopfield | Neural networks and physical systems with emergent collective computational abilities[END_REF][START_REF] Koiran | Dynamics of discrete time, continuous state hopfield networks[END_REF][START_REF] Demircigil | On a model of associative memory with huge storage capacity[END_REF][START_REF] Krotov | Dense associative memory for pattern recognition[END_REF] which benefit from a large storage capacity [START_REF] Krotov | Dense associative memory for pattern recognition[END_REF], can possibly perform hetero-associative concept binding [START_REF] Schlag | Learning to reason with third order tensor products[END_REF][START_REF] Schlag | Learning associative inference using fast weight memory[END_REF] and produce fast and flexible information retrieval [START_REF] Ramsauer | Hopfield networks is all you need[END_REF].

Experiments

In this section, we explore the potential of our meta-learnt synaptic update rule with respect to the three properties of the meta-RL problem exposed in section 2. Namely, 1) efficiency, 2) capacity and 3) generality of the produced learning algorithm. We compare it with three state-of-the-art meta-RL models based on different adaptation mechanisms: RL 2 [48] a memory-based algorithm based on training a GRU-cell to perform Reinforcement Learning within the hidden space of its recurrent unit, MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] which performs online gradient descent on the weights of three-layer MLP and PEARL [START_REF] Rakelly | Efficient off-policy meta-reinforcement learning via probabilistic context variables[END_REF], which performs probabilistic task inference for conditioning policy. Details for each experimental setting are further discussed in S.I. and the code can be found at https: //github.com/mathieuchal/metods2022. To first illustrate that learnt synaptic dynamics can support fast behavioral adaptation, we use a classic experiment from the neuroscience literature originally presented by Harlow [START_REF] Harry | The formation of learning sets[END_REF] and recently reintroduced in artificial meta-RL in [START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF] as well as a heavily-benchmarked MuJoCo directional locomotion task (see Fig. 3). To behave optimally in both settings, the agent must quickly identify the task rule and implement a relevant policy : The Harlow task consists of five sequential presentations of two random variables placed on a one-dimensional line with random permutation of their positions that an agent must select by reaching the corresponding position. One value is associated with a positive reward and the other with a negative reward. The five trials are presented in alternance with periods of fixation where the agent should return to a neutral position between items. In the MuJoCo robotic Ant-dir experiment, a 4-legged agent must produce a locomotive policy given a random rewarded direction.

Efficiency: One-shot reinforcement learning and rapid motor control

Harlow: Since values location are randomly permuted across presentations within one episode, the learner cannot develop a mechanistic strategy to reach high rewards based on initial position. Instead, to reach the maximal expected reward over the episode, the agent needs to perform one-shot learning of the task-contingent association rule during the first presentation. We found that even a very small network of N=20 neurons proved to be sufficient to solve the task perfectly. We investigated the synaptic mechanism encoding the agent policy. A principal component analysis reveals a policy differentiation with respect to the initial value choice outcome supported by synaptic specialization in only a few time-steps (see Figure 1-c). We can further interpret this adaptation in terms of sharp modifications of the Hopfield energy of the dynamic weights (Figure 3-e). Finally, the largest synaptic variations measured by the sum of absolute synaptic variations occur for states that carry a non-null reward signal (see S.I). These results suggest that the recursive hebbian update combined with reward feedback is sufficient to support one-shot reinforcement learning of the task association rule.

MuJoCo Ant-dir: We trained models for performing the task on an episode of 200 timesteps and found that MetODS can adapt in a few time-steps, similar to memory-based models such as RL 2 , (Figure 3-f) thanks to its continual adaptation mechanism. By design, MAML and PEARL do not present such a property, and they need multiple episodes before being able to perform adaptation correctly. We still report MAML performance after running its gradient adaptation at timestep t=100. We further note that our agent overall performance in a single episode is still superior to MAML performance reported in [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF] when more episodes are accessible for training. We further tested the systematicity of MetODS learnt reinforcement program on a partially observable Markov decision process (POMDP): An agent must locate a target in a randomly generated maze while starting from random locations and observing a small portion of its environment (depicted in Figure 4). While visual navigation has been previously explored in meta-RL [START_REF] Duan | Rl2: Fast reinforcement learning via slow reinforcement learning[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF][START_REF] Miconi | Differentiable plasticity: training plastic neural networks with backpropagation[END_REF], we here focus on the mnemonic component of navigation by complexifying the task in two ways, we reduce the agent's visual field to a small size of 3x3 pix. and randomize the agent's position after every reward encounter. The agent can take discrete actions in the set {up,down,left,right} which moves it accordingly by one coordinate. The agent's reward signal is solely received by hitting the target location, thus receiving a reward of 10. Each time the agent hits the target, its position is randomly reassigned on the map (orange) and the exploration resumes until 100 steps are accumulated during which the agent must collect as much reward as possible. Note that the reward is invisible to the agent, and thus the agent only knows it has hit the reward location because of the activation of the reward input. The reduced observability of the environment and the sparsity of the reward signal (most of the state transitions yield no reward) requires the agent to perform logical binding between distant temporal events to navigate the maze. Again, this setting rules out PEARL since its latent context encoding mechanism erases crucial temporal dependencies between state transitions for efficient exploration. Despite having no particular inductive bias for efficient spatial exploration or path memorization, a strong policy emerges spontaneously from training. Ablation study and variations: We explored the contribution of the different features combined in MetODS update rule, showing that they all contribute to the final performance of our meta-learning model. First, we tested the importance of the element-wise tuning of plasticity in weight-based learning models and note that while it adversely affects MAML gradient update, it greatly improves MetODS performance, suggesting different forms of weights update. Second, we verified that augmenting recursivity depth S was beneficial to performance, consistently with Harlow results. Third, we noted that transforming the rightmost vector of the writing equation in 5 with a linear projection (Linear in figure 4, see S.I for full experimental details) yields major improvement while non-linear (MLP) does not improve performance. Finally, we additionally test the capability of the learnt navigation skills to generalize to a larger maze size of 10 × 10 pix. unseen during training. We show that MetODS is able to retain its advantage (see figure 4 and table 5 for results).

Capacity : Maze exploration task

Agent

Generality : Motor control

Finally, we test the generality of the reinforcement learning program learnt by our model for different continuous control tasks:

MetaWorld: First, we use the dexterous manipulation benchmark proposed in [START_REF] Yu | Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning[END_REF] using the benchmark suite [START_REF]Garage: A toolkit for reproducible reinforcement learning research[END_REF], in which a Sawyer robot is tasked with diverse operations. A full adaptation episode consists in N=10 rollouts of 500 timesteps of the same task across which dynamic weights are carried over. Observation consists in the robot's joint angles and velocities, and the actions are its joint torques. We compare MetODS with baseline methods in terms of meta-training and meta-testing success rate for 3 settings, push, reach and ML-10. We show in Fig. 6 the meta-training results for all the methods in the MetaWorld environments. Due to computational resource constraints, we restrict our experiment to a budget of 10M steps per run. While we note that our benchmark does not reflect the final performance of previous methods reported in [START_REF] Yu | Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning[END_REF] at 300M steps, we note that MetODS test performance outperforms these methods early in training and keeps improving at 10M steps, potentially leaving room for improvement with additional training. (see S.I for additional discussion). Finally, we note that all tested approaches performed modestly on ML10 for test tasks, which highlights the limitation of current methods. We conjecture that this might be due to the absence of inductive bias for sharing knowledge between tasks or fostering systematic exploration of the environment of the tested meta-learning algorithms.

Robot impairment:

We also tested the robustness of MetODS learnt reinforcement programs by evaluating the agent ability to perform in a setting not seen during training: specifically, when partially impairing the agent motor capabilities. We adopt the same experimental setting as section 5.1 for the Ant and Cheetah robots and evaluate the performance when freezing one of the robots torque. We show that our model policy retains a better proportion of its performance compared to other approaches. These results suggest that fast synaptic dynamics are not only better suited to support fast adaptation of a motor policy in the continuous domain, but also implement a more robust reinforcement learning program when impairing the agent motor capabilities. In this work, we introduce a novel meta-RL system, MetODS, which leverages a self-referential weight update mechanism for rapid specialization at the episodic level. Our approach is generic and supports discrete and continuous domains, giving rise to a promising repertoire of skills such as oneshot adaptation, spatial navigation or motor coordination. MetODS demonstrates that locally tuned synaptic updates whose form depends directly on the network configuration can be meta-learnt for solving complex reinforcement learning tasks. We conjecture that further tuning the hyperparameters as well as combining MetODS with more sophisticated reinforcement learning techniques can boost its performance. Generally, the success of the approach provides evidence for the benefits of fast plasticity in artificial neural networks, and the exploration of self-referential networks.

Broader Impact

Our work explores the emergence of reinforcement-learning programs through fast synaptic plasticity. The proposed pieces of evidence that information memorization and manipulation as well as behavioral specialization can be supported by such computational principles 1) help question the functional role of such mechanisms observed in biology and 2) reaffirms that alternative paradigms to gradient descent might exists for efficient artificial neural network control. Additionally, the proposed method is of interest for interactive machine learning systems that operates in quickly changing environments and under uncertainty (bayesian optimization, active learning and control theory). For instance, the meta RL approach proposed in this work could be applied to brain-computer interfaces for tuning controllers to rapidly drifting neural signals. Improving medical applications and robot control promises positive impact, however deeper theoretical understanding and careful deployment monitoring are required to avoid misuse.

Defining the weight parameters W of MetODS as dynamic variables lifts the optimization problem (2) into a functional space of control functions parameterized by θ. Hence, meta-optimizing the control necessitates the estimation of gradients with respect to θ over the space T and for any possible trajectory π t in Π. Interestingly, previous meta RL approaches have performed policy gradient optimization by sampling a single policy trajectory π t ∼ M θ (τ ) over M multiple tasks, showing that it is sufficient to obtain correct gradient estimates on θ. We proceed in the same way, by estimating the gradient policy update integrated over the space of tasks as mini-batches over tasks.

∂ ∂θ E τ ∼µ T E π∼µ θ,τ,t π R(τ, π)] ≈ τ1,...,τn T t=0 ∂ log π t (a t |W t , θ) ∂θ r τi (a t , s t ) (7)
Additionally, the memory cost of storing synaptic weights trajectories instead of hidden activity in a network of N neurons is O(N 2 ) instead of O(N ). This might lead to prohibitively large memory requirements for training with BPTT [START_REF] David E Rumelhart | Learning internal representations by error propagation[END_REF] over long episodes. We present in S.I an alternative solution to train the model through the discrete adjoint sensitivity method, leveraging the work of [START_REF] Betancourt | The discrete adjoint method: Efficient derivatives for functions of discrete sequences[END_REF] yielding a memory cost of O(1). The agent's log-policy total derivative with respect to θ can be computed as the solution of an augmented adjoint problem [START_REF] Chalvidal | Go with the flow: Adaptive control for neural ODEs[END_REF].

Gradient policy update

We define the evolution of the agent policy in task τ up to step T as the stochastic policy process (π t ) t≤T in the space of policy Π with measure µ θ,τ,1•••T π and write (W t ) t≤T the trajectory of weights, such that:

π t t≤T = π t (•|W t , s t , θ) t≤T ∼ µ θ,τ,1•••T π (8)
We also recall the definition of R(τ, π) as the average accumulated reward under the realisation of (π t ) t≤T for task τ . Then, The policy gradient update used to train MetODS can be written as the average gradient of θ with respect to µ T and µ θ,τ,1

•••T π : ∂ ∂θ E τ ∼µ T E π∼µ θ,τ,1•••T π R(τ, π)] = E τ ∼µ T T t=0 ∂ ∂θ E πt,Pτ r τ (a t , s t ) (9)
Then for any t ≤ T , we can rewrite the gradient of the average, using the log-policy trick:

∂ ∂θ E πt,Pτ r τ (a t , s t ) = E πt,Pτ r τ (a t , s t ) ∂ log π t (a τ t |W t , θ) ∂θ (10)
As specified in section 3.3, we do not sample over the policy distribution π t and probability transition P τ to estimate the inner expectation in [START_REF] Vilalta | A perspective view and survey of meta-learning[END_REF]. Instead, we rely on a single evaluation, which yield, combining ( 9) with (10):

∂ ∂θ E τ ∼µ T E π∼µ θ,τ,1•••T π R(τ, π)] ≈ T t=0 r τ (a t , s t ) ∂ log π t (a τ t |W t , θ) ∂θ (11)
By sampling over tasks, this last equation allows us to write the following gradient estimator

∂ ∂θ E τ ∼µ T E π∼µ θ,τ,1•••T π R(τ, π)] ≈ 1 M M i=0 T t=0 ∂ log π t (a τi t |W t , θ) ∂θ r(a τi t , s τi t ) (12)

Discrete adjoint system

With the previous notation, let us define the total gradient function ∂J ∂θ as the gradient of our objective function:

∂J ∂θ (θ) = 1 M T t=0 M i=0 ∂ log π t (a τi t |W t , θ) ∂θ r(a τi t , s τi t ) (13)
To clarify, we shall introduce the intermediary cost notation:

c t (W , θ, t) = M i=0 log π t (a τi t |W t , θ)r(a τi t , s τi t ) (14)
and we note the following update equation

W t+1 = δ(W t , θ) (15)
This identify a discrete dynamical system with finite sum and differentiable cost, whose gradient can be computed mediating the introduction of an adjoint dynamical system presented in section 2 of [START_REF] Betancourt | The discrete adjoint method: Efficient derivatives for functions of discrete sequences[END_REF] . Defining (ν t ) t≤T the adjoint sequence, the general gradient equation can be computed as:

∂J ∂θ (θ) = ∂c 0 ∂W 0 -ν 0 • ∂δ ∂W 0 -ν 0 † ∂W 0 ∂θ + T i=1 ∂c t ∂θ -ν † t • ∂δ ∂θ (16)
Applying formula [START_REF] Wade G Regehr | Short-term presynaptic plasticity[END_REF] to MetODS yields the following gradient formula:

∂J ∂θ (θ) = ∂c 0 ∂W 0 -ν 0 + T i=1 ∂c t ∂θ -ν † t • ∂δ ∂θ (17)
where (ν t ) t≤T follows the following update rule backwards:

ν T = 0 ν t-1 = ν t -∂ct ∂Wt ( 18
)
10 Experiment details

Meta-training algorithm:

We show in our experiments, that the all meta-parameters θ = [α, κ, β, f , g, ] can be jointly optimized with two policy gradient algortihms. We use policy gradient methods to meta-train the synaptic parameters: Advantage Actor-critic algorithm (A2C) [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF] where we consider that temporal dependancies are crucial to solve the task, and we use PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] as a sequential policy optimization over fixed rollouts for the motor control experiment. Additionally, to reduce noise in policy gradient updates, we further show that it is possible to learn a dynamic advantage estimate of the Generalized Advantage Estimation (GAE) [START_REF] Schulman | Highdimensional continuous control using generalized advantage estimation[END_REF] and that we can meta-learn it as a second head of the MetODS layer output v (s) . This fact confirms that tuned hebbian-updates are also a sufficient mechanism to keep track of a policy value estimate.

Plasticity parameters: In all our experiments, α consists in a real-valued matrix of R n×n initialized with independent normal distribution N (µ = 0, σ = 1e -3). Similarly, multi-step weighting parameters (κ

(l) s ) l≤s and (β (l)
s ) l≤s can be stored as entries of triangular inferior matrices of R S×S and are initialized with N (µ = 0, σ = 1e -2).

Embedding and read-out: At each time-steps, inputs to the feed-forward embedding function f consist in a concatenation of new observable/state s t as well as previous action and reward a t-1 and r t-1 . After adaptation procedure, policy is read-out from the last activation vector v (S) by a feed-forward function g which output statistics of a parameterized distribution in the action space (categorical with Softmax normalization in the discrete case or Gaussian in the continuous case). Both input and output mappings f and g consist in 2-layer Perceptrons with dense connections initialized with orthogonal initialization and hyperbolic tangent non-linearities. Setting: We consider the directional rewards task proposed in [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] with the Ant and Cheetah robots, as a more complex test of rapid adaptation. We apply standard RL practice for continuous control by parameterizing the stochastic policy as a product of independent normal distributions with fixed standard deviation σ = 0.1 and mean inferred by the agent network. A training meta-episode consists in a single rollout of 200 steps with random sampling of the reward direction for each episode.

Harlow task

Robot impairment: We partially impaired the agent motor capabilities by "freezing" one of the robot actuator. Namely we consistently passed a value of zero to the the right_back_leg in Ant (coordinate 8 in OpenAI Gym XML asset files) and ffoot in Cheetah (coordinate 6 in OpenAI Gym XML asset files).

Continual adaptation: Finally, we further investigated whether the adaptation mechanism was also adjustable after performing initial adaptation by testing trained agents over 5 consecutive episodes with randomly changing rewarded direction while retaining weight state across episodes. We show in Fig 10 the average reward profile of the three meta-RL agents over 1000 test episodes, where MetODS adapt faster and better with respect to reward variations. Maze generation: The maze environments are created following Prim's algorithm [START_REF] Clay | Shortest connection networks and some generalizations[END_REF] which randomly propagates walls on a board of N × N cells. We additionally add walls to cell location were no walls has been created at any of the neighboring 8 cells to avoid null inputs to the agent. The reward and agent locations are selected at random at the beginning of an episode. The reward location does not change during the episode while the agent is restarting from a uniform random location after every encounter with reward.

Maze Navigation task

Models capacity:

In this experiment, since model memory capacity seems to directly impact final performance, we specifically control for the number of learnable parameters in order to compare models and fix a training budget of 10M env. steps. Since MAML does not have a continual adaptation mechanism, we perform gradient adaptation every 20 time-steps during an episode, in order to balance noisiness of the gradient and rapidity of exploration. We show in Table 2 the mean over 1000 episodes of the accumulated reward of agents trained on 8 × 8 mazes and tested on different maze sizes N ∈ [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF][START_REF] Brenden | Building machines that learn and think like people[END_REF][START_REF] Cobbe | Leveraging procedural generation to benchmark reinforcement learning[END_REF][START_REF] Thrun | Lifelong Learning Algorithms[END_REF][START_REF] Clune | Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF]. Agents performances are highly varying within a size setting due to differences in maze configurations and across maze sizes due to increasing complexity. However, MetODS and RL 2 are able to generalize at least partially to these new settings (see table 2). MetODS outperforms RL 2 in every setting, and generalizes better, retaining an advantage of up to 25% in accumulated reward in the biggest maze. Additionally, we investigated the selectivity of neurons of MetODS plastic weight layer with respect to spatial location, to investigate whether there could be emergence of activation patterns resembling those of grid cells found in the mammalian entorhinal cortex. We measured the selectivity of each neuron for specific agent locations by measuring the average activation rate normalized by the number of agent passages in different maze configurations. Interestingly, we found that without any explicit regularization, some neurons displayed sparse activation and consistent remapping between maps.

Meta-world

Experiments: In order to run experiments and baseline of MetaWorld [START_REF] Yu | Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning[END_REF], we used the training routines and functions offered in the python library GARAGE [START_REF]Garage: A toolkit for reproducible reinforcement learning research[END_REF]. We ran two different types of experiments:

1. ML1: In this setting, we restrict the task distribution to a single type of robotic manipulation while varying the goal location. The meta-training "tasks" corresponds to 50 random initial agent and goal positions, and meta-testing to 50 heldout positions. We tested our model on the reach-v2 and push-v2 tasks. 2. ML10: This set-up tests generalization to new manipulation tasks, the benchmark provides 10 training tasks and holds out 5 meta-testing tasks.

Training details: We used PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] for training our model, with clip range r = 0.2 and 10 inner optimization steps maximum, for its good empirical performance and in order to compare with other methods with recurrent computation scheme. We kept the experimental settings from the original paper with N=10 episodes of 500 time-steps and sampled proximal gradient update in a meta-batch size of 25 trajectories. To estimate value function, we both tested a feedforward neural network trained at each iteration of the meta-training or an additional MetODS network, which resulted in similar training results.

Figure 1 :

 1 Figure 1: Meta-Reinforcement Learning as a transport problem and MetODS synaptic adaptation: a)Associating any task τ in T to its optimal policy π * τ in Π can be regarded as finding an optimal transport plan from µ T to µ π * with respect to the cost -R. Finding this transport plan is generally an intractable problem. b) Meta-RL approximates a solution by defining a stochastic flow in the policy space Π conditioned by the current task τ and driving a prior distribution µπ 0 of policies π0 towards a distribution µ θ,τ,t

3. 1

 1 Learning reflexive weights updates

Figure 2 :

 2 Figure 2: A layer of MetODS updates its weights by recursive applications of read and write operations based on neural activations v (s) and synaptic traces W (s) .

Figure 3 :

 3 Figure 3: a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. In the Harlow task, we conduct an ablation study by either reducing the number of recursive iterations (S=1) or removing the trainable plasticity weights α resulting in sub-optimal policy. In Ant-dir we compare our agent training profile against MAML and RL 2 . e) We can interpret the learned policy in terms of a Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy E W t (v1, v2) = -v T 1 Wtv2 along two principal components of the vector trajectory vt. In the first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map, which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during a single episode of the Ant-dir task.

Figure 4 :

 4 Figure 4: a) Examples of generated maze configurations. The mazes consist in 8 × 8 pix. areas, with walls and obstacles randomly placed according to a variation of Prim's algorithm [80] with target location randomly selected for the entire duration of the episode (star). The agent receptive field is highlighted in red. b) Comparisons with MAML and RL 2 and effect of element-wise plasticity α. c) Variations of MetODS writing mechanism as well as depth S of recursivity yield different performances (see. S.I for further details)

Figure 6 :

 6 Figure 6: Left Meta-training results for MetaWorld benchmarks. Subplots show tasks success rate over training timesteps. Average meta-test results for MetODS is shown in dotted line. Right Cumulative reward of the Ant and Cheetah directional locomotion task. For each condition, results are normalized against the best performing policy

Figure 7 :

 7 Figure 7: Harlow task: a) An episode of the Harlow task. From left to right: Actions, visual field of the agent (blue squares:fixation target, red squares: values), Rewards (red;negative,light blue: fixation reward, dark blue:value reward) and sum of absolute synaptic variation ∆W per neuron. b) Distribution of weights before and after adaptation over 5 presentations.c) Three instances of the dynamic weights Wt after adaptation that solved the Harlow task. Every synaptic configuration presents differences but perform optimally.

10. 2 Figure 8 :

 28 Figure 8: Reward profiles of MAML, RL 2 and MetODS over 5 consecutive episodes with randomly varying rewarded direction. In Cheetah, MetODS slightly overperforms RL 2 and in Ant, MetODS is the only approach able to reach performance comparable to first episode in subsequent episodes.

Figure 9 :

 9 Figure 9: Meta-training rewards curves of the maze experiment presented in section 5.2

Figure 10 :

 10 Figure 10: Heatmap of activation rate of three neurons of a trained model across 10 different maze maps normalized by agent passage. Activation patterns show selectivity to particular positions within an episode and strong remapping across maze configurations.

  Performance of Meta-RL models tested at convergence (1e 7 env. steps). MetODS better explores the maze as measured by the average number of steps before 1st reward and the success rate in finding the reward at least once. It then better exploits the maze as per the accumulated reward. (* We assign 100 to episodes with no reward encounter.)

		1st rew. * (↓) Success (↑) Cum. Rew. (↑) Cum. Rew (Larger) (↑)
	Random	96.8 ± 0.5	5 %	3.8 ± 8.9	3.7 ± 6.4
	MAML	64.3 ± 39.3	45.2%	14.95 ± 4.5	5.8 ± 10.3
	RL 2	16.2 ± 1.1	96.2%	77.7 ±46.5	28.1 ± 29.7
	MetODS	14.7 ± 1.4	96.6%	86.5 ± 46.8	34.9 ± 34.9
	Figure 5:

Table 1 :

 1 General information: As specified in section 3 and 5, we test a single model definition for all experiments in this work, with one layer of dynamic weights W t . This layer consists in a dense matrix of size n ∈ N with learnt or random initialization. Our model lightweight parametrization of the synaptic update rule makes it a very parameter efficient technique, which can perform batchcomputation and be ported to GPU hardware to accelerate training. We refer readers to table1for specific details of each experiment presented in section 5. In addition to this pdf file, we provide the code for training METODS agents in experiments presented in section 5 at https:/ Summary of training hyper-parameters for the four experiments presented in this work.

	FEATURE -EXPERIMENT	HARLOW	GYM MUJOCO	MAZE	META-WORLD
	DYNAMIC LAYER SIZE n	20	100	200	100
	INPUT SIZE i	12	134 (ANT) / 27 (CHEETAH)	15	I
	OUTPUT SIZE o	2	8 (ANT)/ 6 (CHEETAH)	4	8
	EMBEDDING f	[i × 32, σ, 32 × n]	[i × 64, σ, 64 × n]	[i × 32, σ, 32 × n]	[i × 32, σ, 32 × 64, σ, 64 × n]
	READ-OUT g	[n × 32, σ, 32 × o]	[n × 64, σ, 64 × o]	[n × 32, σ, 32 × o]	[n × 64, σ, 64 × o]
	NON-LINEARITIES	TANH	TANH	TANH	TANH
	INIT. W0	N (0, 1e -3)	LEARNED	LEARNED	LEARNED
	# NUM. OF EPISODES	5 (MAX)	1	1	10
	LENGHT OF 1 EPISODE	250 (MAX)	200	100	500
	META-TRAINING ALG.	A2C	A2C	A2C	PPO
	LEARNING RATE	5E-4	1E-4	5E-4	5E-4
	META-BATCH-SIZE	50	50	20	25
	DISCOUNT FACTOR λ	9E-1	9E-1	9.9E-1	9.9E-1
	GAE	1.	9.5E-1	9.5E-1	9.5E-1
	VALUE FUNCTION COEFF.	4E-1	4E-1	4E-1	-
	ENTROPY REG. FACTOR	3E-2	3E-2	1E-2	1E-2

Table 2 :

 2 Average accumulated reward over 1000 episodes for different maze sizes for agents trained for N=8.

	MAZE SIZE	MAML	RL 2	METODS	REL IMP. TO RL 2
	6	21.0 ± 18.4 149.3 ± 66.7 169.1 ± 66.1	13%
	8	14.9 ± 4.5	72.1 ± 45.6	87.3 ± 48.3	20%
	10	5.7 ± 7.9	28.1 ± 29.7	34.9 ± 34.9	21%
	12	3.9 ± 6.9	11.1 ± 15.8	13.9 ± 19.8	25%

36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Acknowledgments and Disclosure of Funding

This work was supported by ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A-0004), OSCI-DEEP ANR (ANR-19-NEUC-0004), ONR (N00014-19-1-2029), and NSF (IIS-1912280 and EAR-1925481). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV) via NIH Office of the Director grant S10OD025181. The authors would like to thank the anonymous reviewers for their thorough comments and suggestions that led to an improved version of this work.

Supplementary Information : Meta-Reinforcement Learning with Self-Modifying Networks


Amandine Aftalion
email: amandine.aftalion@ehess.fr

Etienne Sandier
email: sandier@u-pec.fr

Solitons and solitonic vortices in a strip

Introduction

The instability of solitons is a phenomenon which has been widely studied both from a mathematical and a physical point of view. A soliton is a one dimensional solitary wave propagating at constant velocity. The simplest situation is actually the case of zero velocity. Solitons have been observed in a large variety of nonlinear media [START_REF] Dauxois | Physics of solitons[END_REF][START_REF] Newell | Solitons in mathematics and physics[END_REF]. The instability has been investigated in nonlinear optics, crystals, and more recently in ultra cold atomic vapors and polaritons. However a full characterization of the instability of solitons is still missing.

Physical motivation

Ultra cold atoms can be controlled at will and therefore allow to study collective behaviours. They are described by a wave function whose phase plays the role of orientation in magnetization. A characteristic feature is the observation of defects [START_REF] Chevy | Solitons with a twist[END_REF]. The simplest defects in cold atoms are dark solitons which are solitary waves that maintain their shape; they correspond to an envelope having a density dip with a π phase shift across its density minimum. Other types of defects are vortices or vortex rings in which the order parameter winds around a hole or a closed loop where the density vanishes. The study of solitons [START_REF] Gravejat | Asymptotic stability of the black soliton for the Gross-Pitaevskii equation[END_REF][START_REF] Bethuel | Asymptotic stability in the energy space for dark solitons of the Gross-Pitaevskii equation[END_REF][START_REF] Tao | Why are solitons stable?[END_REF], traveling waves [START_REF] Bethuel | Travelling waves for the Gross-Pitaevskii equation I[END_REF][START_REF] Bethuel | Travelling waves for the Gross-Pitaevskii equation II[END_REF][START_REF] Mariş | Traveling waves for nonlinear schrödinger equations with nonzero conditions at infinity[END_REF][START_REF] Bellazzini | Finite energy traveling waves for the Gross-Pitaevskii equation in the subsonic regime[END_REF][START_REF] Sánchez | Existence and nonexistence of traveling waves for the Gross-Pitaevskii equation in tori[END_REF] and vortex rings [START_REF] Roberts | Dynamics of vortex rings[END_REF] in the whole space has been at the center of many mathematical works.

Matter waves dark solitons can be created in experiments by means of various methods [START_REF] Frantzeskakis | Dark solitons in atomic Bose-Einstein condensates: from theory to experiments[END_REF]: phase imprinting, density engineering, quantum state engineering which is a combination of the previous two, matter wave interference and dragging an obstacle through a condensate. Several experimental groups attempted to study solitons by imposing a phase shift in an elongated condensate for respectively bosonic atoms, rubidium [START_REF] Burger | Dark solitons in Bose-Einstein condensates[END_REF] and sodium [START_REF] Lamporesi | Spontaneous creation of Kibble-Zurek solitons in a Bose-Einstein condensate[END_REF][START_REF] Donadello | Observation of solitonic vortices in Bose-Einstein condensates[END_REF] and for fermionic atoms (lithium) [START_REF] Yefsah | Heavy solitons in a fermionic superfluid[END_REF][START_REF] Bulgac | Quantized superfluid vortex rings in the unitary fermi gas[END_REF][START_REF] Ku | Motion of a solitonic vortex in the BEC-BCS crossover[END_REF]. From the first picture, they thought they had observed solitons [START_REF] Lamporesi | Spontaneous creation of Kibble-Zurek solitons in a Bose-Einstein condensate[END_REF][START_REF] Yefsah | Heavy solitons in a fermionic superfluid[END_REF]. Further investigations were needed to fully understand the phenomena: in the case of lithium, they realized it was not a soliton but thought it was a vortex ring [START_REF] Bulgac | Quantized superfluid vortex rings in the unitary fermi gas[END_REF], until [START_REF] Ku | Motion of a solitonic vortex in the BEC-BCS crossover[END_REF] argued that in fact it was a single straight vortex called solitonic vortex; in sodium it was also confirmed it was not a soliton but a solitonic vortex [START_REF] Donadello | Observation of solitonic vortices in Bose-Einstein condensates[END_REF].

A solitonic vortex is a vortex in a channel whose transverse size is much smaller than its length and has the same asymptotic phase profile as a soliton in the transverse direction. Indeed, by virtue of the transverse boundaries, the solitonic vortex is exponentially localized in the longitudinal direction on the length scale of the transverse dimension. This is a remarkable property since the energy density of isolated vortices or of vortex-antivortex pairs decays algebrically in the absence of boundaries [START_REF] Bethuel | Asymptotic stability in the energy space for dark solitons of the Gross-Pitaevskii equation[END_REF][START_REF] Bethuel | Travelling waves for the Gross-Pitaevskii equation I[END_REF][START_REF] Bethuel | Travelling waves for the Gross-Pitaevskii equation II[END_REF][START_REF] Jones | Motions in a Bose condensate. iv. axisymmetric solitary waves[END_REF]. Moreover, in the absence of boundaries, vortex-antivortex solutions do not exhibit any phase difference at infinity. Mathematically, as we will see, the exponential localisation and the phase difference between the two ends of the channel result from the infinite and periodic structure of image vortices. In a finite channel, because the vortex loses its long-range nature beyond the transverse length scale, this has justified the name "solitonic vortex" introduced in [START_REF] Brand | Solitonic vortices and the fundamental modes of the "snake instability[END_REF][START_REF] Komineas | Solitons, solitonic vortices, and vortex rings in a confined Bose-Einstein condensate[END_REF][START_REF] Toikka | Asymptotically solvable model for a solitonic vortex in a compressible superfluid[END_REF].

In the experiments, it may be that the solitonic vortex is due to the decay of a soliton after what is called a snake instability. It was observed that the soliton is stable only at sufficiently low particles number or high aspect ratio [START_REF] Komineas | Solitons, solitonic vortices, and vortex rings in a confined Bose-Einstein condensate[END_REF]. The mode associated with snake instability is the solitonic vortex [START_REF] Brand | Solitonic vortices and the fundamental modes of the "snake instability[END_REF]. The metaphor of snake instability has been used to refer to the bending of the solitonic wave front and the decay of the soliton into closed loops of vortices. In a series of studies, Komineas et al [START_REF] Komineas | Solitons, solitonic vortices, and vortex rings in a confined Bose-Einstein condensate[END_REF][START_REF] Komineas | Nonlinear waves in a cylindrical Bose-Einstein condensate[END_REF][START_REF] Komineas | Vortex rings and solitary waves in trapped Bose-Einstein condensates[END_REF][START_REF] Komineas | Vortex rings and Lieb modes in a cylindrical Bose-Einstein condensate[END_REF] analyze how the soliton bifurcates according to the channel length. In some intermediate cases, the soliton initially deforms to become a pair of vortex-antivortex or a vortex ring in 3d. This structure eventually decays into a stable solitonic vortex. The numerics reveal that the vortex-antivortex pair or vortex ring is unstable, but it is sufficiently long lived to be observed both in the numerics and the experiments.

From a mathematical point of view, Rousset and Tzvetkov, [START_REF] Rousset | A simple criterion of transverse linear instability for solitary waves[END_REF], Theorem 3.3, have proved the transverse instability of solitons in the whole space. Nevertheless, nothing is known about the mode of destabilization and whether it turns into a solitonic vortex or a pair of vortices in dimension 2, or a vortex ring in dimension 3. Nothing has been analyzed in a strip yet.

Main result

The aim of this paper is to determine the ground state of the Gross Pitaevskii energy when both a phase imprinting condition and a reduction of density are imposed. In particular, we want to discriminate the cases where the soliton can be a stable minimizer. We will restrict to static solutions, that is for a zero velocity. In the following, we will set the problem in two dimensions.

Let Ω d = R × (-d, d) be a strip. We will consider the width d as a parameter. Let u(x, y) be a minimizer of

E d (u) = Ω d 1 2 |∇u(x, y)| 2 + 1 4 (1 -|u(x, y)| 2 ) 2 dx dy, (1)
under the conditions ū(x, y) = u(-x, y) and u(-x, -y) = -u(x, y).

(

) 2
Here the bar denotes the complex conjugate. The first equation corresponds to what is called the phase imprinting process: one half of the domain is exposed to a blue detuned laser beam that causes a phase shift of the order parameter [START_REF] Brand | Solitonic vortices and the fundamental modes of the "snake instability[END_REF][START_REF] Komineas | Solitons, solitonic vortices, and vortex rings in a confined Bose-Einstein condensate[END_REF]. The second condition corresponds to the fact the density is depleted at a desired location [START_REF] Fritsch | Creating solitons with controllable and near-zero velocity in Bose-Einstein condensates[END_REF], here the origin. In order to impose the fact that the wave function vanishes at the origin, we have decided to impose imparity of the wave function. These two conditions are equivalent to Re u(-x, y) = Re u(x, y), Im u(-x, y) = -Im u(x, y),

Re u(x, -y) = -Re u(x, y), Im u(x, -y) = Im u(x, y).

Because of the finite energy condition, |u| tends to 1 at infinity, and by the parity, imparity conditions, in fact we will show that the behaviour at infinity is lim

x→-∞ u(x, y) = -i and lim x→+∞ u(x, y) = i, (5)
for either u or ū. Any ground state is a solution of

-∆u + u(1 -|u| 2 ) = 0 (6)
with the conditions (2)-( 5) and ∂u/∂n = 0 on y = ±d. There are special solutions to this equation:

• solitons which are independent of y and are thus one dimensional solutions, i tanh(x/ √ 2),

• solitonic vortices which have a zero at the origin and a degree 1.

We are going to prove that these solutions characterize the ground states.

Theorem 1. There exists a minimizer u of (1) under the conditions [START_REF] Newell | Solitons in mathematics and physics[END_REF]. The behaviour at infinity is given by ( 5) either for u or ū. If d is sufficiently small, the minimizer is independent of y and is the soliton solution i tanh(x/ √ 2). If d is sufficiently large, any minimizer has a vortex of degree +1 or -1. Taking the complex conjugate if necessary, we may assume the vortex has degree +1. If u d (x, y) denotes such a minimizer in

Ω d = R × (-d, d), and if we let v d (x, y) = u d (dx, dy), (7)
then v d , defined in Ω 1 , has the following behaviour:

• the map v d converges as d → +∞, in C 1 (K) for any compact subset K of Ω1 \ {0}, to the map e iϕ 0 , where

ϕ 0 = arctan sinh( πx 2 ) sin( πy 2 )
.

• as d → +∞,

E d (u d ) = π log d + γ -π log π 4 + o(1). ( 9
)
where the constant γ is defined in [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF] as

γ = lim r→+∞ (E 1 (u 0 , B(0, r)) -π log r) ,
where u 0 is the unique radial solution of

-∆u = u(1 -|u| 2 ) in R 2 of degree +1.
Note that it is equivalent to use the width d as a parameter or to put a coefficient 1/d 2 in front of the term (1 -|u| 2 ) 2 using the change of function [START_REF] Bethuel | Travelling waves for the Gross-Pitaevskii equation I[END_REF].

The first part of the Theorem is proved using the Maximum Principle in narrow domains which provides uniqueness. For the second part, we introduce the stream function w 0 , which is such that ϕ 0 is its imaginary part, and the exponential of the harmonic conjugate of ϕ 0 is its modulus. More precisely, h 0 = log |w 0 |, where

w 0 (z) = i tanh π 4 z (10)
and h 0 is the harmonic conjugate of ϕ 0 . Once the functions for the strip are computed, the proof follows the main ideas of the analysis of vortices in the seminal book by Bethuel, Brezis, Hélein [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF] and the analysis of vortex balls by Sandier [START_REF] Sandier | Lower bounds for the energy of unit vector fields and applications[END_REF] and Jerrard [START_REF] Jerrard | Lower bounds for generalized Ginzburg-Landau functionals[END_REF].

Further questions

Let us point out that without the second condition (2), there is a minimizing sequence whose energy tends to 0, namely, for large h, the functions u h (x) = e iπx 2h which are defined in (-h, h) and extended by i and -i outside. Therefore, to have the existence of a minimizer, we have to impose extra conditions, and in particular one corresponding to a density depletion in the experiments. We have chosen imparity of the wave functions but probably other conditions compatible with H 1 functions and density depletion could be sought.

As d is increased, the soliton loses stability. A further question would be to analyze the eigenfunctions of the linearized operator around the soliton solution, and in particular understand whether the first direction of instability is around a vortex/antivortex solution.

Related issues include the study of minimizers of the energy in the product spaces R n × M k , where M k is a compact Riemannian manifold under an L 2 constraint. It is proved in [START_REF] Terracini | The nonlinear Schrödinger equation ground states on product spaces[END_REF] that when the L 2 norm of the solution is sufficiently small, then the ground states coincide with the corresponding R n ground states, that is in our case, the soliton for a small torus. They also prove that above a critical mass the ground states have nontrivial M k dependence.

The problem studied in this paper corresponds to what is known as zero velocity. Further works could be performed by minimizing the energy with fixed velocity or minimizing the energy with fixed momentum. The analysis of periodic solutions in the torus has been the topic of a very nice recent paper [START_REF] Sánchez | Existence and nonexistence of traveling waves for the Gross-Pitaevskii equation in tori[END_REF], where according to the size of the period, a mountain pass solution is constructed. Such methods could probably be generalized here in the case of the strip.

In the experimental papers [START_REF] Claude | Taming the snake instabilities in a polariton superfluid[END_REF] and [START_REF] Koniakhin | Stationary quantum vortex street in a driven-dissipative quantum fluid of light[END_REF], in a one dimensional channel of polariton superfluids, the soliton breaks into arrays of vortex-antivortex. The mathematical analysis is probably similar though the equation there includes a forcing term and a dissipation term. The stationary one dimensional solutions have been studied in [START_REF] Hakim | Metamorphoses of the flow past an obstacle of a resonantly-driven bistable polariton fluid[END_REF] and the analysis of the the solitonic vortices in this context is an interesting open question.

Proof of Theorem 1

Existence of a minimizer and asymptotic properties

Using the soliton as a comparison function, we find that the minimal energy is finite and is bounded above by a constant times d. From any minimizing sequence, we may extract a subsequence which converges weakly in H 1 loc to a limiting u which satisfies the symmetry assumptions and therefore is a minimizer of (1) under conditions [START_REF] Newell | Solitons in mathematics and physics[END_REF]. It also satisfies Neumann boundary conditions on the lines {y = ±d}.

From the energy minimality, |u| ∞ ≤ 1 in the strip and by elliptic regularity, the gradient of u is uniformly bounded in the strip, and tends to 0 as x → ±∞. This first implies that |u| → 1 as x → ±∞: indeed, the gradient bound implies that if |u(z)| < 1 -η for some η > 0, then |u| < 1 -η/2 on a ball of radius r η > 0, and therefore the energy of u on such a ball is bounded below by some c η > 0. This cannot happen for arbitrarily large z since the energy is finite. Hence |u(x, y)| → 1 as x → ±∞.

Moreover, since the gradient tends to 0 in the infinite direction, we have that, as [START_REF] Newell | Solitons in mathematics and physics[END_REF] imply that u(x, 0) is purely imaginary. Since |u(x, 0)| tends to 1 as x → ±∞, it follows that for any x such that |x| is large enough depending on η > 0, we have that |u(x, y) -i| < η or |u(x, y) + i| < η. Therefore u(x, y) converges to either i or -i as x → ±∞ and, using (2) again, that is u(x, y) = ū(-x, y), the limits are opposite of one another.

x → ±∞, u(x, •) -u(x, 0) L ∞ (-d,d) → 0. But conditions
Remark 1. Let us point out that because of (2) we have u(0, 0) = 0, therefore the energy minimum is not zero. Mathematically, the imparity condition (the second condition in (2)), can be viewed as a way to exclude an energy minimum of zero. It is an open question to choose a space of minimization including the first condition of (2) with limits i and -i at plus and minus infinity, and such that u vanishes somewhere.

Uniqueness for small d

As a first step, we want to prove that when d is small, the minimizer is purely imaginary, that is u 1 = Re u ≡ 0. We have, in the strip,

-∆u 1 + u 1 (|u| 2 -1) = 0, (11)
together with Neumann boundary conditions on the lines {y = ±d}. Because of ( 3)-( 4), u 1 (x, 0) has to be both odd and even so that we have u 1 (x, 0) = 0.

From the Neumann boundary conditions, we have ∂ y u 1 (x, ±d) = 0. We define ũ1 (x, y) = u 1 (x, y) for y ∈ (0, d) and ũ1 (x, y) = u 1 (x, d -y) for y ∈ [d, 2d). It follows that ũ1 is a solution of (11) on the strip R × (0, 2d) such that ũ1 = 0 on the boundary y = 0 and y = 2d.

We know that ũ1 tends to 0 at infinity since u tends to ±i. Moreover, it follows from the equation, as proved in [START_REF] Bethuel | Asymptotics for the minimization of a ginzburg-landau functional[END_REF], that |u| ≤ 1. Therefore, ũ1 is a solution of -∆ũ 1 + c(x)ũ 1 = 0 with c(x) uniformly bounded. For d small enough, the Maximum Principle in narrow domains [START_REF] Berestycki | On the method of moving planes and the sliding method[END_REF][START_REF] Berestycki | Generalizations and properties of the principal eigenvalue of elliptic operators in unbounded domains[END_REF][START_REF] Berestycki | The principal eigenvalue and maximum principle for second-order elliptic operators in general domains[END_REF] applies and implies that ũ1 cannot reach a positive maximum or a negative minimum in R × (0, 2d): the proof consists in writing the equation satisfied by w = ũ1 /g where g = cos(αy) where α = π/(8d) is chosen such that its square is bigger than the L ∞ bound of c(x). We have

-∆w -2 ∇g g • ∇w + (α 2 + c(x))w = 0.
Since ∂w/∂n is zero on the boundary and w tends to zero at infinity, and satisfies an equation for which the Maximum Principle holds, w cannot reach a positive maximum or a negative minimum and is identically zero. Therefore ũ1 is identically zero and u is purely imaginary. Knowing that the solution is purely imaginary, its energy on each line {y} × R can be bounded below by that of the soliton i tanh(x/ √ 2), which minimizes the energy among purely imaginary competitors satisfying the boundary conditions. Integrating with respect to y, we deduce that the energy of u is bounded below by 2d times the energy of the soliton, and is bounded above by the same quantity since u is a minimizer. It follows that ∂ y u is identically zero, and that the restriction of u to any horizontal line is a (possibly translated) soliton. Since ∂ y u = 0, the translation is independent of y, and since u(0, 0) = 0, hence u is the soliton.

Upper bound for the energy of a vortex for large d and introduction of the solitonic vortex

We now focus on the asymptotics of minimizers of E d for large d. We begin by computing an upper-bound for the minimal energy, for which a matching lower-bound will be proven below.

To prove the upper bound, we construct a test function which is an approximate solution with a vortex at the origin. A first natural test function is u(z) = iz if |z| ≤ 1 and u(z) = iz/|z| if |z| > 1. It is not difficult to check that this yields the correct upper-bound but only up to a O(1) term. Indeed, the integral on the strip, outside the unit disk is, in polar coordinates, The inner integral is equivalent to 2d/r 2 as r → +∞, hence is convergent, but this does not provide the optimal constant.

In order to be more precise, we construct a test function with a vortex of degree one satisfying Neumann boundary conditions. This can be obtained by superimposing the known solution for a vortex without boundaries with those of image vortices. This provides a condition of no normal flow to the boundary for the gradient of the phase. While a single straight wall requires a single image vortex, the two parallel walls of the channel generate a doubly infinite array of image vortices. The problem can be solved on the complex plane by introducing the meromorphic function w having zeros (resp. poles) at the location of vortices of degree +1 (resp. -1) [START_REF] Greengard | Potential flow in channels[END_REF][START_REF] Barton | Elements of Green's functions and propagation: potentials, diffusion, and waves[END_REF][START_REF] Toikka | Asymptotically solvable model for a solitonic vortex in a compressible superfluid[END_REF]. Then the phase is the argument of w and the harmonic conjugate of the phase is the logarithm of its modulus. The advantage of considering the latter is that it has the same gradient energy as the phase but is single valued.

The solution w for the strip with a vortex at z 0 is given in [START_REF] Greengard | Potential flow in channels[END_REF] for a strip (0, H) and is sinh π

2H (z -z 0 ) sinh π 2H (z -z0 )
.

It is found using a conformal mapping between the strip and the disc. Mapping the interval (0, H) to (-d, d) yields for the strip Ω d with a vortex of degree +1 at z = ia

w a (z) = sinh π 4d (z -ia) sinh π 4d (z + ia + 2id) = i sinh π 4d (z -ia) cosh π 4d (z + ia) , (12)
so that, if a = 0,

w 0 (z) = i tanh πz 4d . ( 13
)
The function w is plotted in Figure 1, and we see that it behaves as a soliton far away from the core. The phase S of a function -i log α+iβ γ+iδ is such that tan S = βγ-αδ αγ+βγ so that the phase ϕ a is given from ( 12) by

tan ϕ a (x, y) = sinh( πx 2d ) cos( πa 2d ) sin( πy 2d ) -cosh( πx 2d ) sin( πa 2d ) . ( 14
)
This formula yields that the phase difference between x = -∞ and x = +∞ is related to the location of the vortex and is

π -π a d .
So in order to have a phase difference of π (between -i and i), the vortex has to be at the origin, that is a = 0. Then, the phase ϕ 0 and its harmonic 15)

h 0 (x, y) = log tanh( π(x + iy) 2d ) = 1 2 log cosh( πx 2d ) -cos( πy 2d ) cosh( πx 2d ) + cos( πy 2d ) . ( 16
)
The test function u is now obtained as follows : first, we let u(x, y) = v(x/d, y/d), so that v is defined in Ω 1 instead of Ω d . We let ε = 1/d and choose ρ > 0 which will be eventually chosen small, but large compared to ε. We have, as noted earlier,

E d (u) = F ε (v) := 1 2 Ω 1 |∇v| 2 + 1 4ε 2 1 -|v| 2 2 . ( 17
)
In D ρ = Ω 1 \ B ρ , we define v in agreement with the discussion above, that is v = e iϕ 0 , where ϕ 0 is defined in [START_REF] Lamporesi | Spontaneous creation of Kibble-Zurek solitons in a Bose-Einstein condensate[END_REF], with d chosen equal to 1. We thus have, in D ρ ,

|∇v| 2 = |∇ϕ 0 | 2 = |∇h 0 | 2 .
To compute the energy of v in D ρ , we first note that the potential term in [START_REF] Yefsah | Heavy solitons in a fermionic superfluid[END_REF] vanishes since |v| = 1 there, and then we integrate by parts as in [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF] to find that

Dρ |∇v| 2 = Dρ |∇h 0 | 2 = ∂Bρ h 0 ∂ ν h 0 ,
where ν is the normal to the boundary pointing toward the center of the ball, and where we have used the fact, easily deduced from [START_REF] Donadello | Observation of solitonic vortices in Bose-Einstein condensates[END_REF], that h 0 = 0 on ∂Ω 1 . Note also that, still from ( 13), the derivative of h 0 with respect to x tends to 0 at infinity exponentially fast so that there are no boundary terms at infinity.

From [START_REF] Frantzeskakis | Dark solitons in atomic Bose-Einstein condensates: from theory to experiments[END_REF] we have, for small ρ,

w 0 (ρe iθ ) = i π 4 ρe iθ 1 + O(ρ 2 ) .
Therefore, as ρ → 0,

h 0 (ρe iθ ) log( πρ 4 ), ∂ ν h 0 (ρe iθ ) - 1 ρ .
It follows after some calculations that, for ρ small,

1 2 Dρ |∇h 0 | 2 = -π log( πρ 4 ) + o(1). (18)
To define v inside B ρ , we note that, since h 0 is the harmonic conjugate to ϕ 0 and since ∂ ν h 0 -1/ρ on ∂B ρ , the phase φ 0 is close to θ + c as ρ → 0, for some constant c. We may then define v in B ρ to be the minimizer of F ε in B ρ with Dirichlet boundary condition e i(θ+c) on the boundary, modified slightly to match the definition of v outside B ρ . Following [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF], such a minimizer will have energy γ + π log(ρ/ε) + o(1) if ε/ρ is small and thus, adding (18) we find.

E d (u) = F ε (v) = π log ρ ε + γ -π log ρπ 4 + o(1) = π log d + γ -π log π 4 + o(1), (19)
where o(1) is small if both ρ and ε/ρ are small. Both are true if d is large and if we choose, for instance, ρ = 1/ √ d.

Lower bound and proof of Theorem 1 completed

Let u d be a minimizer for the energy on the strip Ω d . We have

E d (u d ) ≤ E d (u)
, where u was constructed in the previous section and satisfies [START_REF] Ku | Motion of a solitonic vortex in the BEC-BCS crossover[END_REF]. We let as above ε = 1/d and v ε (x, y) = u d (dx, dy).

Then E d (u d ) = F ε (v ε ), so that F ε (v ε ) ≤ π log 1 ε + γ -π log π 4 + o(1). (20)
We use the following covering property, which can be deduced straightforwardly from ( [START_REF] Sandier | Lower bounds for the energy of unit vector fields and applications[END_REF], [START_REF] Jerrard | Lower bounds for generalized Ginzburg-Landau functionals[END_REF]). Proposition 2. Assume v ε satisfies [START_REF] Jones | Motions in a Bose condensate. iv. axisymmetric solitary waves[END_REF] and that |v| tends to 1 as x → ±∞. Then, for any ε < r < 1, the strip Ω 1 = R × (-1, 1) may be covered by closed disjoints disks D 1 , . . . , D k of total radius r, such that |v ε | > 1/2 outside the disks and such that the following lower bound holds on each disk:

F ε (v ε , D i ) ≥ π|n i | log r ε -C . (21)
Here n i (r) is the degree of the mapping v ε /|v ε | from the circle ∂D i to the unit circle, or zero if D i is not contained in the strip.

Note that, since v ε (0, 0) = u d (0, 0) = 0, there is at least one disk and the origin belongs to one of the disks.

For any fixed r > 0, comparing [START_REF] Jones | Motions in a Bose condensate. iv. axisymmetric solitary waves[END_REF] with [START_REF] Brand | Solitonic vortices and the fundamental modes of the "snake instability[END_REF], we find that

i |n i | ≤ log(1/ε) + C log(r/ε) -C . It follows that i |n i | ≤ 1 if ε is small enough (corresponding to d large enough)
, and then that there is at most one disk with nonzero degree, and that, if it exists, its degree is either +1 or -1. We claim that 1. A disk, call it D 0 , such that the degree on ∂D 0 = ±1 exists. 2. D 0 must be the disk containing the origin.

To prove the second claim we argue by contradiction. If 0 ∈ D 0 , then D 0 is disjoint from D 1 = {-z | z ∈ D 0 }. But the energy of v ε on D 1 is the same as its energy on D 0 , from (2), therefore the total energy of v ε is at least twice that on D 0 , which contradicts the upper bound if ε is small enough.

To prove the first claim we argue again by contradiction. Assume all disks have degree zero, then choose a circle of radius t centered at the origin and not intersecting the other disks -this is possible if r < 1. Then the degree n of v ε on {|z| = t} must be zero. This is impossible because n is odd: Indeed, denoting by φ the phase of v ε and θ the polar angle on the circle, the conditions (2) mean that, modulo 2π, we have φ(π -θ) = -φ(θ) and φ(θ

+ π) = φ(θ) + π. It follows that 0 -π φ = π 0 φ , π 0 φ = π + 2kπ,
and therefore the integral of φ over the interval (0, 2π) belongs to 2π+4πZ or, equivalently, the degree n is odd. We assume below without loss of generality that the degree is +1

The rest of the lower bound follows the well-known line of arguments found in [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF]. Comparing again [START_REF] Jones | Motions in a Bose condensate. iv. axisymmetric solitary waves[END_REF] and [START_REF] Brand | Solitonic vortices and the fundamental modes of the "snake instability[END_REF], we find that for any choice of r, the energy of v ε outside a disk of radius smaller than r which contains the origin remains bounded as ε → 0. Therefore, modulo a subsequence, v ε converges outside the origin (weakly in H 1 loc (Ω 1 \ {0} to be precise) to a map v 0 which must be of constant modulus 1, must be harmonic, and must satisfy a Neumann boundary condition on ∂Ω 1 . These properties follow from the energy bound satisfied by v ε and the fact that it is a minimizer, hence a critical point, of F ε . The degree condition above further implies that in fact v 0 is the canonical harmonic map in Ω 1 with Neumann boundary conditions and a single singularity of degree +1. We deduce that v 0 = e iϕ 0 is exactly as above. The fact that the convergence holds in fact in C 1 norm, locally in Ω1 \ {0} is proved in [START_REF] Bethuel | Ginzburg-Landau vortices[END_REF], Theorem VI.1.

Then, for any fixed ρ > 0, we have

lim inf ε→0 F ε (v ε , D ρ ) ≥ 1 2 Dρ |∇ϕ 0 | 2 = -π log( πρ 4 ) + o ρ (1),
where o ρ (1) denotes a quantity which tends to 0 when ρ → 0. On the other hand, since v ε converges to e iϕ 0 on ∂B ρ , its energy in B ρ is bounded below by that of the minimizer of F ε on B ρ with boundary data e iϕ 0 , which is equal to that of the minimizer with boundary condition e iθ , up to a o ρ (1) error. Thus

lim inf ε→0 F ε (v ε , B ρ ) ≥ γ + π log ρ ε + o ρ (1).
Adding the two lower bounds we find

F ε (v ε ) ≥ π log 1 ε + γ -π log( π 4 ),
which matches [START_REF] Jones | Motions in a Bose condensate. iv. axisymmetric solitary waves[END_REF] as ε → 0 and thus concludes the proof of Theorem 1.

Computation of the renormalized energy of a vortex-antivortex pair

We have seen that for d small, the ground state is a soliton, and for d large, a solitonic vortex. In addition to the minimizers described above, and as we mentioned in introduction, experimental evidence and numerical simulations indicate that in the case of intermediate d, the soliton solution destabilizes into a vortex-antivortex pair. It is not clear how to prove the existence of such solutions, or wether they are the solution of some variational problem. However we are going to provide some computations for a vortexantivortex solution. A rigorous proof might follow the so-called "gluing" approach initiated in [START_REF] Pacard | Linear and Nonlinear Aspects of Vortices: The Ginzburg-Landau Model[END_REF] for Ginzburg-Landau vortices or from the Moutain Pass solutions of [START_REF] Bellazzini | Finite energy traveling waves for the Gross-Pitaevskii equation in the subsonic regime[END_REF]. Proposition 3. When d is large, there is a test function with a vortex at (0, d/2) and an anti-vortex at (0, -d/2) whose energy is asymptotically

2π log d -2π log( π 2 ) + 2π log(sin πa d ) + 2γ + o(1). (22)
We compute the energy of an approximate solution u a in Ω d = R×(-d, d), having a vortex located at p + = (0, a) and an antivortex at p -= (0, -a).

The assumption is that d is large compared to the size of the vortex core, and that the distance between vortices as well as their distance to the boundary is also large with respect to the size of the core. Denoting by ρ the core size, this means that ρ a and ρ d -a.

We approximate u a outside B(p + , ρ) ∪ B(p -, ρ) by a map of modulus one, i.e. we set u a = e iϕa there, with ϕ a denoting the phase, defined only modulo 2π. Then the energy density for u a reduces to |∇ϕ a | 2 and therefore, since we are looking for an energy minimizing solution with fixed vortices, we assume ϕ a is harmonic. Under the assumption [START_REF] Toikka | Asymptotically solvable model for a solitonic vortex in a compressible superfluid[END_REF], the energy inside the core may be assumed to be independent of the precise vortex location and is for two vortices 2γ + 2π log(ρ) + o(1).

Therefore we will now consider the energy outside the core as a function of a.

The harmonic conjugate function to ϕ a , call it h a , is single-valued. Since ϕ a satisfies a Neumann condition on ∂Ω d , the function h a satisfies a Dirichlet condition. In principle h a could take two different values on the top and bottom boundary, but this would give an infinite energy. Therefore the value is the same and h a solves the following equation: Note that the exact equation inside the vortex cores is unimportant for this calculation, since we wish to compute the energy away from the cores. This is why we introduce δ-functions.

∆h a = 2π(δ p + -δ p -) in Ω d h a = 0 on ∂Ω d (25)
There is an explicit formula for h a that corresponds a to single vortex in a double interval. We can in fact deduce it from [START_REF] Roberts | Dynamics of vortex rings[END_REF]: the meromorphic function corresponding to a vortex of degree +1 at ia and a vortex of degree -1 at -ia is w a /w -a , hence h a (x, y) = log |W (x + iy)|, where W (x + iy) = sinh π 2d x cos π 2d (y -a) + i cosh π 2d x sin π 2d (y -a) sinh π 2d x cos π 2d (y + a) + i cosh π 2d x sin π 2d (y + a)

, which may also be written as

W (z) = sinh π 2d (z -ia) sinh π 2d (z + ia) . ( 26
)
The function W is plotted in Figure 2. Let us point out that the function W is the same as the one for a single vortex in a with 4d replaced by 2d, which is consistent with the image vortices. The gradient of the phase is plotted in Figure 3.

To compute the energy outside the core we integrate by parts as in [

h a ∂ ν h a ,
where ν is the normal to the boundary point toward the center of the ball. Note that h a tends to 0 at infinity and its derivative with respect to x tends to 0, so that there are no terms at infinity. From [START_REF] Komineas | Vortex rings and Lieb modes in a cylindrical Bose-Einstein condensate[END_REF]

Let us point out that in the small ρ approximation, this energy has a maximum for a = d/2, but on the other hand if we want to have the same structure as the soliton, that the limits are ±i at ±∞, the only value of a is a = d/2. Indeed the formula for the phase is the same as ( 14) with d replaced by d/2 and a replaced by d/2 -a. Note that in the whole space, the vortex anti-vortex solution does not have a phase difference between two infinite directions. We recollect ( 24) and ( 27) to find [START_REF] Komineas | Solitons, solitonic vortices, and vortex rings in a confined Bose-Einstein condensate[END_REF]. We see that when d is large, ( 22) provides a higher energy than ( 9), but it may be that for intermediate d, the vortex anti-vortex solution is the ground state.

Figure 1 :

 1 Figure 1: The function w 0 for a vortex at the origin.

Figure 2 :

 2 Figure 2: The function W for a vortex/antivortex pair.

Figure 3 :

 3 Figure 3: The gradient of the phase for a vortex, antivortex pair.

  we have, for small r, ∂B(p + , ρ), ∂ ν h a /∂ν = 1/ρ. It follows after some calculations that, for ρ small,

	and on 1 2 Dρ	|∇h a | 2 = -2π log(	πρ 2d	) + 2π log(sin	πa d	) + o(1).
	W (ia + re iθ ) = i	π 2d	re iθ sin	πa d	-1	+ O(r 2 ),
	W (-ia + re iθ ) -1 = -i	π 2d	re iθ sin	πa d	-1	+ O(r 2 ).
	Therefore,	h a (ia + re iθ ) = log(	πρ 2d	) -log(sin	πa d	)

Acknowledgements: The authors would like to thank Frédéric Chevy for discussions on the physics of solitons and solitonic vortices. This project was supported by the grant CNRS-Prime TraDisQ1D.