1 statistical physics meets computational neuroscience 1 2 the low-dimensional manifold hypothesis 3 2.1 In machine learning 3 2.2 In computational neuroscience 5 3 recurrent neural neural networks and attractors 7 3.1 Discrete and continuous attractors in recurrent neural networks 7 3.2 Hopfield model: multiple point-attractors 9 3.2.1 Ingredients of the model 10 3.2.2 Model details and properties 13 3.2.3 Why is it important to study simple models? 15 3.3 Representation of space in the brain 17 3.3.1 Hippocampus 17 3.3.2 Place cells and place fields 18 3.4 Storing a single continuous attractor 21 3.4.1 Tsodyks and Sejnowsky's model 22 3.4.2 CANN and statistical physics: Lebowitz and Penrose's model 23 3.4.3 Continuous attractors and population coding 26 3.5 Why is it necessary to store multiple continuous attractors? 28 3.6 The case of multiple continuous attractors 29 3.6.1 Samsonovich and McNaugthon's model 30 3.6.2 Rosay and Monasson's model 30 3.7 Issues with current theory 34 3.8 Experimental evidences for continuous attractors 37 3.8.1 Head-direction cells 37 3.8.2 The fruit fly central complex 38 3.8.3 Grid cells 39 3.8.4 Prefrontal cortex 40 3.8.5 Other examples 40 4 optimal capacity-resolution trade-off in memories of multiple continuous attractors 43 4.1 Introduction 43 4.2 The Model 44 4.3 Learning the optimal couplings 45 4.4 Results of numerical simulations 48 4.4.1 Couplings obtained by SVM 49 4.4.2 Finite temperature dynamics (T > 0) 49 xiii xiv contents 4.4.3 Zero temperature dynamics (T = 0) and spatial error 52 4.4.4 Comparison with Hebb rule 55 4.4.5 Capacity-Resolution trade-off 55 4.5 Gardner's theory for RNN storing spatially correlated patterns 56 4.6 Quenched Input Fields Theory 62 4.6.1 Replica calculation 64 4.6.2 Log. volume and saddle-point equations close to the critical line 67 4.6.3 Large-p behavior of the critical capacity 69 5 spectrum of multi-space euclidean random matrices 73 5.1 Introduction 73 5.2 Spectrum of MERM: free-probability-inspired derivation 75 5.2.1 Case of the extensive eigenvalue -k=0 76 5.2.2 Case of a single space (L=1) 77 5.2.3 Case of multiple spaces (L = αN) 78 5.3 Spectrum of MERM: replica-based derivation 80 5.4 Application and comparison with numerics 83 5.4.1 Numerical computation of the spectrum 83 5.4.2 Merging of density "connected components": behavior of the density at small α 84 5.4.3 Eigenvectors of MERM and Fourier modes associated to the ERMs 85 6 towards greater biological plausibility 89 6.1 Introduction 89 6.2 Border effects 91 6.3 Positive couplings constraint 92 6.3.1 Couplings obtained by SVMs with positive weights constraint 93 6.3.2 Stability obtained by SVMs with positive weights constraint 94 6.3.3 Adding the positive weights constraint in Gardner's framework 94 6.4 Variants of the place cell model 101 6.4.1 Dilution 101 6.4.2 Place fields of different volumes 102 6.4.3 Multiple place fields per cell in each space 103 6.4.4 Putting everything together 104 6.5 Individuality of neurons 105 6.6 Non uniform distribution of positions 107 6.7 Comparison between SVM and theoretical couplings 109 6.8 Dynamics of learning 110 6.9 Learning continuous attractors in RNN from real images 113 7 conclusions 119 a appendix-chapter 4 125 a.1 Support vector machine learning 125 contents xv a.2 Estimation of critical capacity from SVMs results 129 a.3 Recovering Gardner results in case of one position per map 129 a.4 Computation of Ξ(U) 131 a.5 Dependence on φ 0 and D. 132 b appendix-chapter 5 135 b.1 Resolvent, Blue function and R-transform 135 b.2 Free probability theory in a nutshell 137 c appendix-chapter 6 139 c.1 SVM algorithm with sign-constrained synapses 139 c.2 Analytical details on the individuality of neurons 141 c.2.1 Spectrum of MERM: multi-populations of neurons 141 c.2.2 Quenched PF theory: multi-populations of neurons 144 c.2.3 Spectrum of MERM: multi-populations of neurons (multiple PFs per neuron on a map) 149 c.2.4 Quenched PF theory: multi-populations of neurons (multiple PFs per neuron on a map) 153 c.3 Adatron algorithm

A C K N O W L E D G M E N T S

After a very long journey the time for thanks has come, which makes me a little sad because it is a concrete sign that this beautiful adventure has come to an end.

In fact, I spent three wonderful years in the Physics Laboratory of the ENS (and in general in Paris) that have deeply changed me (hopefully for the better) both from a scientific and personal point of view, and all this is mainly due to the colleagues and friends with whom I shared this experience.

So, I take this opportunity to thank them, even if those who know me are aware that I am not good at expressing my feelings in words. I hope to be able to communicate even a small part of the deep sense of gratitude they deserve. I can only begin by deeply thanking my supervisor Rémi Monasson who opened the doors to this trip. I am truly grateful to him first of all for making me passionate about neuroscience and for showing me how statistical physics can be a really powerful weapon in the study of problems at first sight far from physics. I thank him for the continuous technical and personal support that I have received during these years and that I wish every Ph.D. student to have. It is obvious to say that without him this work would be nothing but a distant mirage and for this I will always be extremely grateful! A sincere thanks also goes to Simona Cocco and to all the fantastic colleagues with whom I had the pleasure to work in the same research group during these years. All of them were always available to discuss about science and this helped me a lot, but I also wanted to express my gratitude to them for their friendship. All this has made the working group actually become a second family and the laboratory a second home and for this I am extremely thankful to everyone. I would also like to take this occasion to thank Alessandro Treves and his students for the good times spent in Trieste, which were very stimulating from a scientific and human point of view. Moreover, a special thanks goes to my office mates, especially for tolerating me during these three long years, but also for the countless scientific discussions that have been crucial for me.

Thanks also to my mentor and tutor, respectively Francesco Zamponi and Jean-Pierre Nadal, for having followed the evolution of this work and for their advice during this path. Thanks a lot even to the members of the committee for accepting with great enthusiasm to correct my thesis and for the valuable suggestions that have certainly improved the quality of this work.

iii Furthermore, I would like to take this opportunity to warmly thank all my friends, from the Parisians to the Romans and the ones of my homeland (Ciociaria) without whom I would certainly not be the person I am but above all for having supported me all these years.

Finally, it is my duty and pleasure to thank my whole family for always being supportive in my choices and for the affection I receive from them constantly, especially thanks to my twin brother Pietro with whom I have shared most of my life (and will continue to do so). iv A B S T R A C T How sensory information is encoded and processed by neuronal circuits is a central question in computational neuroscience. In many brain areas, the activity of neurons is found to depend strongly on some continuous sensory correlate; examples include simple cells in the V1 area of the visual cortex coding for the orientation of a bar presented to the retina, and head direction cells in the subiculum or place cells in the hippocampus, whose activities depend, respectively, on the orientation of the head and the position of an animal in the physical space.

Over the past decades, continuous attractor neural networks were introduced as an abstract model for the representation of a few continuous variables in a large population of noisy neurons. Through an appropriate set of pairwise interactions between the neurons, the dynamics of the neural network is constrained to span a low-dimensional manifold in the high-dimensional space of activity configurations, and thus codes for a few continuous coordinates on the manifold, corresponding to spatial or sensory information.

While the original model was based on how to build a single continuous manifold in an high-dimensional space, it was soon realized that the same neural network should code for many distinct attractors, i.e., corresponding to different spatial environments or contextual situations. An approximate solution to this harder problem was proposed twenty years ago, and relied on an ad hoc prescription for the pairwise interactions between neurons, summing up the different contributions corresponding to each single attractor taken independently of the others. This solution, however, suffers from two major issues: the interference between maps strongly limit the storage capacity, and the spatial resolution within a map is not controlled.

In the present manuscript, we address these two issues. We show how to achieve optimal storage of continuous attractors and study the optimal trade-off between capacity and spatial resolution, that is, how the requirement of higher spatial resolution affects the maximal number of attractors that can be stored, proving that recurrent neural networks are very efficient memory devices capable of storing many continuous attractors at high resolution.

In order to tackle these problems we used a combination of techniques from statistical physics of disordered systems and random matrix theory. On the one hand we extended Gardner's theory of learning to the case of patterns with strong spatial correlations. On the other hand we introduced and studied the spectral properties of a new ensemble of v random matrices, i.e., the additive superimposition of an extensive number of independent Euclidean random matrices in the high-density regime.

In addition, this approach defines a concrete framework to address many questions, in close connection with ongoing experiments, related in particular to the discussion of the random remapping hypothesis and to the coding of spatial information and the development of brain circuits in young animals. Finally, we discuss a possible mechanism for the learning of continuous attractors from real images. vi R É S U M É La manière dont l'information sensorielle est codée et traitée par les circuits neuronaux est une question centrale en neurosciences computationnelles. Dans de nombreuses régions du cerveau, on constate que l'activité des neurones dépend fortement de certains corrélats sensoriels continus ; on peut citer comme exemples les cellules simples de la zone V1 du cortex visuel codant pour l'orientation d'une barre présentée à la rétine, et les cellules de direction de la tête dans le subiculum ou les cellules de lieu dans l'hippocampe, dont les activités dépendent, respectivement, de l'orientation de la tête et de la position d'un animal dans l'espace physique.

Au cours des dernières décennies, les réseaux neuronaux à attracteur continu ont été introduits comme un modèle abstrait pour la représentation de quelques variables continues dans une grande population de neurones bruités. Grâce à un ensemble approprié d'interactions par paires entre les neurones, la dynamique du réseau neuronal est contrainte de s'étendre sur une variété de faible dimension dans l'espace de haute dimension des configurations d'activités, et code ainsi quelques coordonnées continues sur la variété, correspondant à des informations spatiales ou sensorielles.

Alors que le modèle original était basé sur la construction d'une variété continue unique dans un espace à haute dimension, on s'est vite rendu compte que le même réseau neuronal pouvait coder pour de nombreux attracteurs distincts, correspondant à différents environnements spatiaux ou situations contextuelles. Une solution approximative à ce problème plus difficile a été proposée il y a vingt ans, et reposait sur une prescription ad hoc pour les interactions par paires entre les neurones, résumant les différentes contributions correspondant à chaque attracteur pris indépendamment des autres. Cette solution souffre cependant de deux problèmes majeurs : l'interférence entre les cartes limitent fortement la capacité de stockage, et la résolution spatiale au sein d'une carte n'est pas contrôlée.

Dans le présent manuscrit, nous abordons ces deux questions. Nous montrons comment parvenir à un stockage optimal des attracteurs continus et étudions le compromis optimal entre capacité et résolution spatiale, c'est-à-dire comment l'exigence d'une résolution spatiale plus élevée affecte le nombre maximal d'attracteurs pouvant être stockés, prouvant que les réseaux neuronaux récurrents sont des dispositifs de mémoire très efficaces capables de stocker de nombreux attracteurs continus à haute résolution. Afin de résoudre ces problèmes, nous avons utilisé une combinaison de techniques issues de la physique statistique des systèmes désordonnés et de la théorie des matrices vii aléatoires. D'une part, nous avons étendu la théorie de l'apprentissage de Gardner au cas des modèles présentant de fortes corrélations spatiales. D'autre part, nous avons introduit et étudié les propriétés spectrales d'un nouvel ensemble de matrices aléatoires, c'est-àdire la superposition additive d'un grand nombre de matrices aléatoires euclidiennes indépendantes dans le régime de haute densité.

En outre, cette approche définit un cadre concret pour répondre à de nombreuses questions, en lien étroit avec les expériences en cours, liées notamment à la discussion de l'hypothèse du remapping aléatoire et au codage de l'information spatiale et au développement des circuits cérébraux chez les jeunes animaux.

Enfin, nous discutons d'un mécanisme possible pour l'apprentissage des attracteurs continus à partir d'images réelles.

viii P U B L I C AT I O N S

This manuscript comprises the research work that the author (Aldo Battista) has conducted during the last three years under the supervision of Prof. Rémi Monasson at the Laboratoire de Physique de l'École Normale Supérieure, and includes the following published as well as original results.

Journal articles

- [START_REF] Battista | Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks[END_REF] Aldo Battista and Rémi Monasson. « Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks.» In: Physical Review Letters 124.4(2020), p.048302.

- [START_REF] Battista | Spectrum of multispace Euclidean random matrices[END_REF] Aldo Battista and Rémi Monasson. « Spectrum of multispace Euclidean random matrices. » In: Physical Review E 101.5(2020), p.052133.

ix O V E R V I E W O F T H E C H A P T E R S

This thesis work is organized as follows.

-First of all, in Chapter 1 we discuss the close link between statistical physics and computational neuroscience, thus justifying our approach.

-Afterwards, we start providing motivations for our work by discussing in detail the low-dimensional manifold hypothesis, the cornerstone of the whole thesis. That's the content of Chapter 2.

-Once we have clarified the approach and motivations of the thesis, we are going to introduce its subject in Chapter 3, i.e., recurrent neural networks and attractors. This pedagogical introduction to the subject is always accompanied by links with statistical physics and experimental evidences in the field of neuroscience that justify the development of the models themselves.

-In Chapter 4 we go into the heart of the thesis, where we propose and study in detail a new theory with the aim of solving the problems on continuous attractor neural networks to store multiple manifolds presented in the previous Chapter 3. In particular this has given us the opportunity to generalize Gardner's classic theory for the capacity of the perceptron to the case of patterns with strong spatial correlation.

-The theory presented in Chapter 4 led to the introduction of a new ensemble of random matrices, i.e., the superimposition of an extensive number of independent random Euclidean matrices in the high-density limit. Chapter 5 is devoted to a detailed study of the spectral properties of this new class of matrices.

-In Chapter 6 we report additional details on the theory studied in Chapter 4. We also propose a generalization of the results found so far, in particular by inserting biological constraints into the model. Links with ongoing experiments are made as well as another application of our setting is shown in the context of storing continuous attractors in a recurrent neural network starting from real images.

-Finally, Chapter 7 contains a summary of the results obtained in this work as well as indications for lines of research to follow in the near future. Since this is a thesis in theoretical (statistical) physics with applications to computational neuroscience, it is worth reminding the reader of the strong link, both from an historical and methodological point of view, between these two apparently very distant fields of research. This is the purpose of this short Chapter.

Statistical physics was born from thermodynamics towards the end of the nineteenth century and later developed in the field of condensed matter physics [START_REF] Hansen | Theory of simple liquids[END_REF][START_REF] Kittel | Introduction to solid state physics[END_REF][START_REF] Parisi | Theory of simple glasses: exact solutions in infinite dimensions[END_REF].

The main objective of this research area is to deduce easily measurable macroscopic quantities as a result of microscopic laws and also to explain collective phenomena such as phase transitions [START_REF] Huang | Statistical Mechanics[END_REF][START_REF] Davidovich Landau | Statistical physics: theory of the condensed state[END_REF][START_REF] Parisi | Statistical field theory[END_REF]. Its spirit can be summed up in the concise and famous P.A. Anderson phrase of 1972: "more is different", or in other words, when we move from an individual level of description of nature, i.e., a single molecule of water, to a collective level, i.e., a bottle full of water, new and non-trivial phenomena appear, i.e., liquid-solid transition.

In the case of thermodynamics and condensed matter physics, there are basically two reasons for the development of statistical physics:

-what is observed in experiments is usually on a macroscopic scale, so one is generally interested in macroscopic quantities; -it is often not possible to calculate microscopic quantities because of the huge number of parameters.

A statistical approach therefore manages to reconcile well what is needed with what is feasible.

Systems with multiple levels of description are omnipresent in nature, which is why the tools of statistical physics in recent decades have been used in many research fields, even seemingly far from physics, i.e., sociology [START_REF] Castellano | Statistical physics of social dynamics[END_REF], economics [START_REF] Bourgine | Cognitive economics: an interdisciplinary approach[END_REF][START_REF] Rosario | Introduction to econophysics: correlations and complexity in finance[END_REF], finance [START_REF] Bouchaud | Theory of financial risk and derivative pricing: from statistical physics to risk management[END_REF], biology [START_REF] Bialek | Biophysics: searching for principles[END_REF], immunology [START_REF] Agliari | Cancerdriven dynamics of immune cells in a microfluidic environment[END_REF][START_REF] Altan-Bonnet | Quantitative immunology for physicists[END_REF][START_REF] Biselli | Organs on chip approach: a tool to evaluate cancer-immune cells interactions[END_REF], route planning [START_REF] Ho Yeung | From the physics of interacting polymers to optimizing routes on the London Underground[END_REF] and neuroscience [START_REF] Daniel | Modeling brain function: The world of attractor neural networks[END_REF][START_REF] Christopher | The physics of brain network structure, function and control[END_REF].

In particular, this thesis concerns applications of statistical physics to the field of neuroscience, i.e., the study of the brain.

It is now evident the union between these two fields that seemed at first sight very different: the brain is formed by microscopic units, the neurons, which are interconnected in a network, via synapses, and each of them follows its own local microscopic dynamics and from this we want to deduce the global states. This problem is very reminiscent of magnetic systems in physics in which the spins (magnetic moments) interact with each other so that long-range ferromagnetic order can arise from the local couplings between them: the spins are schematized as binary variables, i.e., they can be in both up and down state, as well as a simplified model of a neuron can be active or silent. There is therefore a direct mapping between the celebrated Ising model [START_REF] Ising | Beitrag zur theorie des ferromagnetismus[END_REF], which is the milestone for magnetic system models, and the schematic representations of neural networks [START_REF] Brian | The organisation of neurones: a cooperative analogy[END_REF].

To complete the analogy just think that real neural networks have randomness factors both in the network structure ("quenched noise") and in the response of each neuron ("fast noise", equivalent to a temperature parameter). Therefore, all the mathematical tools developed by statistical physicists can be used to investigate neuroscience problems.

It is therefore no coincidence that many works in the literature have tried to address computational neuroscience problems with statistical physics techniques such as models of neural networks that reproduce observations of brain activity or that are able to perform specific functions [START_REF] Daniel | Modeling brain function: The world of attractor neural networks[END_REF][START_REF] Dayan | Theoretical neuroscience: computational and mathematical modeling of neural systems[END_REF][START_REF] Gerstner | Neuronal dynamics: From single neurons to networks and models of cognition[END_REF].

Today, applications of techniques from the 1980s of statistical physics of disordered systems, i.e., spin glasses, to real and artificial neural networks [START_REF] Daniel J Amit | Statistical mechanics of neural networks near saturation[END_REF][START_REF] Kirkpatrick | Optimization by simulated annealing[END_REF] are a very hot topic for two main reasons:

-the first is to understand the success of machine learning algorithms, particularly in the field of deep learning [97]. In fact, in recent years these algorithms have brought to the state of the art performance in several fields such as image [107, [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF][START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups[END_REF], natural language processing [START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], text translation [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Lample | Phrase-based & neural unsupervised machine translation[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF], computational medical diagnosis [START_REF] Esteva | Dermatologist-level classification of skin cancer with deep neural networks[END_REF] and artificial image/video generation [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF];

-the second comes from the incredible improvement of experimental techniques in neurobiology, such as electrophysiological and fluorescence-based functional recordings of neurons, that allow now to study in a quantitative way what in the 1980s was only speculative [START_REF] Sompolinsky | Computational neuroscience: beyond the local circuit[END_REF].

Manifold learning's approach has as a key assumption the low-dimensional manifold hypothesis which states that although many data are a priori high-dimensional, in reality they are intrinsically of much lower dimension. There are several reasons in favour of this hypothesis, for example:

-this can result from physical laws such as translation, rotation, change of scale and so on. If we consider a real image, whose dimensionality (the number of pixels constituting the photo) is usually very high and we apply the above mentioned transformations, it is clear that all the variants of the starting photo are linked together by a number of parameters much smaller than the number of pixels, i.e., each parameter corresponding to a particular physical law. So in the end we have that the different photos live in a space of dimensionality much lower than the a priori very high number of pixels, see Fig. 1(a);

-moreover, if we consider a classic data-set used in machine learning like MNIST [START_REF] Lecun | MNIST handwritten digit database[END_REF], see Fig. 1(b), it is possible to estimate the intrinsic dimension, that is the the low-dimensional manifold hypothesis number of variables needed in a minimal representation of the data, of its digits by looking at the number of local transformations required to convert a number into another one of its variants, and it turns out that this number is 10 [63], much less than the number of pixels of the images that make up MNIST, equal to 784.

The same goes for a more complicated example, let's consider a data-set consisting of photos of a person's face in different poses, Fig. 1(c); each picture is made of, say, 1000 × 1000 pixels. It is clear that this data-set is a very small subset of all possible colored pictures, which is defined by a 3 • 10 6 -dimensional vector 1 . The reason is that, for a given face, there are only ∼ 50 varying degrees of freedom (the position of all muscles), a very small number compared to 10 6 [START_REF] Lecun | Learning Hierarchies Of Invariant Features[END_REF]. Hence, all data points lie in a (non-linear) manifold, of very low dimension compared to the one of the initial pixel space.

We can therefore generally conclude that even though data may be high-dimensional, very often the number of relevant dimensions is much smaller.

The low-dimensional manifold hypothesis explains (heuristically) why machine learning techniques are able to find useful features and produce accurate predictions from data-sets that have a potentially large number of dimensions (variables) bypassing the curse of dimensionality problem. In fact, when the dimension of data increases, the volume of the configuration space grows so fast (exponentially) that the available data become sparse and this sparsity is problematic for any method that requires statistical significance. The fact that the actual data-set of interest really lives in a space of low dimension, means that a given machine learning model only needs to learn to focus on a few key features of the data-set to make decisions. However, these key features may turn out to be complicated functions of the original variables. Many of the algorithms behind machine learning techniques focus on ways to determine these (embedding) functions [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF].

A simple example of application of this hypothesis comes from the context of supervised learning [START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Goodfellow | Machine learning basics[END_REF]97], that is the fitting of input-output relation from examples, with neural networks for high-dimensional data classification because if the data actually live in manifolds of much smaller dimension, it is necessary to classify the manifolds [START_REF] Chung | Classification and geometry of general perceptual manifolds[END_REF], see Fig. 1(a).

in computational neuroscience

Low-dimensional representations of high-dimensional data are not restricted to machine learning, and are encountered in other fields, in particular, computational neuroscience [1,[START_REF] Gallego | Neural manifolds for the control of movement[END_REF][START_REF] Wang | Attractor network models[END_REF].

In fact, what is typically done in a neuroscience experiment is to measure with electrodes the activity of neurons in a real neural network, see Fig. 2(a). By looking at the population of measured neurons one can often find that the activity of the network can be explained in terms of relative activation of groups of neurons, called neural modes or cell assemblies, see Fig. 2(b). This means that even if the network activity is a priori high-dimensional, if one looks at its trajectory in the space of neural configurations as a function of time, it will be confined to live in a linear (or even non linear) manifold of much smaller dimension D N, being N the number of neurons that make up the network, see Fig. 2(c) and Fig. 2(d).

It is now legitimate to ask what the dimensions of the manifold mean. One of the most reliable hypothesis is that these collective coordinates generated by the neural network activity in the D-dimensional manifold encode sensory correlates, i.e., they encode some external stimulus, as the orientation of a bar presented to the retina [START_REF] David | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF][START_REF] David | Receptive fields and functional architecture of monkey striate cortex[END_REF], and can be used for example from the motor cortex to make decisions and/or produce actions [START_REF] Gallego | Neural manifolds for the control of movement[END_REF].

the low-dimensional manifold hypothesis Also related to this is the fact that low-dimensional continuous attractors provide a paradigm for analog memories, in which the memory item is represented by an extended manifold, i.e., the cognitive map of a place in a certain context, see Section 3.3.

A crucial question to answer is therefore, as we will investigate in detail in the next Chapters, how to engineer the connections in a network of interconnected neurons such as to map a low-dimensional dynamics to the high-dimensional one of the units that make up the net [START_REF] Barak | Mapping low-dimensional dynamics to highdimensional neural activity: A derivation of the ring model from the neural engineering framework[END_REF]. Once we have presented in Chapter 2 the fundamental hypothesis on which this work is based, we can introduce the subject of this thesis, i.e., recurrent neural networks and attractors, through a pedagogical illustration of a series of models introduced in the field of computational neuroscience. These models are always presented together with the experimental evidence that led to their formulation and the connections with statistical physics are also explained in detail, see Chapter 1 for a discussion on the link between statistical physics and computational neuroscience. The aim of this Chapter is therefore to place the work of this thesis in a very precise context within the literature, stressing the problems of the current theory and therefore the need for it.

discrete and continuous attractors in recurrent neural networks

Let's start by defining in a pictorial way what a recurrent neural network (RNN) is, what an attractor is, and the difference between discrete and continuous attractors1 .

A RNN is a kind of non-linear dynamical system defined by a set of N activity variables (neurons) σ i , i = 1, . . . , N, interconnected via pairwise synapses {W ij } (in the following we will never consider the case of self-connections, i.e., W ii = 0, ∀ i), where, depending on the models, both neurons and synapses can assume binary or continuous values and also respect from time to time different constraints of biological nature that we will discuss later, see Fig. 3(a).

In addition, the activity variables are updated over time, which can also be considered discrete or continuous as the circumstances require, following a non-linear dynamics dictated by the connectivity matrix W (or even by external fields), whose choice obviously defines the network properties in a crucial way.

The state of the RNN can therefore be represented by a point evolving in a very highdimensional space, the space of neural configurations of dimension N. In particular we will be interested in studying the trajectory of this point after a long time, especially in the case where the dynamics of the network remains blocked on different fixed points recurrent neural neural networks and attractors depending on the initial condition of the neurons activity variables: these fixed points are the celebrated attractors.

According to the choice of the synapses (without considering the presence of any external fields) we can have different scenarios for the structure of these fixed points:

-we can have that the different fixed points (specific configurations of the network activity variables where the dynamics get stucked) are isolated from each other and divided by attraction basins, that define according to the initial condition which will be the fixed point to which the dynamics of the network will converge: this is the case of discrete attractors, also called point attractors or 0-dimensional attractors, see Fig. 3(b);

-moreover, we can also have situations where the attractors instead of being isolated points, are composed of a continuous set of fixed points (manifolds) living in a Ddimensional space, where typically D N: this is the case of continuous attractors, see Chapter 2. Also here it is possible to have several continuous attractors as fixed points of the same network and divided by attraction basins, where, however, now depending on the initial condition of the network in an attraction basin, the dynamics of the RNN can remain blocked at any point of the relative attractor, see Fig. 3(c). As we discussed in Section 2.2, the important thing is to understand the physical meaning of the collective coordinate r which represents the state of the network onto the continuous attractor, that is a D-dimensional vector.

It is important to note that the dynamics really gets stucked2 to a fixed point if it is noise free (deterministic), i.e., zero temperature Glauber dynamics [START_REF] Roy | Time-dependent statistics of the Ising model[END_REF], otherwise there will be fluctuations around the fixed points that will depend on the level of the neural noise. In particular, as we will see later, with the right temperature it is possible to spontaneously generate for the state of the network transitions between one fixed point and another in the case of discrete attractors and the same is true also in the case of continuous attractors where, however, in addition to transitions between different attractors, a diffusive dynamics of the collective coordinate r on the attractors themselves is present as well.

Questions that we will answer in the following concern how to engineer the choice of synapses in order to build attractors with ad hoc (biological) properties and in particular what is the maximum number of attractors that can be stored in a recurrent network. We will also give below a strong emphasis to the experimental evidence (direct and indirect) of these mechanisms in the brain, especially in the context of memory, where memories correspond to the above mentioned attractors.

hopfield model: multiple point-attractors

Undoubtedly the milestone in this field of research is the seminal work of J.J. Hopfield in 1982 [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF] where the model named after him was formulated.

He showed that the computational properties used by biological organisms or for the construction of computers may emerge as collective properties of systems that have a large number of simple equivalent components (or neurons).

In practice J.J. Hopfield had proposed a practical way to choose connections in a RNN with many neurons in order to build multiple 0-dimensional attractors, showing that a very simple model of interacting binary neurons was able to have non-trivial collective properties, in particular to build autoassociative memories.

Basically there are two main ways to store information on a device: addressable and autoassociative memory.

recurrent neural neural networks and attractors -The first way consist in comparing input search data (tag) against a table of stored data, and returns the matching one [START_REF] Pagiamtzis | Content-addressable memory (CAM) circuits and architectures: A tutorial and survey[END_REF];

-the second is any type of memory that enables one to retrieve a piece of data from only a tiny sample of itself.

The Hopfield model together with all the models we will see in the following are autoassociative memories, and are particularly important to study because they are more biologically plausible than addressable ones.

Moreover, this model has remarkable properties as the robustness to the removal of a certain number of connections, the ability to correct patterns (memories) presented with errors, the ability to store patterns with a time sequence and recall them in the right order, although the single elementary components had independent dynamics without a clock that synchronized them [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF].

Before we discuss the Hopfield model in detail, let us recall the fundamental ingredients of biological inspiration that led to its formulation.

Ingredients of the model

The model is an extremely simplified schematization of a real neural network, where the basic units are binary spins 3 inspired by the Ising model of ferromagnetism from statistical physics [START_REF] Ising | Beitrag zur theorie des ferromagnetismus[END_REF], see Chapter 1. These units or better neurons are therefore binary variables, i.e., σ i = ±1 4 , where the state -1 represents a silent neuron while the state +1 an active neuron.

In reality, neurons are more complicated because they are cells whose activity is given by the difference in potential between the inner and outer part of their membrane. Typically this potential difference is about -70 mV when the neuron is silent, and when the neuron becomes active a very sharp electrical wave (action potential) localized in time is emitted and propagated along the axon which is the output of the neurons. The duration of this wave is of the order of a millisecond and if we properly discretize the time it is possible to distinguish if the neuron is active or not depending on the presence of this wave, see Fig. The different neurons then communicate with each other via these electrical signals that pass through the connections between the neurons called synapses 5 . The important thing about synapses is that when electric waves arrive at the end of the axons and come into contact with them, they may or may not amplify this signal depending on the strength and type of synaptic interaction. There are basically two types of synapses, excitatory (positive) and inhibitory (negative), the former tend to amplify the signal, while the latter tend to weaken it. Moreover, a neuron is connected to many other neurons and therefore receives from all of them the different electrical signals coming from axons and then modulated by synapses at a fixed time. What happens in the neuron is therefore a weighted sum of these signals which is then compared to a threshold, if this threshold is exceeded the neuron will emit an action potential, so it will be active, otherwise it will remain in its 5. From now on we will also use the terms synapses, weights, connections and interactions indiscriminately.

recurrent neural neural networks and attractors resting state, so it will be silent. This dynamics can be represented by the following simple equation:

σ i (t + 1) = sign j =i W ij σ j (t) -θ i , (3.1)
where σ i (t + 1) represents whether neuron i at the discrete time step t + 1 is active or silent, W ij represents the synapse between neuron i and neuron j which can be both excitatory and inhibitory, θ i represents the threshold associated with neuron i and sign is the sign function defined as:

sign(x) :=    -1, if x < 0 +1, otherwise . (3.2)
Once we have defined the dynamics of the model in Eq. (3.1), starting from an initial configuration for the activity of the neurons and a choice for the synaptic coefficients everything is well defined. The next step is then how to determine the choice of synapses. To do this J.J. Hopfield took inspiration from D. Hebb's seminal work "The organization of behavior" of 1949, in particular the idea that "neurons that fire together wire together" [START_REF] Olding | The organization of behavior: A neuropsychological theory[END_REF] or rather, quoting him directly: let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability . . . . When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.

In practice the idea is that if neurons have a tendency to be synchronously active together, then the connection between them will be strengthened over time. If, on the other hand, the opposite happens, i.e., that there is no synchronicity in the activity of some neurons, the connections between them will be weakened.

It is interesting to note that this mechanism is different from what we typically have in physics where the connections in an interacting system are usually given in the Hamiltonian in a static way, while here they are dynamically modified during dynamics through this mechanism of feedback of neuron activity on synapses. In this case it is as if the Hamiltonian of the system self-modifies itself according to the dynamics it has produced in previous times.

It is also important to mention that since Hebb's original theoretical formulation, many experimental studies, both in vitro and in vivo, have investigated the physiological basis of synaptic potentiation. The long-lasting strengthening of the synaptic connection between two neurons is called long-term potentiation, or LTP [START_REF] Tim | A synaptic model of memory: longterm potentiation in the hippocampus[END_REF][START_REF] Tim | Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path[END_REF][START_REF] Morris | Synaptic plasticity and learning: selective impairment of learning rats and blockade of long-term potentiation in vivo by the N-methyl-D-aspartate receptor antagonist AP5[END_REF][START_REF] Nabavi | Engineering a memory with LTD and LTP[END_REF]. In the hippocampus, a region of the brain that we will later discuss in Section 3.3, the bestknown mechanism that enables LTP is the transduction of electrical signals into chemical ones that activate the potentiation mechanisms in both the pre-synaptic and post-synaptic neurons, mediated by the N-methyl-D-aspartate (NMDA) receptor complex [START_REF] Stephen J Martin | Synaptic plasticity and memory: an evaluation of the hypothesis[END_REF]. Note that a similar mechanism occurs for the weakening of synapses, called long-term depression or LTD. It should be noted, however, that synaptic plasticity is still the subject of much research and several plausible mechanisms have been proposed, see as an example [START_REF] Aaron D Milstein | Bidirectional synaptic plasticity rapidly modifies hippocampal representations independent of correlated activity[END_REF].

Model details and properties

Now that we have described all the ingredients properly we can define the Hopfield model in detail.

Let's suppose we want to store a certain set of neuron activity configurations (patterns), where by store we mean that these configurations must be the fixed points for the dynamics established in Eq. (3.1). These patterns are defined as ξ µ i = ±1, where i is the neuron index, which goes from 1 to N as usual, and µ is the pattern index, which ranges from 1 to P. The individual elements of the patterns are sampled randomly and independently of each other. ξ µ i = +1 corresponds to an active neuron, while ξ µ i = -1 corresponds to a silent neuron.

Formalizing therefore in mathematical terms the idea of D. Hebb we can write the following prescription for the connectivity matrix of the RNN that must memorize these patterns obtaining so the famous Hebb rule 6 :

W ij = 1 N µ ξ µ i ξ µ j , (3.3)
where this matrix is of rank P and where the hebbian mechanism is clear, in fact, if two neurons in a pattern are in the same state the connection between them is strengthened, otherwise weakened.

The question to ask now is that if with this ad hoc choice of the weights, Eq. (3.3), the patterns are actually fixed points of the dynamics (3.1) 7 .

In the case P = 1 it is easy to understand why this works because this situation is equivalent to the Curie-Weiss model of statistical physics [START_REF] Huang | Statistical Mechanics[END_REF][START_REF] Kardar | Statistical mechanics of fields[END_REF][START_REF] Parisi | Statistical field theory[END_REF][START_REF] Reichl | A modern course in statistical physics[END_REF][START_REF] Reif | Fundamentals of statistical and thermal physics[END_REF][START_REF] Sethna | Statistical mechanics: entropy, order parameters, and complexity[END_REF][START_REF] Tuckerman | Statistical mechanics: theory and molecular simulation[END_REF], which is basically a mean field version of the Ising model where we take all interactions 6. Note that thus the network presents both excitatory and inhibitory synapses. 7. For simplicity we consider the case without external fields, θ i = 0, ∀i.

recurrent neural neural networks and attractors equal to W ij = 1 N , ∀ i = j. So we know that dynamics (3.1) will take the network to a state with non-zero magnetization, that is:

1 N i σ i = +1 or 1 N i σ i = -1 , (3.4)
depending on the initial activity configuration. It's interesting to note that in this situation the network will be organized in the configuration with all 1s or all -1s after some time, i.e., we have stored the pattern with all the components equal to ξ µ i = -1 and the pattern with all the components equal to ξ µ i = 1 (the network state after a long time has maximum overlap with one of the two memorized patterns). Now that we have seen how to store trivial configurations we can go further and try to understand how to memorize more interesting patterns. It can be done by multiplying the couplings in the Curie-Weiss model by a global gauge: this is the Mattis model [START_REF] Huang | Statistical Mechanics[END_REF][START_REF] Kardar | Statistical mechanics of fields[END_REF][START_REF] Parisi | Statistical field theory[END_REF][START_REF] Reichl | A modern course in statistical physics[END_REF][START_REF] Reif | Fundamentals of statistical and thermal physics[END_REF][START_REF] Sethna | Statistical mechanics: entropy, order parameters, and complexity[END_REF][START_REF] Tuckerman | Statistical mechanics: theory and molecular simulation[END_REF]. In particular, now we choose interactions like

W ij = 1 N ξ 1 i ξ 1 j , (3.5)
where by a simple change of variables we have stored a richer pattern, like the ones we would like to store in the Hopfield model. In fact, also in this case the dynamics (3.1) will bring the network to a magnetized (with maximum overlap) state, for the same reason as in the Curie-Weiss model, but now on the pattern with components ξ 1 i or the opposite one according to the initial condition of the network.

The idea behind the Hopfield model then is that if the patterns to memorize are orthogonal enough so that they don't interact too much 8 , we could put together many Mattis models (one for each pattern to store) and memorize simultaneously in the same network multiple patterns, from this comes the rule in Eq. (3.3).

In addition, the dynamics in Eq. (3.1) is simply a zero temperature updating rule that is equivalent to minimize the following energy 9 [START_REF] Daniel | Modeling brain function: The world of attractor neural networks[END_REF][START_REF] Hertz | Introduction to the theory of neural computation[END_REF]:

E[{σ i }] = - 1 2 i<j W ij σ i σ j + i θ i σ i . (3.6)
We immediately recognize that this is the energy of a spin glass (frustrated 10 disordered magnetic systems) with local fields generated by the local thresholds {θ i }. So we know that this dynamics will converge after some time to the nearest local minimum depending on the initial condition, and the question is whether or not these minima are related to the patterns we want to store, see Fig. 5(a). The answer to this question was found first numerically by Hopfield himself [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF] and then analytically by the seminal work of Amit, Gutfreund and Sompolinsky (AGS) [START_REF] Daniel J Amit | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF] using techniques of statistical physics of disordered systems, i.e., the replica method [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Konrad | Spin glasses[END_REF][START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]]. This answer is affirmative, in the limit of large N (thermodynamic limit), if the number of patterns to be mamorized divided by the number of neurons that make up the network is not too big, more precisely if α = P N , that is called load, is less than the critical capacity 11 α c 0.138 (without considering the presence of external fields, i.e., θ i = 0, ∀i), see Fig. 5

(b).

Moreover, AGS [START_REF] Daniel J Amit | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF] generalized the dynamics of the system from deterministic, see Eq. (3.1), to stochastic by introducing a temperature T such that the system obeys the detailed balance according to the Hamiltonian (3.6).

In this setting, always in the absence of external fields, they were able to find a phase diagram in the thermodynamic limit that includes different phases. In particular at high temperatures we have a paramagnetic phase (PM) in which the spins are substantially random and uncorrelated, therefore an uninteresting phase. If instead we look at lower temperatures we see the presence of two phases, the ferromagnetic phase (FM) and the spin glass phase (SG). At not too high values of α we find ourselves in the FM phase where the minimums of the energy (3.6) actually correspond to the stored patterns. If instead we consider values of α too big we enter the SG phase in which the local minima of (3.6) have nothing to do with the stored patterns and then our network stops being an associative memory, this is due to the fact that when we want to store to many patterns in the connectivity matrix at a certain point the interference between them becomes so strong to generate this catastrophic loss of memories, see Fig. 5(c).

Why is it important to study simple models?

Before moving on to the next topic it is important to stress that the Hopfield model is extremely simplified and far from achieving a good level of biological realism 12 , for example the neurons are schematized as simple spins, the connectivity matrix is symmetric and this is not true in general in real neural networks, the network is a priori fullyconnected while generally biological networks are quite sparse, there is no constraint on the sign of weights while in neuroscience there is a clear difference between excitatory neurons that are connected by positive synapses and inhibitory neurons connected by 11. This is trivially true in the case of large N and finite P, so α = 0 [START_REF] Hertz | Introduction to the theory of neural computation[END_REF]. 12. For more sophisticated and biologically plausible models of RNNs that store descrete attractors see for instance [START_REF] Brunel | Modeling memory: what do we learn from attractor neural networks?[END_REF][START_REF] Mézard | Solvable models of working memories[END_REF][START_REF] Mongillo | Learning in realistic networks of spiking neurons and spike-driven plastic synapses[END_REF]. recurrent neural neural networks and attractors negative synapses according to Dale's rule [START_REF] Hh Dale | Pharmacology and nerve endings[END_REF] and there is no adaptation mechanism in the dynamics of the network [START_REF] Daniel | Modeling brain function: The world of attractor neural networks[END_REF][START_REF] Dayan | Theoretical neuroscience: computational and mathematical modeling of neural systems[END_REF][START_REF] Gerstner | Neuronal dynamics: From single neurons to networks and models of cognition[END_REF][START_REF] Hertz | Introduction to the theory of neural computation[END_REF].

Nevertheless, this model is recognized as a milestone in the field of theoretical neuroscience because it is one of the few paradigms that combine Hebb's rule with attractors. It is also interesting because with this model it is possible to engineer the connectivity matrix to allow different states of activity for the network. Moreover, after the work of AGS [START_REF] Daniel J Amit | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF] the Hopfield model paved the way for the application of spin glass theory [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF] beyond physics (together with simulated annealing [START_REF] Kirkpatrick | Optimization by simulated annealing[END_REF] in computer science and engineering) starting to create a strong sociological impact in the statistical physics community towards theoretical biology and computer science.

Nonetheless, generalizations of the Hopfield model are nowadays a hot topic of research both in regards to neuroscience problems [START_REF] Fachechi | Dreaming neural networks: forgetting spurious memories and reinforcing pure ones[END_REF], but also machine learning thanks to the mapping between this model and (Restricted) Boltzmann Machines [START_REF] Agliari | Parallel retrieval of correlated patterns: From Hopfield networks to Boltzmann machines[END_REF][START_REF] Agliari | Multitasking associative networks[END_REF][START_REF] Tubiana | Emergence of compositional representations in restricted Boltzmann machines[END_REF] or even immune networks applications [START_REF] Agliari | Immune networks: multitasking capabilities near saturation[END_REF][START_REF] Agliari | A thermodynamic perspective of immune capabilities[END_REF].

representation of space in the brain

So far we have discussed in detail the Hopfield model (Section 3.2) which, as we have seen, is the milestone of recurrent neural network models to store multiple discrete attractors, on the other hand we have introduced motivations for another class of attractors found for example in computational neuroscience, i.e., continuous attractors, see Section 2.2 and Fig. 3(c).

Before going on with the models on continuous attractor neural networks, however, it is necessary to give the reader more specific biological motivations. For this reason in the following we introduce the hippocampus, a very important part of the brain where the presence of the mechanisms discussed in Section 2.2 is hypothesized in particular for the representation of space in the brain [START_REF] Treves | Computational analysis of the role of the hippocampus in memory[END_REF][START_REF] Tsodyks | Attractor neural network models of spatial maps in hippocampus[END_REF], where the activity of a recurrent network of noisy neurons (CA3 region of the hippocampus) that is high-dimensional, encodes for a continuous variable of low dimension as the position in an environment of an animal, and then we focus on the specific case of the special neurons that perform this task, i.e., place cells [START_REF] Edvard | Place cells, grid cells, and the brain's spatial representation system[END_REF].

For other examples in the brain of the presence of the continuous attractor mechanism see Section 3.8.

Hippocampus

The hippocampus is a part of the mammalian brain that is located in the medial temporal lobe and is part of the limbic system, see Fig. 6. It is composed of about 30 million neurons in humans and about 0.3 million neurons in rats. All vertebrate species, including reptiles and birds, have an homologous region. This part of the brain is one of the most studied by psychologists and neuroscientists because of its crucial role in spatial navigation and episodic memory.

The first evidence we have of the use of the hippocampus in a memory process was observed in the famous case of the patient H.M., who suffered from severe anterograde amnesia after receiving bilateral ablation of the hippocampus as a treatment for epilepsy [START_REF] Beecher | Loss of recent memory after bilateral hippocampal lesions[END_REF]. This first evidence was followed by many others that confirmed the strong correlation between hippocampal damage/injury and problems with declarative, i.e., involving conscious recall, memory formation and consolidation by human patients [START_REF] Amaral | The hippocampus book[END_REF][START_REF] Zola-Morgan | Human amnesia and the medial temporal region: enduring memory impairment following a bilateral lesion limited to field CA1 of the hippocampus[END_REF].

recurrent neural neural networks and attractors

In rodents, the hippocampus has been highly studied because of its role in spatial navigation and memory, i.e., the process of memorizing and recalling the cognitive map associated with an environment in a certain context. The term "cognitive map", coined by the American psychologist Edward Tolman [START_REF] Edward C Tolman | Cognitive maps in rats and men[END_REF], refers to an allocentric representation of the surroundings embedded in a Euclidean metric, which enables navigation through the cognition of the spatial distances between locations and objects. Tolman himself gave the first strong experimental evidence for a map-based navigation system in rodents by showing the ability of the rats to devise shortcuts to a known prize position. [START_REF] Edward C Tolman | Studies in spatial learning. I. Orientation and the short-cut[END_REF]. In general, the role of the hippocampus in the formation and recall of cognitive maps (spatial memories) has been demonstrated many times in experiments that required specific positions to be memorized in an environment. An important example is the Morris water maze where a rat, who is an able swimmer but dislikes being in the water, is trained to swim to an hidden platform in a specific location within a pool of milky water. An healthy animal quickly learns the position of the platform in the sense that the average time to reach the goal decreases rapidly with the number of trials [START_REF] Richard | Spatial localization does not require the presence of local cues[END_REF]. Morris and co-workers compared the results of healthy rats to those in which the hippocampus had previously been injured, showing that in the latter case the performance was drastically reduced [START_REF] Richard | Place navigation impaired in rats with hippocampal lesions[END_REF]. Now without going into too much detail about the anatomy of the hippocampus, for which we refer to [START_REF] Andersen | The hippocampus book[END_REF], it is important to say that a sub-part of it, called CA3, is practically a network of neurons with many recurrent connections, like those of the models discussed in Section 3.1, so that it is natural to try to formulate a mathematical model for this part of the brain [START_REF] Treves | Computational analysis of the role of the hippocampus in memory[END_REF][START_REF] Tsodyks | Attractor neural network models of spatial maps in hippocampus[END_REF].

Place cells and place fields

The fundamental connection between the hippocampus, including the CA3 region, and spatial navigation is due to the crucial discovery in 1971 by O'Keefe (who won the Nobel Prize in physiology or medicine for this finding in 2014) and Dostrovsky of a population of pyramidal cells 13 (a type of excitatory cells) which are active only when an animal is in specific positions of an environment [START_REF] John | The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat[END_REF]. The sharp firing specificity of these neurons granted them the name of "place cells", and their spatial receptive fields were named "place fields", see Fig. 13(a).

If, for example, we consider a rat exploring a square arena there will be a neuron in the hippocampus that will be very active when the rat is at some specific position and practically silent outside, and the same goes for other neurons with other positions in the environment. The area of space such that the neuron is highly active is exactly the place field 14 associated with that particular neuron (place cell) [START_REF] Nakazawa | NMDA receptors, place cells and hippocampal spatial memory[END_REF], see Fig. 7 15 .

From the seminal discovery of place cells the number of research works exploded in the following years in order to characterize their different properties. In a given environment the locations of the place fields corresponding to the different place cells are randomly positioned and the whole population of place cells is able to cover the whole surroundings [START_REF] John | Place units in the hippocampus of the freely moving rat[END_REF][START_REF] Matthew | Dynamics of the hippocampal ensemble code for space[END_REF]. Another very important property of place cells is the stability of their activity over time, in the sense that after an animal has memorized an environment, i.e., has associated place fields to some place cells (cognitive map), if it returns to visit the same setting after weeks the correspondence between place cells and place fields is almost unchanged [START_REF] Lt Thompson | Place cells and silent cells in the hippocampus of freely-behaving rats[END_REF]. In addition, the place fields are quite robust to small disturbances and transformations of external landmarks [START_REF] Hafting | Microstructure of a spatial map in the entorhinal cortex[END_REF][START_REF] Leutgeb | Progressive transformation of hippocampal neuronal representations in "morphed" environments[END_REF][START_REF] Robert | The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells[END_REF].

14. Note that in the case of animals like rats the place fields are two-dimensional while in the case for example of bats, which can fly, the latter are three-dimensional as they have access to an additional spatial dimension [START_REF] Michael | Representation of three-dimensional space in the hippocampus of flying bats[END_REF].

15. We recommend the reader to watch the attached video in which the activity of some place cells in a rat is recorded while exploring a track: https://www.youtube.com/watch?v=lfNVv0A8QvI. The measurements were made in the Wilson lab at MIT.

recurrent neural neural networks and attractors

Figure 7 -Schematic of a rat exploring a square arena. The place fields corresponding to two particular place cells of the rat are shown. When the rat is located inside a place field the corresponding place cell will be very active, otherwise it will be almost silent.

Moreover, it is known that external inputs, like visual inputs, are very important for the formation of a place field by a place cell but are not necessary to maintain their activity. This property is clear in the experiment of Quirk and collaborators [START_REF] Gregory | The firing of hippocampal place cells in the dark depends on the rat's recent experience[END_REF], who showed that the correspondence between place cells and place fields in an environment stored by a rat was almost unchanged if the light was turned off (no visual input) so that the animal was still able to maintain a good correlation between its position and the activity of the place cells measured. This mechanism of the rodent to integrate its linear and angular self-motion in order to maintain correlation between where it is and where the cells should activate is called path integration [START_REF] Bruce L Mcnaughton | Path integration and the neural basis of the'cognitive map[END_REF].

Finally, it is worth discussing the experiment of Buzsaki and colleagues [START_REF] Diba | Forward and reverse hippocampal place-cell sequences during[END_REF], who showed that the activation sequences of groups of place cells can be generated internally by the network itself and is not necessarily due to physical inputs, in fact, they found that before performing an action in a known environment a rat activates in order, during a planning phase of the action, the same set of place cells that then will actually activate during the motion (but in a time scale about 20 times less than the physical time).

A similar mechanism occurs if the planned action is successful, with the difference that this time the sequence of place cells activity is reproduced in reverse. This process is called backward replay and is associated with a mechanism of plasticity of the synapses (learning) in the sense that the activity of the neurons is reproduced in order to strengthen/weaken the connections for example with a Hebb mechanism, see Section 3.2, or more sophisticated ones, i.e., bidirectional synaptic plasticity [START_REF] Aaron D Milstein | Bidirectional synaptic plasticity rapidly modifies hippocampal representations independent of correlated activity[END_REF]. So it is important to stress that the activity of place cells is not only generated by external physical inputs but can be generated internally by the network.

Thanks to said properties, namely spatial selectivity, stability over time, and so on, the place cells population has been proposed as a suitable candidate for the neurological basis of the cognitive map [START_REF] John | The hippocampus as a cognitive map[END_REF].

We have therefore seen that the CA3 region of the hippocampus can be anatomically schematized by a recurrent neural network in which neurons (place cells) enjoy special properties such as the fact that their activity is linked to the position of an animal in physical space, recalling the mechanisms proposed in Section 2.2 and Fig. 3(c) in the context of continuous attractors [START_REF] Treves | Computational analysis of the role of the hippocampus in memory[END_REF][START_REF] Tsodyks | Attractor neural network models of spatial maps in hippocampus[END_REF]. How is it possible to design a model that reproduces this phenomenology? This is the purpose of the next Section 3.4.

storing a single continuous attractor

We thus begin to present models of recurrent neural networks that qualitatively reproduce what was discussed in Section 3.3, i.e., how to store a continuous attractor in a RNN.

There is certainly no shortage in literature of models of this type called continuous attractor neural networks (CANN) in which a large and noisy population of neurons can reliably encode "positions" in low-dimensional sensory manifolds and continuously update their values over time according to input stimuli [12, 31, 81-83, 237, 265].

The first model ever of this type is the one of Amari [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF] of 1977 in which we consider a recurrent neural net composed of N neurons arranged in a ring structure and connected through excitatory couplings whose strength decays with the distance between the neurons (therefore local excitation), together with another neuron connected with all the others that is excited from these but that in turn inhibits all the others (global inhibition). From these two fundamental ingredients of local excitation and global inhibition, given an initial condition of the network through an external input, it is possible to generate a localization of the activity (bump or cell assembly). A mechanism of this type explains well what happens in the head-direction cells system or in the ellipsoid body of the flies, see Section 3.8, in which a ring attractor encodes the angle of the head of an animal (one recurrent neural neural networks and attractors dimensional continuous attractor), in which simultaneously active neurons encode for close angles while the others are silent.

Without going too much into the details of Amari's model, for which we refer to the original article [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF], in the following we will focus on another CANN model of fundamental importance that contains the same ingredients of the ring attractor one but has been developed specifically to explain the phenomenology of place cells and place fields presented in Section 3.3, namely the model of Tsodyks and Sejnowsky [START_REF] Tsodyks | Associative memory and hippocampal place cells[END_REF].

Afterwards we will focus on how to translate these concepts into a statistical physics framework and at the end we will discuss the importance of the concept of continuous attractor.

Tsodyks and Sejnowsky's model

Here we consider the CANN model introduced by Tsodyks and Sejnowsky [START_REF] Tsodyks | Associative memory and hippocampal place cells[END_REF] in 1995 as a model of the hippocampus CA3 network that produce place selective activity in an environment, see Section 3.3.

Let's so consider a recurrent neural network defined by the following rate equations:

dv i dt = -v i + g j =i W ij v j + I i , (3.7)
W ij = W 0 exp - |r i -r j | ∆ -W 1 , (3.8)
where v i is the average spiking rate 16 of the neuron i, I i is it's external sensory input, g is the gain function of the neurons (like a ReLU function), W ij is the strength of the synaptic coupling between the neurons i and j coding for locations (centers of the place fields) r i and r j respectively in the physical space and ∆ is the diameter of the place fields, see Fig. 7. The excitatory connections in the model are mainly local and the coupling matrix W ij is an exponential function of the distance between their place fields centers. The uniform inhibitory inputs W 1 can be considered as a global feedback inhibition.

If in Eq. (3.7) we do not consider the g term, we get a relaxation equation, this means that after some time the firing rate of all the neurons is zero. What makes the system non-trivial is the fact that the neurons are connected via the couplings {W ij }, and possibly an {I i } external input. If we choose the couplings intelligently so that to have a mapping between the neurons in the network and the position they encode in physical space by choosing excitatory couplings if the distances in physical space are small and inhibitory otherwise, we will be able to create a bump of activity that means that neurons that are active at the same time encode for close positions in physical space. Moreover, this bump can move due to the introduction of an eventual noise term in the dynamics (3.7) or to weak inputs {I i }, thus defining a way to move along the attractor.

3.4.2 CANN and statistical physics: Lebowitz and Penrose's model Now let's try to put together the ideas just presented and formalize a very simple model with which we can make a mapping with a classic model of statistical physics, the Lebowitz and Penrose's model [START_REF] Joel | Rigorous treatment of the Van Der Waals-Maxwell theory of the liquid-vapor transition[END_REF].

Hence, we start as usual with defining a set of N binary neurons, which can take either σ i = 1 or σ i = 0 depending on whether they are active or silent. Moreover, these neurons are interconnected through an appropriate matrix of connectivity {W ij } (without self-connections), see Fig. 3(a).

The weights of the net in this case must be chosen in order to store a continuous variable such as the position of an animal in an environment, as we have seen in Section 3.3. To do this we start by defining a simplified cognitive map model to be memorized, that is, we consider a square environment 17 of unitary area and with periodic boundary conditions 18 where we associate to each neuron a place field 19 , i.e., a fixed radius disk 20 located in a random position r i of the environment, see for example one of the two environments in Fig. 14(b).

It is important to note that there is no relationship between the arrangement of the different place fields in an environment and the arrangement of the corresponding place cells in the neural network also because, as we will discuss in Section 3.5, the same network must store different environments in which the different place fields associated with the same neuron do not have any correlation (global random remapping).

17. Here we focus on the two-dimensional case in analogy with the place cells and place fields in rodents discussed in Section 3.3, but the following is trivially generalizable to any finite dimension D.

18. The use of periodic boundary conditions is justified only in the modeling of head-direction cells where the attractor to be memorized is actually a ring in D = 1, see Section 3.8. Already in the case of place cells this assumption is no longer valid because the presence of edges (walls) in an environment is important, see Section 3.8, nevertheless we will continue to use this assumption for theoretical convenience.

19. In general it is not true that in a given environment each neuron has a place fields, in fact typically about thirty percent of neurons have it while the others do not (silent cells) [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF], for now we do not take into account this aspect for simplicity.

20. From experiments it is known that place fields can have different shapes, but for the moment we consider place fields all the same and circular with the same radius for simplicity.

recurrent neural neural networks and attractors

Once an environment model has been defined, it is necessary to associate the possible locations with a corresponding network activity status. If we consider a specific position r on the map we will consider that the neuron i is active if the distance between the position r and the center of the corresponding place field r i is less than the radius of the place field, i.e., the position is within the place field associated with neuron i, see Fig. 7.

To memorize an environment we mean that the set of patterns associated to the different positions, which are obviously spatially correlated by construction, must be fixed points of the network dynamics and this defines a continuous attractor, see for example one of the manifolds in Fig. 3(c). Now it is important to understand how to choose the synapses of the network to memorize this continuous attractor. In the simple case of the Mattis model, see Section 3.2, we had that interactions were only a function of the specific pattern to store, see Eq. (3.5), but now we are interested in memorizing all the different positions associated with the spatial map.

The standard way to do this, following what already seen in Tsodyks and Sejnowski's model, is to consider a connectivity matrix in which the synapses between neuron i and neuron j are a function of the distance between the respective place fields in the environment with centers located in r i and r j , that is:

W ij = w |r i -r j | , (3.9)
where | • | denotes the distance in the map. If w is sufficiently excitatory at short distances and inhibitory at long ones, a bump state spontaneously emerges, in which active neurons tend to code for nearby positions in the map. Classic examples of kernels w are for instance exponential, step function or Gaussian ones.

It is important to note that the mechanism presented in Eq. (3.9) is reminiscent of Hebb's rule as the concept that neurons that fire together wire together remains valid. In fact, neurons that have place fields nearby will often be active together so we need to strengthen the connection between them, while neurons whose place fields are far apart will hardly be active together so it is better to have a weak connection between them.

To start studying the distribution of neural activity configurations associated to this kind of model what is missing is the definition of a dynamics associated to the network. We consider here a scheme of the same type already used in the Hopfield model, see Section 3.2, but with the introduction of a finite temperature, given by the following formula that explain how neuronal states are updated stochastically according to the probabilities

Prob σ i (t + 1)|{σ j (t)} = 1 1 + exp -1 T 2σ i (t + 1) -1 ( j( =i) W ij σ j (t) -θ i ) , (3.10)
where T is a temperature parameter to be set such that a bump of activity may form and sustain itself 21 .

The thresholds {θ i } in this dynamics are simply effective fields that keep the activity of the network at a fixed level typically giving an inhibitory contribution (they are negative and somehow schematize the effect of the interneurons) and are necessary because the connections between place cells in Eq. (3.9) are instead excitatory (positive).

It is interesting to note that a model of the kind just presented is very similar to a very old model of statistical physics introduced in 1966 by Lebowiz and Penrose to explain the liquid/vapor transition [START_REF] Joel | Rigorous treatment of the Van Der Waals-Maxwell theory of the liquid-vapor transition[END_REF][START_REF] Monasson | Transitions between spatial attractors in place-cell models[END_REF].

Consider a D-dimensional lattice, whose N sites x i can be occupied by a particle (σ i = 1), or left empty (σ i = 0) 22 . The energy of a configuration {σ i } is given by the Ising-like Hamiltonian

E {σ i }, {x i } = - i<j w |x i -x j | σ i σ j , (3.13)
where w is a positive and decaying function of its argument, i.e., of the distance between sites. At fixed number of particles and low enough temperature T translation invariance on the lattice is spontaneously broken: particles tend to cluster in the x-space, and form a high density region (liquid drop) surrounded by a low-density vapor. The density profile of this "bump" of particles hardly fluctuates, but its position can freely move on the lattice, and defines a collective coordinate for the microscopic configuration of particles, see Fig. 8.

21. Note that in the null temperature limit, i.e., T = 0, we obtain a deterministic dynamics of the type:

σ i (t + 1) = Θ j =i W ij σ j (t) -θ i , (3.11)
where Θ is the Heaviside step function defined as:

Θ(x) :=    0, if x < 0 1, otherwise . (3.12)
So exactly the same dynamics as the Hopfield model rewritten for spins 0 or 1 instead of ±1, see Section 3.2.

22. The fact that the particles are on a grid here whereas previously we have seen that place fields have random positions in the environment does not change the phenomenology of the model in the large N limit.

Fixing the number of particles here is equivalent to find in our model of CANN an inhibitory threshold that keeps the number of active neurons in the network constant.

Continuous attractors and population coding

From the neuroscience point of view, the existence of a collective coordinate, weakly sensitive to the high stochasticity of the microscopic units, is central to population coding theory.

It should now be clear the equivalence between the RNN models to store a continuous attractor and the Lebowiz and Penrose model. The key in both cases is in fact to have synapses that are functions of some distance, in the first case the distance between the place fields in the environment and in the second case the distance between the particles. Moreover, the fact of creating a liquid droplet at low temperatures in the second case is equivalent to create a bump of activity in the first case in which the neurons that are active at the same time code for nearby positions in the map.

Since the system is invariant under translation, the bump of activity can spread both because of the temperature in the dynamics and because of a weak external input and explore all the positions associated with the environment which are equivalent to each other (no preferred positions), this means that with this choice of the connectivity matrix we have actually managed to store a continuous attractor. Moreover, the shape of the bump is determined by the Hamiltonian of the system but its position is totally random. So we got exactly what we wanted, a model of a population of noisy neurons in which we have the emergence of a collective coordinate (the position of an animal in an environment) that is robust to the error of the single neuron, i.e., a robust encoding. Obviously the number of coded coordinates is equal to the dimension of the stored map and it is also important to note that the activity bump may persist in the absence of external input, exactly as we saw for the place cells in Section 3.3.

In addition, we notice that we can have two points of view with this kind of models, either we look at a fixed time to the activity of the neuron population and we see the emergence of this collective variable coding for the position in the map, see Fig. 9, or we look at the activity of a single neuron and we see that this is active when the distance of the collective variable, i.e., the position of the animal in the environment, is close to the center of the place field of the neuron considered, which is instead what we typically see in experimental measurements, see Section 3.3. In general models of CANNs have many computationally appealing properties, such as efficient population decoding, smooth tracking of moving objects, and implementing parametrical working memory. The computational advantages of CANNs and their successes in modeling brain functions have suggested that CANNs serve as a canonical model for neural information representation [START_REF] Wu | Continuous attractor neural networks: candidate of a canonical model for neural information representation[END_REF].

recurrent neural neural networks and attractors 3.5 why is it necessary to store multiple continuous attractors?

We have therefore seen in detail how to store a continuous attractor in a recurrent neural network, see Section 3.4, that could correspond to the memorization of a cognitive map of an environment by a network of place cells, see Section 3.3. In reality, however, the story is more complicated than this because the hippocampus needs to contain simultaneously several cognitive maps corresponding to different environments and also, for the same environment, to different contexts. But first of all, how do the different cognitive maps relate to each other? For different environments, place fields associated to different place cells can re-position in a supposedly random way, a property called "global remapping" 23 [121, 143, 175, 185], or rather, given a certain place cell this can encode positions (have a place field) in different environments and the location of the different place fields, corresponding to the same cell, seems to be totally random without any correlation between the various maps.

The fact that the different maps are unrelated because of random remapping will be one of the key elements to model a recurrent neural network that stores multiple continuous attractors, see Section 3.6, because in this way the configurations of neurons encoding different positions in different environments are orthogonal to each other with high probability, this recalls the case of the Hopfield model, i.e., multiple discrete attractors, where random (orthogonal) patterns were chosen in order to limit the interference between them, see Section 3.2, here the idea is exactly the same.

An important example of this mechanism is provided by the experiment of Alme and collaborators [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF] in which rats are trained to store eleven different rooms. Once these environments have been memorized the animals are able to recall the cognitive maps according to the room they are in, and the different maps present the above mentioned random remapping phenomenon for the place fields in the various rooms associated to the same place cell. This experiment shows how the hippocampus can store several maps at the same time, raising the natural theoretical question, obviously very complicated to study experimentally and of which we do not have biologically plausible estimates, on the maximum number of continuous attractors that a network of this type can store (critical capacity), which we will start to deal with in Section 3.6. It has to be said, however, that despite the difficulty of experimentally measuring the network capacity, we know that, for example, wild rats (and not only laboratory ones) are able to navigate perfectly in many environments, so it is reasonable to think that they are able to store a large number of cognitive maps in their hippocampus.

Moreover, the specific positioning of place fields are flexible and might shift or entirely re-arrange upon drastic changes in external landmarks and boundaries [START_REF] James J Knierim | Place cells, head direction cells, and the learning of landmark stability[END_REF][START_REF] Robert | The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells[END_REF], odors [START_REF] Timothy A Allen | Nonspatial sequence coding in CA1 neurons[END_REF], or even abstract variables such as contextual conditions or the task to be performed [START_REF] Timothy A Allen | Nonspatial sequence coding in CA1 neurons[END_REF][START_REF] Jezek | Theta-paced flickering between place-cell maps in the hippocampus[END_REF][START_REF] Kelemen | Dynamic grouping of hippocampal neural activity during cognitive control of two spatial frames[END_REF][START_REF] David | Hippocampal place cells, context, and episodic memory[END_REF].

Finally, it is important to note that it is possible to have different cognitive maps also depending on the sensory modalities used by the animal, for example Geva-Sagiv et al. [START_REF] Geva-Sagiv | Hippocampal global remapping for different sensory modalities in flying bats[END_REF] showed in bats that they used different cognitive maps (which differed through global remapping) for the same environment depending on whether the animal used as external inputs the visual one or echolocation. Other examples where global remapping is observed by changing or combining different input modalities, such as visual and path integration, include experiments performed in virtual reality where one has full control over the different input modalities to which the animal is subject [56].

At this point it is therefore obvious that the next step is to try to combine the Hopfield model for multiple discrete attractors, see Section 3.2, with what we have seen in Section 3.4 for single continuous attractors in order to have a model that can reproduce the phenomenology just discussed. This will be the purpose of the next Section 3.6.

the case of multiple continuous attractors

We discussed in Section 3.4 how to store a single D-dimensional attractor in a recurrent network, but it is clear from Section 3.5 that it may be necessary to have a model of a RNN that stores many continuous attractors in the same connectivity matrix. To solve this problem Samsonovich and McNaugthon presented the following RNN model [START_REF] Samsonovich | Path integration and cognitive mapping in a continuous attractor neural network model[END_REF] in 1997 that is kind of a mix between the Hopfield model to store multiple discrete attractors, see Section 3.2, and the model presented in Section 3.4 to store a single continuous attractor [START_REF] Samsonovich | Path integration and cognitive mapping in a continuous attractor neural network model[END_REF].

Then we will present a version of this model inspired by statistical physics, that is the model of Rosay and Monasson [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF], with which it is possible to study in particular the critical capacity of RNN that store multiple continuous attractors, that is the maximum number of D-dimensional attractors L that can be stored in a network of N neurons.

recurrent neural neural networks and attractors

Samsonovich and McNaugthon's model

In the models presented for a single continuous attractor, like the model of Tsodyks and Sejnowsky (Section 3.4), we have seen how to memorize a map it is essential to choose connections between place cells so that they are a function of the distance between the centers of the place fields in the map of the respective neurons, see Eq. (3.8). Now instead of storing only one map, we want to memorize L of them in the same connectivity matrix, where the different maps differ because of the random remapping phenomenon discussed in Section 3.5, i.e., given a place cell, this encodes a position (has a place field) in each map and these positions are totally random and uncorrelated in the different maps. Based on this fact and also the Hebb rule presented in the Hopfield model where the contributions of the different patterns were summed in the connectivity matrix, see Eq. (3.3), Samsonovich and McNaugthon proposed the following rule for the couplings matrix of a RNN that must store multiple continuous attractors [START_REF] Samsonovich | Path integration and cognitive mapping in a continuous attractor neural network model[END_REF]:

W ij = L =1 exp - |r i -r j | 2 ∆ 2 , (3.14)
where = 1, . . . , L and r i is the location of the place-field center of the cell i in the map . Moreover, using a network dynamics of the type defined in Eq. (3.7) they showed that when one cognitive map is retrieved, the activity is organized as a coherent 2D bump on the corresponding map, while looking scattered and uninformative on all the other maps (how it must be given that at a fixed time an animal is in a specific map position and also because of the random remapping the different maps are orthogonal to each other).

Rosay and Monasson's model

From the model discussed above we can therefore understand that to store multiple continuous attractors in one RNN we can in general use the rule defined in Eq. (3.9) for each of the maps separately and then consider as connectivity matrix the sum of the connectivity matrices of the single environments, obtaining in this way, assuming each one of the L maps contributes equally to the learning process, a rule of type

W ij = L =1 w |r i -r j | , (3.15)
where r i is the center of the place field of neuron i in environment .

In particular, what we mean to store more than one map in the same connectivity matrix is precisely the fact that if we consider the dynamics of network 24 (3.10) with rule (3.15) at some time we will have a bump of activity in one of the maps (the map of which we are recalling some specific position) and basically noise in the others or rather that the fixed points associated with dynamics (3.10) at the zero temperature are all the configurations of the neurons built starting from a position r in one of the maps in which the place cells associated with the place fields close to r are active and the others silent. Obviously we will have the bump only in one of the maps at a fixed time because the different maps are orthogonal to each other (random remapping), i.e., the activity patterns that encode for the positions of a specific map will substantially give noise in the other maps or better that the patterns encoding for the different positions in the different maps are orthogonal with high probability.

In the following we consider the model of Rosay and Monasson25 [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF], inspired by the model of Lebowitz and Penrose (Section 3.4), in which the issue of storing multiple continuous attractors in a RNN is set as a statistical physics problem.

So we consider a network of binary neurons where the N place cells are modeled by binary units σ i equal to 0 (silent state) or 1 (active state). These neurons interact together through excitatory couplings {W ij }. Moreover, they interact with inhibitory interneurons, whose effect is to maintain the total activity of the place cells to a fraction f of active cells (global inhibition). We also assume that there is some stochasticity in the response of the neurons, controlled by a noise parameter T . All these assumptions come down to considering that the network states are distributed according to the Gibbs distribution associated to the Hamiltonian

E W [{σ i }] = - i<j W ij σ i σ j , (3.16)
restricted to configurations of spins {σ i } such that

i σ i = f N . (3.17)
We want to store L + 1 environments in the coupling matrix, indexed by , each defined as a random permutation π of the N neurons' place fields that are initially arranged on a D-dimensional grid (the case of only one map is reminescent of the Lebowitz and recurrent neural neural networks and attractors Penrose's model). This schematize the random remapping of place fields from one map to the other. With this definition, an environment is said to be stored when activity patterns localized in this environment are stable states of the dynamics. In other words, the configurations where active neurons have neighbouring place fields in this environment are equilibrium states. To make this possible, we assume a Hebbian prescription for the couplings {W ij } that is a straightforward extension of the Hopfield synaptic matrix, see Section 3.2, to the case of quasi-continuous attractors. This rule is described as follows:

-additivity:

W ij = L =1 W ij
where the sum runs over all the environments.

-potentiation of excitatory couplings between units that may become active together when the animal explores the environment:

W ij = 1 N if d ij d c , 0 if d ij > d c , (3.18)
where d ij is the distance between the place field centers of i and j in the environment . d c represents the distance over which place fields overlap. This distance is chosen in such a way that each neuron i is connected to the same number of other neurons j, regardless of the spatial dimension D. If ŵ N is this number: ŵ ( 1) is the fraction of neurons to which each neuron is connected. So for example

d c = ŵ 2 N in D = 1 and d c = ŵN π in D = 2. The 1
N factor in Eq. (3.18) ensures that the total input received by a cell remains finite as N goes to infinity. 26 were able to show in this model that in the thermodynamic limit it is possible to store an extensive number of maps L (at fixed dimension D) with a recurrent neural network composed of N neurons if the ratio α = L N is less than a critical capacity α c (D), where this quantity is of order 1. It's important to note that while in the Hopfield model, see Section 3.2, the load parameter α was defined as the ratio between the number of patterns P to store divided by the number of neurons N, now here we have the number of maps L at the numerator (each of which corresponds a priori to many patterns, one for each position in the map to store).

Rosay and Monasson

We show in Fig. 10 the phase diagram associated with this model in the case D = 1, f = .1 and ŵ = .05 27 . It is then interesting to make a comparison of this phase diagram with the one concerning the Hopfield model, see Fig. 5(c). In both phase diagrams there are different phases depending on temperature, in the Rosay and Monasson case in particular the clump phase (CL), the paramagnetic phase (PM) and the spin glass phase (SG). This CL phase is the equivalent in the Hopfield model of the ferromagnetic phase, and is the 26. See [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF] for all the details on the replica computation which led to finding the phase diagram of the model. This computation is somehow a generalization of the AGS one for the standard Hopfield model, see Section 3.2.

27. The phase diagram remains qualitatively similar to variations in model parameters, see [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF].

region of the phase diagram interesting to consider in order for the neural network to function as associative memory. The neural configurations allowed in the CL region are those in which a bump of activity is able to form in a random position of one of the maps, in the other phases this is not possible. At high noise level (high T), the system is in the PM phase, where no coherent representation is formed and the population activity is comparable to random. Finally, when the number of stored maps exceeds the critical capacity (α c (D, T )) the system falls into the SG phase, a behavior characterized by the presence of many local minima where the effective noise induced by the competition between maps freezes the activity and no spatial or map selectivity is achieved.

In addition, from a qualitative comparison with the Hopfield model it seems here that the system is more robust against noise, in fact, the ferromagnetic region of the Hopfield model has a triangular shape while the CL phase in this case is more squared, suggesting that the storage of correlated patterns is more robust against noise than completely random patterns. Besides the static properties of the model, Rosay and Monasson have also studied its dynamical features [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity[END_REF] that are much richer than in the standard the Hopfield setting. First of all, in the case of a single environment, i.e., α = 0, we have the emergence of a quasi-particle (the bump) with an activated diffusion, expected from what we have discussed in Section 3.4; furthermore, in the case of several maps, i.e., α > 0, in addition to recurrent neural neural networks and attractors the diffusion of the bump within a map we have as well spontaneous transitions between the different maps 28 , see Fig. 11 29 .

In fact, a dynamics at a moderate noise level will most of the time tend to move in the space of the configurations in one of the manifolds (composed of a set of spatially related fixed points) dictated by the initial condition, but occasionally it may jump to another set of fixed points corresponding to another map, see Fig. 3(c). In the case of the Hopfield model instead the situation was much simpler because the different patterns corresponded to different isolated fixed points so there may be only, with a sufficient level of noise, drastic jumps between a fixed point and another, see Fig. 3(b).

issues with current theory

The type of models discussed in Section 3.6 represent the state of the art to date on how to store multiple continuous attractors in a recurrent neural network by using the ad hoc prescription defined in Eq. (3.15) for the pairwise interactions between neurons, despite the presence of the following important problems both from a practical and theoretical point of view.

1. First of all, due to interferences (crosstalk) between the different manifolds, the attractors are not continuous any longer, and are effectively discrete. How to practically learn smooth attractors and ensure a prescribed level of spatial resolution, i.e., in a nutshell the typical distance between nearest neighbour fixed points in every attractor, is an open issue.

In fact, as soon as we want to store more than one map in the RNN, the activity bump gets stuck in some preferred locations in the retrieved map due to the interferences coming from the other non-retrieved maps [START_REF] Cerasti | The spatial representations acquired in CA3 by self-organizing recurrent connections[END_REF]. In other words, rule (3.15) does not define truly continuous attractors, as large barriers oppose the motion of the bump along the map [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity[END_REF].

To better understand this phenomenon, we refer to the case of having to memorize only one continuous attractor. In this case, in fact, as we have seen in Section 3.4, there is invariance under translation, in the sense that if we consider two neurons coding for two positions (so having their place fields centered in that positions) on the map, the interaction between them depend only on the distance between these positions, see Eq. (3.9), consequently, an activity bump can be located in any position of the environment at the same cost, see Eq. (3.13). But as soon

28.

A full characterization of the phenomenology of these spontaneous transitions can be found in [START_REF] Monasson | Transitions between spatial attractors in place-cell models[END_REF].

29. See also the video at https://journals.aps.org/prl/supplemental/10.1103/PhysRevLett.115. 098101. Note that the circles of different colors represent the centers of the place fields whose corresponding place cells are active at some time in the different maps. as we try to store in the same connectivity matrix more than one map with rule (3.15), and consider a bump of activity that has formed in one of the maps, we will break this invariance under translation because in the connections there will be the contributions of the other maps (that we are not recalling) that will act as a noise term, this can be described with the fact that in the case of a single manifold the bump is travelling in an effective flat free energy landscape while in the case of multiple maps this landscape is rough due to this crosstalk term, see Fig. 12 and [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity[END_REF]. This means that if we are at a low level of neural noise (low temperature) the bump will remain stuck in one of the free energy minima, in other words we have no longer stored a continuous attractor but only a discrete set of positions that correspond to the minima of this free energy. 2. This issue of spatial resolution is also unclear from a theoretical point of view. Capacity calculations [START_REF] Francesco | Attractor neural networks storing multiple space representations: a model for hippocampal place fields[END_REF][START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF] require that a bump can form in any of the memorized maps, in at least one position: they offer no guarantee about the existence of other memorized positions, that is, we have no explicit control over which positions we actually have stored in each map (what are precisely the fixed points of the neural dynamics?).

3.

Moreover, theoretical studies [START_REF] Francesco | Attractor neural networks storing multiple space representations: a model for hippocampal place fields[END_REF][START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF][START_REF] Monasson | Transitions between spatial attractors in place-cell models[END_REF] show that the maximal capacity corresponding to the prescription (3.15) is very low, see Fig. 10. It is reasonable to expect that the optimal storage capacity could be much higher: a ∼ 15-fold increase was found from the Hebb-rule critical capacity, 0.138 [START_REF] Daniel J Amit | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF], to the optimal capacity, α c = 2 [START_REF] Gardner | The space of interactions in neural network models[END_REF] in the case of 0-dimensional attractors, corresponding to the Hopfield model [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF].

Where by optimal capacity we mean that the weights of the neural net have been chosen to maximize the number of attractors (number of maps at fixed number of stored positions per map) to memorize, rather than making an a priori choice for the couplings and then see how many maps can be memorized with that choice following the Hebb-like rules (3.15) approach.

4.

Optimal learning could also provide detailed insights on the statistical structure of the neural couplings {W ij }, which could be compared to the physiological distribution of synaptic connections [START_REF] Brunel | Is cortical connectivity optimized for storing information?[END_REF].

The purpose of this thesis is to solve these problems. In fact, as we shall see in the next Chapters, we will show how to achieve optimal storage of continuous attractors and study the optimal trade-off between capacity and spatial resolution, that is, how the requirement of higher spatial resolution affects the maximal number of attractors that can be sustained in a RNN.

experimental evidences for continuous attractors

Before presenting a solution to the problems posed in Section 3.7, it is important to provide other examples in the brain of neurons whose activity depends strongly on some continuous sensory correlates in addition to the place cells that we have already discussed in detail, see Sections 3.3 and 3.5.

Head-direction cells

The place cells we have already described are not the only example of neurons in the brain showing a sharp spatial selectivity in their activation properties. In fact, after their discovery, the head-direction (HD) cells in the septal presubiculum were found [START_REF] Jb Ranck | Head direction cells in the deep layer of dorsal presubiculum in freely moving rats[END_REF][START_REF] Jeffrey S Taube | The head direction signal: origins and sensory-motor integration[END_REF]. These cells have the surprising property of being active or silent depending on the orientation of a rodent's head with respect to a reference direction of the environment, see Fig. 13(b).

Neurons responsive to the direction of the motion have later been discovered in other regions, such as the entorhinal cortex [START_REF] Sargolini | Conjunctive representation of position, direction, and velocity in entorhinal cortex[END_REF], the anterior and lateral dorsal thalamic nuclei [START_REF] Sj Mizumori | Directionally selective mnemonic properties of neurons in the lateral dorsal nucleus of the thalamus of rats[END_REF], the lateral mammillary nucleus [START_REF] Sharp | Lesions of the mammillary body region severely disrupt the cortical head direction, but not place cell signal[END_REF], the retrosplenial cortex [START_REF] Longtang L Chen | Head-direction cells in the rat posterior cortex[END_REF] and the striatum [START_REF] Sidney | Spatial and behavioral correlates of striatal neurons in rats performing a self-initiated navigation task[END_REF], suggesting that the directional signal can be calculated in regions of the brain outside the hippocampal formation [START_REF] Van Strien | The anatomy of memory: an interactive overview of the parahippocampal-hippocampal network[END_REF].

Since HD cells fire allocentrically and depending only on the ongoing direction of the animal, and not on the specific location within the environment, they have been interpreted as the "compass" used for navigation in the cognitive map [START_REF] Jeffrey S Taube | The head direction signal: origins and sensory-motor integration[END_REF]. HD cells primarily rely on external landmarks to represent the motion direction [START_REF] Jeffrey S Taube | Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis[END_REF][START_REF] Jeffrey S Taube | Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations[END_REF], although they are known to respond to self-motion cues [START_REF] Edward | Head direction cells in rats with hippocampal or overlying neocortical lesions: evidence for impaired angular path integration[END_REF] as well as contextual conditions [START_REF] Jacob | An independent, landmark-dominated head-direction signal in dysgranular retrosplenial cortex[END_REF] when visual information is unavailable or unreliable [START_REF] James J Knierim | Place cells, head direction cells, and the learning of landmark stability[END_REF].

recurrent neural neural networks and attractors

The fruit fly central complex

It is also very important to mention experimental evidence in animals simpler than rodents, here in fact we will focus on the fruit fly, in fact even insects are excellent in space navigation but at the same time they are much simpler than rodents and this allows to study the neural circuits really in detail with state-of-the-art experimental techniques such as two-photon calcium imaging in head-fixed walking flies and optogenetics in order to get a mechanistic description of the latter. In particular here we will discuss neurons that are equivalent to HD cells previously discussed for rodents giving the first direct experimental evidence of the mechanism of continuous attractors in the brain.

Specifically, let's consider a part of the brain of the fruit fly, called central complex, common with some variations to a large class of insects, and in particular let's focus on a sub-part called ellipsoid body, where are arranged in a ring a type of neurons named compass cells or E-PG neurons (similar to HD cells).

Looking at the activity of these neurons Seelig et al. [START_REF] Johannes | Neural dynamics for landmark orientation and angular path integration[END_REF] have identified that this ring structure presents a single activity bump that acts as an abstract representation of the fly orientation and where the activity of the network is confined to move on this ring in the neural activity space. Moreover, this representation persists in the absence of visual and self-motion cues.

Then, using the fact that in this context we have a compass network with a strong topography, that it is possible to monitor the activity of entire populations of cell types, that it is possible to selectively perturb the activity of specific neurons and that one has access to both structural and functional connectivity, Kim et al. [START_REF] Soo | Ring attractor dynamics in the Drosophila central brain[END_REF] have been able to demonstrate that the neural circuitry of the ellipsoid body of the fly must be implemented by a recurrent network with local excitation and uniform inhibition, i.e., the same type of models discussed in Section 3.4, giving for the first time direct evidence of this mechanism in the brain.

Another important question to be understood from a mechanistic point of view is how the bump formed in the compass network can move in the absence of visual stimuli (in the dark). To do this Turner et al. [START_REF] Turner-Evans | Angular velocity integration in a fly heading circuit[END_REF], always using two-photon calcium imaging and electrophysiology in head-fixed walking flies, were able to identify in the central complex another population of neurons, called P-ENs, that simultaneously encodes heading and angular velocity, and is excited selectively by turns in either the clockwise or counterclockwise direction. They showed how these mirror-symmetric turn responses combine with the neurons' connectivity to the E-PG neurons to create a mechanism for updating the fly's heading representation when the insect turns in absence of visual inputs.

It should now also be clear that the study of animals as simple as drosophila is essential to understand the functioning of neural circuits from a mechanistic point of view (much more complicated thing to do in more complex brains like those of mammals) and in fact this line of research is proving extremely fruitful today [START_REF] Gruntman | The computation of directional selectivity in the Drosophila OFF motion pathway[END_REF][START_REF] Soo Kim | Generation of stable heading representations in diverse visual scenes[END_REF]. In addition, mechanisms similar to those presented for insects have been successful even in more complex animals such as mammals (bats) [START_REF] Finkelstein | Dynamic control of cortical head-direction signal by angular velocity[END_REF].

Grid cells

Another key example is the one provided by the grid cells discovered by E. and MB. Moser (also Nobel Prize winners together with O'Keefe in 2014 for this finding) in the medial entorhinal cortex (mEC) [START_REF] Fyhn | Spatial representation in the entorhinal cortex[END_REF][START_REF] Hafting | Microstructure of a spatial map in the entorhinal cortex[END_REF]. These neurons have the particular property of exhibiting triangular, periodic grid-like spatial selectivity, see Fig. 13(c).

The grid cells can be described by the orientation and period of the grid. These cells enjoy a property called "topography" which consists in the fact that neighboring neurons in the mEC have similar period, or grid spacing, while this increases along the dorsoventral axis of the cortex [START_REF] Vegard Heimly Brun | Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex[END_REF]. Moreover, grid cells have the fundamental property of not changing their mutual relationship in different environments, namely that the superimposition of their firing fields is constant regardless of external conditions, since the firing fields translate and rotate coherently in different familiar environments [START_REF] Fyhn | Hippocampal remapping and grid realignment in entorhinal cortex[END_REF].

The independence of spatial encoding from the context of grid cells, as opposed to the complex variability of place cells, has led to their interpretation as a putative substrate for the representation of a universal metric for navigation [START_REF] Hafting | Microstructure of a spatial map in the entorhinal cortex[END_REF][START_REF] Bruce L Mcnaughton | Path integration and the neural basis of the'cognitive map[END_REF][START_REF] John | Dual phase and rate coding in hippocampal place cells: theoretical significance and relationship to entorhinal grid cells[END_REF][START_REF] Yoon | Specific evidence of low-dimensional continuous attractor dynamics in grid cells[END_REF]. A contextindependent metric is essential to perform path integration, i.e., the process of updating one's cognition of self-location based on the estimation of linear and angular direction and velocity from proprioception and vestibular information, which allows for navigation in a known environment even in the absence of visual guidance, as we had already mentioned in Section 3.3.

Moreover, since anatomically mEC projects into the hippocampus, it is thought that the weighted sum of grid cells activity can produce localized activity (place field) in the hippocampus and also that changes in these weights may be responsible for the phenomenon of random remapping. The process of formation of place cells from grid cells is an hot topic of study today, in fact, for example, based on the above, it is not clear why in newborn animals are observed place cells before the emergence of grid cells. Experiments in which the activity of neurons is measured simultaneously in mEC and hippocampus could help to better understand this fundamental problem [START_REF] Bush | What do grid cells contribute to place cell firing?[END_REF].

It is important to note, from the point of view of mathematical models of recurrent neural networks that store continuous attractors, that the key difference between single activity bump models such as place cells or even head-direction cells and models with multiple bumps (in this case the bumps are located on the triangular lattice), lies on the fact that in the first case we have a global inhibition mechanism, while in the other case we must have an inhibition that depends on the distance, so a local and not global inhibition (just having a local excitation is not enough) [START_REF] Si | Continuous attractor network model for conjunctive position-by-velocity tuning of grid cells[END_REF][START_REF] Spalla | Can grid cell ensembles represent multiple spaces?[END_REF].

Prefrontal cortex

Indirect evidence for the continuous attractor mechanism is present as well in the prefrontal cortex (PFC), a part of the mammalian brain that subserve higher cognitive functions like decision making and working memory, during the delay period in an oculomotor delayed response task.

Indeed, Wimmer at al. [START_REF] Wimmer | Bump attractor dynamics in prefrontal cortex explains behavioral precision in spatial working memory[END_REF] analyzed behavioral and electrophysiological data from monkeys performing a spatial working memory task and tested predictions from a continuous attractor hypothesis, see Sections 2.2 and 3.4, for spatial memory maintenance. Their analysis support the idea that PFC activity represents spatial memories in a fixed-shape bump of activity that is used for guiding behavior, but is liable to cumulative encoding errors as a result of random fluctuations.

Thus validating the concept of prefrontal persistent activity as the basis of spatial working memory, supporting the continuous (or finely discretized) nature of spatial memory encoding in PFC, and being consistent with bump attractor dynamics mediating cognitive function in the cortex.

Other examples

Other examples for the mechanism of continuous attractors in the brain include:

-simple cells in the V1 area of the visual cortex coding for the orientation of a bar presented to the retina, see Hubel and Wiesel (Nobel prize winners in physiology or medicine in 1981) experiments [START_REF] David | Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[END_REF][START_REF] David | Receptive fields and functional architecture of monkey striate cortex[END_REF];

-border cells [START_REF] John | Geometric determinants of the place fields of hippocampal neurons[END_REF] responsive to the walls of an environment were reported in several regions of the hippocampal formation, see Fig. 13(d);

-speed cells [START_REF] Kropff | Speed cells in the medial entorhinal cortex[END_REF] that are neurons which firing rates depend on an animal's speed through its environment;

See [START_REF] James | Attractor dynamics of spatially correlated neural activity in the limbic system[END_REF][START_REF] Wu | Continuous attractor neural networks: candidate of a canonical model for neural information representation[END_REF] for nice reviews on the topic.

O P T I M

A L C A PA C I T Y-R E S O L U T I O N T R A D E -O F F I N M E M O R I E S O F M U LT I P L E C O N T I N U O U S AT T R A C T O R S
After Chapter 3 it should be clear that recurrent neural networks are powerful tools to explain how attractors may emerge from noisy, high-dimensional dynamics. We study in this Chapter how to learn the ∼ N 2 pairwise interactions in a recurrent neural network with N neurons to embed L manifolds of dimension D N, showing that the capacity, i.e., the maximal ratio L/N, decreases as | log | -D , where is the error on the position encoded by the neural activity along each manifold, or in other words, that recurrent neural networks are flexible memory devices capable of storing a large number of manifolds at high spatial resolution. These results rely on a combination of analytical tools from statistical physics of disordered systems and random matrix theory, extending Gardner's classical theory of learning to the case of patterns with strong spatial correlations.

All the findings presented in this Chapter have been published in [START_REF] Battista | Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks[END_REF].

introduction

At this stage it should be evident the objective of the thesis, namely how to store multiple continuous attractors in a recurrent neural network (RNN). Furthermore, after the reading of Chapter 3, the reasons for this kind of research should be understood from the point of view of computational neuroscience, see Section 3.5, as well as the problems with the current theory on the subject should be clear, see Section 3.7. Recall in fact that in the last twenty years no theoretical progress has been made on this fundamental problem despite the presence of several critical aspects of popular models such as:

-due to interferences between the different manifolds stored in the same network, the attractors are not continuous any longer, and are effectively discrete. How to practically learn smooth attractors and ensure a prescribed level of spatial error , where by spatial error we indicate the typical distance on the maps between two nearby positions where the activity bump can get stuck (actually memorized map positions), is an open issue.

-Moreover, theoretical studies [START_REF] Francesco | Attractor neural networks storing multiple space representations: a model for hippocampal place fields[END_REF][START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Phase diagram[END_REF][START_REF] Monasson | Transitions between spatial attractors in place-cell models[END_REF] show that the critical capacity (maximal number of maps that can be memorized in a RNN) corresponding to the standard Hebbian-like prescription (3.15), an ad hoc learning rule for the pairwise interactions between neurons summing up the different contributions corresponding to optimal capacity-resolution trade-off in memories of multiple ca each single attractor taken independently of the others, see Section 3.6, is very low. In addition, these studies do not convey any information about the spatial error with which the different continuous attractors are encoded in the sense that it is not clear in the different maps which and how many positions are really stored.

Is it possible to define not a priori learning rules for the network connectivity matrix in order to have control over the spatial error with which the different maps are actually stored? Also, once this is done, is it possible to understand the optimal trade-off between capacity and spatial error? Answering these questions is of fundamental importance also because one might be interested in very different scenarios, i.e., to store in a RNN a few maps but with very small spatial error or maybe to store many maps in a coarser way. In the following we present a theory in order to bypass these problems.

the model

To answer the questions mentioned in Section 4.1 and Section 3.7 we introduce the following model with which we want to study the optimal storage of multiple quasicontinuous manifolds with prescribed spatial error in a RNN with N binary neurons (σ i = 0, 1) and real-valued, oriented connections W ij , same scheme as Figs. 3(a) and (c).

Inspired by the phenomenology of place cells and place fields, see Section 3.3, we define in this context a map as a D-dimensional cube with unitary volume and periodic boundary conditions1 in which each neuron i = 1, . . . , N of the RNN has a randomly located input (place) field (PF), i.e., a D-dimensional sphere of volume φ 0 < 1, centered in position r i , see one of the two maps in Fig. 14(b).

We want than to store in the network L = α N (as usual we define the parameter α = O(1) as the load, see Section 3.6) maps that differ through random rearrangements of the PF center positions, r i , = 1, . . . , L (see Fig. 14(b)). Imposing random remapping between the different maps is essential to make them orthogonal as we already discussed in Section 3.5, where by orthogonal maps we mean that the activity configurations of the neurons coding for the positions of the different environments are orthogonal to each other, and thus avoid as much interference between them as possible. Now instead of trying to memorize the different maps with a priori learning rules as Hebbian-like ones (3.15), see Section 3.6, where by a priori we mean here that the learning of the maps through the choice of the connectivity matrix is done without having an explicit control on the positions actually stored in the different environments (what are the fixed points of the network dynamics in the different maps?), we approximate each map through a collection of p random positions r ,µ , µ = 1, . . . , p, and we will define in the next Section 4.3 how to store in the connectivity matrix of the RNN explicitly these positions. The spatial error , which is an index to the accuracy of how each map is stored, is defined as the typical distance between nearest neighbor positions on an environment, see Fig. 14(a), and in this setting where we have approximated the maps with p random positions it scales as p -1 D [START_REF] Chandrasekhar | Stochastic problems in physics and astronomy[END_REF]. In this way we can have control on the accuracy of the coding because increasing p correspond to decrease the spatial error , and in particular we will be able to store quasi-continuous attractors in the large p limit.

At this stage for every position in every map, r ,µ , we extract an N-dimensional pattern of activity to be stored in the network: the neuron i is active (σ ,µ i = 1) if the distance between this position and the center of the PF associated with the neuron i in map , |r ,µr i |, is smaller than the PF radius r c , and silent (σ ,µ i = 0) otherwise, see Fig. 14(b). Also this way of building the patterns to memorize is clearly inspired by the mechanism of place cells and place fields discussed in Section 3.3. We end up with a data-set of p × L, N-dimensional, binary patterns {σ ,µ i }, where i is the neuron index, the environment index and µ the index of the position to be stored in map .

It is essential to note that each pattern built in this way corresponds to a bump state for the network in the sense that precisely by construction, given a position in a certain environment, the related pattern has active neurons corresponding to PFs in that map that are close to the position itself (those that contain it), while the other neurons whose PFs are far away from the position (those that do not contain it), are silent.

Then, by storing a map in a RNN we mean that the patterns associated to the sampled positions on that map must be fixed points of the network dynamics and we will see in the next Section 4.3 how to learn the connections between neurons for this to be possible.

learning the optimal couplings

How do we store the different maps, that is the patterns generated in the previous Section 4.2, in the RNN?

Before discussing this we need to define a dynamics for the network. In this specific case we consider the standard sequential zero temperature Monte Carlo dynamics [START_REF] Luciano | Scientific programming: C-language, algorithms and models in science[END_REF][START_REF] Frenkel | Understanding molecular simulation: from algorithms to applications[END_REF][START_REF] Roy | Time-dependent statistics of the Ising model[END_REF][START_REF] Tuckerman | Statistical mechanics: theory and molecular simulation[END_REF]:

σ i (t + 1) = Θ j =i W ij σ j (t) , (4.1)
where Θ is the Heaviside step-function and at every time step t + 1 we choose uniformly at random the neuron i to update among the N ones, exactly the same kind of dynamics encountered in the previous Chapter 3 with 0 or 1 spins and without considering this time optimal capacity-resolution trade-off in memories of multiple ca (a) the local external threshold fields, in this case we don't need any threshold because as we will see the connectivity matrix will have both positive and negative interactions and this is enough to keep the network activity fixed. Moreover it is important to recall, as we saw in Section 3.2, that such a noiseless dynamics converges from any initial configuration of the neurons to its nearest fixed point, i.e., a configuration of the network neurons activity such that the dynamics remain stuck.

r 1 r 2 r 3 × × × × × × × × × ε (b)
At this point we want to choose minimal conditions for the connectivity matrix W such that the patterns generated in the previous Section 4.2 are fixed points of the neural dynamics (4.1), that is they are stored in the RNN. This property is equivalent to impose for every pattern in the data-set

∆ ,µ i ≡ 2σ ,µ i -1 j =i W ij σ ,µ j > 0 , ∀ i , , µ , (4.2)
where ∆ ,µ i is defined as the stability of the neuron i for the pattern associated with the map in position µ.

If we now look carefully at the definition of the RNN it is easy to realize that since we don't have self-connections (W ii = 0 , ∀ i), if we focus on a single neuron i and wonder what is the connectivity matrix row W i , an (N -1)-dimensional vector, that satisfies (4.2), we immediately realize that this problem is a perceptron problem [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Rosenblatt | Principles of neurodynamics: Perceptions and the theory of brain mechanisms[END_REF] with neuron i as output and the other N -1 neurons as input, see Fig. 15(a). In particular we have N independent perceptron problems to solve, one for each output neuron i, so that to consider all the rows of the connectivity matrix that after learning will be non-symmetric a priori. The important thing is that even though we have N distinct problems to solve, learning is done on the same data-set with the only difference being to change the output and input neurons in the different perceptrons, which after the learning phase will not be uncorrelated and will produce a well defined dynamics for the whole network.

In order for the patterns to be fixed points of the dynamics according to (4.2) we have to solve the N classification problems associated to the different perceptrons. In fact, if we look at the p × L binary patterns in the (N -1)-dimensional input space of one of the perceptrons, we see that the patterns are divided into two classes: the class that includes the patterns in which the output neuron is active and the class in which it is silent. Choosing the row of the connectivity matrix W i that satisfies (4.2) for a fixed output neuron i is equivalent to find the hyperplane that linearly separates the two classes. This problem, if solvable, admits infinite solutions (all solutions that can be reached by the perceptron algorithm [START_REF] Engel | Statistical mechanics of learning[END_REF][START_REF] Hertz | Introduction to the theory of neural computation[END_REF]), and among all these solutions we are interested in finding the hyperplane that maximizes the distance between the two classes, this problem therefore corresponds to a convex optimization problem with an unique solution [START_REF] Boyd | Convex optimization[END_REF], where this optimal distance is called maximal stability [START_REF] Jk Anlauf | The adatron: an adaptive perceptron algorithm[END_REF][START_REF] Krauth | Learning algorithms with optimal stability in neural networks[END_REF] or in machine learning terminology hard margin, because this particular solution to the perceptron problem is equivalent to a support vector machine (SVM) with a linear kernel and hard margin [START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Cortes | Support-vector networks[END_REF][START_REF] Scholkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], see Fig. 15

(b).

This choice of W i ensure the biggest basins of attraction in the pattern space, i.e., robustness against thermal noise [START_REF] Daniel J Amit | Retrieval phase diagrams for attractor neural networks with optimal interactions[END_REF][START_REF] Krauth | Learning algorithms with optimal stability in neural networks[END_REF][START_REF] Krauth | Basins of attraction in a perceptron-like neural network[END_REF], because the optimal hyperplane ensures that every perceptron is as robust as possible in the classification of the two classes (smallest generalization error).

To put in formulas these concepts, we are interested in finding the connectivity matrix W that maximizes the stabilities of the patterns to be stored

κ = max W min {i=1...N, =1...L,µ=1...p} 2σ ,µ i -1 j( =i) W ij σ ,µ j , (4.3)
optimal capacity-resolution trade-off in memories of multiple ca where all the rows of W are normalized to 1, i.e., j( =i) W 2 ij = 1 ∀ i, for details on the SVM algorithm see Appendix A.1.

( ) 1
( ) 0 1 σ 1 σ 4 σ 3 σ 5 σ 2 W 12 W 15
It is interesting to note that usually SVMs are used in supervised learning (fit of inputoutput relation from examples typically in high dimensions) problems where in general the output is simply a label associated to the input [START_REF] Chung | Classification and geometry of general perceptual manifolds[END_REF] while in this specific setting we are using the same technique in an unsupervised (find statistical features of data for clustering, dimensional reduction and so on) way in the sense that here the output neurons are part of the activity patterns to be stored that are composed of input and output together.

results of numerical simulations

Now that we have well defined the model in Section 4.2 and the dynamics of the network together with the learning rule in Section 4.3 we can start studying the results of simple numerical simulations and see if qualitatively the typical characteristics of RNNs that store continuous attractors are reproduced, see Sections 3.4 and 3.6.

Couplings obtained by SVM

First, we report some qualitative features of the couplings obtained by SVMs after the training process (Section 4.3) on the specific data discussed in Section 4.2.

-As shown in Fig. 16(a) and (c) the couplings W ij are correlated with the distances d ij = |r ir j | between the PF centers of the neurons i and j in the different maps .

Note that the dependence on distance is less marked in the single environments as the number L of maps increases, due to the interferences between the maps encoded in the same connectivity matrix, see Section 3.6.

-In order to sustain a bump state with average activity φ 0 (obviously for construction the average activity of the patterns to memorize is equal to the area of the PFs), couplings are excitatory at short distances, up to roughly the radius r c of the PF, and inhibitory at longer ones. The sign of the couplings can be intuitively understood. Two neurons at short distances have largely overlapping PFs: their activities are likely to be equal, and having a large coupling helps increasing the stability, see (4.2). If the distance is bigger than r c , the activities are likely to be different, hence inhibitory (negative) couplings would increase the stability, see Fig. 16(a) and (c).

-Moreover, in Fig. 16(b) and (d) we show the histograms of the couplings and it seems that the amplitudes decay with N. In agreement with [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF] we expect the average values and standard deviations to scale, respectively, as 1/N and 1/ √ N, see Sections 4.6 and 6.7.

Finite temperature dynamics (T > 0)

Second, once the coupling matrix {W ij } has been learned, we may perform Monte Carlo simulations to investigate the behavior of the network. Instead of considering the natural dynamics of the network without noise (4.1), we implement a noisy dynamical scheme, the same introduced in Section 3.4, but now without the threshold terms (see Section 4.3), where neuronal states are updated stochastically according to the probabilities [START_REF] Luciano | Scientific programming: C-language, algorithms and models in science[END_REF][START_REF] Frenkel | Understanding molecular simulation: from algorithms to applications[END_REF][START_REF] Roy | Time-dependent statistics of the Ising model[END_REF][START_REF] Tuckerman | Statistical mechanics: theory and molecular simulation[END_REF] Prob

σ i (t + 1)|{σ j (t)} = 1 1 + exp -1 T 2σ i (t + 1) -1 j( =i) W ij σ j (t) , (4.4)
with T as a temperature parameter to be set such that a bump of activity may form and sustain itself. This happen when T is comparable to, or smaller than the stability κ of the network. All that in order to show the diffusion of the activity bump within a map and optimal capacity-resolution trade-off in memories of multiple ca the transitions between maps [START_REF] Monasson | Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity[END_REF][START_REF] Monasson | Transitions between spatial attractors in place-cell models[END_REF][START_REF] Samsonovich | Path integration and cognitive mapping in a continuous attractor neural network model[END_REF] as already seen previously in Sections 3.1 and 3.6. We illustrate the dynamical properties of the model with two examples2 :

(b) (a) (c) (d)
-first, we consider a network with N = 1000 neurons, in which we store one map (L = 1) in dimension D = 2 and with average activity φ 0 = .3. We consider then the case in which the learning is done on a small number of points, p = 30, resulting in a large value of the stability (as we will see in fact the stability of the network decreases as the constraints that the connectivity matrix must satisfy increase, so L and p), κ = 1.7. And then the case in which p = 300 is higher, and the stability is smaller: κ = .6. Our noise parameter T is set to .8 to allow the bump to form in both cases.

In the large κ case, the bump, which is possible to visualize by looking at which are the PFs that correspond to the active neurons at a given time, gets stuck very quickly in one of the p training positions, depending on the initial configuration, see videos LargeKappaL1.mp4 and LargeKappaL1Bis.mp4. In the small κ case, the bump diffuses on the map, see video SmallKappaL1.mp4. For larger p, the bump can easily travel through the environment, with a large diffusion coefficient; in contrast, in the small p case, the stability landscape is very rough (only the stored positions have a very high positive stability value, the others can have a priori negative stabilities, which is why the dynamics remain locked, see Fig. 34(a)) and the bump is stucked close to the stored positions.

-In the second example we consider the case of L = 2 maps and p = 150 points. The other parameters have the same values as in the first example, i.e., the stability is fixed to κ = .5. In the video SmallKappaL2.mp4 we see that, as κ is small, the bump diffuses in one maps and sporadically jumps to the other map, as it should be due to the fluctuations induced by the temperature and a reasonable number of stored positions per map. In fact, since the two maps are orthogonal by construction, if we focus on which are the PFs that correspond to the active neurons at a given time, we observe that in one of the two maps these are clustered and therefore form a bump of activity (the position in one of the maps we are recalling at that time), while in the other map we have that these PFs are basically random, as it should be given that the bump must be located in one of the maps only 3 . Moreover, since the dynamics has a noise level, it is possible to observe the transitions of the bump from one map to another simply by noticing that at a given time the network activity has PFs clustered in one map and random in the other, and when a transition occurs the opposite happens.

different environments in the connectivity matrix of the neural network and the blue circles represent the centres of the PFs whose corresponding neurons are active at a fixed time (the PFs whose neurons are silent at a certain time are not shown). Also note the presence of periodic boundary conditions in the maps that can sometimes make the bump not easy to visualize.

3. The case of having activity bumps located at the same time in multiple maps is possible if they are correlated. Since in our model the PFs are random in the different maps (global remapping) this situation is unlikely to be possible by construction.

optimal capacity-resolution trade-off in memories of multiple ca Moreover, the fact that the stored attractors can be continuous depends primarily on the number of place cells encoding the map and the size of the PFs, as these parameters define the possible number of distinct patterns and therefore the positions that can be stored in the map itself. Once the map is defined, the dynamical properties of the activity bump are basically fixed by the number of positions memorized per map and if the difference between the points is a fraction of the bump's width, the discretization in practice is no longer perceived. To test if our model is able to do the tracking of moving inputs we made a simulation where, starting from a RNN of N = 1000 neurons, we learn L = 2 maps in D = 1, where the size of the PFs is fixed at φ 0 = .2 and where are stored p = 250 positions per map. Then we go to study the dynamics of the network with a temperature T = .5 and also add an external input that acts on individual neurons in the following way:

I i (t) = exp(- |x i -x(t) | φ 0
), where x(t) represents the center of the input at a given time in one of the two maps. From Fig. 17 it is clear that the center of the activity bump 4 (blue points) follows the input (orange curve) very well, even if the input is abruptly moved from one map to another. From this example we can see that the continuous tracking of moving inputs is achieved in our model.

Zero temperature dynamics (T = 0) and spatial error

Now instead, if we are interested in calculating the spatial error with which the maps have been stored, we consider the natural dynamics of the network (4.1). Starting from an initial activity configuration, we track the system dynamics for at most N 2 Monte Carlo steps (N sweeps), and retain the visited configuration with the minimum number of violated constraints, i.e., with the highest number of non-negative stabilities 5

∆ i = (2σ i -1) j( =i) W ij σ j 0 . ( 4.5)
We generate L environments and p points in each of them, learn the coupling matrix corresponding to these p × L patterns. We then pick at random a position in one of the learned maps, and use that position to construct the initial activity configuration of the dynamics. After the dynamics described above is done we keep the final configuration and use it to decode the final position on that map, as the center of mass of the PFs (on the map) of the active neurons in the final configuration. The distance between this 4. See the next paragraph for details on how to decode the position of the center of the bump from network activity.

5. The choice of N sweeps as a maximal simulation time is empirical: we do not find that significantly better results are obtained by increasing this bound. Actually, the dynamics often ends up in a fixed point with stabilities ∆ i 0 for all neurons i in much less sweeps. Nevertheless, it is necessary to set a maximum number of sweeps because the learned connectivity matrix is not a priori symmetrical and therefore the convergence of dynamics is not ensured. estimated position and the initial one (taking care of the periodic boundary conditions), after averaging over many initial positions (100 in the figures showed), defines the spatial error with which the maps have been stored (the spatial error is roughly the same in all the environments since p is the same in all maps and the positions are random).

ℓ = 1 ℓ = 2
Fig. 18(a) 6 shows that becomes quite large as L increases if we use Hebbian-like learning rules (3.15) for the connectivity matrix. However, with the maximal-stability learning rule (Section 4.3), the spatial error can be tuned at will by varying p. For a fixed p, remains remarkably stable as the load increases until its critical value is reached. This is in sharp contradistinction with the Hebb rule case (3.15), for which quickly increases with the number of maps. The p patterns form a discrete approximation of the map, with average spatial error scaling as = p -1/D , i.e., as the typical distance between neighboring points [START_REF] Chandrasekhar | Stochastic problems in physics and astronomy[END_REF], see Fig. It is therefore clear now that by increasing p we can control the accuracy of the coding by decreasing the spatial error . It is obvious, however, that requiring small for the maps has a cost in terms of how many maps can be stored because there are more 6. In this figure we show only the case of an exponential kernel (where the parameters have been chosen such that to have zero spatial error in case of a single map) for the Hebbian rule, but it can be seen from Fig. 19 that for example a Gaussian kernel has similar performance as the exponential one. In general it is true that the specific value of the spatial error depends on the choice of the kernel (and its parameters), however the important point to note is that regardless of the kernel used, a Hebbian type rule has a spatial error that increases with the number of maps while the key to our approach is to have a method that can maintain the same level of spatial error even when storing multiple maps (obviously until the critical capacity is reached). constraints that must be taken into account by the connectivity matrix. The goal now is therefore to understand this trade-off between capacity and resolution.

Comparison with Hebb rule

Before studying how the optimal stability κ defined in Eq. ( 4.3) and obtained from SVMs (Section 4.3) behaves as a function of α = L N and p, as a sanity check, we show that it is always much higher than the maximal stability achievable with Hebbian-like rules (3.15), even after the optimization over the interaction kernel w. In fact, this should be true by construction, see Section 4.3.

In order to show that we consider the cases of an exponential kernel,

w(d) = a e -d/b -1 , (4.6)
and of a Gaussian kernel,

w(d) = a e -d 2 /b -1 . (4.7)
We then optimize over a and b; the value of the negative offset at large distance is arbitrary, since couplings are normalized row by row. Results for a typical sample are shown in Fig. 19. The stability for the best kernel w is always much lower (and negative in the examples considered here) than the optimal stability κ found with SVM.

Capacity-Resolution trade-off

The optimal stability κ (4.3) is shown in Fig. 20(a) as a function of the load α and of the number p of prescribed fixed points. As expected, κ(α, p) is a decreasing function of α and p: increasing the number of maps or enforcing finer spatial error reduces the stability.

The value of the load α at which κ(α, p) vanishes defines the critical capacity α c (p), that is, the maximal load sustainable by the network as a function of the required spatial error. Fig. 20(a,inset) shows that α c (p) decreases proportionally to 1/p at low p, and then much more slowly as p grows, see Appendix A.2 for details on how we estimated numerically the critical capacity.

For small p, all L × p patterns are roughly independent, and we have α c (p)

α c (1)
p , where α c (1) is the capacity of the perceptron with independent, biased patterns having a fraction φ 0 of active neurons [START_REF] Gardner | The space of interactions in neural network models[END_REF]. As p gets large, substantial redundancies between the p patterns within a map appear, as nearby positions define similar patterns, see Fig. 14, and the capacity is expected to decrease less quickly with p. The cross-over takes place at p c.o. ∼ 1/φ 0 , see Fig. 20(b). The nontrivial behavior of α c (p) when p p c.o. will be characterized in the analytical study in Section 4.5 and Section 4.6.

gardner's theory for rnn storing spatially correlated patterns

As we have just discussed in Section 4.4, increasing p corresponds to decrease the spatial error , but we can easily understand that increasing p, and therefore the number of constraints that must be satisfied by the connectivity matrix has a cost in terms of how many maps the RNN can store. In order to try to understand this trade-off between capacity and spatial error analytically we used, as a first attempt, Gardner's approach developed in the 1980s [START_REF] Gardner | The space of interactions in neural network models[END_REF][START_REF] Gardner | Optimal storage properties of neural network models[END_REF].

The idea is to look at the space of all possible connectivity matrices that have our patterns, see Section 4.2, as fixed points of the network dynamics (4.1), i.e., all possible hyperplanes that linearly separate the two classes for all N perceptrons, see Section 4.3. In this feasible set of solutions there will be as well the one that corresponds to the connectivity matrix with maximal stability κ, namely the solution with the maximum margin for every perceptron, see Section 4.3. When we increase α = L N at fixed p the logarithm of the volume of this set of solutions decrease until it shrinks to a single point, as we will explain in detail below. When this happens we found the critical capacity because adding a new constraint would make the problem impossible to solve. So here we are going to extend the Gardner theory for the capacity of the perceptron (SVM with linear kernel and hard margin) to the case of continuous attractors.

We shall consider the now usual RNN consisting of N neurons, with p × L binary (0 or 1) patterns {σ µ i } constructed by drawing randomly p positions in each of the L environments, see Section 4.2, so that the resulting patterns are spatially correlated and to be stored as fixed points of the network dynamics

σ i (t + 1) = Θ j =i W ij σ j (t) . (4.8)
For the storage of the patterns a strong characterization of fixed points is provided by

∆ ,µ i ≡ 2σ ,µ i -1 j =i W ij σ ,µ j > κ , ∀ i, , µ (4.9)
which ensures a finite radius of attraction for κ > 0 [START_REF] Gardner | The space of interactions in neural network models[END_REF][START_REF] Gardner | Optimal storage properties of neural network models[END_REF][START_REF] Thomas | Domains of attraction in neural networks[END_REF][START_REF] Krauth | Learning algorithms with optimal stability in neural networks[END_REF][START_REF] Krauth | Basins of attraction in a perceptron-like neural network[END_REF][START_REF] Krauth | The roles of stability and symmetry in the dynamics of neural networks[END_REF]. Then we can choose to normalize W's by enforcing the "spherical" constraint

j =i W 2 ij = 1 , ∀ i , (4.10)
optimal capacity-resolution trade-off in memories of multiple ca for each row of the matrix of couplings.

The volume in the space of couplings {W ij } that corresponds to admissible solutions of the storage problem, i.e., the partition function, can be written, up to a normalization factor, in the form [

109] Z = ∞ -∞ N i =j dW ij i δ j( =i) W 2 ij -1 i, ,µ Θ (2σ µ i -1) j( =i) W ij σ µ j -κ (4.11)
and is equal to the product of the N single-site volumes Z i , with i = 1, . . . , N. So we may focus for example on the volume associated with i = 1:

Z 1 = ∞ -∞ N j=2 dW j δ j 2 W 2 j -1 ,µ Θ (2σ µ 1 -1) j 2 W j σ µ j -κ , (4.12)
where W j ≡ W 1j .

Using the replica method [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Mezard | Information, physics, and computation[END_REF][START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF][START_REF] Nishimori | Statistical physics of spin glasses and information processing: an introduction[END_REF][START_REF] Urbani | Statistical Physics of glassy systems: tools and applications[END_REF], we compute the average of log Z 1 over the patterns as

log Z i = lim n→0 Z n i -1 n , (4.13)
where . represent the quenched average over the data distribution, so we get

Z n 1 = ∞ -∞ j,a dW ja a δ j 2 W 2 ja -1 ,µ,a Θ (2σ µ 1 -1) j 2 W ja σ µ j -κ , (4.14)
where a = 1, . . . , n is the replica index.

Let us now introduce the integral representation of the Heaviside functions and exploit the statistical independence of the different maps so that we can write

Z n 1 = ∞ -∞ j,a dW ja a δ j W 2 ja -1 χ(W) αN (4.15)
where L = α N and with χ(W) equal to

p µ=1 dr µ N j=1 dr j ∞ κ µ,a dt µa ∞ -∞ µ,a d tµa 2π e i µ,a tµa t µa j e -i µ,a tµa (2σ µ 1 -1)W ja σ µ j , (4.16)
where rµ denotes the p prescribed locations in the environment, and r j the N PF centers of the neurons in the map (these variables must be integrated on the map, i.e., D-dimensional cube with periodic boundary conditions).

Let Φ(r) be the indicator function of the PF centered in 0: Φ = 1 if |r| < r c , where r c is the radius of the PF (with dr Φ(r) = φ 0 ), and 0 otherwise. Let Γ (r) = dr Φ(r ) Φ(rr ) be the correlation function of Φ and σ(r 1rµ ) = 2Φ(r 1rµ ) -1. Given p points rµ , µ = 1, . . . , p drawn uniformly at random in space, we define the p × p Euclidean random matrix 7 [START_REF] Mézard | Spectra of Euclidean random matrices[END_REF] with entries

Γ µ,ν R ≡ {r µ } = Γ rµ -rν -φ 2 0 . (4.17)
We first carry out explicitly the integrals over the PF centers with indices j = 2, 3, ..., N, leaving the integrals over r 1 and all rµ in χ(W) so that we obtain

χ(W) = p µ=1 dr µ dr 1 ∞ κ µ,a dt µa ∞ -∞ µ,a d tµa 2π e i µ,a tµa t µa ×e -iφ 0 µ,a,j W ja tµa σ(r 1 -r µ ) e -1 2 µ,ν,a,b,j W ja W jb Γ µ,ν ( R) tµa tνb . ( 4.18)
We can now introduce the order parameters

m a = j 2 W ja , ∀a (4.19)
and

q ab = j 2 W ja W jb , a < b (4.20)
to be added with the standard trick of inserting the integral representation of Dirac delta functions and fix their values through integration and rewrite where ûa are the Lagrange multipliers enforcing the spherical constraints, ma the ones that enforce the definition of m a and qab the ones that enforce the definition of q ab , and they simply come from the integral representation of the Dirac delta functions.

Z n 1 = ∞ -∞ ja
It is possible then to rewrite χ(W) as

χ({m a , q ab }) = µ dr µ dr 1 ∞ κ µ,a dt µa √ 2π ∞ -∞ µ,a d tµa √ 2π e -1 2 µ,ν,a,b q ab Γ µ,ν ( R) tµa tνb e -i µ,a m a φ 0 tµa σ(r 1 -r µ )+i µ,a tµa t µa , (4.22)
7. See Chapter 5 for details on Euclidean random matrices.

where due to translation invariance, the integral over r 1 is irrelevant, and we can set r 1 = 0.

We can also define from (4.21) the quantity Y({ ûa , ma , qab }) as

∞ -∞ ja dW ja e a ûa 2 (1- j W 2 ja ) e a ma (m a - j W ja ) e a<b qab (q ab - j W ja W jb ) (4.23)
so that we can write (4.21) as

Z n 1 = a d ûa 4πi a d ma dm a 2πi a<b d qab dq ab 2πi Y({ ûa , ma , qab }) χ({m a , q ab }) αN . (4.24)
It is possible now make the Replica Symmetric (RS) Ansatz [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF] (expected to be valid since the domain of suitable couplings is convex, see Section 4.3) on the structure of the order parameters so that ma = m , ∀ a , (4.25

) qab = q , a < b , (4.26)
and the same for the conjugate variables and as well the Lagrange multipliers associated to the normalization of the weights.

We can therefore write within the limit of large N and small n:

1 nN log Y = û 2 - q q 2 + lim n→0 1 n log ∞ -∞ a dW a e -û 2 a W 2 a -q a<b W a W b -m a W a . (4.27)
It is now possible to compute the Gaussian integrals over {W a } and take the small n limit so we get:

1 nN log Y = û 2 - q q 2 - 1 2 log ( û -q) + m2 - q 2( û -q) . ( 4.28)
We are now able to calculate the remaining integrals over the order parameters and their conjugated variables with the saddle-point method. So we start by writing the saddle-point equations relative respectively to m, û and q from the above equation:

m = 0 , (4.29) 1 û - q = 1 -q , (4.30) q = -q (1 -q) 2 . (4.31)
By solving this equations we finally find:

m = 0 , (4.32) q = -q (1 -q) 2 , (4.33) û = 1 -2q (1 -q) 2 . (4.34)
By injecting these results in (4.28) we found that when q → 1, so when the volume associated with the admissible solutions is reduced to one point and therefore we are at critical capacity, we have:

1 nN log Y 1 2(1 -q) . ( 4

.35)

As for the term χ this can be rewritten in the RS ansatz and using a Gaussian integral trick as:

χ = µ dr µ µ dz µ √ 2π exp -1 2 µ,ν z µ Γ( R) -1 µ,ν z ν det Γ( R) (4.36) × ∞ κ µ dt µ ∞ -∞ µ d tµ 2π e -1 2 (1-q) µ,ν tµ Γ µ,ν tν e i µ tµ (z µ √ q+t µ -mφ 0 σ(r 1 -r µ ))) n .
After performing the Gaussian integral in { tµ }, taking the small n limit and the q → 1 limit we get:

log χ n - 1 2(1 -q) µ dr µ µ dz µ √ 2π exp -1 2 µ,ν z µ Γ( R) -1 µ,ν z ν det Γ( R) × min {t µ κ} µ,ν t µ -(z µ + mφ 0 σ(r 1 -rµ ) Γ( R) -1 µ,ν t ν -(z ν + mφ 0 σ(r 1 -rν ) . (4.37)
Putting all together we get:

Z n 1 -1 nN 1 2(1 -q) 1 -α µ dr µ µ dz µ √ 2π exp -1 2 µ,ν z µ Γ( R) -1 µ,ν z ν det Γ( R) × min {t µ κ} µ,ν t µ -(z µ + mφ 0 σ(r 1 -rµ ) Γ( R) -1 µ,ν t ν -(z ν + mφ 0 σ(r 1 -rν ) . ( 4

.38)

optimal capacity-resolution trade-off in memories of multiple ca

We finally obtain the expression for the critical capacity

α c (κ, p) = max m α c (m; κ, p),
where α c (m; κ, p) is the load α cancelling the terms inside the curly brackets in (4.38).

We can rewrite this final expression for the maximal load at fixed κ and p like

α c (κ, p) = 1/ min m E p ( R, Z, m; κ, φ 0 ) R,Z (4.39)
where the minimum is taken over m = j( =i)

W ij . In the formula above, • denotes the average over the vectors R = (r 1 , ..., rp ) of p positions rµ drawn uniformly at random in the D-dimensional cube with periodic boundary conditions, and Z = (z 1 , ...z p ) drawn from the multivariate centered Gaussian distribution with R-dependent covariance matrix

Γ µ,ν ( R) = Γ rµ -rν -φ 2 0 .
Here, Γ (d) is the overlapping volume between two PF, whose centers are at distance d from one another, hence, Γ (0) = φ 0 . The function E p in Eq. (4.39) is defined through

E p ( R, Z, m; κ, φ 0 ) = min {t µ κ} µ,ν t µ -(z µ + mφ 0 σ(r µ ) Γ( R) -1 µ,ν t ν -(z ν + mφ 0 σ(r ν ) (4.40)
where σ(d) = 1 if d < r c , -1 otherwise and we have set r 1 = 0 for traslation invariance. r c is the radius of the PF, i.e., the smallest number such that Γ (2 r c ) = 0.

In practice, computing α c (κ, p) from Eq. (4.39) is quite involved from a numerical point of view, as it requires to solve the p-dimensional semi-definite quadratic optimization problem [START_REF] Boyd | Convex optimization[END_REF][START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF] in Eq. (4.40), as well as to average over the random vectors R and Z. This can be accurately done for small enough p, with results in excellent agreement with the SVM simulations, see Fig. 21.

Notice that as already mentioned in Section 4.4, for p = 1, our calculation reproduces Gardner's critical capacity α c (κ, 1) for independent and biased patterns [START_REF] Gardner | The space of interactions in neural network models[END_REF]. This is expected as spatial correlations between patterns within a map appear when p 2, see Appendix A.3.

quenched input fields theory

In order to compute the capacity α c (κ, p) with the Gardner approach in Section 4.5 we have to solve a p-dimensional constrained quadratic optimization problem, depending on p correlated Gaussian random variables, see Eqs. In other words Eq. (4.39) seems, unfortunately, intractable for large p. The intricate dependence on p, i.e., showing up through the Gaussian correlations between the p random fields z µ in Eq. (4.40), stems from the average (in each map ) over the N PF centers {r i } at fixed positions {r ,µ }.

To avoid introducing these correlations and having an explicit dependence on the parameter p, we consider an alternative calculation scheme, where the p positions in each map are averaged out, while keeping the L × N centers quenched.

To further simplify the calculation we neglect in the effective action all terms of order 3 in the couplings {W ij } following closely [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF] but, as we will discuss later, this Gaussian approximation is expected to be exact in our setting in the large-p limit.

A potentially interesting feature of this approach is that it holds at fixed PFs, instead of averaging over them as in Section 4.5, and could be applied to specific situations, i.e., sets of PFs measured in experiments.

optimal capacity-resolution trade-off in memories of multiple ca

Replica calculation

Starting from the replicated volume Z n 1 in (4.15), we now perform first the average over each one of the p locations in χ(W) in (4.16) as follows

dr µ exp   -i a tµa (2σ ,µ 1 -1) j 2 W ja σ ,µ j   = exp -i a m a tµa - 1 2 a,b tµa (q ab -m a m b ) tµb + O( t3 ) (4.41)
where we have reintroduced the map index to underline that the PFs are kept fixed here. The order parameters in the formula above are

m a = j 2 W ja 2 Γ |r j -r 1 | -φ 0 (4.42)
and

q ab = j,k 2 W ja W kb Γ |r j -r k | , (4.43)
to be introduced as in Section 4.5.

We then simplify the calculation with two approximations:

-We truncate the expansion in powers of t in (4.41) to the second order, and omit all higher order terms. This amounts to approximate the distribution of couplings {W ij } (at fixed PFs) by a Gaussian. This approximation is valid only if the couplings fluctuate weakly around their means, which is the case in the large-p limit, see Section 4.6.3.

-We also neglect the dependence of the order parameters m and q above on the map . The histograms of the overlaps q measured by SVM are shown in Fig. 22.

As can be seen from the figure, the distribution of overlaps is not concentrated in the large-N limit at fixed p. Therefore, while m a = m a and q ab = q ab is a valid Ansatz for the saddle-point equations of the log. partition function (due to the statistical equivalence between the maps), we expect Gaussian fluctuations to be relevant even in the infinite-N limit. However, as p increases, these fluctuations are smaller and smaller, and are asymptotically negligible. The order parameters then reduce to, after summation over the maps = 1, . . . , L,

m a ≡ 1 L L =1 m a = j 2 W ja 2 C 1j {r j } -φ 0 (4.44)
and

q ab ≡ 1 L L =1 q ab = j,k 2 W ja W kb C jk {r j } . (4.45)
The N × N multi-space Euclidean random matrix C appearing in the expressions above is defined by

C jk {r i } = 1 L L =1 Γ |r j -r k | . (4.46)
In the following, we denote by ρ(λ) the density of eigenvalues λ of C. This density is self-averaging when the PFs are randomly drawn in the large L, N double limit. Its resolvent, defined as

g(U) = dλ ρ(λ) λ + U , (4.47)
where the integral runs over the support of ρ(λ), is solution of the implicit equation

U = - 1 g(U) + k =0 α Γ (k) α + g(U) Γ (k) , (4.48)
where the Γ (k)'s are the components of the Fourier transform of Γ on the D-dimensional infinite reciprocal cube, as we will see in detail in Chapter 5 8 . Within the RS Ansatz, the overlap matrix q ab is fully characterized by its diagonal and off-diagonal elements that we denote by, respectively, s and q:

s = i,j 2 C ij [W 1i W 1j ] , q = i,j 2 C ij [W 1i ] [W 1j ] . (4.49)
where, as in Section 4.5, the brackets denotes the average over the random patterns, and the square parenthesis stand for the average over all couplings satisfying the inequalities (4.9).

8.

In Chapter 5 we will show (Section 5.2) how to compute the expression for the implicit equation of the resolvent g(U), that is Eq. (4.48), associated to the matrix C defined in Eq. (4.46), see the result in Eq. (5.21). An alternative derivation of the same result is presented in Section 5.3. It is important to note however that the notations in Chapter 4 and Chapter 5 are slightly different in the sense that g(U) in the former corresponds to s(z) in the latter. Following closely [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF], we obtain the expression of the average logarithm of the volume,

log Z 1 N = - 1 2 q q + sŝ + m m + û - 1 2 dλ ρ(λ) log(2 û + (2ŝ -q)λ) + qλ 2 û + (2ŝ -q)λ + m2 Ξ 2(2ŝ -q) + α p Dz log H z q -m 2 -m + κ √ s -q (4.50)
where Dz denotes the Gaussian measure,

H(x) = ∞ x Dz = 1 2 erfc( x √
2 ), and the • Lagrange parameters enforce the definitions of the order parameters ( û enforces the normalization condition over the rows of the W matrix). The quantity Ξ is a function of the argument 9

U = 2 û 2ŝ - q , (4.51)
and is defined as

Ξ(U) = j,k 2 H j U Id + C -1 jk H k with H j = 2 C 1j -φ 0 , (4.52)
9. The physical meaning of U is to represent the eigenvalues of the multi-space Euclidean random matrix associated to the different environments, similar to the pattern correlation matrix in [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF]. Always in the same reference it is possible to better understand the relationship between U and κ on the critical line, see for example the Eq. ( 17) associated, where ν corresponds to our U. Moreover, in Section 3.4 of [START_REF] Monasson | Properties of neural networks storing spatially correlated patterns[END_REF] (a work prior to [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF]) it is shown how ν is also found from the modified Hebb rule to take into account the correlations. The presence of ν is not specific of the replicas, but is also found in simpler approaches (Hebb + denoising) and allows to interpolate between K equal to the identity matrix for very high ν and K equal to the inverse of the correlation matrix of the patterns for ν → 0 according to the constraints that one puts on denoising, see Eq. (3.31) of [START_REF] Monasson | Properties of neural networks storing spatially correlated patterns[END_REF].

where Id is the identity matrix of size N -1. In the above equation, the inverse is intended over the (N -1)-dimensional restriction of the matrix U Id + C to entries j, k 2. The quantity Ξ(U) can be calculated, we report in Appendix A.4 the details, with the result:

Ξ(U) = 1 + 4 U - 4 g(U)
. As α reaches its maximal value (at fixed κ and p), the set of couplings satisfying the inequalities (4.9) shrink to a single solution, and we expect s, q to reach the same value according to Eq. (4.49). We therefore look for an asymptotic expression for 1 N 2 log Z in Eq. (4.50) when

= s -q , ( 4.54)
is very small and positive 10 . In this regime, we expect the conjugated Lagrange parameters to diverge as inverse powers of . More precisely, calling

ˆ = 2ŝ -q , (4.55)
we assume that

ˆ = V and q = T 2 , (4.56)
as → 0. To the leading order, we obtain

1 N log Z 1 = F(α) 2 + O | log | , (4.57)
where F(α) is the extremum over m, q, U, V, T of F(α; m, q, U, V, T ) that is equal to

V q + U - m 2 Ξ(U) + T 1 - 1 V dλ ρ(λ) λ λ + U -αp(q -m 2 ) ∞ x dz √ 2π e -z 2 2 (z -x) 2 (4.58)
10. Note that here is defined as the difference between the order parameters s -q, where we restrict our analysis to the critical line α c (κ, p) at fixed p so that s -q = → 0 + . Hence, in this case is different from the spatial error considered in the rest of the manuscript which is defined in the limit of p large as ∼ p -1 D .

optimal capacity-resolution trade-off in memories of multiple ca with x = m-κ √ q-m 2 and U defined in (4.51). Note that, in order to obtain (4.58), the saddle point equation over m in (4.50) was derived and solved for m explicitly. Extremizing over U, T, V, we obtain

V = dλ ρ(λ) λ λ + U , (4.59)
T = -q + U - m 2 Ξ(U) dλ ρ(λ) λ λ + U , (4.60
)

1 + m 2 Ξ(U) 2 dΞ dU = q + U - m 2 Ξ(U) dλ ρ(λ) λ (λ + U) 2 .
(4.61)

Note that the derivative of Ξ with respect to U can be easily computed from the derivative of g with respect to U according to Eq. (4.53). Following the implicit equation over g in Eq. (4.48), we find

dg dU (U) = 1 k =0 α Γ (k) (α + g Γ (k)) 2 - 1 g 2 . ( 4

.62)

We may now write the saddle-point equations over q and m, which give, after some elementary manipulation,

α p H(x) = dλ ρ(λ) λ λ + U , (4.63)
m m -κ 1 Ξ(U) -1 = 1 √ 2π x e x 2 /2 H(x) -1 . (4.64)
The three coupled equations (4.61,4.63,4.64) allows one, in principle, to compute q, m, U, and therefore T, V and F(α). In addition, the optimization of log Z in (4.57) over immediately gives F(α) = 0, hence, a fourth equation to determine the critical value of α at fixed κ and p. This last equation read, after the simplification according to Eq. (4.64),

U κ = m 1 Ξ(U) -1 . (4.65)
Resolution of these equations gives access to κ(α, p), in very good agreement with the numerical results obtained with SVM, see Fig. 23(a). Small deviations can, however, be noticed and diminish with increasing p as expected, see Section 4.6.3.

In addition, the order parameters q and m are shown as functions of p in Fig. 24, in good agreement with SVM results for large p ( p c.o. ). Furthermore, the value of p at which the confluence between the results from the quenched theory and SVM takes place is a decreasing function of the PF size φ 0 and of the map dimension D, see Appendix A.5.

Large-p behavior of the critical capacity

We now focus on the maximal capacity, obtained when κ → 0. According to (4.65), U vanishes, and equations (4.63,4.64) as a well as the implicit Eq. (4.48) on the resolvent g give a set of two coupled equations for x and the resolvent g:

1 g = k =0 Γ (k) 1 + g p H(x) Γ (k) , (4.66
)

1 - 4 g = x √ 2π H(x) e x 2 /2 . (4.67)
from which the capacity can be computed as a function of the number p of points,

α c (p) = 1 p H(x) . ( 4

.68)

optimal capacity-resolution trade-off in memories of multiple ca (b) (a) In practice, we can choose x at will, compute g from (4.67), then p from (4.66), and, finally, α c (p) from (4.68), see Fig. 23(a,inset) for the results of the numerical resolution of these equations.

Remark that equation (4.66) can be rewritten as

p H(x) = G g p H(x) with G(y) = k =0 y Γ (k) 1 + y Γ (k) . ( 4

.69)

According to dimensional analysis, the large momentum scaling of the Fourier coefficients is given by

Γ (k) ∼ φ 2 0 k φ 1 D 0 D+1 = φ 1-1 D 0 k D+1 , (4.70)
where k = |k| and D is the dimension of the physical space. We deduce that, for large arguments y,

G(y) ∼ A 1 (D) φ D-1 D+1 0 y D D+1 with A 1 (D) = d D u |u| D+1 + 1 . (4.71)
In addition, using the asymptotic expansion of the erfc function, we have

x √ 2π H(x) e x 2 /2 1 - 1 x 2 (4.72)
for large x.

Combining these expressions allows us to obtain the asymptotic relation between x and y, y We then deduce the asymptotic scaling of the critical capacity

1 D+1 = 4 A 1 (D) φ D-1 D+1 0 x 2 . ( 4
α c (p) ∼ A(D) φ -(D-1) 0 (log p) D (p → ∞) , (4.75)
with

A(D) = 1 8 D A 1 (D) D+1 . (4.76)
Equation (4.75) is the main result of this thesis. Informally speaking, the very slow decay of the critical capacity with p, see Fig. 23(a,inset), means that recurrent neural nets can efficiently store multiple spatial maps, even at high spatial resolution. More precisely, enforcing a strong reduction of the spatial error, such as → 2 , results in a moderate drop of the maximal sustainable load, α c → α c /2 D .

In addition, the capacity is predicted to be a decreasing function of the PF size in dimensions D = 2, 3, but not in dimension D = 1. This asymptotic statement is qualitatively corroborated by SVM results, even for moderate values of p, see Fig 23(b).

Moreover, the scaling for x in Eq. (4.74) entails the following relation between the order parameters q and m in the large-p regime,

q m 2 -1 ∼ 1 2 log p . ( 4

.77) optimal capacity-resolution trade-off in memories of multiple ca

To interpret the consequences of the equation above, we consider a set of replicated couplings, {W ia }. For any random position r in map defining the pattern σ, we define the rescaled and centered random variable

B r|{W ia } = 1 m a 2σ i -1) j 2 W ja σ j -m a . (4.78)
By definition of the order parameter m, the average value of B vanishes:

B r|{W ia } r = 0 . (4.79)
Equation (4.77) implies that the variance of B is

B r|{W ia } 2 r 1 2 log p , (4.80)
as p gets large and the load takes its maximal value (critical capacity). In other words, the standard deviation of B scales as (log p) -1/2 for large p. We thus expect that the k th cumulant of B will scale as (log p) -k/2 . Under this assumption, the distribution of the stability t has mean value m and fluctuations of the order of ∆t = m/ log p. These fluctuations are negligible in the large-p limit, since resolution of the saddle-point equation (4.61) shows that

m D 4 - D 2 256 (log p) 3 + o 1 (log p) 3 (4.81)
at the critical point. Hence, ∆t ∼ (log p) -1/2 is smaller and smaller as p increases, and the distribution of t is well approximated by a Gaussian in the large-p limit. The Gaussian approximation obtained by discarding all powers of t of order 3 in Eq. (4.41) in our quenched PF theory is therefore expected to be exact in this limit. This Chapter is dedicated to a detailed investigation of the spectral properties of the random matrix introduced in Section 4.6, namely in Eq. (4.46), given by the superimposition of an extensive number of independent random Euclidean matrices in the high-density limit. More specifically, we calculate analytically its resolvent both with free probability theory techniques and with the replica method from statistical physics of disordered systems. The results for the spectrum and eigenmodes are shown in the particular case of the model presented in Section 4.2, and are corroborated by numerical simulations.

All the findings presented in this Chapter have been published in [START_REF] Battista | Spectrum of multispace Euclidean random matrices[END_REF].

introduction

In the twenty years following their introduction, Euclidean Random Matrices (ERM) have been studied in a variety of contexts in physics [START_REF] Brézin | Applications of random matrices in physics[END_REF][START_REF] Mézard | Spectra of Euclidean random matrices[END_REF][START_REF] Se Skipetrov | Eigenvalue distributions of large Euclidean random matrices for waves in random media[END_REF] and mathematics [START_REF] Bordenave | Eigenvalues of Euclidean random matrices[END_REF][START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF][START_REF] El | The spectrum of kernel random matrices[END_REF]. Examples of applications of ERM include the theoretical description of vibrations in topologically disordered systems [START_REF] Ts Grigera | On the high-density expansion for Euclidean random matrices[END_REF][START_REF] Ts Grigera | Vibrations in glasses and Euclidean random matrix theory[END_REF][START_REF] Parisi | Euclidean random matrices, the glass transition and the boson peak[END_REF], wave propagation in random media [START_REF] Goetschy | Non-Hermitian Euclidean random matrix theory[END_REF][START_REF] Se Skipetrov | Eigenvalue distributions of large Euclidean random matrices for waves in random media[END_REF], relaxation in glasses [START_REF] Ciliberti | Anderson localization in Euclidean random matrices[END_REF], Anderson localization [START_REF] Amir | Localization, anomalous diffusion, and slow relaxations: A random distance matrix approach[END_REF] and many more [START_REF] Goetschy | Euclidean random matrices and their applications in physics[END_REF]. There is also no lack of ERM applications in the modeling of biological networks [START_REF] Grilli | Metapopulation persistence in random fragmented landscapes[END_REF][START_REF] Muir | Eigenspectrum bounds for semirandom matrices with modular and spatial structure for neural networks[END_REF].

While determining the spectral properties of ERM is generally quite involved due to the existence of correlations between the entries of these matrices, a well-understood limit is the so-called high-density regime [START_REF] Bordenave | Eigenvalues of Euclidean random matrices[END_REF][START_REF] Mézard | Spectra of Euclidean random matrices[END_REF]. Assume N points r i are drawn uniformly at random in a bounded space, i.e., the unit D-dimensional hypercube H D , and define the Ndimensional ERM M (1) with entries M (1) ij = Γ (|r ir j |)/N. Here, | • | denotes the Euclidean distance (with periodic boundary conditions over H D ), and Γ is a given function that depends only on the distance | • | and that has a finite range, independent of N. In the large-N limit (for fixed D), the points effectively form a dense, statistically uniform sampling of the hypercube; the eigenmodes of M (1) are well approximated by Fourier plane waves [START_REF] Goetschy | Euclidean random matrices and their applications in physics[END_REF][START_REF] Mézard | Spectra of Euclidean random matrices[END_REF], with eigenvalues

Γ (k) = H D dr e -i 2π k•r Γ (|r|) , (5.1)
where the components of k = (k 1 , k 2 , ..., k D ) are integer-valued.

spectrum of multi-space euclidean random matrices

Hereafter, we consider a novel statistical ensemble of ERMs in the high-density regime obtained by mixing multiple spaces. Instead of having a single set of N random points r i , we consider L such sets, r i , with = 1, ..., L (and i = 1, ..., N as usual). Each index points to a different "space" (hypercube), and for simplicity all points are drawn uniformly at random in the different spaces. We define the superimposition of all the ERM attached to the spaces, with entries

C ij = 1 L L =1 Γ r i -r j .
(5.

2)

The random matrix (5.2) has not yet been considered in statistical physics and is of exactly the same type as the one we encountered in Eq. (4.46) and called Multispace-ERM (MERM).

Let us briefly recall the motivation coming from the field of computational neuroscience that led us to introduce (Section 4.6) and study in detail MERM, more precisely, the need to understand how the hippocampal place-cell network [START_REF] Edvard | Place cells, grid cells, and the brain's spatial representation system[END_REF] can account for multiple cognitive maps, coding for various environmental and contextual situations. From a model perspective the points r i correspond to the positions of the centers of the place field of place cell i in map . The resulting statistical ensemble for MERM is sketched in Fig. 25. An important issue is the maximal number L of maps the hippocampal recurrent neural network (with N neurons) can sustain, more precisely, the maximal ratio

α = L N , (5.3)
called critical capacity. This capacity depends on the dimension of the maps, D N, and of their spatial accuracy (the precision with which N-dimensional neural configurations encode D-dimensional positions along the map). In Section 4.6 we have shown how the critical capacity could be determined from the knowledge of the resolvent of C. A non trivial statistical setting is obtained when the number L of spaces is of the order of the matrix size, N. More precisely, we consider herefater the double infinite size limit L, N → ∞ at fixed ratio α. This choice corresponds to the assumption that the hippocampal network activity can code for many different environments [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF] or different contexts [START_REF] Jezek | Theta-paced flickering between place-cell maps in the hippocampus[END_REF], and operates, as hypothesized for other cortical areas [START_REF] Barbour | What can we learn from synaptic weight distributions?[END_REF][START_REF] Brunel | Is cortical connectivity optimized for storing information?[END_REF][START_REF] Brunel | Optimal information storage and the distribution of synaptic weights: perceptron versus Purkinje cell[END_REF], in a regime close to maximal capacity. This Chapter is organized as follows. The spectrum of MERM is computed using arguments borrowed from free probability theory in 5.2, and re-derived using the replica method in 5.3. Finally we show the results for the spectrum and eigenmodes for the choice of Γ corresponding to Fig. 25 and compare with numerical simulations in 5.4.

The reader unfamiliar with random matrix theory can find in Appendix B a brief summary of the key concepts needed to understand the following. 2), where Γ is a generic function of the distance between points. A possible choice for Γ , inspired from the place cells in neuroscience, is the overlap (common area) between pairs of disks (place fields) of surface φ 0 < 1 and having centers r i in each space .

spectrum of merm: free-probability-inspired derivation

Let us consider an extensive number L of spaces, see (5.3), with

M (L) ij = 1 N L =1 Γ r i -r j , (5.4)
where the points are independently drawn from one space to another and where the single elements of the sum are ERM defined from N points r i drawn uniformly at random in the D-dimensional unit hypercube H D :

M (1) ij = 1 N Γ (|r i -r j |) . (5.5)
Before computing the spectrum of (5.4) we recall how to derive the spectral properties of (5.5) in the high-density regime heuristically, for a mathematically rigorous derivation see [START_REF] Bordenave | Eigenvalues of Euclidean random matrices[END_REF].

For any ERM M (1) , we can always formally write N j=1 M

(1)

ij v j (k) = Γi (k)v i (k) with v i (k) = e ik•r i √ N and Γi (k) = 1 N N j=1
e -ik•(r i -r j ) Γ (|r ir j |).

(5.6)

In the large N limit the phase -ik • (r ir j ) vary weakly between neighboring points r i and r j so that the sum in Eq. (5.6) can be approximated by an integral, thus Γi (k) does not depend anymore on i, becoming an eigenvalue of M (1) , Γ (k) = N Γ0 (k), where

Γ0 (k) = 1 N H D dr e -i 2π k•r Γ (|r|) (5.7)
is the Fourier transform of Γ (|r|). This eigenvalue is associated with the eigenvector

( e ik•r 1 √ N , . . . , e ik•r N √ N ).
In fact, given a Fourier mode k, the associated Fourier eigenvectors define a two dimensional space (linear combinations of sine and cosine corresponding to k), and we denote by v ⊥ the projection of the eigenvector v of M (1) orthogonal to this two dimensional-space. We show the squared norm of v ⊥ in Fig. 26 (averaged over several random realization and for the specific model defined in Fig. 25 but for D = 1 and L = 1) vs N in log-log scale. It is clear from Fig. 25 that the squared norm of the orthogonal projection of v scales as 1 N ; hence, each orthogonal squared component scales as 1 N 2 , and each orthogonal component as ± 1

N . This indicates that the eigenvectors of M (1) are well approximated by Fourier modes (to the order 1 √ N ) in the large N limit.

Case of the extensive eigenvalue -k=0

We would like to compute the resolvent (Stieltjes transform) of M (L) using arguments from free-probability theory [START_REF] Livan | Introduction to random matrices: theory and practice[END_REF][START_REF] James | Free probability and random matrices[END_REF][START_REF] Nica | Lectures on the combinatorics of free probability[END_REF][START_REF] Voiculescu | Free random variables[END_REF]. Heuristically, asymptotic freeness between the different ERMs relies on the fact that their eigenvectors basis are mutually incoherent. In the N → ∞ limit, the eigenvalues of M (1) in space are given by (5.1) with associated eigenvectors of components v i (k) e i 2π k•r i / √ N [START_REF] Bordenave | Eigenvalues of Euclidean random matrices[END_REF][START_REF] Goetschy | Euclidean random matrices and their applications in physics[END_REF][START_REF] Mézard | Spectra of Euclidean random matrices[END_REF]. All ERMs defined in the sum in (5.4) have mutually incoherent eigenbasis only if we restrict the analysis to the subspace orthogonal to the uniform mode attached to k = 0, shared by all the spaces. Though this argument is not rigorous, we expect this restriction to allow us to find all the eigenvalues of M (L) , except the one corresponding to the asymptotically uniform eigenvector. Furthermore it is easy to determine the leading behavior (when N is large) of the eigenvalue of M (L) corresponding to k = 0. As the corresponding eigenvector is expected to have all components equal to N -1/2 , we find that the corresponding eigenvalue is extensive in N and approximately equal to Λ = N α Γ (0). For the matrix C the corresponding eigenvalue is

z ext = Λ α = N Γ (0).
From now on we concentrate on calculating the spectrum of M (L) corresponding to k = 0; the term "resolvent" will refer to the resolvent in the k = 0 subspace.

Case of a single space (L=1)

The resolvent of M (1) is defined as

1 s 1 (z) = 1 N Trace M (1) -z Id -1 (1)
,

1. To avoid ambiguity with the standard results presented in Appendix B.1 we must specify that the resolvent considered in this work s(z) is defined as minus the resolvent used in the common notation in the literature, i.e., s(z) = -g(z), obviously nothing changes if we are consistent with the notation.

where • (1) stands for the average over the distribution of the matrix (5.5). It is easy to rewrite the resolvent when N 1,

s 1 (z) = - 1 zN N + ∞ =1 k =0 (|k| N) Γ (k) 1 z = - 1 z - 1 N z γ 1 z (5.9) with γ(u) = k =0 u Γ (k) 1 -u Γ (k) (5.10)
and where the sum runs over Z D without the k = 0 term.

Case of multiple spaces (L = αN)

We now consider the case of M (L) . It's resolvent s L (z) is defined as

s L (z) = 1 N Trace M (L) -z Id -1 (L) , (5.11)
where • (L) stands for the average over the distribution of the matrix (5.4), can be computed through the following steps, as explained in Appendix B.2:

1. Invert (functionally) the resolvent s 1 (z) of M (1) 2 : we first rewrite (5.9) into the following implicit equation for the inverse resolvent:

z 1 (s) = - 1 s - 1 N s γ 1 z 1 (s)
.

(5.12)

We then send N to infinity in the above equation, and obtain that z 1 (s) = -1/s in this limit. Using this expression for the argument of the γ function in (5.12) we obtain the 1 N -correction to the inverse resolvent:

z 1 (s) = - 1 s - γ(-s) N s . (5.13)
2. Compute the R-transform of M (1) , defined through

R 1 (s) ≡ z 1 (-s) - 1 s . (5.14)
Note the unusual presence of a minus sign in the argument of z 1 in the above equation, due to the fact that our resolvent is defined as the opposite of the standard resolvent [START_REF] Livan | Introduction to random matrices: theory and practice[END_REF]. Using (5.13), we obtain

R 1 (s) = γ(s) N s + o 1 N . ( 5
R L (s) = α γ(s) s + o(1) , (5.17)
where the corrections o(1) vanish when both N, L → ∞ at fixed ratio α.

4.

Write the functional inverse resolvent of M (L) through

z L (s) = R L (-s) - 1 s = - 1 + α γ(-s) s .
(5.18)

5. Compute the resolvent s L (z) of M (L) . From (5.18) and (5.10) we find the implicit equation satisfied by s L :

z = α k =0 Γ (k) 1 + s L Γ (k) - 1 s L . (5.19)
Note that for what is needed in Section 4.6 we are interested in the spectral properties of the matrix C with entries

C ij = 1 L L =1 Γ r i -r j = 1 α M (L) ij .
(5.20)

Obviously, the resolvent s of C is related to the resolvent s L of M (L) through the equation s(z) = α s L (α z), see Appendix B.1. Hence we obtain our fundamental implicit equation for the resolvent of C 4 :

z = k =0 α Γ (k) α + s Γ (k) - 1 s .
(5.21)

3.

Here the use of free-probability is crucial, see Appendix B.2. In fact, the important property to note is that since the different ERM corresponding to single environments in the subspace k = 0 have orthogonal eigenbasis with high-probability due to random remapping (they are asymptotically free), the R-transform of the MERM (R L (s)) is simply given by the sum of the R-transforms of the single ERM (R 1 (s)), which are equal to each other, so R L (s) = L R 1 (s).

4. That is the result we announced in Section 4.6. Note that compared to Eq. (4.48) there is only a change of notation, what was previously g(U) has now become s(z).

spectrum of multi-space euclidean random matrices

spectrum of merm: replica-based derivation

Here we re-derive the implicit equation (5.19) for the resolvent of M (L) defined in (5.11) using the replica method coming from statistical physics of disordered systems. We start by rewriting the definition of the resolvent, see Appendix B.1, as

s L (z) = 2 N ∂ z log det M (L) -z Id -1 2 (L) , (5.22)
where • (L) it's still the average over the distribution of the matrix (5.4). With this repre-

sentation the determinant det M (L) -z Id -1
2 can be expressed as a canonical partition function:

Z L (s) = det M (L) -z Id -1 2 = i dφ i √ 2π exp z 2 i φ 2 i - 1 2 ij φ i M (L) ij φ j , (5.23)
where i, j go from 1 to N. Notice that it is legitimate to adopt a real-valued Gaussian representation for the inverse square root of the determinant. Each ERM M (1) is a correlation matrix, and have real, non-negative eigenvalues; consequently, M (L) , which is the sum of correlation matrices, also has real and non-negative eigenvalues.

Resolvent (5.22) can be calculated using the replica trick [START_REF] Mezard | Information, physics, and computation[END_REF][START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF][START_REF] Nishimori | Statistical physics of spin glasses and information processing: an introduction[END_REF]:

s L (z) = 2 N ∂ z log Z L (s) (L) = 2 N ∂ z lim n→0 1 n log Z L (s) n (L) (5.24)
with

Z L (s) n (L) = ia dφ a i √ 2π exp z 2 a i (φ a i ) 2 exp - 1 2 a ij φ a i M (L) ij φ a j (L) , (5.25)
where we have replicated the system n times, i.e., a goes from 1 to n.

In order to perform the average in (5.25) we rewrite (5.4) by considering the -th space ERM in its eigenbasis:

M (L) ij = 1 N L =1 Γ r i -r j = k =0 v ki Γ (k) v kj , (5.26)
where goes from 1 to L, and the sum over k discards the k = 0 extensive mode because as discussed in the previous Section 5.2. The eigenvector components, v ki

1 √ N sin 2π k • r i , 1 √ N cos 2π k • r i , are real due to the symmetry Γ (k) = Γ (-k). Hence we get exp - 1 2 a ij φ a i M (L) ij φ a j (L) = exp - 1 2 a, ,k =0 Γ (k) i v ki φ a i 2 (L)
. (5.27)

spectrum of merm: replica-based derivation 81

We now use the Stratonovich trick to linearize

( i v ki φ a i 2 : exp - 1 2 a, ,k =0 Γ (k) i v ki φ a i 2 (L) = a,k =0 du a ,k √ 2π × exp - 1 2 a,k =0 u a ,k 2 exp -i a,k =0 Γ (k) u a k i φ a i v ki (L)
.

(5.28)

Using the fact that v ki = 0 and v ki v k j = 1 N δ ij δ kk it is easy to perform the average in the above equation, with the result

exp -i a,k =0 Γ (k) u a k i φ a i v ki (L) = exp - 1 2 a,b k =0 Γ (k)q ab u a k u b k (5.29)
where we have defined the overlap q ab as

q ab = 1 N i φ a i φ b i (5.30)
to be fixed through

1 = a b d qab dq ab 2πi N exp N a b qab q ab - a b qab i φ a i φ b i . ( 5.31)
We can finally write Z L (s) n (L) as

a b d qab dq ab 2πi N exp N log a dφ a √ 2π exp z 2 a (φ a ) 2 - a b qab φ a φ b + a b qab q ab + α log k =0,a du a k √ 2π exp - 1 2 k =0,a (u a k ) 2 - 1 2 k =0 a b Γ (k)q ab u a k u b k .
(5.32)

The Gaussian integrals over φ a and u a k can be easily computed. We then make the Replica Symmetric (RS) Ansatz on the structure of the order parameters q ab and their conjugate variables qab , so that q ab = r + (q -r)δ ab (5.33) and qab = r + ( qr)δ ab .

(5.34)

The integrals over q, r, q and r are then estimated using the saddle-point method valid for large N, and then taking the small n limit. The resulting expression for the resolvent of (5.4) is

s L (z) = 2∂ z opt q,r, q, r lim n→0 lim N→∞ 1 nN log Z L (s) n (L) = 2∂ z opt q,r
, q, rf(q, r, q, r) , (5.35) where f is the free energy density equal to

f(q, r, q, r) = qq - 1 2 rr - α 2 k =0 log 1 + Γ (k)(q -r) + Γ (k)r 1 + Γ (k)(q -r) - 1 2 log 2 q -r -z - r 2(2 q -r -z)
.

(5.36)

The saddle-point equations obtained by optimizing f(q, r, q, r) with respect to q, r, q and r read

q = - r 2 q -r -z 2 + 1 2 q -r -z , r = - r 2 q -r -z 2 , q = α 2 k =0 Γ (k) 1 + Γ (k)(q -r) - r Γ (k) 2 1 + Γ (k)(q -r) 2 , r = -α k =0 r Γ (k) 2 1 + Γ (k)(q -r) 2 .
(5.37)

This system of equations admits r = r = 0 as a solution, which gives, according to (5.35), the following implicit equation satisfied by s L (z): .38) This equation is identical to (5.19) obtained using free probability theory.

z = α k =0 Γ (k) 1 + s L Γ (k) - 1 s L . ( 5
It's interesting to note that since we consider the solution corresponding to r = r = 0, an annealed computation where we directly bring the average of the different realizations of M (L) inside the logarithm of the partition function, instead of keeping it out as in our version of the quenched computation that led us to use the replica method, it would have brought to the same result (obviously the quenched computation is a cleaner way to proceed) [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF].

Moreover, we should as well observe that in this setting we have not needed to use the asymptotic freeness for the different ERMs M (1) as we did in the free-probability computation of the previous Section 5.2, so in this sense this approach seems more powerful. In fact, we can also use this method to study the case of MERM where the statistical features of the L ERM's are non-independent from space to space (in this case we can't use free probability because we lose the asymptotic freeness between the different ERMs because the entries of these are correlated with each other), see Section 6.5.

application and comparison with numerics

Numerical computation of the spectrum

We now aim at solving the implicit equation (5.21) satisfied by the resolvent of C. We show in Fig. 27(a) the representative curve of z as a function of s around the pole at the origin (s = 0). A set of forbidden disjoint intervals, z ∈ [z

(m) -, z (m)
+ ], with m = 1, ..., M is found, which cannot be reached for real-valued s; the number M of these intervals is a decreasing function of the ratio α. When z lies in one of these intervals, we look for a solution to equation (5.21) with

s = s r + i s i , (5.39)
where the imaginary part s i is strictly positive. For z = x + i , the density of eigenvalues at x is given by ρ(x) = lim →0 s i (z)/π by virtue of well-known properties of the Stieljes transform, see Appendix B.1. From now on we will indicate with z the eigenvalue and with ρ(z) the correspondent density, bearing in mind the → 0 limit.

The implicit equations fulfilled by s r and s i for z ∈ [z

(m) -, z (m)
+ ], with m = 1, ..., M read

z = k =0 α 2 Γ (k) α + s r Γ (k) 2 + s i Γ (k) 2 ,
(5.40)

1 s 2 r + s 2 i = k =0 α Γ (k) 2 α + s r Γ (k) 2 + s i Γ (k) 2 , (5.41)
and can be solved numerically. Figure 28 shows the density of eigenvalues for various values of α. We observe the presence of the disconnected intervals [z

(m) -; z (m)
+ ] corresponding to non-zero density ρ(z), referred to as "connected components" below. These connected components originate from the discrete spectrum of ERM (with eigenvalues labelled by k) and progressively merge as α increases (Fig. 27(b)). We now discuss the mechanism leading to merging in the large |k|, small α regime.

5.4.2

Merging of density "connected components": behavior of the density at small α For small α, we look for a solution of equation (5.21) near the poles, so that to consider only a value k = 0 in the sum over the modes:

z(k) = α Γ (k) α + s(k) Γ (k) - 1 s(k) . ( 5

.42)

We find then s(k) such that dz(k) ds(k) = 0, i.e., where the resolvent has singularities (eigenvalues), obtaining: 5.43) this implies that the spectrum has the edges located at: .44) This means that when α become sufficiently small the spectrum develop a connected component in correspondence of every k = 0 centered in

s ± (k) = - α Γ (k) 1 ± √ α , (
z ± (k) = Γ (k) α 1 ± 2 √ α . ( 5
z k = 1 2 (z -(k) + z + (k)) = Γ (k) α and of half-width 1 2 (z + (k) -z -(k)) = 2 Γ (k) √ α .
In order now to understand how the density of eigenvalues behaves inside these connected components we look to a solution of equation (5.42) of the form

s(k) = s r (k) + i s i (k) ,
(5.45) so that to finally obtain the parametric equations for the density ρ(z) of eigenvalues equal to z:

ρ(x; k) = α 3 2 π Γ (k) 1 -x 2 , z(x; k) = Γ (k) α 1 + 2x √ α , (5.46)
where x ∈ [-1; 1]. This solution makes sense only for the modes k and ratios α such that the local semi-circle distributions attached to two contiguous eigenvalues do not overlap. More precisely, the ratio α should be smaller than

α merging (k) ( Γ (k) -Γ (k c )) 2 4( Γ (k) + Γ (k c )) 2 , (5.47)
where k c is the momentum vector corresponding to the closest eigenvalue to Γ (k). This formula gives the values of the ratios at which the small connected components of ρ(z) (Figs. 27(b) and 28) successively merge, and is asymptotically correct for large |k|.

When α is sufficiently large, all connected components have merged into a single continuous, semi-circle distribution, as could be expected from the vanishing correlation between the matrix elements of C, centered in z 1 = 1 2 (z -+ z + ) = Γ1 and of half-width

1 2 (z + -z -) = 2 Γ2 /α, with Γ1 = k =0 Γ (k) and Γ2 = k =0 Γ (k) 2 .

Eigenvectors of MERM and Fourier modes associated to the ERMs

We briefly discuss here the properties of the eigenvectors of MERM. We consider a connected component of eigenvalues originated from the same ERM eigenvalue (labelled by k), see previous Subsection 5.4.2. To quantify how much the MERM eigenvectors v are related to the 2L eigenvectors (Fourier modes) of the L ERMs, we write

v i = L =1 γ 1 √ N cos (2π k • r i ) + δ 1 √ N sin (2π k • r i ) + R i , (5.48)
where γ and δ are the projection coefficients onto the 2L ERMs eigenvectors and R is the component of v orthogonal to this subspace 5 .

The distributions of the coefficients γ , δ and of the norm of R are shown in Fig. 29 in the case L = 5 and for increasing values of N. We observe that 5. R is equivalent to v ⊥ encountered in Section 5.2 in the case of single ERM (L = 1). spectrum of multi-space euclidean random matrices -the magnitude of γ and δ seems to be independent of N (Fig. 29(a)), which implies that these coefficients remain finite as N → ∞. Conversely, the projections of v on Fourier modes attached to a momentum k = k vanishes with increasing N, see Fig. 29(b). Hence, v retains some coherence with the 2L eigenvectors of the ERMs attached to the connected component even in the infinite size limit (provided L remains finite).

z z z z ⇢(z) ⇢(z) ⇢(z) ⇢(z) ↵ = 1 ↵ = 10 ↵ = .02 ↵ = .1 z z z z ⇢(z) ⇢(z) ⇢(z) ⇢(z) ↵ = 1 ↵ = 10 ↵ = .02 ↵ = .1 ↵ = 10 ↵ = 1 ↵ = .1 ↵ = .02
-the norm of R seems to get peaked as N grows around a non-zero value. Therefore, v has a substantial component outside the 2L-dimensional subspaces spanned by the ERM eigenmodes.

Notice that the magnitudes of the γ, δ coefficients and of the norm of R are related to each other through γ 2 = δ 2 = (1 -R 2 )/L to ensure the normalization of v. The results above were derived for finite L and large N; in the double scaling limit where both L, N are large at fixed ratio α, we find that the coefficients γ, δ of the projections on the Fourier modes attached to the connected component also scale as N -1/2 , in accordance with the number of those modes. In this Chapter we give additional details of the model presented in Chapter 4 and also generalize the latter to take into account some constraints of biological nature.

`, ` `, `|R| ⇢( `, `) ⇢( `, `) ⇢(|R|) (a) (b) (c) `, ` `, `|R| ⇢( `, `) ⇢( `, `) ⇢(|R|) (a) (b) (c) `, ` `, ` ⇢( `, `) ⇢( `, `) ⇢(|R|) (a) (b) (c) `, ` ⇢( `, `) ⇢( `, `) (a) (b) (a) (b) (c)
In particular, we try to understand to what extent the main result of this thesis, the scaling found in Eq. (4.75), is robust to variations of the model. We also try to propose links of our theory with ongoing experiments. Finally, we present another application of our setting to the case of storing continuous attractors in recurrent neural networks starting from real images. Not all the results presented in this Chapter can be considered at a final stage and this will give us the opportunity to present in the Conclusions different lines of research to follow in the near future, see Chapter 7.

Part of the findings presented in this Chapter have been published in [START_REF] Battista | Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks[END_REF] and [START_REF] Battista | Spectrum of multispace Euclidean random matrices[END_REF].

introduction

In Chapter 4 we found a non-trivial result suggesting that recurrent neural networks are very efficient devices in the storage of continuous attractors, contrary to what it seemed in standard models based on Hebbian-like learning rules of the type (3.15), see Section 3.7. Despite this positive result, there is still work to be done especially in view of the fact that the model presented in Chapter 4 should be a schematization of the recurrent network of place cells in CA3 of the hippocampus, see Sections 3.3 and 3.5, in fact, we have not considered different constraints of biological nature that this type of network must meet, such as:

-take into account the border effects instead of simple periodic boundary conditions as they are crucial in spatial navigation to identify the presence of walls in an environment [START_REF] Barry | The vector cell model of place cell firing and spatial memory[END_REF], also remember the presence of cells with this specific function, the border cells, see Section 3.8;

-make explicit the difference between excitatory and inhibitory synapses in the connectivity matrix according to Dale's principle [START_REF] Hh Dale | Pharmacology and nerve endings[END_REF], which basically states that a neuron performs the same chemical action at all of its synaptic connections to other cells, regardless of the identity of the target cell. In particular, it is known that the connections between place cells are only excitatory and the network activity is kept fixed thanks to the presence of the interneurons that are purely inhibitory and whose only function is to balance the positive contribution of place cells;

towards greater biological plausibility -consider more realistic models of place cells in which place fields can have different sizes [START_REF] Syed A Hussaini | Increased size and stability of CA1 and CA3 place fields in HCN1 knockout mice[END_REF][START_REF] Mizuseki | Activity dynamics and behavioral correlates of CA3 and CA1 hippocampal pyramidal neurons[END_REF], take into account the presence of silent cells [START_REF] Lt Thompson | Place cells and silent cells in the hippocampus of freely-behaving rats[END_REF], not all place cells have a place field in a given environment, indeed only about thirty percent do, and also consider that a place cell can have more than one place field in a given environment, which is what usually happens when considering large maps [START_REF] André A Fenton | Unmasking the CA1 ensemble place code by exposures to small and large environments: more place cells and multiple, irregularly arranged, and expanded place fields in the larger space[END_REF][START_REF] Park | Ensemble place codes in hippocampus: CA1, CA3, and dentate gyrus place cells have multiple place fields in large environments[END_REF][START_REF] Rich | Large environments reveal the statistical structure governing hippocampal representations[END_REF];

-take into account that in the context of place cells and fields, it is known that neurons have some individuality, that is, retain some properties in the different environments. In particular it was reported experimentally [START_REF] Sung | The statistical structure of the hippocampal code for space as a function of time, context, and value[END_REF][START_REF] Rich | Large environments reveal the statistical structure governing hippocampal representations[END_REF] that each place cell has its own propensity to have a place field per square meter: many neurons have very low propensity values, i.e., have no place field at all in many maps, and few neurons that have very high propensity and therefore tend to code almost all maps even with more than one place field connected component per map;

-consider the sparse nature of synapses [START_REF] Jose Guzman | Synaptic mechanisms of pattern completion in the hippocampal CA3 network[END_REF] and neuron activity observed in CA3 [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF];

-consider that the positions to be stored in a map as well as the centers of the input (place) fields do not necessarily have to be distributed uniformly in the different environments. The latter is due to the lack of homogeneity of sensory inputs (visual, auditory, olfactory and so on) and the former due to the trajectories of the animal, which may prefer to spend more time in some areas than in others (reward areas) [START_REF] Bourboulou | control of hippocampal spatial coding resolution by local visual cues[END_REF][START_REF] Hollup | Accumulation of hippocampal place fields at the goal location in an annular watermaze task[END_REF][START_REF] Mamad | Place field assembly distribution encodes preferred locations[END_REF]. Linked to this is also the fact to consider that maps can be stored with different levels of spatial error.

It is therefore important to understand if the results found so far in Chapter 4 remain valid also in the case of the above mentioned variations of the starting model. Moreover, since the theory presented in Section 4.6 has been developed with quenched place fields perhaps it is possible to propose some links with experiments in which we have access to place fields dispositions in different environments, maybe in order to better understand the phenomenon of random remapping, see Section 3.5.

In addition, another interesting topic to discuss is the learning dynamics of the network. In fact, so far we have always considered offline SVM algorithms (in which the network always has access to the whole data-set during the learning phase), instead of online procedures (in which instead patterns are presented one at a time) that are definitely more biologically plausible. This is a fundamental step in the perspective of wanting to study the dynamics of learning in a recurrent neural network that progressively matures continuous attractors and that could be connected to experiments on cognitive maps formation performed on new born animals [START_REF] Farooq | Emergence of preconfigured and plastic time-compressed sequences in early postnatal development[END_REF].

Finally, we show an application of our setting in a different context of the place cells network, in particular to understand how to store continuous attractors in a recurrent neural network starting from real images [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF].

border effects

Until now, periodic boundary conditions (PBCs) have been considered for the maps in order to simplify the analytical calculations, i.e., translation invariance, see Section 4.2.

Here we try to discuss numerically what is the effect of removing PBCs in the model. In particular, now for environment we really consider D-dimensional cubes and not torus, the borders of the maps are rigid and any place field near the edge is a cut sphere according to the proximity from the walls.

In Fig. 30 we compare using SVMs the optimal stability κ defined in Eq. ( 4.3) as a function of the load α = L N and of the number p of prescribed fixed points in case the maps do not have PBCs with the case they do.

From simulations it seems that the scaling found in the setting with PBCs is also valid if they are removed. However, at α and p fixed, κ increases slightly in the case PBCs are not present. This can be explained heuristically by noting that when we don't have PBCs the possible configurations of the patterns on the map borders decreases compared to the case of PBCs, so as p increases, the redundancy in the patterns starts to be seen first. Even if the result of the simulations seems quite intuitive, unfortunately an analytical treatment in case of removing PBCs becomes much more complicated.

The main problem is that removing the PBCs breaks the invariance under translation in the model 4.2 and therefore, in the computation with Gardner approach, see Section 4.5, things get more complicated. In theory we could compute numerically the same equations (4.39) but having to explicitly solve the integral on r 1 , that is not irrelevant any longer, and also the expression of the matrix (4.17), that is

Γ µ,ν {r µ } = Γ rµ -rν -φ 2 0 , (6.1)
becomes more complicated since the latter is no longer a standard ERM because it is no longer a function only of the distance between the positions, but also of their center of mass since the invariance under translations is broken, what happens at the edge of the environment is different from what happens at the bulk.

The computation with quenched place fields in Section 4.6 does not seem possible as well in this case for the same reason. We do not know how to calculate the spectrum of towards greater biological plausibility the matrices of single environment that are no longer ERM because of the presence of the edges, consequently we do not have access to the spectrum of the MERM

C jk {r i } = 1 L L =1 Γ |r j -r k | , ( 6

positive couplings constraint

Here we study the effect of the constraint on the sign of the couplings in the model presented in Chapter 4, in fact, as it is well known in neuroscience the connections between pyramidal cells, place cells in our specific case, are only excitatory.

So in the following we generalize our model in order to take into account this constraint and explicitly considering threshold terms {θ i } in the RNN that effectively represents the inhibitory contribution coming from the network of interneurons that serve to keep the network activity fixed.

Somehow, what we are trying to do here is a generalization of the maximal stability perceptron approach with simple random patterns and sign-constrained synapses [START_REF] Barbour | What can we learn from synaptic weight distributions?[END_REF][START_REF] Brunel | Is cortical connectivity optimized for storing information?[END_REF][START_REF] Brunel | Optimal information storage and the distribution of synaptic weights: perceptron versus Purkinje cell[END_REF] to the case of patterns with strong spatial correlation.

In the case of random patterns it is known that the connectivity of a recurrent neural network with the constraint of positivity on the weights is quite sparse [START_REF] Brunel | Is cortical connectivity optimized for storing information?[END_REF], i.e., large fraction of zero synaptic weights ("potential" synapses), and compatible with the physiological distribution of synaptic connections observed in CA3 [START_REF] Jose Guzman | Synaptic mechanisms of pattern completion in the hippocampal CA3 network[END_REF].

Moreover, always in the case of random patterns it is known that the capacity of a recurrent neural network with the constraint on the sign of the weights is half of the same quantity calculated for a not constrained RNN [START_REF] Daniel J Amit | The interaction space of neural networks with sign-constrained synapses[END_REF][START_REF] Daniel J Amit | Perceptron learning with sign-constrained weights[END_REF][START_REF] Campbell | On the storage capacity of neural networks with sign-constrained weights[END_REF][START_REF] Nadal | On the storage capacity with sign-constrained synaptic couplings[END_REF][START_REF] Raju | Sign-constrained synapses and biased patterns in neural networks[END_REF][START_REF] Wong | Competitive attraction in neural networks with sign-constrained weights[END_REF]. What happens in the case of spatially correlated patterns? Before starting the analytical treatment we try to understand numerically what the introduction of this constraint implies in our case. To do so we use a generalization of the SVM algorithm with the restriction of having only positive weights, the details of which are discussed in Appendix C.1.

Couplings obtained by SVMs with positive weights constraint

Hence, we report some qualitative features of the couplings obtained by SVMs with the positive couplings constraint, see Section 4.4 for a comparison with the unconstrained case.

-As shown in Fig. 31(a) and (c) the couplings W ij are, also in this case, correlated with the distances d ij = |r ir j | between the PF centers of the neurons i and j in the different maps . Note that, also here, the dependence on distance is less marked as the number L of maps increases, due to the interferences between the maps.

-In this case the couplings are always positive, so the role of the threshold θ i < 0 is fundamental to support bump states with an average activity φ 0 .

-Histograms of the couplings in Fig. 31 (b) and (d) show that the amplitudes decay with N. Also in this case we get the average values and standard deviations to scale, respectively, as 1/N and 1/ √ N. These results recall the ones already found in [START_REF] Barbour | What can we learn from synaptic weight distributions?[END_REF][START_REF] Brunel | Is cortical connectivity optimized for storing information?[END_REF][START_REF] Brunel | Optimal information storage and the distribution of synaptic weights: perceptron versus Purkinje cell[END_REF] for the distribution of synaptic weights in the case of random patterns and comparison with data in the cortex, suggesting also in this setting that synaptic connectivity in a RNN that should memorize continuous attractors (like the CA3 network in the hippocampus) may be optimized to store their number in a robust towards greater biological plausibility fashion. Also here, in fact, it seems that a large fraction of weights is zero suggesting a rather sparse connectivity compatible with what found in [START_REF] Jose Guzman | Synaptic mechanisms of pattern completion in the hippocampal CA3 network[END_REF].

It is worth noting that the weights are of the same order in module both in the standard case of SVM (histograms in Fig. 16 (b) and(d)) and the one where we introduce a constraint on the sign of the couplings (histograms in Fig. 31 (b) and(d)). So the contribution coming from the thresholds must compensate that part of the connections that cannot be negative in the second case. The bias terms however schematize the input coming from a population of inhibitory neurons (the interneurons) whose function is only to keep fixed the activity of the network, so it is reasonable to think that this contribution can compensate the fact that now we consider only excitatory neurons keeping the system biologically feasible, we are simply separating the role of the neurons of interest, that are the place cells (only excitatory), from the population of interneurons schematized by the thresholds.

Stability obtained by SVMs with positive weights constraint

In Fig. 32 we compare the optimal stability κ pos defined as

κ pos = max {W ij 0,θ i } min {i=1...N, =1...L,µ=1...p} 2σ ,µ i -1 j( =i) W ij σ µ j + θ i (6.3)
with the positive weights constraint and where the rows of the connectivity matrix are normalized as usual to unit, with the one in Eq. (4.3) of standard SVMs as a function of the load α = L N and of the number p of prescribed fixed positions per map (in the case of SVMs with the weights sign constraint α has been multiplied by a factor of 2). It is clear that the numerically calculated curves coincide. This means that the capacity obtained in the constrained case turns out to be half of the unconstrained one. This result is remarkable since the size of the space of possible couplings is strongly reduced with this restriction and we lose only a finite fraction in term of capacity.

Adding the positive weights constraint in Gardner's framework

After the numerical evidences just shown, here we are going to extend Gardner theory for the capacity of the perceptron with sign-constrained synapses [START_REF] Daniel J Amit | The interaction space of neural networks with sign-constrained synapses[END_REF] to the case of continuous attractors. The substantial differences with the computation we have presented in Section 4.5 are the fact to consider the integrals over the weights {W ij } from 0 to ∞ instead of the whole R and also the introduction of the threshold terms {θ i }. Nevertheless we report below all the details. that the resulting patterns are spatially correlated. The stability of the i component of the pattern that correspond to position µ in the environment is given by 1

∆ µ i = (2σ µ i -1) j( =i) W ij √ N σ µ j + θ i , (6.4)
where W ij 0 and θ i is the threshold term. The training set is said to be stored if all the patterns have stabilities larger than some threshold κ 0 2 .

1.

To notice that here we normalize the rows of the connectivity matrix to N instead of 1 but we have scaled the weights of √ N, therefore nothing changes regarding the computation presented in Section 4.5. Moreover, as we have seen in Section A.1, the normalization of weights is irrelevant in our setting and we can choose it arbitrarily.

2. From here until the end of the computation we will call κ what was previously called κ pos to simplify the notation.

The volume in the space of couplings that corresponds to the admissible solutions of the storage problem, is

Z = ∞ 0 N i =j dW ij i δ j( =i) W 2 ij -N i, ,µ Θ (2σ µ i -1) j( =i) W ij √ N σ µ j + θ i -κ (6.5)
and is equal to the product of the N single-site volumes Z i , with i = 1 . . . , N.

So we may focus for example on the volume associated with i = 1:

Z 1 = ∞ 0 N j=2 dW j δ j 2 W 2 j -N ,µ Θ (2σ µ 1 -1) j 2 W j √ N σ µ j + θ -κ , (6.6)
where W j ≡ W 1j and θ ≡ θ 1 . Using the replica method [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF][START_REF] Urbani | Statistical Physics of glassy systems: tools and applications[END_REF], we compute the average of log Z 1 over the patterns. Introducing integral representations of the Heaviside functions and exploiting the statistical independence of the different maps, we write the average of the n th power of the volume,

Z n 1 = ∞ 0 j,a dW ja a δ j W 2 ja -N χ αN (6.7)
where a = 1, . . . , n is the replica index, and

χ = p µ=1 dr µ N j=1 dr j ∞ κ µ,a dt µa ∞ -∞ µ,a d tµa 2π e i µ,a tµa (t µa -θ(2σ µ 1 -1)) × j e -i √ N µ,a tµa (2σ µ 1 -1)W ja σ µ j , (6.8)
where rµ denotes the p prescribed locations in the environment, and r j the N PF of the neurons in the map, as in Section 4.5.

We first carry out explicitly the integrals over the PF with indices j = 2, 3, . . . , N, leaving the integrals over r 1 and all rµ in χ. We then introduce, as in Section 4.5, the order parameters

m a = 1 √ N j 2 W ja (6.9)
and e a<b qab (Nq ab -j W ja W jb ) χ αN (6.11)

q ab = 1 N j 2 W ja W jb , (6.10)
where we have used the integral representation of the Dirac-delta functions.

Still following what seen in Section 4.5, we consider Φ(r) to be the indicator function of the PF centered in 0: Φ = 1 if |r| < r c , where r c is the radius of the PF (with dr Φ(r) = φ 0 ), and 0 otherwise. Let Γ (r) = dr Φ(r ) Φ(rr ) be the correlation function of Φ and σ(r 1rµ ) = 2Φ(r 1rµ ) -1. Given p points rµ , µ = 1, . . . , p drawn uniformly at random in space, we define the p × p Euclidean random matrix with entries

Γ µ,ν R ≡ {r µ } = Γ rµ -rν -φ 2 0 . (6.12)
We can then rewrite χ as

χ = µ dr µ dr 1 ∞ κ µ,a dt µa √ 2π ∞ -∞ µ,a d tµa √ 2π e -1 2 µ,ν,a,b q ab Γ µ,ν ( R) tµa tνb
×e -i µ,a (m a φ 0 +θ) tµa σ(r 1 -r µ )+i µ,a tµa t µa . (

Due to translation invariance, the integral over r 1 is irrelevant, and we can set r 1 = 0.

We also define from (6.11):

Y ≡ ∞ 0 j,a dW j,a e a ûa 2 (N-j W 2
ja ) e a ma ( √ Nm a -j W ja ) e a<b qab (Nq ab -j W ja W jb ) . (

It is possible now to make the RS Ansatz (expected to be valid also here since the domain of suitable couplings is still convex) on the structure of the order parameters and their conjugate variables.

We can therefore write within the limit of large N and small n:

1 nN log Y = û 2 + q q 2 + lim n→0 1 n log ∞ 0 a dW a e -û 2 a W 2 a + q a<b W a W b -m a W a (6.15)
where we have changed q inq for simplicity. After having inserted in the exponent of the above expression ( a W a ) 2 and used a Gaussian integral trick, is it possible to solve the integral, obtaining in the small n limit:

1 nN log Y = û 2 + q q 2 + 1 2 log π 2 - 1 2 log ( q + û) + m2 + q 2( q + û) + Dz log erfc m - √ qz √ 2 √ q + û , (6.16)
where Dz is the Gaussian measure and erfc

y = 2 √ π ∞ y dx e -x 2 .
We shall now concentrate on the saddle-point equations to determine m, û and q as a function of q. We shall assume that as q → 1, so when the space of solutions reduces to the optimal coupling matrix, q q+ û diverges and m √ q → 0, and verify that the solution satisfies this condition. In this limit it is possible to solve Dz log erfc m-

√ qz √ 2 √ q+
û because when z < 0 we can use the following asymptotic expansion erfc y e -y 2 √ πy , and when z > 0 we get trivially a constant. In the end we can write:

1 nN log Y = û 2 + q q 2 - 1 4 log ( q + û) + m2 + q 4( q + û) - 1 4 log ( q) + const . (6.17)
It is possible then to write the saddle-point equations relative respectively to m, û and q from the above equation:

m = 0 , (6.18)
1 2 = m2 + 2 q + û 4( q + û) 2 , (6.19)
q 2 = 1 4 q + m2 + q 4( q + û) 2 . ( 6.20)
By solving this equations we finally find:

m = 0 , (6.21) q
= -2 + 3q + -4 + q(4 + q) 4(1 -q) 2 , (6.22)
û = 4 + q 2 -q(6 + -4 + q(4 + q)) 4(1 -q) 2 , (6.23)
so that our assumptions are indeed consistent. By injecting these results in (6.17) we find that when q → 1, we have:

1 nN log Y 1 4(1 -q) . (6.24) µ dr µ µ dz µ √ 2π e -1 2 µ,ν z µ Γ( R) -1 µ,ν z ν det Γ( R) min {t µ κ} × (6.27) µ,ν t µ -(z µ + (mφ 0 + θ)σ(r 1 -rµ ) Γ( R) -1 µ,ν t ν -(z ν + (mφ 0 + θ)σ(r 1 -rν ) .
After absorbing into m either φ 0 , either θ we finally obtain the expression for the critical capacity α c (κ, p) = max m α c (m; κ, p), where α c (m; κ, p) is the load α cancelling the terms inside the curly brackets in (6.27).

By comparing the last expression with the one found in Eq. (4.38) it is therefore clear that in this case the capacity is then only half of the one in which we do not consider the restriction on the sign of weights, see Section 4.5.

It is important to note that since the introduction of this constraint brings in the final equations only the difference of a factor 2, the results found in the theory with quenched place fields, see Section 4.6, remains valid also in this setting and therefore the scaling in Eq. (4.75) it is still preserved, despite the introduction of this strict constraint of biological nature. In other words this restriction limits the capacity of a finite factor not causing any problem to the theory developed so far.

variants of the place cell model

In order to see how much the scaling of α c (p) in Eq. (4.75) is robust against the choice of the parametrization Φ(r) of the manifolds, we are going to show that reducing the number of active neurons in each map, allowing for variations in the sizes of the PFs from neuron to neuron and consider place cells with more than one spatial connected components per map do not affect it.

The function Γ we have considered so far corresponds to the simple model defined in Fig. 25. In a unit cube H D in D dimensions with periodic boundary conditions, a set of N positions r i (centres of D-dimensional spheres, place fields, of volume φ 0 < 1) are drawn uniformly and independently at random for each map . The term Γ r ir j entering in the correlation matrix (5.2) is simply the overlap (common volume) between the two place fields in the same environment, see Fig. 25. We consider below three variants of this model of interest for a more realistic modeling of the CA3 recurrent place cells network.

Dilution

Let us first consider single-space ERM in which a fraction ρ 0 of the N place cells (chosen at random among 1, . . . , N) carry vanishingly small place fields, and the remaining cells have normally place fields of volume φ 0 whose centers are randomly located on the map, see Fig. 33(a). All the entries of the ERM M (1) ij = Γ (|r ir j |)/N such that i or j belongs to the first subset (with point-like PFs) are equal to zero. We are left with a block matrix of dimension (1 -ρ 0 )N × (1 -ρ 0 )N, equal to the ERMs considered so far with the model of Fig. 25, see Chapter 5. As a consequence, in the large N limit, the eigenvalues of this block-ERM are equal to ρ 0 Γ (k), while the remaining eigenvalues are equal to zero.

The resolvent of this diluted version of ERM in the high-density regime has the same form as (5.9):

s 1 (z) = - 1 zN ρ 0 N + ∞ =1 k =0 (|k| ρ 0 N) Γ (k) 1 z + (1 -ρ 0 )N = - 1 z - 1 N z γ 1 z (6.28)
where

γ(u) = k =0 u ρ 0 Γ (k) 1 -u ρ 0 Γ (k) . ( 6.29)
towards greater biological plausibility

The computation of the functional inverse of the resolvent (blue function) of the dilute MERM can be done as in the standard case, see Chapter 5, and we get:

z = k =0 α ρ 0 Γ (k) α + s ρ 0 Γ (k) - 1 s . ( 6.30)
We can now solve equation (6.30) in order to get the density of eigenvalues. The agreement with the spectrum obtained from numerical simulations is excellent, see Fig. 33(d).

Place cells with vanishingly small place fields serve to model the fact that not all place cells take part in the coding of a given environment (a neuron that presents vanishingly small PF can be seen as a silent cell because it is never active). The hippocampus has about 30M neurons in humans and .3M neurons in rodents, so it is reasonable to think that to effectively store an environment we do not need to use all the cells available, in fact in the experiments it is clear that about 40% of the cells measured in the hippocampus are actually place cells, i.e., they have at least one PF in the environment [START_REF] Lt Thompson | Place cells and silent cells in the hippocampus of freely-behaving rats[END_REF].

Place fields of different volumes

We now discuss the case of a multinomial distribution of place fields volumes. We consider first that, in each space, a fraction ρ 1 of the N PFs have volume φ 1 , while the remaining fraction ρ 2 = 1 -ρ 1 have volume φ 2 , see Fig. 33(b). For every space we build a matrix composed of 4 blocks:

M (1) = 1 N   Γ 11 Γ 12 Γ 21 Γ 22   , (6.31)
where the block Γ ab is a ρ a N × ρ b N ERM depending on the overlaps between PFs of volumes φ a and φ b , and with a, b taking values 1 or 2.

We look for eigenvectors of M (1) of components v i (k) ∝ e i 2π k•r i multiplied by α a for the site i in the fraction ρ a , with a = 1, 2. We obtain the following eigen-system: with γ a (r) being the indicator function of the place field of volume φ a3 . We find α a ∝ γa (k) and λ(k

   ρ 1 Γ11 (k) α 1 + ρ 2 Γ12 (k) α 2 = λ(k) α 1 ρ 1 Γ21 (k) α 1 + ρ 2 Γ22 (k) α 2 = λ(k) α 2 . ( 6
) = ρ 1 ( γ1 (k)) 2 + ρ 2 ( γ2 (k)) 2 .
This result immediately extends to more than two PFs types. If we have K finite (as N → ∞) types of PFs, with associated volumes φ a and fractions ρ a , with a = 1, . . . , K, the eigenvalue of ERM attached to the momemtum k is given by Place cells with PFs of different volumes are particularly important in the study of heterogeneous environments, i.e., environments in which there are areas of interest (food, water, etc.) and areas of disinterest for the animal that stores it. Typically the areas of interest have a higher density of PFs with a small size, while the areas of disinterest have less PFs but with a larger size [START_REF] Bourboulou | control of hippocampal spatial coding resolution by local visual cues[END_REF]. Obviously the size of PFs determines the accuracy with which different parts of the environment are stored, as they determine the width of the activity bump.

λ(k) = K a=1 ρ a ( γ(k)) 2 . ( 6

Multiple place fields per cell in each space

We extend the above setting to the case of multiple place fields per cell in each space. More precisely, we assume that for each place cell i = 1, . . . , N, there are c centers r i,m of PFs, with m = 1, . . . , c in each space , see Fig. 33(c); we assume that c remains finite as N, L are sent to infinity. The associated MERM is defined as follows

C ij = 1 L L =1 c m,m =1 Γ r i,m -r j,m . (6.35)
To better understand what happens in this setting we consider the limit case of a single map:

M (1) ij = 1 N c m,m =1 Γ r i,m -r j,m . (6.36)
towards greater biological plausibility

In the high-density regime the eigenvectors of this ERM have components approximately equal to v i (k) ∝ m e i 2π k•r i,m , with eigenvalues given by c Γ (k) (for k = 0).

The only change to the functional inverse of the correspondent MERM resolvent is Γ (k) → c Γ (k), so that we obtain:

z = k =0 α c Γ (k) α + s c Γ (k) - 1 s . ( 6.37)
We have solved equation (6.37) in order to get the density of eigenvalues; results are in excellent agreement with numerics, see Fig. 33(f).

Place cells that have multiple PFs serve to model large environments in which this phenomenon is observed [START_REF] André A Fenton | Unmasking the CA1 ensemble place code by exposures to small and large environments: more place cells and multiple, irregularly arranged, and expanded place fields in the larger space[END_REF][START_REF] Park | Ensemble place codes in hippocampus: CA1, CA3, and dentate gyrus place cells have multiple place fields in large environments[END_REF][START_REF] Rich | Large environments reveal the statistical structure governing hippocampal representations[END_REF]. Since the locations in an environment are stored by the entire population of place cells encoding it, having multiple PFs is not harmful as long as there are no problems in decoding the different positions and it may be more convenient to avoid unnecessary cells that are silent in a given map having to take part in encoding it. One is used to think that a place cell can only have one PF in an environment because typically the experiments were done in small rooms where it was not possible to observe this quite common feature. Moreover, the fact that place cells can have more than one PFs in a given environment is not related to grid cells because PFs have centers placed randomly and not on a triangular grid covering the whole environment, see Section 3.8.

Putting everything together

As already discussed, in the CA3 region of the hippocampus, neurons may have place fields in some environment and none in other maps, which corresponds to the dilute model presented above. In addition, we have introduced other variants, in which the radius of place fields varies or a place field is made of more than one connected spatial component, as seen in large environments [START_REF] Rich | Large environments reveal the statistical structure governing hippocampal representations[END_REF]. While the variants of the model considered here lead to different densities of eigenvalues z, the behaviours of these densities for z → 0 and α → 0 seem qualitatively robust, which suggests that the storage capacity of recurrent neural networks is a robust property of the space-to-neural activity encoding.

In fact, the quenched place field theory in these cases extends in a trivial way 4 with respect to the computation presented in Section 4.6, indeed the only irrelevant differences in these situations are the average activity of the network that is different and the spectrum of the associated MERMs which do not change its behaviour for small eigenvalues 4. The new kind of disorders introduced here always enter the correlation matrix that defines the order parameters of the single environments, which also remains in the ERM class. Therefore the picture presented in Fig. 22 remains unchanged in the large p limit where self-averaging properties still apply.

individuality of neurons 105

c = 2 r ℓ i,1 r ℓ i,2 (a) (b) (c) Diluition Spheres of different sizes Multiple spheres z z z ⇢(z) ⇢(z) ⇢(z) n = r ℓ i,1 r ℓ i,2 (a) (b) (c)

Diluition

Spheres of different sizes Multiple spheres

z z z ⇢(z) ⇢(z) ⇢(z) n = 2 r ℓ i,1 r ℓ i,2 (b) (c) Diluition Spheres of different sizes Multiple spheres z z z ⇢(z) ⇢(z) n = 2 r ℓ i,1 r ℓ i,2 (b) (c) Spheres of different sizes Multiple spheres z z ⇢(z) ⇢(z) ↵ = 1 ↵ = 1, ⇢ 0 =.4 = 1 ⇢ 0 =.2 c c c ↵ = 1, ⇢ 0 =.2 = 1 ⇢ 0 c = 2 r ℓ i,1 r ℓ i,2 (a) (b) (c) Diluition Spheres of different sizes Multiple spheres z z z ⇢(z) ⇢(z) ⇢(z) n = r ℓ i,1 r ℓ i,2 (a) (b) (c)

Diluition

Spheres of different sizes Multiple spheres so that the scaling in Eq. (4.75) remains valid even in these contexts of greater biological relevance.

z z z ⇢(z) ⇢(z) ⇢(z) n = 2 r ℓ i,1 r ℓ i,2 (b) (c) Diluition Spheres of different sizes Multiple spheres z z z ⇢(z) ⇢(z) n = 2 r ℓ i,1 r ℓ i,2 (b) (c) Spheres of different sizes Multiple spheres z z ⇢(z) ⇢(z) ↵ = 1 ↵ = 1, ⇢ 0 =.4 = 1 ⇢ 0 =.2 c c c ↵ = 1, ⇢ 0 =.2 = 1 ⇢ 0 (a) (b) (c) (d) (e) (f )

individuality of neurons

Until now we have always considered that the different place cells in our recurrent neural network all behave the same way, in the sense that they all have the same propensity to have place fields in different maps. As the experiments in [START_REF] Sung | The statistical structure of the hippocampal code for space as a function of time, context, and value[END_REF][START_REF] Rich | Large environments reveal the statistical structure governing hippocampal representations[END_REF] show, however, it doesn't seem to be properly so, in fact each place cells looks to have its own propensity to have a place field per square meter and in particular this propensity seems to be maintained in different environments. In the following we try to extend our model taking this towards greater biological plausibility aspect into consideration together with the variants of the place cells model presented in the previous Section 6. [START_REF] Agliari | Multitasking associative networks[END_REF] In particular, here we want to study the consequences of non-independence between the elementary ERMs composing the MERM on the density of eigenvalues taking into account a biologically plausible statistics for PF propensities that will automatically imply in an elegant way the constraint on the sparsity in the activity of the neurons observed in CA3 [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF]. Moreover, since in this case the elements in the same position of the ERMs corresponding to different environments are correlated, we can no longer use the free probability theory approach to derive the spectral properties of the associated MERM (we no longer have the property of asymptotic freeness), but fortunately the calculation can be performed in this case using the replica method, see Sections C.2.1 and C.2.3.

Technically, in the following we generalize the computations presented in Section 4.6 and 5.3 to consider these more biologically relevant settings and also this time try to understand if the scaling found in (4.75) remains preserved even in case of taking into account the individuality of the place cells.

Let's consider therefore a simple schematization of what said above, that is to consider a network of N neurons that are divided in M = O(1) groups with β ρ N neurons each and M ρ=1 β ρ = 1. Neurons belonging to the same group have c ρ PFs per map of area φ ρ,m with m = 1, . . . , c ρ . In this way we explicitly consider that there are neurons that have many place fields (of various areas) in the different maps and others that have few or none.

The MERM that we have to take in consideration in this setting is the following:

C ij = 1 L L =1 c i m=1 c j m =1 Γ (i,m),(j,m ) r i,m -r j,m , (6.38)
where Γ (i,m),(j,m ) r i,mr j,m represents the overlap in the map of the spatial components m and m associated with place cells i and j with areas that depend on the group of neurons to which they belong.

Before proceeding to the generalization of the quenched place field theory it is necessary to calculate the resolvent of (6.38), the details of the computation can be found in Appendices C.2.1 and C.2.3 and putting together these results we obtain the following system of equations that allows us to find this resolvent:

     g(U) = M ρ=1 q ρ , U = - β ρ q ρ + k =0 α cρ m=1 Γ (k) ρ,m α+ M ν=1 q ν c θ m =1 Γ (k) θ,m , ∀ρ , (6.39)
where we defined an overlap q ρ for each group of neurons, ρ = 1, . . . , M. Now using this result and following closely Appendices C.2.2 and C.2.4 we can generalize to this case the theory presented in Section 4.6 and obtain finally the asymptotic scaling of the critical capacity in the large p limit:

α c (p) ∼ A(D) (log p) D M ρ=1 β ρ c ρ m=1 φ 1-1 D ρ,m D (p → ∞) . (6.40)
It is interesting to note that in case D = 1 the critical capacity does not seem to depend on the size of the PFs and also that we find in the limit cases trivially the results of 4.6, C.2.2 and C.2.4.

non uniform distribution of positions

Another trivial but at the same time important generalization of our theory is to consider the case of variable spatial error from map to map. In fact, as already discussed in Section 4.1, it is legitimate to think of storing in a RNN different environments with different levels of accuracy depending on the needs and importance (some maps or some areas in a map may be more important to remember for an animal as they may correspond to points of interest such as home, food or a reward in general).

This can be done simply by substituting p with its average value over the maps in Eq. (4.75) fo the model presented in Chapter 4 and suggests that the fraction of maps with finest spatial resolution should not exceed ∼ D when → 0, in order not to affect too much the critical capacity, thus keeping unchanged the results found so far.

Moreover, while we have assumed in Chapter 4, for the sake of simplicity, that the distribution of positions was statistically uniform across space, there is no need for this to be so in practice. Experiments have shown that spatial representations of environments are enriched in PFs close to spots of interests (such as water pots [START_REF] Hollup | Accumulation of hippocampal place fields at the goal location in an annular watermaze task[END_REF] or objects [START_REF] Bourboulou | control of hippocampal spatial coding resolution by local visual cues[END_REF]) with respect to void regions. We report in the following numerical simulations showing that increasing the density of prescribed positions in regions of the physical space allows us to carve specific attractors in the neural activity space, representing preferentially those towards greater biological plausibility regions. This result is compatible with recent studies establishing the link between PFs distribution and behavioral place preference [START_REF] Mamad | Place field assembly distribution encodes preferred locations[END_REF].

More precisely, up to now we have considered for simplicity that the p positions were drawn uniformly at random to produce statistically homogeneous maps, i.e., without preferred positions. It is straightforward to extend this setting to the case of heterogeneous densities of prescribed positions.

Figure 34 shows the spatial distribution of stabilities for homogeneously scattered points (a) and an heterogeneous repartition of points, densely packed in a subregion (diagonal) (b). In the latter case, the strong heterogeneity in the local distribution of stabilities will favor the location of the bump along the zones with a major density of positions. As a consequence, a 1D-attractor is effectively built in the D = 2-dimensional space. In order to better understand this mechanism we show an example of Monte Carlo simulation SmallKappaL1Hetero.mkv 5 that corresponds to the heterogeneous distribution 5. The mentioned example can be viewed at: https://journals.aps.org/prl/supplemental/10.1103/ PhysRevLett.124.048302. As in the videos presented in Section 4.4, also here the red crosses represent the of positions shown in Fig. 34(b) and that was obtained with the same parameters as in SmallKappaL1.mp4, see Section 4.4, but with 150 out of the p = 300 positions drawn on the diagonal of the map; the stability of the network was κ = .5.

Actually in this case it is expected that the dynamics of the network is favored to move along the diagonal of the map, i.e., the favorite patterns are those that correspond to have active place cells associated with PFs near the diagonal of the map, or in other words PFs such that if we calculate their center of mass (center of the bump) that corresponds to the position decoded at a given time, this is near the diagonal. This happens because the positions to be stored in the map are concentrated on the diagonal (heterogeneous distribution of positions) and after the learning process they have typically high stability values compared to the a priori not stored positions, see Fig. 34(b). Since the dynamics of the network is favored towards patterns that have a high stability value, the bump is confined to move mostly on the diagonal, in this sense we say that we have learned a 1D attractor from a 2D map. Moreover, considering that patterns coding for positions near the diagonal are favoured in dynamics, the activity of place cells whose PFs are near the diagonal is higher 6 . Although the results of numerical simulations seem reasonable, also here, as in Section 6.2 for the periodic boundary conditions, an analytical treatment in which we consider that the positions inside the maps are not homogeneously distributed becomes much more complicated for the same reason, i.e., lack of invariance under translations that technically makes hard the calculation of the ERM spectrum of single environment [START_REF] Parisi | EUCLIDEAN RANDOMMATRICES: SOLVEDAND OPEN PROB-LEMS[END_REF].

comparison between svm and theoretical couplings

Interestingly, the quenched PF theory developed in Section 4.6 can be applied to any particular set of PFs, not necessarily homogeneously distributed over space; knowledge of the PF characteristics, i.e., from experimental measurements, allows us to determine the multispace correlation matrix C defined in Eq. (4.46) and to make specific predictions. We show here a proof of principle of this approach where we compare the couplings found with SVMs and the ones given by our quenched PF theory on synthetic data, see Section 4.2.

Figure 35 shows how the couplings {W ij } depend on the size of the network, N, and of the distances between the PF centers of neurons i, j in the maps. We generally find that the couplings W ij obtained by SVMs and the "thermal" averages [W ij ] predicted by the stored positions of the different environments in the connectivity matrix of the neural network and the blue circles represent the centres of the PFs whose corresponding neurons are active at a fixed time (the PFs whose neurons are silent at a certain time are not shown).

6. Maybe from the video it doesn't seem so because PFs cover 30% of the area of the environment so even place cells with PFs quite far from the diagonal can be active.

towards greater biological plausibility quenched PF theory, see Section 4.6, for a fixed set of PF centers are in excellent agreement, see equations [START_REF] Barak | Mapping low-dimensional dynamics to highdimensional neural activity: A derivation of the ring model from the neural engineering framework[END_REF] and [START_REF] Barbour | What can we learn from synaptic weight distributions?[END_REF] in [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF] for details on the calculation of the average couplings and associated standard deviations.

-Both sets of couplings have mean values scaling as 1 N and standard deviations scaling as 1 √ N .

-The sign of couplings depend on the distance between their PF centers in the maps.

We get excitatory couplings for distances up to the radius of the PFs (r c = φ 0 π in D = 2), and inhibitory interactions for larger distances.

Furthermore, a direct application of our quenched PF theory could be to investigate in detail the phenomenon of global random remapping. During all this work we have always assumed random remapping between cognitive maps associated to different environments, i.e., orthogonal maps in which a given place cell presents place fields in different environments in totally random positions without any correlation with other maps. It is difficult to think that this is completely true especially in the case of two maps associated to very similar environments, see for example [START_REF] Romani | Continuous attractors with morphed/correlated maps[END_REF]. It might therefore be interesting to compare the theoretical MERM spectrum assuming random remapping with a numerical spectrum built from real data for example from the Alme et al. [START_REF] Charlotte B Alme | Place cells in the hippocampus: eleven maps for eleven rooms[END_REF] experiment and see if there are really differences in the spectrum compared to the assumption of totally random place fields. A technical problem in this sense would be that the numbers presented for example in the above mentioned experiment are very far from the thermodynamic limit with which the theory was developed, in fact, only about N = 30 place cells are measured simultaneously in the different rats that have memorized only L = 11 environments, so it will be crucial to understand the finite size effects while waiting for better data. In fact, thanks to new generation electrodes (neuropixels) or calcium imaging techniques it is now possible to simultaneously record the activity of thousands of neurons for long periods of time and therefore it is in principle possible to estimate the correlation matrix from PFs measured in different environments and reach L and N values in accordance with the thermodynamic limit with which the theory was developed.

dynamics of learning

Another extremely interesting topic in which here we limit ourselves to providing a first step, is the study of the learning dynamics of the network model presented in Chapter 4. More precisely the dynamics of the weights of a recurrent neural network such that starting from discrete attractors, i.e., maps in which few positions have been stored, progressively we get to store continuous attractors, i.e., increasing the number of memorized (the maximal distance achievable in unit square with periodic boundary conditions). Couplings were then put in the corresponding bins for all maps, and the averages and standard deviations were plotted as functions of the bin centers. Average couplings and associated standard deviations with quenched PF theory were computed with [START_REF] Barak | Mapping low-dimensional dynamics to highdimensional neural activity: A derivation of the ring model from the neural engineering framework[END_REF] and (24) of [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF], with the substitution α c → p α c as the number of patterns is here p × L.

towards greater biological plausibility positions per map thus decreasing sufficiently the spatial error with which the maps are encoded.

In particular here we show that the number of presentations of the patterns needed to stabilize a map is approximately proportional to p.

In fact, the SVM algorithm exposed in Sections 4.3 and A.1 is offline: all the patterns are available to the learning procedure at all times, which is not biologically realistic. As a preliminary attempt to understand how maps are learned, we have implemented an online learning scheme, which is an adaptive version of the perceptron algorithm (adatron) [START_REF] Jk Anlauf | The adatron: an adaptive perceptron algorithm[END_REF], see Appendix C.3 for details. In this procedure, patterns are presented one after the other. We may choose the order of presentation, as well as the learning rate η. In order to study the time needed for the algorithm to store a map, we have run the learning scheme in the simplest case of a single environment (L = 1). We have monitored the stability of the network during this learning phase as a function of the number of training rounds, a round corresponding here to the presentation of all the patterns in the data-set. In Fig. 36, we show that the number of rounds needed to stabilize a map is roughly proportional to p/η (for a fixed ratio p/N).

It would be interesting to relate this finding to biological results. Let us remark that the presentation of repeated rounds considered here could be realistic for an animal exploring the same environment several times, in particular a 1D corridor in which the sequence of visual inputs remains roughly unchanged from one exploration to another. Experiments show that changes in the environment (insertion of one object) lead to the production of a new representation, which is stabilized over 4-5 explorations, see [START_REF] Bourboulou | control of hippocampal spatial coding resolution by local visual cues[END_REF].

Moreover, it would be extremely important to study plausible learning rules that could ultimately elucidate how the network progressively maturates to account for more and more fixed points and eventually defines a quasicontinuous attractor, as seems to be the case during the first weeks of development in rodents [START_REF] Farooq | Emergence of preconfigured and plastic time-compressed sequences in early postnatal development[END_REF]. In fact, the authors showed that when a newborn rat need to learn a new environment, initially it seems that its representation of the map is more similar to a discrete attractor (a map where few positions are stored), but as the days pass it looks like that the attractors become progressively more continuous (this is due to the fact of starting to store more positions in the same map so as to reduce the spatial error with which the map is stored).

Finally, another important study with practical application would be the analysis of the diffusion coefficient of the bump of activity (which can be measured experimentally, see [START_REF] Stella | Hippocampal reactivation of random trajectories resembling Brownian diffusion[END_REF]) to tie it with the number of positions stored in a map and also understand the time it takes to store these positions online. From measurements of the activity bump during sleep of a rodent that has previously explored an environment we may be able to link these quantities and have access to a deeper mechanism of learning the maps themselves.

learning continuous attractors in rnn from real images

In this Section we present an application of the model presented in Chapter 4 in a different context than seen so far, that is how to memorize continuous attractors in a recurrent neural network starting from real images, inspired by the work of Zou et al. [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF]. In particular the authors of [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF] have proposed a biologically plausible scheme of a neural system that stores continuous attractors based on two fundamental steps.

-The first is how to generate high-level representations of objects such that the correlations between the neural representations reflect the semantic representations between them, using for this purpose a deep neural network trained on a large number of real images [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF].

towards greater biological plausibility -The second is how to learn correlated patterns in an RNN, using an orthogonal Hebb rule [START_REF] Srivastava | Overcoming catastrophic interference in connectionist networks using Gram-Schmidt orthogonalization[END_REF], where as usual the representations between the neural patterns are encoded in the weights of the network.

In the end they were able to show with this procedure that if the images presented to the network are linked by a continuous feature a continuous attractor is efficiently stored in a RNN.

In the following we present preliminary results in which instead of the orthogonal Hebb rule we use SVM learning, see Sections 4.3 and A.1, to memorize correlated patterns.

Moreover, besides showing results for a single continuous attractor stored in a RNN, as in [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF], we will try also to understand if it is possible to store more than one in the same connectivity matrix (in this case two).

Let's start by discussing what we have done in detail. So far we have always learned continuous attractors starting from ad-hoc patterns, in the sense that by construction, see Section 4.2, near positions in an environment had similar neural configurations (with a large overlap), while positions corresponding to different environments have neural configurations with almost no overlap (this is due to the random remapping between the different maps).

If we consider now the case of real images this situation is not so immediate, consider for example the data shown in Fig. 37 where there are photos of objects (a guitar and a sax) on a black background and rotated by different angles, in particular we have an image per angle, then p = 360 data per object 7 .

As already noted by Zou et al. [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF] however, representations of images of the same category in different positions, maybe rotated by a certain angle, can be more different (low overlap between the pixel vectors associated with the corresponding images) than representations of images belonging to different categories. This is due to the fact that the initial representations do not have a semantic meaning but only a spatial one, so for example two guitars rotated by 90 degrees can have almost no overlap. It is therefore important to extract from the data high-level representations so that data belonging to the same category always have very correlated representations while data belonging to different categories always have quite orthogonal representations, as it should be.

Following closely [START_REF] Zou | Learning a continuous attractor neural network from real images[END_REF], in order to extract these representations with a semantic meaning we use the convolutional deep neural network VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] that has been trained on a very large number of real images mimicking somehow the dorsal visual pathway 7. Actually the images that we will use in the following have a lower resolution that is 224 × 224 × 3 pixels, the factor 3 represents the RGB color scale, and where the values of the different pixels are normalized between 0 and 1 so that to fit the input of the deep neural network that we are going to use [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. So now we present the images of the data-set in Fig. 37 to VGG16 extracting all the high-level representations. Using our standard notation we have therefore patterns of size N = 4096 corresponding to two maps L = 2, one for the guitar and one for the sax, where we have a certain level of spatial resolution given by the number of data per map, p = 360, one image per angle, and we would like to store these data in a recurrent neural network with the procedure used so far with SVMs, see Sections 4.3 and A.1 8 .

towards greater biological plausibility

Once built the data-set we can perform the learning with SVMs and then try to study the dynamics of the associated recurrent neural network with the finite temperature scheme defined in Section 4.4.

First of all we show in Fig. 38 the results of the network dynamics after learning a single map, the one associated with the guitar, in which we display the findings in the bi-dimensional space corresponding to the first two principal components extracted from the training data with PCA [START_REF] Wold | Principal component analysis[END_REF].

It is interesting to note that although we projected the data in a two-dimensional space these seem to be arranged on a circle, so a one-dimensional manifold, this makes sense since the different guitars differ from each other only by one variable (an angle).

We start the dynamics from an initial configuration corresponding to one of the initial images and set a temperature of the order of the network stability. If we look at the evolution of the network state in the space where we projected the training data, it appears that this remains confined to the manifold defined by the data themselves, as it should be, see Fig. 38. Moreover, it seems as well that close points in the manifold correspond to guitars rotated more or less of the same angle.

Even more interesting is the case of storing two manifolds in an RNN at the same time, considering the data-set with both guitar and sax images. Also here we show after the learning phase with SVMs the results of Monte Carlo simulations at finite temperature. The first important thing to note in the examples that will follow is that after projecting the data of the two objects on the first two principal components obtained from PCA carried out on all the data together, the data corresponding to different classes are clustered and this shows that therefore the high-level representations obtained from VGG16 are able to distinguish the different categories going to capture the semantic meaning of the different objects.

In addition, in the videos Tsmall.mp4 and Thigh.mp4 9 we show two examples of the dynamics initialized from the same image but starting from different temperatures. In the 9. The video of the simulation at lower temperature can be seen at: https://www.dropbox.com/s/ e14acuw075k3ggi/T _ small.mp4, while the one corresponding to higher temperature can be seen at: https: //www.dropbox.com/s/7kds8vs4g1lhx7d/T _ high.mp4. In the above videos we present in the left panel the data projections in the two-dimensional space of the first two principal components extracted from the entire data-set. The blue dots correspond to the images of the guitars while the orange dots correspond to the images of the sax, the red star instead corresponds to the state of the network projected on the same space that evolves over time, a frame of the videos corresponds to a Monte Carlo sweep. While the central and right panels show the normalized overlap of the network state at a certain time with respect to the images of the data-set as a function of the angle of rotation of the guitar and the sax respectively. The red star present at every time in one between the central and right panels corresponds to the data-set image with greater overlap with respect to the current network state, identifying the location of the bump in one of the two maps. case of the simulation at lower temperature the dynamics remains confined to diffuse on the starting manifold, while in the example with a little higher temperature in addition to seeing diffusion on the individual manifolds we see as well transitions between them as we already saw in Section 4.4.

So in the end it seems, at least from this preliminary analysis, that our method is able to store continuous attractors in RNN even in more generic situations instead of the artificial data studied in the rest of the thesis. Moreover, we believe that this method allows also here to overcome the issues presented in Section 3.7, provided that we have extracted good representations of the data to store.

C O N C L U S I O N S

In this Chapter we sum up the work that has been done in this manuscript by summarizing the main results obtained and by giving a cue for future lines of research.

Let us begin by summarizing the results of this thesis.

Certainly the main finding of this work has been to show how to achieve optimal storage of multiple continuous attractors in recurrent neural networks together with the study of the optimal trade-off between capacity and spatial resolution, that is, how the requirement of higher spatial resolution affects the maximal number of attractors that can be stored, thus overcoming the theoretical problems left unsolved in the last twenty years, see Section 3.7.

Using a combination of state-of-the-art statistical physics and random matrix theory tools, we showed that the capacity decreases very slowly with the spatial error, more precisely as the inverse D th power of the logarithm of the spatial error, where D is the dimension of the manifolds, see Section 4.6.

This non trivial scaling proves that recurrent neural networks are very efficient memory devices capable of storing many continuous attractors at high resolution.

Moreover, also if the motivations of this work come from the field of computational neuroscience, in particular how to model the CA3 recurrent network in the hippocampus, this setting gave us the opportunity to generalize Gardner's theory, a milestone in statistical physics of disordered systems, to the case of patterns with strong spatial correlation. The study of data with structure is nowadays a very hot topic instead of considering in theoretical approaches simple random patterns, not only in the field of neuroscience but also the one of machine learning [START_REF] Goldt | Modelling the influence of data structure on learning in neural networks[END_REF].

In addition, in this work we have introduced a novel statistical ensemble for Euclidean random matrices (ERM), see Chapter 5, where the element i, j of the matrix depend on the distances between representative points of i and j in more than one space. Using a combination of heuristic assumptions, analytical and numerical calculation, we have shown that the high-density limit is non trivial when the number L of spaces and the size N of the matrix are sent to infinity, with a fixed ratio α = L/N. We have analytically studied the density of eigenvalues of this Multiple-space-ERM (MERM) ensemble, based on free-probability identities and on the replica method.

conclusions Even though we have introduced MERM in a specific setting we hope that this ensemble of random matrices will find applications and be of interest in other fields, i.e., in applied mathematics or in information theory. In particular, our results could be used for functions Γ with a dependence on the pairwise distances different from the ones considered in this work.

Finally, as we began to discuss in Chapter 6, there are several lines of research to pursue in the near future based on these results.

-An important aspect that we have not considered in this work is the extension to recurrent neural networks with continuous neurons, for example threshold linear [START_REF] Schönsberg | Efficiency of local learning rules in threshold-linear associative networks[END_REF][START_REF] Treves | Threshold-linear formal neurons in auto-associative nets[END_REF], instead of simple binary units. A generalization in this sense would not only be useful in view of more biologically plausible models, that would also allow to investigate another type of remapping seen in other regions of the hippocampus as CA1, namely rate remapping [START_REF] Fyhn | Hippocampal remapping and grid realignment in entorhinal cortex[END_REF], but also for example in the application we have shown in Section 6.9 about learning continuous attractors from real images where high-level representations extracted from deep neural networks have usually positive continuous components.

-Moreover, from a theoretical point of view, it would be very interesting to try to understand how to solve the problem of breaking the invariance under translation in the calculation of ERM spectrum in the high-density limit, with consequent applications in our model to the study analytically of maps without periodic boundary conditions (Section 6.2), heterogeneous distributions of positions (Section 6.6) and non-uniform distributions of the PFs in the different maps.

-Taking into account the fact that the different maps are not independent but can have correlations both in terms of PFs centers and positions is certainly an important problem to study, see [START_REF] Romani | Continuous attractors with morphed/correlated maps[END_REF]. Numerically it is easy to implement, just generate maps that are properly correlated with each other, for example assuming that only a fraction of PFs remap or that the different maps are generated from a common one in which the PFs centers are slightly shifted with a displacement extracted from a Gaussian distribution centered in zero and with a variance smaller than the diameter of the PFs. It is also possible to generate positions in different maps on similar trajectories, but this adds correlation between the patterns in different maps only if they have correlated PFs. In general it makes sense to assume maps with correlated PFs in order to store similar environments (with similar external inputs such as visual, auditory, etc.) and also to assume similar trajectories in similar maps as there may be common points of interest where an animal is interested to go (such as food or water). Analytically we are now working on this problem for a better understanding of the phenomenon of random remapping that is not clear is completely random especially in the case of very similar environments, see Section 6.7. Our theory is developed at quenched PFs and this allows us in principle to consider more complicated cases of random PFs provided that we can analytically calculate the spectral properties of the associated MERM (which is not trivial when taking into account correlations). Nevertheless, introducing correlations in general means to have patterns even more similar to each other compared to the case studied in the thesis and this can only increase the capacity of the network (with patterns that contain less information content due to these redundant representations).

-Furthermore, always making reference to how much seen in Section 6.7, it could be interesting to qualitatively compare the theoretical weights with the synapses measured experimentally in CA3 [START_REF] Jose Guzman | Synaptic mechanisms of pattern completion in the hippocampal CA3 network[END_REF], in particular to try to understand if the real connectivity matrix is more compatible with the fact to have the connections proportional to the correlation matrix between the patterns (Hebb rule) or, as the quenched PF theory predicts, see in particular the Eqs. ( 23) and ( 24) in [START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF], as the inverse of the correlation matrix (optimal connectivity).

-A crucial aspect that we have started to discuss in Section 6.8 is the development of a biologically plausible learning rule for the study of the learning dynamics of a RNN, that could be compared with experimental data of newborn animals that progressively mature continuous attractors starting from discrete ones [START_REF] Farooq | Emergence of preconfigured and plastic time-compressed sequences in early postnatal development[END_REF]. In Section 6.8 we started to study an online algorithm but still far from being compared with experimental data. This kind of study could be useful in order to answer questions such as how much time is needed to mature a continuous attractor, what is a reasonable value to choose for the learning rate, what is a reasonable initial configuration of the weights to choose, how should the environment be explored from an animal to memorize it more efficiently, how much does previous knowledge of other environments help to memorize a new one and so on. Moreover, in this context we can try to answer a question about the fact that when an animal is facing the learning of a new environment that can be similar to an old environment, it should decide whether to learn the new environment as a new map, or learn it as a reinforcement of an old map. A key ingredient in this direction can be found in [START_REF] Allegra | Differential Relation between Neuronal and Behavioral Discrimination during Hippocampal Memory Encoding[END_REF] where the authors show how an attention modulator (coming from the dentate gyrus which is an area external to the hippocampus) allows to discriminate between similar maps depending on whether or not it is behaviorally important for the animal.

-It would also be extremely interesting to study models in which the propensities of place cells are stochastic and not deterministic as in Section 6.

C ij = 1 L L =1 Φ i (r) Φ j (r) ( ) r , (7.1)
where the average • ( ) r is taken over the random positions r of points in the manifold corresponding to map . What matters in the parametrization of the manifolds and determines the asymtotics of the capacity is the behaviour of the density of eigenvalues of C, ρ(λ), for eigenvalues λ → 0. It would be very interesting to classify the possible behaviours of ρ and obtain the corresponding scalings of α c (p) for example, considering more complicated geometries for the environments than simple cubes, considering in the maps additional dimensions beyond spatial ones, such as variables corresponding to olfactory or auditory stimuli, or considering more general contexts for continuous attractors and not restrict ourselves to cognitive maps as in [START_REF] Chung | Classification and geometry of general perceptual manifolds[END_REF].

-We could also extend the model to study the grid cells, see Section 3.8. To do this we should assume that each neuron has fields that are arranged on a triangular grid instead of having one or a few fields per environment, so a position can activate a specific neuron whenever it is inside one of the fields associated with the grid cell.

Following [START_REF] Spalla | Can grid cell ensembles represent multiple spaces?[END_REF] we could also study the critical capacity of a model of this type in which the weights are not chosen in Hebbian way and try to understand the optimal trade-off between capacity and spatial resolution. While in the case of place cells this analysis is more motivated because it is known that animals store many maps associated with different environments and contexts, in the case of grid cells this is not so obvious as it seems at the moment that they define a single map (a single continuous attractor) and then it may not be necessary to study RNN that stores an extensive number of continuous attractors based on grid cells, but certainly an interesting theoretical problem to study.

-A very important aspect that we have not discussed in this thesis is the effect of the θ rhythm in the hippocampus which is considered essential for memory formation and navigation [START_REF] Mark | Theta-paced flickering between place-cell maps in the hippocampus: A model based on short-term synaptic plasticity[END_REF][START_REF] Romani | Short-term plasticity based network model of place cells dynamics[END_REF][START_REF] Wang | Theta sequences are essential for internally generated hippocampal firing fields[END_REF]. In biologically plausible models of RNN this rhythm is usually modeled with the introduction of an oscillating external field that acts on the dynamics of the neurons, it would be interesting to see how the dynamical properties of our model depend on the introduction of external fields like the latter.

-Finally it would be crucial as well to study in our model both the effect of a possible noise during the learning phase [START_REF] Ky | Neural networks optimally trained with noisy data[END_REF], as we have developed both simulations and theory at zero temperature and from a biological point of view it is a great simplification, and to consider at the same time network dynamics with learning dynamics conclusions 123 (i.e, to consider dynamical synapses) [START_REF] Cc | Dynamical synapses enhance neural information processing: gracefulness, accuracy, and mobility[END_REF][START_REF] Wang | How Short-Term Synaptic Depression Reshapes Dynamics of Continuous Attractor Neural Networks[END_REF], since we have always studied the dynamical properties of RNNs after the learning of the connectivity matrix (which remains quenched during the network dynamics).

From this long to-do list it is clear that there is still a lot of research work to be done, particularly from the point of view of the biological plausibility of our model. Nevertheless, our results allow to give possible theoretical explanations to the fact that animals in real life (and not in the laboratory) are very able to navigate many environments with great accuracy, since they can in principle store in the CA3 network of their hippocampus a large number of cognitive maps with a good level of spatial resolution. This statement, although clear from an experimental point of view, was difficult to show in terms of standard theories on multiple continuous attractor neural networks, so we hope that our results have clarified this fundamental point.

To conclude, it is important to note that throughout the work we have focused on studying a model of the CA3 network without considering its interaction with other relevant areas of the brain. For example, as discussed in Sections 3.3 and 3.8, in the brain it is present the path integrator (PI), capable of integrating proprioceptive, vestibular and visual flow inputs and possibly supported by the grid-cell network in the medial-enthorinal cortex (mEC), that allows the animal to update the neural representation during navigation. It lacks therefore a generalization of our model in order to understand how contextual and PI inputs are combined by the hippocampal network to produce cognitive maps and accurate positional encoding [START_REF] Posani | Integration and multiplexing of positional and contextual information by the hippocampal network[END_REF].

A A P P E N D I X -C H A P T E R 4
The purpose of this Appendix is to facilitate the reading of Chapter 4. Indeed, we report here some details about the support vector machine algorithm, numerical simulations and also the calculations, both for Gardner's approach and for the quenched input (place) field theory. In addition, we propose again some results already presented in Chapter 4 but changing the model parameters in order to show the robustness of our findings.

a.1 support vector machine learning

As seen in Section 4.3 the problem of learning the connectivity matrix in our RNN can be decomposed into N independent problems of support vector machine (SVM) with linear kernel and hard margin [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF][START_REF] Cortes | Support-vector networks[END_REF][START_REF] Saunders | Support vector machine-reference manual[END_REF][START_REF] Scholkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF][START_REF] Vapnik | The nature of statistical learning theory[END_REF]. So let's focus now on one of the SVMs, say the one with output neuron i, and discuss the algorithm in detail.

We begin with the two-class classification problem, see Fig. 15(b), of the form

h i = j =i W ij σ j , (A.1)
where {σ j } are the input components of the SVM, that is a (N-1)-dimensional vector considering that j = i, taking binary values 0 or 1. While {W ij } are all connections that arrive on neuron i and are a priori real numbers (either positive, negative or null), see Fig. 15(a).

The training data-set, see Section 4.2, comprises p × L input binary vectors of components {σ ,µ j } ∈ {0, 1}, = 1, . . . , L, µ = 1, . . . , p and j = i, with corresponding target values 2 {σ ,µ i } -1 ∈ {1, -1}. A new data point is classified according to the sign of h i and we shall assume for the moment that the training data-set is linearly separable in the input space, so that by definition there exists at least one choice of the (N -1)-dimensional vector W i (decision boundary) such that a function of the form (A.1) satisfies h

,µ i = j =i W ij σ ,µ j > 0 for points having 2 σ ,µ i -1 = 1 and h ,µ i < 0 for points having 2 σ ,µ i -1 = -1, so that the stabilities ∆ ,µ i ≡ (2 σ ,µ i -1)h ,µ i
> 0 are positive for all the training data points.

There may of course exist many such solutions that separate the classes exactly, like the ones that can be found from the perceptron algorithm, see Section 4.3. Moreover, as we have already discussed (Section 4.3), among all the possible solutions to this problem that 125 appendix-chapter 4 allow us to correctly classify the training data-set, a good idea is to choose the one with the smallest generalization error (the hyperplane that maximizes the distance between classes). The SVM approaches this problem through the concept of the margin, which is defined to be the smallest distance between the decision boundary and any of the samples. In SVMs the decision boundary is then chosen to be the one for which the margin is maximized, as illustrated in Fig. 15(b).

The perpendicular distance of an input pattern from an hyperplane defined by h i = 0, where h i takes the form (A.1), is given by

|h ,µ i | j =i W 2 ij
, see Fig. 15(b). Furthermore, we are only concerned in solutions for which all input patterns are correctly classified, so that ∆ ,µ i > 0 ∀ , µ. Thus the distance of an input pattern to the hyperplane is given by

(2σ ,µ i -1)h ,µ i j =i W 2 ij = (2σ ,µ i -1) j =i W ij σ ,µ j j =i W 2 ij . (A.2)
The margin we are interested in is determined from the distance to the closest input pattern of the data-set, and we wish to optimize the vector W i in order to maximize this distance. Hence the maximum margin solution is found by solving max

W i 1 j =i W 2 ij min ,µ (2σ ,µ i -1) j =i W ij σ ,µ j , (A.3)
where we have taken the factor Direct solution of this optimization problem would be very complex, and so we shall convert it into an equivalent problem that is much easier to solve. To do this we note that if we make the rescaling W i → κ i W i , then the distance from any input point to the decision surface, given by

(2σ ,µ i -1)h ,µ i j =i W 2 ij
, is unchanged. We can use this freedom to set

(2σ ,µ i -1) j =i W ij σ ,µ j = 1 (A.4)
for the input data that is closest to the surface. In this case, all data points satisfy the constraints

(2σ ,µ i -1) j =i W ij σ ,µ j 1, ∀ , µ . (A.5)
This is known as the canonical representation of the decision hyperplane. In the case of input data points for which the equality holds, the constraints are said to be active, whereas for the remainder they are said to be inactive. By definition, there is always at least one active constraint, because there is always a closest point, and once the margin has been maximized there are at least two active constraints. The optimization problem then simply requires that we maximize

1 j =i W 2 ij
, which is equivalent to minimize

j =i W 2 ij ,
and in the end we have to solve the optimization problem min

W i 1 2 j =i W 2 ij (A.6)
subject to the constraints given by (A.5). The factor of 1 2 in (A.6) is included for later convenience. This is an example of a quadratic programming problem in which we are trying to minimize a quadratic function subject to a set of linear inequality constraints [START_REF] Ben | Lectures on modern convex optimization: analysis, algorithms, and engineering applications[END_REF][START_REF] Boyd | Convex optimization[END_REF][START_REF] Nesterov | Introductory lectures on convex optimization: A basic course[END_REF].

In order to solve this constrained optimization problem, we introduce Lagrange multipliers {λ µ }, with one multiplier for each of the constraints in (A.5), giving the Lagrangian function

L(W i , {λ µ }) = 1 2 j =i W 2 ij - ,µ λ ,µ (2σ ,µ i -1) j =i W ij σ ,µ j -1 . (A.7)
Note the minus sign in front of the Lagrange multiplier term because we are minimizing with respect to W i and maximizing with respect to {λ µ }. Setting the derivatives of L(W i , {λ µ }) with respect to W i equal to zero, we obtain the following conditions

W ij = L =1 p µ=1 λ µ (2σ µ i -1) σ µ j . (A.8)
Eliminating W i from L(W i , {λ µ }) using these conditions then gives the dual representation of the maximum margin problem in which we maximize

L({λ µ }) = L =1 p µ=1 λ µ - 1 2 L ,m=1 p µ,ν=1 (2σ µ i -1)(2σ mν i -1)λ µ λ mν N j( =i) σ µ j σ mν j (A.9)
subject to the constraints λ ,µ 0, ∀ , µ . (A.10)

This optimization problem can be solved using available numerical routines [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF][START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF].

appendix-chapter 4

In the end, the only relevant data points for the learning process are the one corresponding to λ µ > 0 as they define the weights of the network according to the Eq. (A.8), they infact correspond to points that lie on the maximum margin hyperplanes in the input space, see Fig. 15(b), and are the so called support vectors.

Once we have obtained {λ µ } we can then finally find the maximum margin as

κ i = 1 j =i W 2 ij (A.11) using (A.8).
As an illustration of the learning procedure, we show in Fig. 39 how the number of stored patterns (with positive stabilities) grows as a function of the number of iterations of the quadratic optimization algorithm solving (C.3), until all p prescribed patterns are stabilized. We are thus now able to calculate the optimal couplings for an SVM and to find the respective maximal margin κ i . We can then use this algorithm to extract all the rows of A.2 estimation of critical capacity from svms results 129 the RNN connectivity matrix, one for each SVM, which therefore is not a priori symmetric. We can as well derive all the margins κ i ∀ i and at the end the margin associated to the RNN is given by the minimum among all the margins of the single SVMs κ = min i κ i , (A.12) see Section 4.3.

Note that formula (A.8) does not give normalized couplings to the unit and so, after calculating the optimal margin with Eq. (A.11), we should eventually normalize them by hand.

a.2 estimation of critical capacity from svms results

The critical capacity α c (p), that is the value of the load α = L N at which the stability κ(α, p) vanishes or, in other words, the maximal load sustainable by the network as a function of the required spatial error, was estimated as follows from SVM results in all the thesis figures in which it appears.

We computed the optimal stabilities κ (at fixed p) for M different values of the load α (obviously the values of α are chosen in a range such that the classes are linearly separable, i.e., κ > 0), with M generally equal to 20. Then we fitted these points with the empirical function (depending on the parameters a, b, c)

κ = a √ α + b α + c , (A.13)
and extrapolated from the fit the value of the load at which the fitted function vanished; this defined our estimate for α c (p). Note that the small α behaviour in equation (A.13) above can be justified analytically from Gardner's calculation [START_REF] Gardner | The space of interactions in neural network models[END_REF], see Section 4.5.

a.3 recovering gardner results in case of one position per map

Here we show that, when only a single pattern is considered in each map (p = 1), the equation (4.39) is equivalent to the celebrated Gardner critical capacity in the case of biased patterns [START_REF] Gardner | The space of interactions in neural network models[END_REF], where the bias comes from the PF area φ 0 , if for example this area is half the area of the environments, i.e., φ 0 = .5, then the corresponding patterns would be unbiased.

For p = 1, the Euclidean random matrix Γ( R) defined in equation (4.17) reduces to the scalar

Γ( R) 1,1 = φ 0 (1 -φ 0 ) ≡ 1 -M 2 4 , (A.14) a.4 computation of Ξ(U)
Here we explain in detail the calculation of Ξ(U) defined in Eq. (4.52).

Expanding the terms in Ξ(U), we write

Ξ(U) = Ξ 1 (U) + Ξ 2 (U) + Ξ 3 (U) with Ξ 1 (U) = 4 j,k 2 C 1j U Id + C -1 jk C 1k , (A.20) Ξ 2 (U) = φ 2 0 j,k 2 U Id + C -1 jk , (A.21) Ξ 3 (U) = -4 φ 0 j,k 2 C 1j U Id + C -1 jk . (A.22)
Computation of Ξ 1 : Consider the N × N matrix C (N) , with entries C ij for i, j comprised between 1 and N. Let us also define Id (N) as the identity matrix in dimension N, while Id above referred to the identity matrix in dimension N -1. Using block-matrix inversion formulas, we write that

U Id (N) + C (N) -1 11 = 1 U + C 11 - j,k 2 C 1j U Id + C -1 jk C 1k . (A.23)
The left hand side of the equation above is equal, in the large-N limit, to the resolvent g(U) of C defined in (4.47). Using C 11 = Γ (0) = φ 0 and the definition of Ξ 1 (U), we obtain

Ξ 1 (U) = 4 U + φ 0 - 1 g(U) . (A.24)
Computation of Ξ 2 : Let |v + be the normalized vector with N identical components, (v

+ ) i = 1 √ N . We have Ξ 2 (U) = N φ 2 0 v + U Id + C -1 v + . (A.25)
For large N, |v + is the top eigenvector of C, with (extensive) eigenvalue

λ + = N dr Γ (r) = N φ 2 0 . Hence, Ξ 2 (U) = N φ 2 0 × 1 U + Nφ 2 0 → 1 , (A.26)
in the large-N limit (since U remains bounded, see Section 4.6).

appendix-chapter 4 In order to compute g(z), we can rewrite Eq. (B.2) in various forms. First, we note that In particular, R(z) is the generating function of the "free cumulants".

Let us now mention a couple of properties useful for the analysis in Chapter 5. The functions g(z), B(z), and R(z) obey the following scaling relations:

g c A (z) =

free probability theory in a nutshell

Free probability theory is a research field in mathematics started by Voiculescu in 1983 [246, 247]. The initial goal of this theory was to say something about the spectral properties of the sum of two matrices X 1 + X 2 from the knowledge of the spectral properties of the individual ones X 1 and X 2 . Unless the two matrices commute, knowing the spectrum of the individual ones is not sufficient to find the spectrum of the sum. In any case free probability theory identifies a sufficient condition, the so-called asymptotic freeness with which this problem can be faced without resorting to the eigenvectors of the matrices. This notion of asymptotic freeness is a generalization of the concept of statistical independence for random variables when these variables, in our particular case the matrices, do not commute.

Let us briefly recall the basic properties of independent variables. We denote by p x the probability density of the variable x, by g x (z) ≡ e z x = n 0 x n z n /n! its characteristic function, and by r x (z) ≡ ln g x (z) = n 0 c x,n z n its cumulant generating function. For two independent real random variables x 1 and x 2 , the following relations hold: where " * " denotes the convolution operation. We will see that these relations find their equivalents for asymptotically free matrices.

x 1 x 2 =
By definition, two Hermitian matrices X 1 and X 2 are asymptotically free if for all l ∈ N and for all polynomials p i and q i (1 i l), we have [START_REF] Tulino | Random matrix theory and wireless communications[END_REF] p i (X 1 ) Λ = q i (X 2 ) Λ = 0 ⇒ p 1 (X 1 )q 1 (X 2 ) . . . p l (X 1 )q l (X 2 ) Λ = 0 , (B. [START_REF] Daniel J Amit | Retrieval phase diagrams for attractor neural networks with optimal interactions[END_REF] where the expectation value • Λ is defined as

X Λ = 1 N Trace X . (B.17)
The interpretation of the formal definition (B.16) is the following: two matrices are asymptotically free if their eigenbases are related to one another by a random rotation, or said differently, if their eigenvectors are almost surely orthogonal.

From the definition (B.16), it is easy to compute various mixed moments of X 1 and X 2 . Considering Xi = X i -X i Λ that obey X1 Λ = X2 Λ = 0, we obtain from Eq. (B.16):

X 1 X 2 Λ = X 1 Λ X 2 Λ .
(B.18)

c.2 analytical details on the individuality of neurons

Here we show a series of generalizations of the computations presented in Sections 5.3 and 4.6 that will allow us to include in the model introduced in Chapter 4 the individuality of neurons discussed in Section 6.5 together with the variants on the place cell models presented in Section 6.4.

c.2.1 Spectrum of MERM: multi-populations of neurons

As a first step in the above direction we consider the following MERM:

M (L) ij = 1 N L =1 Γ ij r i -r j , (C.7)
in which the Γ function depend explicitly on the indices i and j, i.e., the N neurons are divided in M finite groups with fractions of neurons β µ with µ = 1, . . . , M. Every group of neurons has a specific property, i.e., a fixed area for the place fields φ µ .

We are going to compute the resolvent of M (L) using the replica method.

As in Section 5.3 we start by rewriting the definition of the resolvent as 2 can be expressed as a canonical partition function:

s L (z) = 1 N Trace M (L) -z Id -1 = 2 N ∂ z log det M (L) -z Id
Z L (s) = det M (L) -z Id -1 2 = i dφ i √ 2π exp z 2 i φ 2 i - 1 2 ij φ i M (L) ij φ j , (C.9)
where i, j go from 1 to N.

The resolvent (C.8) can be calculated using the replica trick [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]: where we have replicated the system n times, i.e., a goes from 1 to n.

s L (z) = 2 N ∂ z log Z L (s) = 2 N ∂ z lim

appendix-chapter 6

In order to perform the average in (C.11) we rewrite (C.7) by considering the -th space ERM in its eigenbasis: where we have used the fact that the N neurons are divided in M groups and we have defined an overlap q ab µ per group as notice, in fact, that here the function Γ depend on the indices j and k of the neurons not only for indicating the centers of the PFs in the different maps but also to explicitly distinguish the properties of the different neurons and also φ j varies from neuron to neuron.

M (L) ij = 1 N L =1 Γ ij r i -r j = k =0 v ki Γij (k) v kj , (C.
q ab µ = 1 N i∈β µ N φ a i φ b i , ∀µ (
Now we make the same approximations as Section 4.6 and so we can write the final expression of the order parameters after adding up all the maps:

m a ≡ 1 L L =1 m a = j 2
W ja 2 C 1j {r j } -φ j (C.29) and

q ab ≡ 1 L L =1 q ab = j,k 2
W ja W kb C jk {r j } .

(C.30)

The N × N multi-space Euclidean random matrix C appearing in the expressions above is defined as

C ij = 1 L L =1
Γ ij r ir j . (C.31)

In the following, we denote by ρ(λ) the density of eigenvalues λ of C. This density is selfaveraging when the PFs are randomly drawn in the large L, N double limit. Its resolvent, defined as

g(U) = dλ ρ(λ) λ + U , (C.32)
where the integral runs over the support of ρ, is solution of the following system of equations, see Section C.2.1:

     g(U) = M ρ=1 q ρ , U = - β ρ q ρ + k =0 α Γρ (k) α+ M
θ=1 Γθ (k)q θ , ∀ρ .

(C.33)

appendix-chapter 6

From here on we obtain the same equations of what saw in Section 4.6 except for the quantity Ξ defined as

Ξ(U) = j,k 2 H j U Id + C -1 jk H k with H j = 2 C 1j -φ j , (C.34)
and Id is the identity matrix. In the above equation, the inverse is intended over the N -1-dimensional restriction of the matrix U Id + C to entries j, k 2. in which the N neurons are divided in M finite groups with fractions of neurons β µ with µ = 1, . . . , M. Every group of neurons has a specific property, i.e., c µ PFs per map. The area of the PFs is fixed to φ 0 < 1 for all the neurons for simplicity.

We are going to compute the resolvent of M (L) using the replica method coming from statistical physics of disordered systems.

We start by rewriting the definition of the resolvent as where i, j go from 1 to N. The resolvent (C.62) can be calculated using the replica trick [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]: where we have replicated the system n times, i.e., a goes from 1 to n.

s L (z) = 1 N Trace M (L) -z Id -1 = 2 N ∂ z log det M (L) -z Id
s L (z) = 2 N ∂ z log Z L (s) = 2
In order to perform the average in (C.65) we rewrite (C.61) by considering the -th space matrix in its eigenbasis: Here we are going to extend the Gaussian theory with quenched PF developed in Section 4.6 to the case of multi-populations of neurons and multiple PFs per neuron in the same map. By multi-populations of neurons we mean that the N neurons are divided in M finite groups with fractions of neurons β ρ with ρ = 1, . . . , M and every group of neurons with a specific property, i.e., c ρ PFs with area φ 0 < 1. The N × N multi-space Euclidean random matrix C appearing in the expressions above is defined as

M (L) ij = k =0 v ki Γ (k) v kj , (C.
C ij = 1 L L =1 c i m=1 c j m =1
Γ r i,mr j,m . (C.85)

In the following, we denote by ρ(λ) the density of eigenvalues λ of C. This density is selfaveraging when the PFs are randomly drawn in the large L, N double limit. Its resolvent, defined as

g(U) = dλ ρ(λ) λ + U , (C.86)
where the integral runs over the support of ρ, is solution of the system of equations, see C. Now we make the hypothesis that q ρ = g r ρ with r ρ = O(1) ∀ρ so that can write (C.97) and (C.98) as:

c.3 adatron algorithm

Until now we have always used offline algorithms to solve SVM problems where the patterns to be stored were presented all together to the network in an unrealistic way, using standard packages to solve convex optimization problems [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF][START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], see Section A.1. Here we show a version of the SVM algorithm of online nature in which the patterns are presented one at a time (more biologically plausible) and that would allow us to study questions like what is the best way to present the patterns in order to stabilize them as soon as possible, or even to present the patterns in such a way to represent the realistic trajectories of an animal that explores one after the other the different environments. So this is a first step for the study of learning dynamics in our model. Fortunately there are already in literature algorithms of this type [START_REF] Jk Anlauf | The adatron: an adaptive perceptron algorithm[END_REF][START_REF] Engel | Statistical mechanics of learning[END_REF][START_REF] Krauth | Learning algorithms with optimal stability in neural networks[END_REF] and in particular in the following we will consider the adatron algorithm of which we report here the details.

The adatron algorithm is equivalent to the support vector machine in the sense that it converges to the same solution of the perceptron with optimal stability (maximum ABSTRACT How sensory information is encoded and processed by neuronal circuits is a central question in computational neuroscience. In many brain areas, the activity of neurons is found to depend strongly on some continuous sensory correlate. Over the past decades, continuous attractor neural networks were introduced as an abstract model for the representation of a few continuous variables in a large population of noisy neurons. While the original model was based on how to build a single continuous manifold in an highdimensional space, it was soon realized that the same neural network should code for many distinct attractors. An approximate solution to this harder problem was proposed twenty years ago, and relied on an ad-hoc prescription for the pairwise interactions between neurons. This solution, however, suffers from two major issues: the interference between maps strongly limit the storage capacity, and the spatial resolution within a map is not controlled. In the present manuscript, we address these two issues using a combination of techniques from statistical physics of disordered systems and random matrix theory. We show how to achieve optimal storage of continuous attractors and study the optimal trade-off between capacity and spatial resolution.

MOTS CLÉS

Réseau Neuronal à Attracteur Continu, Physique Statistique, Théorie des Matrices Aléatoires, Neurosciences Computationnelles.

RÉSUMÉ

La manière dont l'information sensorielle est codée et traitée par les circuits neuronaux est une question centrale en neurosciences computationnelles. Dans de nombreuses régions du cerveau, on constate que l'activité des neurones dépend fortement de certains corrélats sensoriels continus. Au cours des dernières décennies, les réseaux neuronaux à attracteur continu ont été introduits comme un modèle abstrait pour la représentation de quelques variables continues dans une grande population de neurones bruités. Alors que le modèle original était basé sur la construction d'une variété continue unique dans un espace à haute dimension, on s'est vite rendu compte que le même réseau neuronal pouvait coder pour de nombreux attracteurs distincts. Une solution approximative à ce problème plus difficile a été proposée il y a vingt ans, et reposait sur une prescription ad hoc pour les interactions par paires entre les neurones. Cette solution souffre cependant de deux problèmes majeurs: l'interférence entre les cartes limitent fortement la capacité de stockage, et la résolution spatiale au sein d'une carte n'est pas contrôlée. Dans le présent manuscrit, nous abordons ces deux questions en utilisant une combinaison de techniques issues de la physique statistique des systèmes désordonnés et de la théorie des matrices aléatoires. Nous montrons comment parvenir à un stockage optimal des attracteurs continus et étudions le compromis optimal entre capacité et résolution spatiale.

KEYWORDS

Continuous Attractor Neural Network, Statistical Physics, Random Matrix Theory, Computational Neuroscience.

xi 1 S

 1 TAT I S T I C A L P H Y S I C S M E E T S C O M P U TAT I O N A L N E U R O S C I E N C E

1 .

 1 The factor 3 comes from the RGB color channels.

Figure 1 -

 1 Figure 1 -(a) The set of variations associated with the image of a dog form a continuous manifold of dimension much lower than the pixels space. Other object images, such as those corresponding to a cat in various poses, are represented by other manifolds in same space. Figure adapted from [58]; (b) Examples of digits corresponding to the number two in the MNIST data-set. Figure taken from [152]. (c) Pictures of a person with various facial expressions. They lie in a very low dimensional manifold of the vector space of pictures with 1000 × 1000 pixels. Figure taken from [139].

Figure 2 -

 2 Figure 2 -(a) Neural modes as a generative model for population activity. The relative area of the blue/green regions in each neuron represents the relative magnitude of the contribution of each cell assembly to the neuron's activity; (b) Spikes from three recorded neurons during task execution as a linear combination of two neural modes; (c) Trajectory of time-varying population activity in the neural space of the three recorded neurons (black). The trajectory is mostly confined to the neural manifold, a plane shown in gray and spanned by the neural modes u 1 and u 2 ; (d) A curved, nonlinear neural manifold, shown in blue. Figure adapted from [86].

Figure 3 -

 3 Figure 3 -(a) Sketch of a recurrent neural network (RNN) with N = 6 neurons and connectivity matrix W; (b) Schematic diagram of the space of configurations of a RNN (Ndimensional space) in which the connections have been chosen to store multiple discrete attractors. The dashed lines indicate the attraction basins between the different fixed points while the arrows are examples of possible trajectories for a deterministic dynamics (converging to the nearest fixed point depending on the initial condition) of the network, for different initial configurations; (c) Same as (b) but in the case of multiple continuous attractors. r is a D-dimensional collective coordinate that describe the "position" along one of the manifolds. Figure adapted from [28].

  4(a) for a schematic representation of two interconnected neurons and Fig. 4(b) for a schematic plot of an action potential.

Figure 4 -

 4 Figure 4 -(a) Schematic representation of two interconnected neurons. The contact areas where the information is transmitted are called synapses. A signal from the presynaptic cell is transmitted through the synapses to the postsynaptic cell. Figure taken from [230]; (b) Approximate plot of a typical action potential shows its various phases as the action potential passes a point on a cell membrane. The membrane potential starts out at approximately -70 mV at time zero. A stimulus is applied at time 1 ms, which raises the membrane potential above -55 mV (the threshold potential). After the stimulus is applied, the membrane potential rapidly rises to a peak potential of 40 mV at time 2 ms. Just as quickly, the potential then drops and overshoots to -90 mV at time 3 ms, and finally the resting potential of -70 mV is reestablished at time 5 ms. Figure taken from [253].

Figure 5 -

 5 Figure 5 -(a) Energy landscape of a Hopfield network, highlighting the current state of the network (up the hill), an attractor state to which it will eventually converge, a minimum energy level and a basin of attraction shaded in green. Note how the update of the Hopfield network is always going down in energy. Figure taken from [254]; (b) Average percentage of errors in the Hopfield model as a function of α = P N at zero temperature and with no external fields. Figure taken from [17]. (c) Phase diagram, temperature T vs load α = P N , of the Hopfield model in the absence of external fields. The continuous line represents the transition line between paramagnetic (PM), ferromagnetic (FM) and spin glass (SG) phases. Figure adapted from [17, 155].

13 .

 13 The name of pyramidal cells comes from their shape.

Figure 6 -

 6 Figure 6 -Main structures of the human and rodent limbic system. (a) Human brain showing the amygdala (green), bed nucleus of stria terminalis (BNST, blue), hypothalamus (yellow), and hippocampus (pink). The hippocampus (pink) attaches to the mamillary bodies (orange) through the fimbria-fornix. Olfactory inputs are received by the olfactory bulbs (MOB, purple). Other structures include the nucleus accumbens (NuAc), ventral tegmental area (VTA), and the periaqueductal gray (PAG). (b) Similar structures are found in rodents. Figure and caption adapted from [221].

Figure 8 -

 8 Figure 8 -Phase diagram of Lebowitz and Penrose's theory of the lquid/vapor transition, in dimension D = 1 (periodic boundary conditions). Insets show the density of particles ρ(x) as a function of position over space, x ∈ [0; 1]. Note the coexistence between the homogeneous and bump states at intermediate temperatures. The location of the bump is arbitrary. Figure taken from [60].

Figure 9 -

 9 Figure 9 -An example network of N neurons (small circles) with 1D continuous attractor dynamics. Local excitatory and global inhibitory connections (not shown) between all neurons stabilize population states that are local activity bumps (i.e., blue bump A or B; gray: transient/unstable activity profiles). Figure adapted from [264].

Figure 10 -

 10 Figure 10 -(a) Phase diagram in the (T, α = L N ) plane in D = 1, f = .1 and ŵ = .05 for the Rosay and Monasson's model. Thick lines: transitions between phases. Figure taken from [166].

Figure 11 -

 11 Figure 11 -Time evolution of a 1D network with N = 1000 units and 2 stored environments observed in a Monte Carlo simulation at T = 0.007, illustrating the coexistence of diffusion within one map and transitions between maps. Each black dot represents an active unit. Both panels represent the same data, they only differ by the ordering of the units along the x-axis (remapping of the place fields). (a) the units i are arranged according to their place field centres in environment 1. (b) they are arranged according to their place field centres in environment 2. The y-axis represents time (in Monte Carlo rounds). Between time 0 and time ∼ 1000 the activity is localized in map 2 and delocalized in map 1. Then it undergoes several transitions between 1 and 2.Between times ∼ 2000 and ∼ 3700, the network is in attractor 1 and the bump diffuses within this attractor. Finally, it ends up and diffuses in map 2. Note the abruptness of the transitions between maps dynamics of quasi-continuous attractors is much richer than in the case of more basic models as the Hopfield network, where the only possible evolution is to transit from one attractor to the other, see Fig. 3(b).Figure taken from [167].

  Figure 11 -Time evolution of a 1D network with N = 1000 units and 2 stored environments observed in a Monte Carlo simulation at T = 0.007, illustrating the coexistence of diffusion within one map and transitions between maps. Each black dot represents an active unit. Both panels represent the same data, they only differ by the ordering of the units along the x-axis (remapping of the place fields). (a) the units i are arranged according to their place field centres in environment 1. (b) they are arranged according to their place field centres in environment 2. The y-axis represents time (in Monte Carlo rounds). Between time 0 and time ∼ 1000 the activity is localized in map 2 and delocalized in map 1. Then it undergoes several transitions between 1 and 2.Between times ∼ 2000 and ∼ 3700, the network is in attractor 1 and the bump diffuses within this attractor. Finally, it ends up and diffuses in map 2. Note the abruptness of the transitions between maps dynamics of quasi-continuous attractors is much richer than in the case of more basic models as the Hopfield network, where the only possible evolution is to transit from one attractor to the other, see Fig. 3(b).Figure taken from [167].

FrFigure 12 -

 12 Figure 12 -Sketch of the effective free energy landscape probed by the bump of neural activity (dashed bump) moving through an environment in the case of storing multiple continuous attractor in a RNN. Figure taken from [167].

Figure 13 -

 13 Figure 13 -Spatial cognition requires having a cognitive map, i.e., an internal representation of the outside world [232]. Single neuron recordings have led to the discovery of various neuronal cell assemblies, each with its own specific function, in the hippocampus and the enthorhinal cortex. The different biological features of each serve different specific functions, and have been found to provide the neural basis of spatial navigation. The activity of a place cell (a), a head direction cell (b), a grid cell (c) and a border cell (d) in environments of different sizes are illustrated in the spike plots from left to right. Figure taken from [150].

Figure 14 -

 14 Figure14-(a) Sketch of a manifold to store in the RNN, see Fig.3(c). We require to memorize p points (left, red crosses) on the manifold, whose separation defines the spatial error . (b) Place fields (PF) of N = 5 neurons in two maps. Each color identifies one neuron; the corresponding PF define the regions (with periodic boundary conditions) in the maps in which the neuron is active. The table lists, for each map, p = 3 activity patterns corresponding to the marked points.

  Figure14-(a) Sketch of a manifold to store in the RNN, see Fig.3(c). We require to memorize p points (left, red crosses) on the manifold, whose separation defines the spatial error . (b) Place fields (PF) of N = 5 neurons in two maps. Each color identifies one neuron; the corresponding PF define the regions (with periodic boundary conditions) in the maps in which the neuron is active. The table lists, for each map, p = 3 activity patterns corresponding to the marked points.

Figure 15 -

 15 Figure 15 -(a) One of the N perceptrons that make up the RNN with N = 5 neurons and where the output neuron is the one correspondent to the index i = 1. (b) Example of a learning problem for a perceptron where the aim is to find the hyperplane separating the two classes, class (0) and class (1), while maximizing its distance from them (in the maximum margin setting). It is clear from the figure that the problem of linearly separating the two classes with an hyperplane admits infinite solutions when the classes are linearly separable.

Figure 16 -

 16 Figure 16 -Couplings obtained after training with SVM for L = 1 (a) and (b) and L = 2 (c) and (d) maps. Left: dependence of the average coupling with the distance between the corresponding neurons; the vertical line locates the radius r c of the place fields. Averages were computed over 500 samples of the p positions per map at fixed PF centers; N = 1000 neurons. Right: histograms of the couplings, for sizes N = 100 (blue) and N = 1000 (orange) Parameters: D = 2, φ 0 = .3 and p × L = N.

Figure 17 -

 17 Figure 17 -Tracking a moving input. The x(t) represent the centers of the bump decoded by the activity of the place cells in the two environments (taking into account the periodic boundary conditions). Every temporal step t in the simulation corresponds to a Monte Carlo sweep (that is to the attempt to update N neurons chosen at random following the dynamics (4.4) that includes the external input) and consequently the plot reported corresponds to a single dynamics, and not to the average on different samples of possible trajectories.

  Fig.18(a)6 shows that becomes quite large as L increases if we use Hebbian-like learning rules(3.15) for the connectivity matrix. However, with the maximal-stability learning rule (Section 4.3), the spatial error can be tuned at will by varying p. For a fixed p, remains remarkably stable as the load increases until its critical value is reached. This is in sharp contradistinction with the Hebb rule case(3.15), for which quickly increases with the number of maps. The p patterns form a discrete approximation of the map, with average spatial error scaling as = p -1/D , i.e., as the typical distance between neighboring points[START_REF] Chandrasekhar | Stochastic problems in physics and astronomy[END_REF], see Fig.18(b).

Figure 18 -

 18 Figure 18 -(a) Spatial error vs number L of two-dimensional maps in a network of N = 1000 neurons. Black: rule (3.15), with w(d) = e -d/0.01 + w 0 , where w 0 < 0 enforces a fraction φ 0 = .3 of active cells. Colors: SVM results for different numbers p of prescribed positions. Line widths show the error bars. (b) spatial error vs number p of positions in a network of N = 1000 neurons storing L = 5 maps, in dimensions D = 1, 2, 3. Lines show the expected scalings ∼ p -1/D in log-log scale.

Figure 19 -

 19 Figure 19 -Stabilities obtained with the Hebb rule with exponential (a) and Gaussian (b) kernels on a given representative sample. The kernel parameters a and b vary from 0 to 10 with a step of .01. Parameter values: N = 1000, D = 2, φ 0 = .3, α = .1 and p = 5. The optimal value of the stability on that sample obtained by SVM is κ .55.

Figure 20 -

 20 Figure 20 -(a) Optimal stability κ as a function of the load α and the number p of positions obtained from SVM. Parameter values: D = 2, φ 0 = 0.3, N = 1000 for SVM. Inset: α c (p) decreases proportionally to 1/p (straight line) at low p, and much more slowly for large p. Dots indicate results from SVM (N = 5000), averaged over 50 samples; the dot size indicates the maximal error bar. (b) Scaling cross-over of α c (p) vs p for different values of φ 0 . The vertical lines correspond to the values of p c.o. ∼ 1 φ 0 . We use for this results D = 2, N = 5000, and we have averaged over 50 different realization of the environments and different realizations of the p positions. Parameter values: N = 5000, p = 100, Samples= 25.

Figure 21 -

 21 Figure 21 -Optimal stability κ as a function of the load α and the number p of positions. Dots, SVM results; dashed lines, Gardner's theory (4.39). Parameter values: D = 2, φ 0 = 0.3, N = 1000 for SVM.

Figure 22 -

 22 Figure 22 -Distributions of the overlaps q for different values of N and p. It is clear that the histograms are roughly Gaussian. We use for this results D = 2, φ 0 = .3, α = .1, and we have averaged over 500 realization of the p positions at fixed PF.

(4. 53 ) 4 . 6 . 2

 53462 Log. volume and saddle-point equations close to the critical line

Figure 23 -

 23 Figure 23 -(a) Optimal stability κ as a function of the load α and the number p of positions. Dots, SVM results; dashed lines dashed-dotted lines, quenched PF theory. Parameter values: D = 2, φ 0 = 0.3, N = 1000 for SVM. Inset: α c (p) vs p. Dots indicate results from SVM (N = 5000), averaged over 50 samples; the dot size indicates the maximal error bar. The dashed-dotted line shows the predictions from the quenched PF theory. (b) Critical capacity obtained by SVM vs φ 0 for different values of D in log-log scale. Parameter values: N = 5000, p = 100, Samples= 25.

Figure 24 -

 24 Figure 24 -Order parameters m (a) and q (b) vs p. Dots, SVM results (N = 2500), averaged over 50 samples; dashed-dotted lines, quenched PF theory. Parameters: D = 2, φ 0 = 0.3, α = 0.02 (top) and 0.05 (bottom), for which up to, respectively, p c 2500 and p c 250 points can be memorized.

3 Figure 25 -

 325 Figure 25 -Basic statistical ensemble of MERM considered in this Chapter, same as the one defined in Eq. (4.46) corresponding to the model defined in Section 4.2. L = 3 sets of N = 6 points, r i , with = 1, ..., L and i = 1, ..., N are drawn uniformly at random in unit squares H 2 (dimension D = 2). Points are represented by crosses, whose colors identify their indices i. The MERM is defined through (5.2), where Γ is a generic function of the distance between points. A possible choice for Γ , inspired from the place cells in neuroscience, is the overlap (common area) between pairs of disks (place fields) of surface φ 0 < 1 and having centers r i in each space .

Figure 26 - 1 .

 261 Figure 26i (v ⊥ i ) 2 vs N in log-log scale for the specific model defined in Fig. 25 but for D = 1, L = 1 and for the top 3 eigenvectors corresponding to k = 1, 2, 3. Each point is averaged over Samples = 500 realizations and the errorbars are simply given by the standard deviation divided by √ Samples -1. The fits are of the form log y = a + b log x.

.15) 2 . 3 . 3 R

 233 That is, calculate the Blue function, see Appendix B.1.5.2 spectrum of merm: free-probability-inspired derivation 79 Compute the R-transform of M (L) through L (s) = L R 1 (s) . (5.16) Using (5.15), we obtain,

Figure 27 -

 27 Figure 27 -(a) z vs s, see (5.21), close to the origin (s = 0), for different values of α. (b) Support of the spectrum for different values of α: black segments show the interval of eigenvalues z with non-zero density ρ(z). Results obtained by taking for Γ the overlap (common length) between segments of length φ 0 = .2, centered in points r i randomly drawn in the unit interval H 1 (D = 1), more precisely Γ (|r ir j |) = φ 0 -|r ir j |.

Figure 28 -

 28 Figure 28 -Density of eigenvalues of C, without the extensive eigenvalue z ext , for various values of α. Orange: results from (5.40). Blue: outputs of numerical diagonalization for N = 2500. Same model as in Fig. 27.

Figure 29 -

 29 Figure 29 -(a) Histogram of the coefficients γ and δ for different values of N. Results correspond to the k = 1 connected component of eigenvalues in dimension D = 1 and for L = 5 spaces, averaged over 50 samples. Same model as in Fig. 27. (b) Histograms of the projections of eigenvectors v to the k = 2 Fourier modes of the ERMs. (c) Histograms of the norm of the orthogonal component R, see (5.48).

. 2 )

 2 see Eq. (4.46) and Chapter 5, and therefore to all the rest.

Figure 30 -

 30 Figure 30 -Optimal stability κ as a function of the load α and the number p of positions obtained with SVMs. Stars, results with PBCs; crosses, results without PBCs. Parameter values: D = 2, φ 0 = 0.3, N = 1000. Inset: α c (p) decreases proportionally to 1/p (straight line) at low p, and much more slowly for large p. Stars indicate results from SVMs (N = 5000), averaged over 50 samples, with PBCs. Crosses indicate the same in the case without PBCs.

Figure 31 -Figure 32 -

 3132 Figure 31 -Couplings obtained after training with SVM with positive weights constraint for L = 1 (a) and (b) and L = 2 (c) and (d) maps. Dependence of the average coupling with the distance between the corresponding neurons; the vertical line locates the radius r c of the place fields. Averages were computed over 500 samples of the p positions per map at fixed PF centers; N = 1000 neurons. Histograms of the couplings, for sizes N = 100 (blue) and N = 1000 (orange) Parameters: D = 2, φ 0 = .3 and p × L = N. As in Section 4.5, the training set consist of p × L binary patterns {σ µ i } constructed by drawing randomly p positions in each of the L environments, defined in Section 4.2, so

. 32 )

 32 In the system above Γab (k) = γa (k) γb (k) with a, b taking value 1 or 2 and γa (k) = H D dr γ a (r) e -i2πk•r (6.33)

. 34 )

 34 It is straightforward to write the resulting self-consistent equation for the MERM resolvent by simply changing Γ (k) → K a=1 ρ a ( γaa (k)) 2 in Eq.(5.21). In Fig.33(e) we show the perfect agreement of this theoretical result with numerical simulations.

Figure 33 -

 33 Figure 33 -Top panels: sketches of the model variants, respectively (a) dilution, (b) place fields of different sizes and (c) multiple place fields per cell in a given map. Bottom panels: (d) Density of eigenvalues for the MERM for φ 0 = .2 with different dilution fractions ρ 0 . (e) Density of eigenvalues of MERM with different fractions ρ φ 0 =.2 of PF with volume φ 0 = .2 and ρ φ 0 =.4 of PF with volume φ 0 = .4 in each space. (f) Density of eigenvalues of MERM for φ 0 = .2 with c = 2 PF for each index i in each map. Parameters: N = 2500, D = 1, α = 1. In all cases we do not show the extensive eigenvalue.

Figure 34 -

 34 Figure 34 -Distribution of local stabilities after the learning of a map with SVMs. (a) Homogeneous case: the p positions of the data-set are drawn randomly. (b) Heterogeneous case: here the data-set has 150 positions on the diagonal of the maps and the other 150 positions are drawn at random. Here, D = 2, φ 0 = .3, N = 1000, p = 300 and L = 1. We show the contour map made from 2500 realization of random positions, for which we evaluate the stabilities of the corresponding patterns. The overall network stabilities (minimal pattern stabilities) in the homogeneous and heterogeneous cases are, respectively, κ .44 and κ .53 for the samples considered here.

Figure 35 -

 35 Figure 35 -Comparison of couplings obtained with SVM and (c) and with the Quenched PF Theory (b) and (d). Top: Dependence of couplings on N. Bottom: Dependence of the couplings on distance;The vertical line locates the radius r c of the PF. These results were obtained for D = 2, φ 0 = .3, α = .1, p = 5; we have averaged over 100 different realizations of the p positions at fixed PF centers for the SVM results. Space was divided in 50 bins with values ranging from 0 to √ 2/2 (the maximal distance achievable in unit square with periodic boundary conditions). Couplings were then put in the corresponding bins for all maps, and the averages and standard deviations were plotted as functions of the bin centers. Average couplings and associated standard deviations with quenched PF theory were computed with (23) and (24) of[START_REF] Monasson | Storage of spatially correlated patterns in autoassociative memories[END_REF], with the substitution α c → p α c as the number of patterns is here p × L.

3 Figure 36 -

 336 Figure 36 -Number of rounds required for adatron to reach the same stability value obtained with the standard SVM algorithm vs. the ratio p η at fixed p N . The values of N range between 1000, 1500 and 2000, while the ones of η between .5, 1 and 1.5. The other parameters are set to L = 1, D = 1, φ 0 = .3, Samples = 100.

Figure 37 -

 37 Figure 37 -Examples of the images that we are going to use for the learning phase. (a) Images of a guitar rotated by different angles on a black background, in particular we will have a figure for each angle from 1 to 360 degrees. (b) The same of (a) but for images of a sax.

tFigure 38 -

 38 Figure 38 -Study of the dynamics of a RNN after learning through SVMs the high-level data representations extracted from VGG16 and associated with the different images of guitars rotated by different angles. The blue dots represent the data on which the learning was done projected on the first two principal components extracted from the data with PCA. The photos associated with the red stars correspond to 2 images of the data-set, in particular it seems that close points in this space correspond to guitars rotated almost of the same angle. In addition, the crosses represent the dynamics of the network starting from one of the images of the data-set and in which we have a temperature of the order of the stability of the network. The colors of the crosses go from dark purple to yellow with the passing of time of the dynamics, in particular we have a cross every Monte Carlo sweep, see Section 4.4.

  over , µ because W i does not depend on them.

Figure 39 -

 39 Figure 39 -Number of patterns with positive stabilities (y-axis) vs number of iterations of the quadratic optimization solver (x-axis) for one map (L = 1) with p = 500 points stored by a network with N = 1000 neurons, see Section 4.2 for details about the model and how the patterns were built. Parameter values: D = 2, φ 0 = .3.

Figure 41 -

 41 Figure 41 -Same results for spatial error and capacity presented in Chapter 4 but here for D = 1, (a) and (c), and D = 3, (b) and (d). The other parameters are identical to the ones used for the equivalent figures in D = 2, see Figs. 18(a), 21 and 23(a).

Bwhere

  A P P E N D I X -C H A P T E R 5This short Appendix is intended to remind the reader of the fundamental notions and quantities of random matrix theory 1 needed to understand Chapter 5, i.e., the study of MERM's spectral properties.b.1 resolvent, blue function and R-transformThe eigenvalues Λ i of a N × N Hermitian matrix A, random or not, are real. Their density,-Λ i ) , (B.1)where • stands for the average over the distribution of the matrix A, can be obtained from the (one-point) resolvent Id is the identity matrix of size N.Using the standard relation lim →0 + 1/(Λ + i ) = P 1/Λiπδ(Λ) (P denotes the Cauchy principal value), we can rewrite Eq. (B.2) asg(Λ + i ) = P ∞ -∞ dΛ p(Λ ) Λ -Λ iπp(Λ) , (B.3)so that p(Λ) may be reconstructed from either the imaginary or the real part of g(Λ + i ):p(Λ) = -1 π lim →0 + Im g(Λ + i ) ,

3 .

 3 det(z Id -A) . (B.7)This expression will be used in the replica-based computation presented in Section 5.Another interesting expression for g(z) is a decomposition in terms of the moments of p(Λ),Λ m = ∞ -∞ dΛ p(Λ)Λ m = 1 N Trace A m . (B.8)For Hermitian matrices, g(z) is an holomorphic function of z ∈ C except for some cuts along the real axis where eigenvalues of A are concentrated. Therefore, we can reconstruct g(z) for all z by analytic continuation of its series expansion general, converges only in the vicinity of |z| → ∞.Other important objects for us are the functional inverse of g(z), also called the Blue function2 , and the R-transform:B(z) ≡ g -1 (z) , (B.10) R(z) ≡ B(z) -1 z . (B.11)Both of them are fundamental objects of the free random variable theory, see Section B.2.

2 .

 2 z/c),B c A (z) = cB A (c z) , (B.12) R c A (z) = cR A (c z) ,where c ∈ C * . So named because it is defined as the functional inverse of the resolvent, the Green function.B.2 free probability theory in a nutshell 137 b.

  • stands for the average over the distribution of the matrix (C.7). With this representation the determinant det M (L) -z Id -1

1 √N sin 2π k • r i , 1 √N 2 .

 112 [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF] where goes from 1 to L, and the sum over k discards the k = 0 extensive mode as discussed in Chapter 5, Γij (k) = γi (k) γj (k) with γi (k) being the Fourier transform of the indicator function of the place field of area φ µ and the eigenvector components, v ki cos 2π k • r i , are real due to the symmetry Γ (k) = Γ (-k). (C.[START_REF] Amir | Localization, anomalous diffusion, and slow relaxations: A random distance matrix approach[END_REF] We now use the Stratonovich trick to linearize( i v ki φ a i γi (k) γi (k) .Using the fact that v ki = 0 and v ki v k j = 1 N δ ij δ kk it is easy to perform the average in the above equation, with the result expγµ (k) 2 (C.[START_REF] Daniel J Amit | The interaction space of neural networks with sign-constrained synapses[END_REF]

2 Wj,k 2 W

 22 exactly what seen in Section 4.6 until the definition of the order parameters that are nowm a = j ja 2 Γ 1j |r 1r j | -φ j (C.27)andq ab = ja W kb Γ jk |r jr k | , (C.28)

c. 2 . 2 . 1 4 j,k 2 C 2 j,k 2 C 1 jk φ k - 2 j,k 2 φ j U Id + C - 1 jkΞ 1 : 11 = 1 U + C 11 -j,k 2 C 38 )Ξ 2 :M ρ=1 β ρ φ 2 ρU + N M ρ=1 β ρ φ 2 ρρ 1 +k 52 ) 1 D+1 = 4 A 1 8 DΓ

 221422212211111112382221521418 Computation of ΞExpanding the terms in Ξ(U) in Eq. (C.34) above, we write Ξ(U) = Ξ 1 (U) + Ξ 2 (U) + Ξ 3 (U) withΞ 1 (U) = 1j U Id + C 1j U Id + C-Consider the N × N matrix C(N) , with entries C ij for i, j comprised between 1 and N. Let us also define Id(N) the identity matrix in dimension N, while Id above referred to the identity matrix in dimension N -1. Using block-matrix inversion formulas, we write thatU Id (N) + C (N) -1 1j U Id + CThe left hand side of the equation above is equal, in the large-N limit, to the resolvent g(U) of C defined in (C.32). Using C 11 = φ 1 and the definition of Ξ 1 (U), we obtainΞ 1 (U) = 4 U + φ 1 -Let |v + be the normalized vector with N identical components, (v + ) i = 1 √ N . For large N, |v + is the top eigenvector of C, with (extensive) eigenvalue λ + = N M ρ=1 β ρ φ 2 ρ .After the computation of the braket of this vector with the matrix of components φ j U Id + C -1 jk φ k , and taking the large N limit we get:Ξ 2 (U) = N pH(x)g M θ=1 Γθ (k)r θ . (C.49)According to dimensional analysis, the large momentum scaling of the Fourier coefficients is D+1 , ∀ρ , (C.50)where k = |k| and D is the dimension of the physical space. Using (C.50), equation (C.49) can be rewritten asp H(x) = G g p H(x)In addition, using the asymptotic expansion of the erfc function, we have x √ 2π H(x) e x 2 /2 1 -1 x 2 (C.53) for large x. Combining these expressions allows us to obtain the asymptotic relation between x and y, y A 1 (D) D+1 . (C.57)in order to show that the hypothesis of r ρ = O(1) ∀ρ is consistent we notice using (C.50) that (C.48) can be rewritten in the following way:β ρ C r ρ φ C is fixed using (C.[START_REF] Brunel | Optimal information storage and the distribution of synaptic weights: perceptron versus Purkinje cell[END_REF]) so that in the end we get: the hypothesis it's verified. Moreover, once we know (C.59) we can rewrite the asymptotic scaling of the critical capacity asα c (p) ∼ A(D)( ρ φ 1 D -1 ρ β ρ ) D (log p) D (p → ∞) ,(C.60)where we recover trivially the result in Section 4.6 for M = 1. Notice also that in D = 1 the critical capacity does not depend on the distribution of the sizes of the PFs. c.2.3 Spectrum of MERM: multi-populations of neurons (multiple PFs per neuron on a map) Let's consider the following MERM: r i,mr j,m , (C.61)

1 2

 1 • stands for the average over the distribution of the matrix (C.61). With representation the determinant det M (L) -z Id can be expressed as a canonical partition function:Z L (s) = det M (L) -z Id ij φ j , (C.[START_REF] Jose | Learning intrinsic dimension and intrinsic entropy of high-dimensional datasets[END_REF]

v ki φ a i 2 . (C. 67 ) 4

 2674 [START_REF] Hh Dale | Pharmacology and nerve endings[END_REF] where goes from 1 to L, and the sum over k discards the k = 0 extensive mode as discussed in Chapter 5. The eigenvector components, v ki cos 2π k • r i,m , are real due to the symmetry Γ (k) = Γ (-k).We now use the Stratonovich trick to linearize ( i v ki φ that v ki = 0 and v ki v k j = 1 N c i δ ij δ kk it is easy to perform the average in the above equation, with the result exp -Quenched PF theory: multi-populations of neurons (multiple PFs per neuron on a

ΓΓ 2 W ja 2 C 2 W

 222 The computation follows exactly what seen in Section 4.6 until the definition of the order parameters that are now|r 1,mr j,m | -c j φ 0 |r j,mr k,m | . (C.82)Now we make the same approximations as Section 4.6 and so we can write the final expression of the order parameters after adding up all the maps: 1j {r j,m } -c j φ 0 ja W kb C jk {r j,m } .(C.84)

4 j,k 2 CΞ 1 : 11 = 1 U + C 11 -j,k 2 C

 421111112 k =0 α Γ (k)c µ α+ Γ (k) ν q ν c ν , ∀ρ . (C.[START_REF] Gardner | The space of interactions in neural network models[END_REF] From here on we obtain the same equations of what we saw in Section 4.6 except for the quantity Ξ defined asΞ(U) = j,k 2 H j U Id + C -1 jk H k with H j = 2 C 1j -c j φ 0 , (C.88)and Id is the identity matrix. In the above equation, the inverse is intended over the N -1-dimensional restriction of the matrix U Id + C to entries j, k 2.c.2.4.1 Computation of ΞExpanding the terms in Ξ(U) in Eq. (C.88) above, we writeΞ(U) = Ξ 1 (U) + Ξ 2 (U) + Ξ 3 (U) with Ξ 1 (U) = 1j U Id + C Consider the N × N matrix C (N) , with entries C ij for i, j comprised between 1 and N. Let us also define Id(N) the identity matrix in dimension N, while Id above referred to the identity matrix in dimension N -1. Using block-matrix inversion formulas, we write that U Id(N) + C (N) -1 1j U Id + C side of the equation above is equal, in the large-N limit, to the resolvent g(U) of C defined in (C.86). Using C 11 = c 1 φ 0 and the definition of Ξ 1 (U), we obtain Ξ 1 (U) = 4 U + c 1 φ 0 -

Γ (k)c ρ 1 +c ρ r ρ 1 + 106 ) 1 D+1 = 4 A 1 (( 8 DA

 111061418 pH(x)g Γ (k) M θ=1 c θ r θ , ∀ρ . (C.102) Summing all the equations (C.102) we get: pH(x)g Γ (k) M θ=1 c θ r θ . (C.103) According to dimensional analysis, the large momentum scaling of the Fourier coefficients is given by Γ (k) k = |k| and D is the dimension of the physical space. Using (C.104), equation (C.103) can be rewritten as p H(x) = G g p H(x)φIn addition, using the asymptotic expansion of the erfc function, we havex √ 2π H(x) e x 2 /2 1 -1 x 2 (C.107)for large x. Combining these expressions allows us to obtain the asymptotic relation between x and y,y ρ c ρ r ρ ) D (log p) D (p → ∞) , A 1 (D) D+1 . (C.111)Now in order to show that the hypothesis of r ρ = O(1) ∀ρ is consistent we notice using (C.104) that (C.102) can be rewritten in the following way:β ρ C r ρ c ρ , ∀ρ , (C.112)where C is fixed using (C.101) so that in the end we get: it's verified. once we know (C.113) we can rewrite the asymptotic scaling of the critical capacity asα c (p) ∼ D (log p) D (p → ∞) ,(C.114)where we recover trivially the result in Section 4.6 for M = 1 and c 1 = 1.

  5, in particular referring to the salt and pepper distribution introduced in [142] by Lee et al.. More generally, our quenched PF theory (Section 4.6) relies essentially on the activityactivity correlation matrix:

conclusions

-

  x 1 x 2 , (B.13) p x 1 +x 2 = p x 1 * p x 2 , (B.14) r x 1 +x 2 = r x 1 + r x 2 ,(B.15)

As it will become clearer during the thesis, in this context we will always use the terms manifold, map, environment and continuous attractor with the same meaning.

Note that in general a dynamical system can have other types of attractors such as periodic or chaotic attractors[START_REF] Vulpiani | Chaos: from simple models to complex systems[END_REF] that we will not consider in this thesis, in fact we will always be interested, both in the case of discrete and continuous attractors, in fixed points where the dynamics of the system remains stuck in the absence of small noise or external fields (stable or indifferent equilibrium points).

In the following we will use the terms spins, neurons, activity variables and units indiscriminately.

All the following can be trivially formulated with the notation σ i = 0, 1 which we will use afterwards.Here we consider the notation σ i = ±1 because the Hopfield model was historically introduced in this way.

Note then that this is a model with continuous and positive neurons, unlike the standard Hopfield model defined with binary units, see Section 3.2.3.4 storing a single continuous attractor

Regarding the hippocampal region CA3 we have only global remapping. In reality, however, there is also another type of remapping, called "rate remapping", in which a place fields vary in the frequency of spikes of the respective place cells without changing its position in the map, which is present in another region of the hippocampus, that is CA1[START_REF] Andersen | The hippocampus book[END_REF] (which we will not consider in the models presented in this thesis).

From now on we will continue to use only binary neurons, not as in the above mentioned model ofSamsonovich and McNaugthon.

See as well the Battaglia and Treves's model[START_REF] Francesco | Attractor neural networks storing multiple space representations: a model for hippocampal place fields[END_REF] which has several points in common with the one we are describing.

The more realistic case of environments in which we remove periodic boundary conditions is discussed in Section 6.2.

The examples mentioned below can be viewed at: https://journals.aps.org/prl/supplemental/10. 1103/PhysRevLett.124.048302. In the attached videos the red crosses represent the stored positions of the

optimal capacity-resolution trade-off in memories of multiple ca

γ a (r) is the same as Φ(r) in Section

4.5 in the case where all the PFs of the different neurons have the same volume in all the maps.

A technical problem is that the representations extracted from VGG16 have real non negative components while in our procedure we need binary patterns, so we had to binarize the data by hand hoping not to lose too much information with this operation.

For a detailed analysis see for example[START_REF] Livan | Introduction to random matrices: theory and practice[END_REF][START_REF] Potters | A First Course in Random Matrix Theory[END_REF].

towards greater biological plausibility

As for the term χ this can be rewritten in the RS ansatz and using a Gaussian integral trick as:

×e -1 2 (1-q) µ,ν tµ Γ µ,ν ( R) tν e i µ tµ (z µ √ q+t µ -(mφ 0 +θ)σ(r 1 -r µ )))

n .

After performing the Gaussian integral in { tµ }, taking the small n limit and the q → 1 limit we get:

Putting all together we obtain:

where M is the average activity of the binary pattern in ±1 notation, i.e., under the change of variable σ(r 1 ) = {0, 1} → ξ = 2σ(r 1 ) -1 = {-1, +1}.

The convex optimization problem to be solved in (4.38) thus amounts to compute

2 , (A. [START_REF] Daniel J Amit | The interaction space of neural networks with sign-constrained synapses[END_REF] where v = m φ 0 and the Gaussian variable z 1 in (4.38) has zero mean and variance Γ( R) 1,1 .

The minimum over t 1 in (A. [START_REF] Daniel J Amit | The interaction space of neural networks with sign-constrained synapses[END_REF]) can easily be determined, with the result

As r1 is drawn uniformly at random, ξ is a random binary variable:

-1 with probability 1 -M 2 .

(A.17)

We get, with the normalized Gaussian variable z = z 1 × 2/ √ 1 -M 2 and the measure Dz = dz/ √ 2π exp(-z 2 /2), 1 α c (v; κ, p = 1) = (A.18)

where v is chosen in order to maximize α c (v; κ, p = 1):

These equations coincide with the results of [START_REF] Gardner | The space of interactions in neural network models[END_REF] up to the change κ → 2κ due to the fact that the neuron activities take here values 0,1 and not ±1.

A.4 computation of Ξ(U)

appendix-chapter 4

Computation of Ξ 3 : As C jk with j, k 2 does not depend on the locations r 1 of the place fields associated to neuron i = 1 in the different maps , we may substitute C 1j in Eq. (A.22) with its average over those positions, equal to φ 2 0 . We obtain

in the large-N limit, see the calculation of Ξ 2 (U).

Expression of Ξ: Gathering the three terms above, we obtain

. (A.28)

a.5 dependence on φ 0 and D.

Here we discuss the dependence of the results found in Section 4.6 on the size of the place fields φ 0 and the dimension of the maps D (as already stated in Section 4.2 the dimension of the place fields is always the same as that of the environments).

In Fig. 40 we show that the value of p such that the results obtained with the quenched PF theory and SVMs match increases as φ 0 decrease.

In fact, an analysis of equations (4.66, 4.67), valid in the small φ 0 limit, indicate that this minimal value of p scales as

and becomes very large as φ 0 becomes small. Realistic values for the place fields area φ 0 are reported in the experimental literature [START_REF] Mizuseki | Activity dynamics and behavioral correlates of CA3 and CA1 hippocampal pyramidal neurons[END_REF] and [START_REF] Syed A Hussaini | Increased size and stability of CA1 and CA3 place fields in HCN1 knockout mice[END_REF] to range between .2 and .3. We stress that place fields are, however, continuous-valued rate fields for real neurons, while, in our model (Section 4.2), they represent binary on/off values. The correspondence between our model and experimental studies relies therefore on the introduction of a cutoff value for the minimal firing rate of place cells; values of φ 0 ranging between .2 and .3 seem to be reasonable in view of [START_REF] Mizuseki | Activity dynamics and behavioral correlates of CA3 and CA1 hippocampal pyramidal neurons[END_REF] and [START_REF] Syed A Hussaini | Increased size and stability of CA1 and CA3 place fields in HCN1 knockout mice[END_REF].

Moreover, Fig. 41 shows the spatial error of trained recurrent neural networks as a function of the number of maps L, the optimal stability κ as a function of the load α = L (dashed-dotted lines) gets closer to SVM (scatter plots) as p increase, the value of p for which quenched PF theory and SVM matches increase as φ 0 decrease. We use for this results D = 2, N = 5000, and we have averaged over 50 different realization of the environments and different realizations of the p positions.

appendix-chapter 5

Note that this last condition is not enough to define asymptotic freeness, since matrices do not commute. For example, from Eq. (B.16), the forth moments read

The free cumulants are defined such that the sum property (B.15) is preserved for the generating function of the free cumulants, the so-called R-transform [START_REF] Jarosz | Random Hermitian versus random non-Hermitian operators-unexpected links[END_REF][START_REF] Tulino | Random matrix theory and wireless communications[END_REF]. Interestingly, the R-transform is simply related to the Blue function (B.10), i.e., the functional inverse of the resolvent g(z), by Eq. (B.11). 3 The R-transform of the sum of two asymptotically free matrices X 1 and X 2 obeys:

Hence, the problem of finding the eigenvalue distribution of the sum of two free random matrices is straightforward. Applying successively Eqs. (B.10), (B.11), and (B.20), one readily infers g X 1 +X 2 from g X 1 and g X 2 . The steps of the algorithm are as follows:

Moreover, the generalization to the sum of an arbitrary number of matrices is trivial.

3. Note that g(z) plays the role of a free characteristic function, see Eqs. (B.8) and (B.9).

C

A P P E N D I X -C H A P T E R 6

In this Chapter we report some details on the numerical, algorithmic and analytical part of the results presented in Chapter 6 in order not to make the latter too heavy to read.

c.1 svm algorithm with sign-constrained synapses

Here we generalize the algorithm of support vector machines presented in Sections 4.3 and A.1 to take into account the constraint on the sign of the synapses. In fact, the basic algorithm does not consider this aspect admitting a priori connections of any sign.

After we have generated a data-set of activity patterns, as in Section 4.2, we want to learn the connections {W ij } of the recurrent neural network that maximize the stability

defined already in Eq. ( 6.3) at fixed α and p. Indeed, as we have previously seen in Sections 4.3 and A.1, this choice of the weights ensure the biggest basins of attraction in the pattern space, i.e., robustness against thermal noise. In order to do that we implement SVM learning [START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Scholkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], but adding this time the positivity constraint on {W ij } and the threshold terms {θ i } (fundamental now that the weights are only positive to make the problem linearly separable).

In practice, for each neuron i, we want to compute the connections {W ij } from the other neurons j (with W ii = 0, ∀ i, no self-connections) and {θ i }, which are solution of the following primal constrained convex optimization problem minimize

We have to solve N such problems to extract all the rows of the coupling matrix and all the threshold terms. The dual form of this problem is maximize

where the {λ µ }'s and {c ij }'s are Lagrange multipliers enforcing respectively the first and the second sets of constraints in defined in Eq. (C.2). This optimization problem can be solved using available numerical routines [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF], as the standard SVM, see Sections 4.3 and A.1.

Once we obtain the {λ µ }'s and {c ij }'s we can compute the connections through

Using the fact that any support vector (data points that lies on the maximal hyperplanes) satisfies

we can determine the value of the threshold parameters {θ i }'s thanks to

where S is the set of support vectors and N S is their number.

We then normalize the rows of the couplings matrix to unity, i.e., j( =i) W 2 ij = 1, and divide by the same number also the thresholds {θ i }'s. Finally, the stability κ pos is computed through the formula (6.3).

We can finally write Z L (s) n as

The Gaussian integrals over φ a and u a k can be easily computed. We then make the Replica Symmetric (RS) Ansatz on the structure of the order parameters q ab µ and their conjugate variables qab µ , so that

and

The integrals over q µ , r µ , qµ and rµ are then estimated using the saddle-point method valid for large N, and then taking the small n limit. The resulting expression for the resolvent s L (z) of (C.7) is

where f({q µ , r µ , qµ , rµ }) is the free energy density equal to

where Γµ (k) = γµ (k) 2 . The saddle-point equations obtained by optimizing f({q µ , r µ , qµ , rµ }) with respect to qµ , rµ , q µ and r µ read ∀µ

This system of equations admits r µ = rµ = 0, ∀µ as a solution, which gives, according to (C.21), the following system of equations satisfied by s L (z):

Note that we are eventually interested in the spectral properties of the matrix C with entries

Obviously, the resolvent s of C is related to the resolvent s L of M (L) through the equation s(z) = α s L (α z). Hence we obtain our fundamental system of equations for the resolvent of C:

Quenched PF theory: multi-populations of neurons

Here we are going to extend the Gaussian theory with quenched PF presented in Section 4.6 to the case of multi-populations of neurons. By multi-populations of neurons we mean that the N neurons are divided in M finite groups with fractions of neurons β ρ with ρ = 1, . . . , M and every group of neurons with a specific property, i.e., a fixed area for the place fields φ ρ < 1.

of Ξ 3 : As C jk with j, k 2 does not depend on the locations r 1 of the place fields associated to neuron i = 1 in the different maps , we may substitute C 1j and C 1k in Eq. (C.37) with their average over those positions, respectively equal to φ 1 φ j and φ 1 φ k . We obtain

in the large-N limit, see calculation of Ξ 2 (U) above.

Expression of Ξ: Gathering the three terms above, we obtain

Large-p behavior of the critical capacity

The rest of the steps are the same presented in Section 4.6, in fact now we can directly write the set of coupled equations for x and the resolvent g:

from which the capacity can be computed as a function of the number p of points,

In practice, we can choose x at will, compute g from (C.45), then p from (C.43) and (C.44), and, finally, α c from (C.46). Now we make the hypothesis that q ρ = g r ρ with r ρ = O(1) ∀ρ so that we can write (C.43) and (C.44) as:

where we have used the fact that the N neurons are divided in M groups and we have an overlap q ab µ per group as

to be fixed through

We can finally write Z L (s) n as

The Gaussian integrals over φ a and u a k can be easily computed. We then make the Replica Symmetric (RS) Ansatz on the structure of the order parameters q ab µ and their conjugate variables qab µ , so that

and

The integrals over q µ , r µ , qµ and rµ are then estimated using the saddle-point method valid for large N, and then taking the small n limit. The resulting expression for the resolvent s L (z) of (C.61) is

where f({q µ , r µ , qµ , rµ }) is the free energy density equal to

The saddle-point equations obtained by optimizing f({q µ , r µ , qµ , rµ }) with respect to qµ , rµ , q µ and r µ read ∀µ

This system of equations admits r µ = rµ = 0, ∀µ as a solution, which gives, according to (C.75), the following system of equations satisfied by s L (z):

Note that we are eventually interested in the spectral properties of the matrix C with entries

Obviously, the resolvent s of C is related to the resolvent s L of M (L) through the equation s(z) = α s L (α z). Hence we obtain our fundamental system of equations for the resolvent of C:

Computation of Ξ 3 : As C jk with j, k 2 does not depend on the locations r 1,m of the place fields associated to neuron i = 1 in the different maps , we may substitute C 1j and C 1k in Eq. (C.91) with their average over those positions, respectively equal to φ 2 0 c 1 c j and φ 2 0 c 1 c k . We obtain

in the large-N limit, see calculation of Ξ 2 (U) above.

Expression of Ξ: Gathering the three terms above, we obtain

Large-p behavior of the critical capacity

The rest of the steps are the same presented in Section 4.6, in fact now we can write the set of equations coupled for x and the resolvent g:

from which the capacity can be computed as a function of the number p of points,

In practice, we can choose x at will, compute g from (C.99), then p from (C.97) and (C.98), and, finally, α c from (C.100).

appendix-chapter 6 margin), but with the big difference that it is an online scheme by the perceptron algorithm itself, more biologically plausible.

Below we report the algorithm, referring to [START_REF] Jk Anlauf | The adatron: an adaptive perceptron algorithm[END_REF][START_REF] Engel | Statistical mechanics of learning[END_REF] for details on the algorithm convergence demonstrations and links with other schemes such as minover or adaline.

Let's consider the case of one of the RNN perceptrons for which we need to find the optimal weights (as usual the different perceptrons are independent so the N problems can be solved separately), see Section 4.3, say the one corresponding to neuron i = 1.

Let's start by writing the net weights in the following way

where x ,µ 1 are called the embedding strengths.

We can choose any value for x ,µ 1 as long as they are non-negative, including the tabula rasa case where they are all null.

Once the problem is initialized we start by presenting to the network one pattern at a time in a sequential way (the algorithm can also be implemented in parallel) and we update the x ,µ 1 according to the following rule δx ,µ 1 = max{-x ,µ 1 , η(1 -∆ ,µ 1 )} (C. [START_REF] Syed A Hussaini | Increased size and stability of CA1 and CA3 place fields in HCN1 knockout mice[END_REF] where the η (learning rate) range must be between 0 and 2 to ensure convergence of the algorithm and ∆ ,µ 1 is defined as usual as ∆ ,µ 1 = (2ξ ,µ 1 -1)

Presenting several times the patterns to the net following this rule we will arrive to convergence when the following conditions (Kuhn-Tucker conditions, see [START_REF] Boyd | Convex optimization[END_REF]) are satisfied, that is either (x ,µ 1 > 0 and ∆ ,µ 1 = 1) or (x ,µ 1 = 0 and ∆ ,µ These conditions are equivalent as those satisfied by the SVM algorithm, in fact once we have reached convergence and normalized to one the weights, we find exactly the same results. In addition, the speed of convergence to the perceptron with maximum stability of this algorithm is exponential.