Filipa França De Barros

Julien Bacqué-Cazenave

Coralie Taillebuis

Gilles Courtand²

Marin Manuel

Hélène Bras

Michele Tagliabue

Denis Combes

François M Lambert

Dr Mathieu Beraneck
email: mathieu.beraneck@u-paris.fr

Conservation of locomotion-induced oculomotor activity through evolution in higher tetrapods

Keywords: Central pattern generators, efference copy, evolution, gaze stabilization, locomotion, mice, motor-to-motor coupling, vestibular, eye movements Highlights

• Adult decerebrated mice exhibit conjugated rhythmic eye movements during treadmill locomotion • Locomotor-induced oculomotor activity occurs in absence of visuo-vestibular inputs • Conserved CPG-based efference copy signal in vertebrates with common features.

eTOC blurb

We report a functional coupling between spinal locomotor and oculomotor networks in the mouse, similar to the one previously described in Amphibians. This is the first evidence for the direct contribution of locomotor networks to gaze control in mammals, suggesting a conservation of the spino-extraocular coupling in higher tetrapods during sustained locomotion.

Summary

Efference copies are neural replicas of motor outputs used to anticipate the sensory consequences of a self-generated motor action or to coordinate neural networks involved in distinct motor behaviors [START_REF] Straka | A New Perspective on Predictive Motor Signaling[END_REF] . An established example of this motor-to-motor coupling is the efference copy of the propulsive motor command that supplements classical visuo-vestibular reflexes to ensure gaze stabilization during amphibian larval locomotion [START_REF] Combes | An intrinsic feed-forward mechanism for vertebrate gaze stabilization[END_REF] . Such feedforward replica from spinal pattern-generating circuits produces a spino-extraocular motor coupled activity that evokes eye movements, spatio-temporally coordinated to tail undulation independently of any sensory signal [START_REF] Lambert | Gaze Stabilization by Efference Copy Signaling without Sensory Feedback during Vertebrate Locomotion[END_REF][START_REF] Bacqué-Cazenave | Temporal Relationship of Ocular and Tail Segmental Movements Underlying Locomotor-Induced Gaze Stabilization During Undulatory Swimming in Larval Xenopus[END_REF] . Exploiting the evolutionary-development characteristic of the frog [START_REF] Straka | A New Perspective on Predictive Motor Signaling[END_REF] , studies in metamorphing Xenopus demonstrated the persistence of this spinoextraocular motor command in adults, and its developmental adaptation to tetrapodal locomotion [START_REF] Uckermann | Spinal Efference Copy Signaling and Gaze Stabilization during Locomotion in Juvenile Xenopus Frogs[END_REF][START_REF] Von Uckermann | Adaptive plasticity of spino-extraocular motor coupling during locomotion in metamorphosing Xenopus laevis[END_REF] . Here, we demonstrate for the first time the existence of a comparable locomotorto-ocular motor coupling in the mouse. In neonates, ex vivo nerve recordings from brainstemspinal cord preparation reveals a spino-extraocular motor coupled activity similar to the one described in Xenopus. In adult mice, trans-synaptic rabies injection in lateral rectus eye muscle labels cervical spinal cord neurons projecting directly to abducens motor neurons.

Finally, treadmill-elicited locomotion in decerebrated preparations [START_REF] Meehan | Decerebrate mouse model for studies of the spinal cord circuits[END_REF] evokes rhythmic eye movements in synchrony with the limb gait pattern. Overall, our data are evidence for the conservation of locomotor-induced eye movements in higher tetrapods. Thus, in mammals as in amphibians, during locomotion CPG-efference copy feedforward signals might interact with sensory feedback to ensure efficient gaze control.

(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.

The copyright holder for this preprint this version posted June 28, 2021. ; https://doi.org/10.1101/2021.06. 26.450039 doi: bioRxiv preprint

Results

Fictive locomotor activity induces ex vivo spino-extraocular motor coupling in neonates in absence of sensory inputs

The ability of locomotor CPG rhythmic activity to elicit spino-extraocular motor coupling was evaluated on isolated ex vivo brainstem-spinal cord preparations of neonatal mice (Figure 1A).

Electrical stimulation of either the first sacral dorsal root (S1Dr, Figure 1A) or the 8th cervical dorsal root (Dr. C8, Figure S1A) evoked episodes of fictive locomotion with the typical coordination pattern between cervical and lumbar ventral roots (Vr) rhythmic activity (Vr; Figure 1A), as previously reported [START_REF] Juvin | Descending respiratory polysynaptic inputs to cervical and thoracic motoneurons diminish during early postnatal maturation in rat spinal cord[END_REF][START_REF] Juvin | Cervicolumbar coordination in mammalian quadrupedal locomotion: role of spinal thoracic circuitry and limb sensory inputs[END_REF] . In this locomotor pattern, paired bursting discharge in homolateral C8 (light green in Figure 1A, B bottom panel) and L2 (light orange in Figure 1A, B bottom panel) Vr was in alternance with homolateral C5 and L5 Vr [START_REF] Juvin | Cervicolumbar coordination in mammalian quadrupedal locomotion: role of spinal thoracic circuitry and limb sensory inputs[END_REF] . In most cases (66%, Figure 1C) this rhythmic cervico-lumbar coordinated locomotor activity was coupled to a bursting activity of the abducens motor nerves (Figure 1A-B) within the same frequency range (0.5-1.5Hz; Figure1D, E) and a latency of ~62ms to C8 Vr burst and ~80ms to L2 Vr burst, respectively (Figure 1F), compatible with a monosynaptic delay [START_REF] Kasumacic | Segmental Organization of Vestibulospinal Inputs to Spinal Interneurons Mediating Crossed Activation of Thoracolumbar Motoneurons in the Neonatal Mouse[END_REF][START_REF] Lambert | Early postnatal maturation in vestibulospinal pathways involved in neck and forelimb motor control: Vestibulospinal Projections to Cervical Spinal Cord in Neonatal Mouse[END_REF] . Inversely, noncoordinated spinal activity failed to evoke a coupled rhythmic discharge in abducens nerves (Figure 1C), which instead displayed a tonic, non-rhythmic activation. Locomotor-related bursts in the bilateral abducens occurred in alternation (Figure 1B, top panel) and were in synchrony with the ipsilateral paired C8-L2 Vr rhythmic discharge, respectively (Figure 1G).

Frequency analysis of abducens discharge in response to S1Dr stimulation (Figure S1C-E) demonstrated that the spino-extraocular coupling was not induced by proprioceptive feedback inputs. In addition, coordinated fictive locomotor episodes evoked by NMDA/5HT spinal application also produced a well correlated rhythmic activity in the abducens motor nerves (Figure 1H -I) in a lower frequency range (0.15-0.25Hz; Figure 1J) and with a preserved phase relationship relative to the Vr burst discharge (Figure 1K). Since all these brainstem-spinal preparations lacked visuo-vestibular sensory inputs as well as cortical, mesencephalic and cerebellar inputs (see methods), this spino-extraocular motor coupling was produced by a locomotor efference copy signal, originating from the spinal motor CPGs.

Spino-extraocular motor coupling depends on the integrity of the cervical, and not lumbar, spinal cord

To determine the respective roles of the cervical and lumbar CPG networks in the origin of locomotor-induced spino-extraocular motor coupling, cervical and lumbar regions were selectively isolated in brainstem-spinal cord preparations with spinal split-baths (Figure 2A).

When the cervical cord was perfused with a calcium-free aCSF (Figure 2A, left) electrical stimulation of the S1Dr evoked typical alternated fictive locomotor activity in the lumbar Vr (Figure 2B bottom traces, Figure 2D upper panel). However, it failed to evoke correlated fictive locomotor activity in the cervical Vr or rhythmic discharge in the abducens motor nerves (Figure 2B upper traces, Figure 2D, lower panel). This experiment demonstrates the instrumental role of cervical motor networks activity in the genesis of the spino-extraocular motor coupling. Next, we tested whether the cervical cord acted as a neuronal relay or was the origin of the spino-extraocular motor coupling. Locomotor-induced rhythmic discharge of abducens motor nerves was recorded in response to electrical stimulation of either S1Dr or C8Dr, in isolated ex vivo preparations exposed to a mid-thoracic section (Figure 2E). After the thoracic section, S1Dr stimulation failed to produce any spino-extraocular motor coupling (Figure 2F) whereas C8Dr electrical stimulation evoked locomotor-induced rhythmic discharge of the abducens motor nerves (Figure 2G). Even though the cervical CPG-related abducens rhythmic activity was recorded at relatively low frequency (Figure 2H, 0,2-0,35Hz, orange circles), it remained in synchrony with C8Vr bursting discharge (Figure 2I and2J). Overall, these results show that, while the activation of the lumbar CPG allowed for a more robust coupling, the locomotor-related activity in abducens motor nerves originated specifically from the cervical CPG.

Last-order pre-motor neurons in cervical cord project to abducens motoneurons

To determine the cervical neuronal relay implicated in the spino-extraocular coupling (Figures 1 and2), transsynaptic retrograde tracing of last-order premotor neuronal populations was performed by rabies virus (RV) injection in the lateral rectus muscle (LRM) of adult mice (Figure 3A). The monosynaptic labeling time window (55h after RV infection of motoneurons innervating LRM, Figure 3A and 3B) revealed several populations of last-order premotor neurons connected directly to RV + abducens motoneurons (visualized by ChAT immunolabeling, Figure 3B). As expected from vestibulo-ocular reflex circuitry, RV + neurons were found in the bilateral medial vestibular nuclei (MVN, VOR neurons, Figure 3C). Moreover, RV + neurons were observed bilaterally throughout the cervical spinal cord (Figure 3D-F) in all preparations (n=3; Figure 3G-I). Average RV infection ratio was of 1 abducens motoneuron contacted by 6 cervical neurons (Figure 3J). RV + cervical neurons were mainly located in the spinal ventral horn or around the central canal (Figure 3K). Notably, at this monosynaptic infection time, no RV + neurons were observed caudally from the C8 segment. An extended infection time (70h) revealed spinal neuronal populations disynaptically connected with abducens motoneurons, scattered throughout the gray matter as well as beyond the cervical region (Figure S3B). This result suggests that inputs from spinal motor networks and sensory spinal circuitry converge at least on the direct abducens spino-extraocular pathway (Figure S3A). Overall, the use of RV trans-synaptic injections revealed a spinal-brainstem pathway likely to support the functional coupling of abducens motor nerve discharge with the rhythmic activity evoked by the cervical-CPG during locomotion in mice.

Coupling between forelimb and eye movements during sustained trot-like locomotion

To determine if locomotor-induced eye movements occur in the absence of visual and vestibular sensory signals, eye movements were recorded in the dark during head-fixed treadmill locomotion in decerebrated adult mice (Figure S4A). Such preparations enable the study of the spinal cord output and sensorimotor integration in the absence of descending supra-spinal motor commands [START_REF] Meehan | Decerebrate mouse model for studies of the spinal cord circuits[END_REF] . Binocular nystagmic-like eye movements were observed during treadmill-evoked bouts of 10-40s sustained locomotion (Figure 4A, expanded section in Figure 4B). This locomotor-induced oculomotor behavior consisted in an alternation of short ocular movement (quick-phase like) in one direction, followed by longer eye movements (slow phase-like) of comparable amplitude in the opposite direction (Figure 4A-C). Eye movements were mostly restricted to the horizontal plane (compare vertical, in light colors, and horizontal components, in dark colors, in Figure 4B and4C) and their orientation remained constant (Figure 4D) throughout the recordings. Left and right eye movements were always conjugated, with similar directionality for both eyes, and appeared well coordinated (see quick and slow eye movement phases on mean cycles of panel C; horizontal binocular coordination on oculogram in Figure 4E).

To determine the temporal relationship between locomotor and ocular movements, the occurrence of both limbs and eye events were compared. The frequency of the oculomotor cycles increased with the frequency of the locomotor cycles (Figure 4F). Moreover, eye and forelimb movements appeared in most instances synchronized (Figure 4G). All recordings shared two main features: first, consistent eye movements only occurred once a trot-like locomotion frequency (2-4Hz ) [START_REF] Lemieux | Speed-Dependent Modulation of the Locomotor Behavior in Adult Mice Reveals Attractor and Transitional Gaits[END_REF] was attained (Figure 4A and4H). Second, the quantification of the event occurrence comparing eye, forelimb and hindlimb movements showed a match between eye and limb movements in about 80% of the cycles (Figure 4I Figure 4K; phase in right panel). In all recordings, eye movements consisted in nystagmic-like movements composed of alternated short (0,066±0,022) and long (0,223±0,095) duration phases (Figure 4L, left panel) lasting 20% and 80% of the oculomotor cycles, respectively (Figure 4L, right panel). Both components had similar amplitudes in range 1-8° (Figure 4M, quick phases -black-, slow phases -gray-, and Figure S4E), leading to different velocities (amplitude/velocity relation on Panel M; slopes of 20.27 and 2.09 for short and long eye movements, respectively; Figure S4F). Overall, eye movements were observed during >80% of trot-like locomotor cycles and closely matched forelimb movements. Locomotor-induced synchronized binocular movements consisted in the alternation of short-duration rapid quick phase-like movements followed by longer duration slow phase-like eye movements in the opposite direction. The results obtained in decerebrated preparations thus confirm the presence of a spino-extraocular motor coupling based on an efference copy from the locomotor cervical CPG described in newborn mice (Figure 1 and2). Additionally, they demonstrate that the persistence of this functional short-latency pathway in adults (Figure 3) ensures a tight coupling between forelimb locomotor movements and eye movements during sustained trot-like locomotion (Figure 4).

Discussion

Conservation of locomotor-induced spino-extraocular motor coupling through vertebrate evolution

Locomotor-induced spino-extraocular motor coupling was initially described in Xenopus frog 2- 6 . During swimming, both larvae and adult frogs exhibited a CPG efference copy-based extraocular bursting activity tightly coupled to the spinal motor rhythmic discharge. This spinoextraocular motor coordination adapted through metamorphosis to produce appropriate horizontal eye movements that counteracted either tail (larva) or limb (adults) movements during strong propulsive swimming [START_REF] Bacqué-Cazenave | Temporal Relationship of Ocular and Tail Segmental Movements Underlying Locomotor-Induced Gaze Stabilization During Undulatory Swimming in Larval Xenopus[END_REF][START_REF] Uckermann | Spinal Efference Copy Signaling and Gaze Stabilization during Locomotion in Juvenile Xenopus Frogs[END_REF][START_REF] Von Uckermann | Adaptive plasticity of spino-extraocular motor coupling during locomotion in metamorphosing Xenopus laevis[END_REF] . The efference copy mechanism was supported by an ascending spinal-brainstem neural pathway connecting spinal neurons to extraocular motoneurons, crossed in larvae and bilateral in adult xenopus [START_REF] Lambert | Gaze Stabilization by Efference Copy Signaling without Sensory Feedback during Vertebrate Locomotion[END_REF][START_REF] Uckermann | Spinal Efference Copy Signaling and Gaze Stabilization during Locomotion in Juvenile Xenopus Frogs[END_REF] . Using newborn and adult mice preparations, the present study reports a locomotor-induced spino-extraocular motor functional coupling that exhibits common features with frogs. First, in both species this coupling is supported by a direct monosynaptic pathway that persists during mouse ontogeny, as previously reported in xenopus. Second, the bilateral distribution of abducens-projecting cervical neurons revealed in mice by RV tracing suggests a comparable organization of the spinal-brainstem pathway between adult frog and mouse, putatively common to tetrapod locomotor systems. Third, the locomotor-induced ocular signal, recorded either ex vivo or in decerebrated animals, was only observed during sustained and vigorous locomotor activity, corresponding to running behavior, a terrestrial equivalent for propulsive undulatory swimming. Ergo, our findings, combined with previous studies in Xenopus, strongly support the hypothesis that the CPG efference copy-based ocular signal is a conserved mechanism in amphibians and mammals, and putatively in all vertebrates. This conserved locomotor-induced ocular behavior is in accordance with the absolute necessity to stabilize the visual field during motion, by performing rapid counteracting eye adjustments in all vertebrates, including humans [START_REF] Land | Eye movements in man and other animals[END_REF] . There is considerable behavioral evidence for the contribution of locomotor-related spinal signals during gaze stabilization in several mammalian species. In cat [START_REF] Rivers | Gaze shifts and fixations dominate gaze behavior of walking cats[END_REF][START_REF] Zubair | Gaze coordination with strides during walking in the cat[END_REF] and in humans [START_REF] Dietrich | Strategies for Gaze Stabilization Critically Depend on Locomotor Speed[END_REF] , locomotor signals are integrated to favor vertical eye movements by projecting gaze towards the direction of locomotion and canceling VOR during walking or running on a treadmill. In monkeys performing circular running in the dark, compensatory horizontal eye movements are favored by non-vestibular velocity signals [START_REF] Solomon | Stabilization of gaze during circular locomotion in darkness. II. Contribution of velocity storage to compensatory eye and head nystagmus in the running monkey[END_REF] . Here, we report new evidence demonstrating the existence of the spino-extraocular motor command in mice that could constitute a neural mechanism sustaining the coupling of locomotion and eye movements, as suspected of other mammalian species.

Adaptation of the CPG efference copy-based mechanism through the vertebrate lineages

Gaze control largely results from the sensory-motor transformations which primarily depend on the integration of visual and vestibular inputs (optokinetic and vestibulo-ocular reflex) [START_REF] França De Barros | Long term visuo-vestibular mismatch in freely behaving mice differentially affects gaze stabilizing reflexes[END_REF] .

Interestingly, no locomotor-induced ocular movements were observed in head-fixed alert mice during sustained treadmill locomotion in the dark (Figure S4), suggesting a sensory-dependent supraspinal control of the CPG efference copy signal in natural conditions, which is suppressed in decerebrated mice. Recent studies in mice reported that the vestibulo-ocular reflex accounts for a dominant part of the conjugated gaze stabilization observed during headfree locomotion [START_REF] Meyer | A Head-Mounted Camera System Integrates Detailed Behavioral Monitoring with Multichannel Electrophysiology in Freely Moving Mice[END_REF][START_REF] Payne | Magnetic eye tracking in mice[END_REF][START_REF] Michaiel | Dynamics of gaze control during prey capture in freely moving mice[END_REF] . VOR-based eye movements were found to reach higher dynamics, compensating naturalistic head movements interspersed with non-conjugated (vergence) and shifting movements relocating gaze during directed head turn [START_REF] Michaiel | Dynamics of gaze control during prey capture in freely moving mice[END_REF][START_REF] Meyer | Two Distinct Types of Eye-Head Coupling in Freely Moving Mice[END_REF] . The spino-extraocular coupling observed in our study consisted in a symmetrical and conjugated ocular pattern. This suggests that the locomotor efference copy signal might contribute to a set of conjugated, eye movements. Whether this mechanism is participating in compensatory and/or anticipatory eye movements during quadrupedal running behaviour remains to be determined. Even though the spino-extraocular motor command is the predominant way to stabilize gaze during swimming in frogs, this mechanism might complement the visuo-vestibular based reflexes during high dynamic locomotor patterns in mice. All in all, the evidence suggests an adaptation of the CPG efference copy-based mechanism through the vertebrate lineages in relation with changes in sensory capacities and locomotor repertoire during evolution. (I) Polar plots of the phase relationships between the abducens and C8Vr before (control, left) abd C8 ipsi before (µ= 345,161º±2,546; r= 0,974) abd C8 contra before (µ= 162,803±4,285; r= 0,943) and after mid-thoracic lesion (after cut, right) abd C8 ipsi after cut (µ= 353,912±2,463; r= 0,999), abd C8 contra after cut (µ= 177,873±4,909; r= 0,939)(n=3 mice).

The phase relationship is conserved after the cut.

(J) Mean absolute latency time between the activity in the C8 Vr and the abducens nerve in control (ctrl) (52,96±4,51ms) and after cut (66,49±5,96ms). There was no significant difference in the delay observed before and after the mid-thoracic cut (t-test, p=0,1038, n=3).

STAR Methods

Key Resources

Ex vivo brainstem-spinal cord preparations

The dissection protocol followed as previously detailed by Kasumacic [START_REF] Kasumacic | Segmental Organization of Vestibulospinal Inputs to Spinal Interneurons Mediating Crossed Activation of Thoracolumbar Motoneurons in the Neonatal Mouse[END_REF] . Briefly, mice were deeply anesthetized by 4% isoflurane inhalation until the loss of the nociceptive reflexes. After being placed in Sylgard-coated Petri dish, they were decerebrated and eviscerated while submerged in an ice-cold (1º-5ºC) oxygenated (95% O2 and 5% CO2) artificial cerebrospinal fluid (aCSF) (in mM: 128 NaCl, 4.5 KCl, 2.5 CaCl2.2H2O, 1.0 MgSO4.7H2O, 1.2 NaH2PO4.H2O, 5 Hepes, 25 NaHCO3 and 11 D-glucose). In a dorsal view, a craniotomy was performed to expose and remove the cerebellum, and the abducens nerves were carefully cut so that they could be recorded. With a ventral approach, a full laminectomy was done to expose the spinal cord and all the dorsal and ventral roots of both cervical sides were cut so that the final brainstem-spinal cord preparation could be extracted. After removing the meninges, the preparation was fixed with the ventral side up and fixed to the Sylgard-coated dish with insect pins and bathed in oxygenated aCSF fluid at room temperature.

Fictive locomotion protocols and recordings

Isolated brainstem-spinal cord preparations had the distal ends of abducens nerves as well as bilateral spinal dorsal motor roots (sacral, lumbar or cervical) simultaneously recorded For electrical stimulation, the electrodes suctioning the dorsal roots were connected to a pulse generator; this allowed the delivery of pulse trains (4Hz for 10s, inter-stimulus interval of 0.25s and at least 5 minutes between pulse trains). Signals were amplified (x10000) by differential AC amplifiers, digitized at 10 KHz (CED 1401; Cambridge Electronic Design, Cambridge, UK), displayed and stored on computer with acquisition software (Spike 2, Cambridge Electronic Design) and finally analysed off-line with customized scripts. The described pulse trains were delivered at the lumbar or at the cervical levels while the abducens nerves were recorded.

To obtain fictive locomotion through pharmacological stimulation, the ex vivo brainstem-spinal cord preparations (n= 2 mice) were bathed in an aCSF infused with glutamatergic receptor agonists; namely NMDA (7,5 µM) and serotonin (5-HT; 15 µM). The recordings were performed as described above.

In a subset of experiments (n=6 mice), the motor coupling was disrupted using a split-bath configuration that allowed reversibly isolating the brainstem from the spinal cord. Isolation was obtained by partitioning the recording chamber with two custom-made plastic walls that traversed the preparation cervical and lumbar regions with spinal split-baths. The cervical spine was immersed in an aCSF 0 Ca2+ solution, in order to completely block transmission at that level, while lumbar root stimulation was performed. To isolate the cervical from the rest of the preparation, a section was made at the thoracic spinal cord and the cervical was then stimulated while the abducens nerves were recorded.

Methods for tracing experiments

Rabies virus (RV) manipulation was done by vaccinated experimenters in a Biosafety level 2 facility. The virus (Challenge Virus Strain, CVS) was produced and concentrated as previously described [START_REF] Gaudin | Rabies virus glycoprotein is a trimer[END_REF] . Aliquots of the virus were stocked at -80ºC and thawed before use. Mice were anesthetized by an i.p. injection of ketamine (Imalgene, 60 mg/kg) and xylazine (Rompun, 10 mg/kg) and deep anesthesia was confirmed by lack of response to interdigital pinching. The left extraocular muscles were exposed through an incision on the skin covering the mouse's eye, surrounding fatty tissue was removed and absorbable hemostatic gelatin sponges were used to curtail bleeding (Spongostan, Ethicon). After its identification, the lateral rectus muscle (LR) was isolated using small hooks and an injector cannula (gauge 33), linked to a 10µL

Hamilton syringe, was inserted. 1µL of the RV was then slowly injected and the needle was left in place for an extra minute to prevent leaking [START_REF] Bras | Prenatal activation of 5-HT 2A receptor induces expression of 5-HT 1B receptor in phrenic motoneurons and alters the organization of their premotor network in newborn mice[END_REF][START_REF] Coulon | Characterization of last-order premotor interneurons by transneuronal tracing with rabies virus in the neonatal mouse spinal cord[END_REF] . Finally, the skin was closed using non-absorbable monofilament suture and mice were put back on their cages under a red light for post-surgery care. Labelling times were narrowed down to obtain RV-infected spinal cord neurons with monosynaptic connections to the abducens nucleus. A 70h labelling window displayed RV+ neurons in the cerebellum (Purkinje cells; Figure S3C), confirming disynaptic connections to that delay. When the inoculation time was lowered to 55h the RV+ neurons were restricted to the brainstem and cervical spinal cord, with no infection on the cerebellum or any adjacent areas.

Immunochemistry

After 55 or 67hh of RV infection, animals were deeply anesthetized with (Imalgene, 60 mg/kg) and xylazine (Rompun, 10 mg/kg) and perfused intracardially with 0.1M phosphate buffer saline (PBS) followed by 4% paraformaldehyde (PFA). Immediately after, the brain was dissected coronally from the midbrain to the medulla oblongata and the spinal cord was removed in a single block. Tissue fixation was let for 4h at 4˚C in 4% PFA, subsequently cryoprotected in 20% sucrose during at least 24h and freeze embedded at -80ºC in Tissue Tek (Sakura, Zoeterwoude, The Netherlands). Transverse sections (30 µm thick) of the brain and cervical, thoracic and lumbar spinal cord were serially cut in cryostat (Microm, Heidelberg, Germany) and collected onto poly-L-lysine-coated slides. These sections were first incubated to avoid nonspecific binding for 1h at room temperature in blocking solution (PBS, Triton X-100, 0,2% bovine-serum albumin and normal donkey serum) and then left at 4˚C for 48h with the anti-RV phosphoprotein [START_REF] Coulon | Characterization of last-order premotor interneurons by transneuronal tracing with rabies virus in the neonatal mouse spinal cord[END_REF] and anti-choline acetyltransferase (ChAT) diluted 1:100 in the blocking solution. After being rinsed thrice in PBS, sections were incubated at room temperature with the secondary antibodies Alexa546 goat anti-mouse (Thermo Fisher Scientific) and Alexa488 donkey anti-goat (Thermo Fisher Scientific) at a 1:200 dilution for 2h.

Finally, sections were rinsed two times with PBS and once with tap water before being mounted (Immu-Mount, Fisher Scientific) under coverslips.

To analyze the distribution of RV+ neurons, the obtained sections were visualized under an upright light microscope (Axioscope, Carl Zeiss, Germany). Images of each section were taken using a CCD camera (AxioCam MR3, Carl Zeiss). These images were afterwards treated in ZEN (Carl Zeiss) imaging software where brainstem RV + /ChAT + motoneurons and spinal cord RV + neurons were counted on each section and plotted from one out of two (60µm apart).

Only neurons with visibly infected nuclei, determined by the presence of Negri bodies, were considered.

The success of the viral infection was verified by the presence of double-positive RV+ and ChAT+ neurons on the abducens nucleus (Figure 3B). Abducens motoneurons simultaneously RV+ and ChAT+ were found in all mice (n=3), ipsilateral to the injection site (left lateral rectus muscle), while motoneurons innervating other eye muscles (Figure 3B) were never marked, confirming the specificity of the infection.

Procedure for decerebration

Procedures for decerebration were previously described by Meehan and colleagues [START_REF] Meehan | Decerebrate mouse model for studies of the spinal cord circuits[END_REF] . Briefly, mice were deeply anesthetized with isoflurane (2-3% in 100% O2) in an induction chamber, and then transferred to a heating pad on the surgery table with a nose cone to maintain anesthesia (1.5-2% in 100% O2). A rectal thermometer (Physiosuite, Kent Scientific) was used to monitor body temperature throughout the surgery as well as a pulse oximeter (MouseSTAT, Kent Scientific) to monitor O2 saturation. Mice were then artificially ventilated via tracheostomy by blunt dissection; first the sternohyoid muscle was exposed, then the upper half of the fibrous membrane between two cartilages was cut and finally, the air delivery mode was quickly switched to ventilator mode (SomnoSuite, Kent Scientific) and a plastic endotracheal tube was inserted into the trachea. After confirming that the pCO2 levels were acceptable (>2.5%), the isoflurane was lowered to 1-1,2%. To avoid excessive bleeding during the decerebration, the left and right carotid arteries were bluntly isolated, ligatured and cauterized using an electric cauterizer (Change-a-tip cauteries, Bovie). The initial midline incision was sutured and, to avoid dehydration, 0.3 ml of sterile lactated Ringer's solution (Braun Medical) were injected subcutaneously. The mouse was placed in a stereotaxic frame and a superficial cut was made across the skin above the midline of the skull. Using a microdrill (Foredom, David Kopf Instruments), the skull was scored and the parietal bones removed.

Decerebration was performed at the confluence of the sagittal and transverse sinus, perpendicular to the midline and at a 40±5º angle (Figure S4A). A surgical scalpel (no.10 blade; Fine Science Tools) was used to perform a swift and gentle slicing motion. The structures rostral to the incision were fully removed to obtain a premammillary and precolliculary decerebration. Finally, a subcutaneous injection of Ringer's lactate (0.3 ml) was applied, and the isoflurane gas anesthesia was decreased to zero. Complete anesthesia withdrawal and muscular tonus occurred 10-20 minutes after isoflurane removal; PCO2 levels rose consequently.

Videoculography and locomotion recordings and analysis

After the decerebration, mice were tightly secured using ear bars and a custom-made mouthpiece onto the motorized treadmill (Panlab, Harvard Apparatus). All recordings were performed in complete darkness. The locomotion was recorded using an infrared video camera (100Hz; Grasshopper3, FLIR Systems, Inc) fixed perpendicular to the left side of the body. To perform binocular video-oculography, the eyes of the mouse were illuminated with infrared emitters attached to 2 CCD cameras (120Hz; ISCAN, Burlington, MA) that were placed symmetrically on each side of the treadmill. Special care was provided to prevent eye dryness by regularly re-applying ophthalmic gel, to ensure that at least one eye could be tracked online. Eye video signals were processed online (ETL-200, ISCAN, Burlington, MA), sampled at 1kHz (CED 1401; Cambridge Electronic Design, Cambridge, UK) and recorded with Spike2 software. Online tracking of the eye movements with set-up fixed cameras allowed verifying the absence of head movements during the recordings. The video-oculography and locomotion cameras were synchronized using trigger signals generated using spike2 software.

Locomotor cycles were tracked from the acquired videos using a markerless pose estimation (DeepLabCut) [START_REF] Mathis | DeepLabCut: markerless pose estimation of user-defined body parts with[END_REF] . Eye movements and locomotion were analyzed offline using DataView software (W.J. Heitler, University of St.Andrews Scotland).

Overall, eye movements coupled with locomotion were observed on n=26 preparations.

Because the spino-extraocular coupling critically depends on the locomotion performed (see results), only preparations which exhibited bouts of sustained (>30s) locomotion at frequency >1Hz were retained (n=9, including 5 with binocular recordings) for final reporting.

Quantification and statistical analysis

A table featuring all the numerical values obtained from the statistical tests performed is given in the Supplementary information. Prism (GraphPad) was used for statistical analysis.

For paired t-tests normality was first evaluated using the D'Agostino & Pearson normality test and Shapiro-Wilk in the case of smaller unpaired two-tailed samples. When testing locomotion frequency and latency as well as the integrated electrophysiological signals, differences between two results were obtained using the unpaired two-tailed Mann-Whitney U-test and Kolmogorov-Smirnov test to compare distributions. To compare several values, the nonparametric Kruskal-Wallis test was processed with a Dunn's multiple comparisons test. To evaluate the correlations between the frequencies of discharge of the locomotor nerves and the extraocular nerves, regression tests (R) and linear regression (r 2 ) were performed.

Regression results were expressed as (R, r²). Circular data was analysed with Oriana 4.02 (Kovach Computing Services, UK) and the phase and strength of coupling were indicated by their mean vector (µ) and its length (r), respectively for non-uniform distributions values (tested with the Rayleigh's uniformity test, p).

The results are expressed as the mean ± SEM and p values threshold determined as * p < 0.05; * * p < 0.01; * * * p < 0.001; * * * * p < 0.0001; ns: non-significant.

França de Barros et al. (which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.

França de Barros et al.

Cerebellum

Floc/PF

  , left histogram), and an eye mismatch (limb movement, no eye movement; Figure 4I, middle histogram) in about 20% of the cycles. Notably, locomotor-induced eye movements never occurred in the absence of forelimb movements (Figure 4I, right histogram). The frequency of the eye movement was closely correlated with the frequency of the forelimb movement (Figure 4J slope = 0.7882; r²=0.43). Left and right eye movements were tightly correlated and synchronized (r 2 =0,98;

Figure 1 . 1 (

 11 Figure 1. Fictive locomotor activity evokes spino-extraocular motor coupling in neonatal brainstem-spinal cord isolated preparations. (A) Schematic of the neonatal mouse brainstem and spinal cord preparation with stimulated and recorded nerve branches (left side). Extracellular nerve recordings (right side, raw traces in light gray, integrated traces in color) of the left (le, light blue) and right (ri, dark blue) abducens nerves (Abd.), the left and right 8th cervical roots (le C8, light green; ri C8, dark green) and the right 5th lumbar ventral root (ri L5, red). Discharges recorded during an episode of fictive locomotion evoked by the electrical stimulation of the S1 dorsal root (stim. S1Dr) with a 4Hz pulse train (black rectangle).

( B )

 B Average cyclic modulation of the discharge activity (integrated trace) from the motor nerves shown in (A) over 10 consecutive locomotor cycles. The ri C8 trace (dark green) was used as the reference to determine locomotor cycles (C) VIIIth nerve-evoked (black arrowhead) abducens motor responses (individual traces in gray; mean in black) occur with a typical disynaptic latency compatible with the activation of the direct VOR pathway. (D) Superimposed abducens activity in response to electrical pulse (black arrowhead) applied on S1Dr (gray traces) and average activity (black trace). No reproducible S1Dr-related activity was observed following locomotor-evoked abducens discharge.(E) Frequency periodograms obtained from FFT (Fast Fourier Transform) analysis of the abducens nerve discharge during the S1Dr evoked fictive locomotion episodes (grey traces represent a single locomotor sequence, the black trace is the average from the 4 grey traces).Periodograms revealed only one magnitude peak at the locomotor frequency (locomotor freq., about 1Hz) and did not shown any peak at the S1Dr stimulation frequency (S1Dr freq., 4Hz), demonstrating the absence of spinal afference-evoked direct response in the abducens discharge.

Figure 2 .

 2 Figure 2. Efference copy signaling responsible for spino-extraocular motor coupling originates from cervical locomotor CPG.

Figure 3 . 3 (

 33 Figure 3. Anatomy of the spino-extraocular pathway

Figure 4 . 4 (

 44 Figure 4. Locomotion-induced eye movements in adult mice

(

  Model 1700 AC amplifiers, A-M Systems, Carlsborg, US) with fire-polished borosilicate glass suction recording electrodes filled with aCSF solution. The electrodes were connected to extracellular amplifiers and the analog signals recorded by an analog/digital interface (CED 1401; Cambridge Electronic Design, Cambridge, UK). Data was acquired with Spike2 software (Spike2, Cambridge Electronic Design) at a frequency of 1000Hz. Fictive locomotion was achieved in brainstem-spinal cord preparations either through electrical or pharmacological stimulation.

  frequency (Hz) Abd frequency (Hz) Abd/sp. Vr freq. reg. (r²) Abd/sp. Vr latency (ms) not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint this version posted June 28, 2021. ; https://doi.org/10.1101/2021.06.26.450039 doi: bioRxiv preprint França de Barros et al.

  França de Barros et al.

  related to principal abducens MN after 72h of infection

Table

	REAGENT or RESOURCE	SOURCE	IDENTIFIER
	Antibodies
	Anti-RV phosphoprotein	Coulon et al 2011	N/A
	Anti-Choline Acetyltransferase(ChAT)	Merck	AB144P
	antibody
	Goat anti-Mouse IgG (H+L),	Thermo Fisher	Cat # A10036
	Superclonal™ Recombinant Secondary	Scientific
	Antibody, Alexa Fluor 546
	Donkey anti-Goat IgG (H+L) Cross-	Thermo Fisher	Cat# A11055
	Adsorbed Secondary Antibody, Alexa	Scientific
	Fluor 488
	Bacterial and Virus Strains
	Challenge Virus Strain, CVS	Gaudin et al., 1992	N/A

Biological Samples x Chemicals, Peptides, and Recombinant Proteins x

  Animals were used in accordance with the European Communities Council Directive 2010/63/EU. All efforts were made to minimize suffering and reduce the number of animals included in the study. All protocols were performed on C57/BL6J male mice. Ex vivo experiments were performed on neonate (2-3 days old; n= 23) following procedures specifically approved by the ethical committee for animal research of the University of Bordeaux. Experiments on adults (8-14 weeks) were approved by the ethical committee for animal research of the University of Aix-Marseille (n=6) and of the university of Paris (n=29).

	Data and code availability
	The datasets generated during this study are available at "Dataset_Barros_2021", Mendeley
	Data, V1, doi: 10.17632/h8nbswgf95.1
	Experimental model and subject details
	Critical Commercial Assays
	x
	Deposited Data
	Dataset_Barros_2021	This study	Database: Mendeley
			doi: 10.17632/h8nbswgf95.1
	Experimental Models: Cell Lines
	x
	Experimental Models: Organisms/Strains
	x

Acknowledgements

The authors thank Patrice Jegouzo for his technical help designing the set up for the semiintact experiments, Patrice Coulon for his input on the rabies virus tracing and the Animotion platform (INCIA) for the expertise in the locomotion analysis. FFB, CT, MT & MB were supported by the Centre National d'Etudes Spatiales, the Centre National de la Recherche Scientifique and the Université de Paris, and by the Agence Nationale de la Recherche (ANR-15-CE32-0007-02). This study contributes to the IdEx Université de Paris ANR-18-IDEX-0001. This work has benefited from the support and expertise of the animal facility of BioMedTech Facilities at Université de Paris (Institut National de la Santé et de la Recherche Médicale Unité S36/Unité Mixte de Service 2009).

Resource availability

Lead contact

Further information and requests for resources should be directed to M. Beraneck (mathieu.beraneck@univ-paris.fr).

Materials availability

This study did not generate any new materials or reagents.

Author contribution

Conceptualization

Declaration of interests

The authors declare no competing interests.

Supplemental information


Mihir Durve

Fernando Peruani

Antonio Celani

Learning to flock through reinforcement

Flocks of birds, schools of fish, insects swarms are examples of coordinated motion of a group that arises spontaneously from the action of many individuals. Here, we study flocking behavior from the viewpoint of multi-agent reinforcement learning. In this setting, a learning agent tries to keep contact with the group using as sensory input the velocity of its neighbors. This goal is pursued by each learning individual by exerting a limited control on its own direction of motion. By means of standard reinforcement learning algorithms we show that: i) a learning agent exposed to a group of teachers, i.e. hard-wired flocking agents, learns to follow them, and ii) that in the absence of teachers, a group of independently learning agents evolves towards a state where each agent knows how to flock. In both scenarios, i) and ii), the emergent policy (or navigation strategy) corresponds to the polar velocity alignment mechanism of the well-known Vicsek model. These results show that a) such a velocity alignment may have naturally evolved as an adaptive behavior that aims at minimizing the rate of neighbor loss, and b) prove that this alignment does not only favor (local) polar order, but it corresponds to best policy/strategy to keep group cohesion when the sensory input is limited to the velocity of neighboring agents. In short, to stay together, steer together.

The spectacular collective behavior observed in insect swarms, birds flocks, and ungulate herds have long fascinated and inspired researchers [1][2][3][4][5][6]. There are many long-standing and challenging questions about collective animal behavior: How do so many animals achieve such a remarkably synchronized motion? What do they perceive from their environment and how do they use and share this information within the group in order to coordinate their motion? Are there any general rules of motion that individuals obey while exhibiting collective behavior? Since the last few decades, these questions have been addressed with systematic field observations coupled with mathematical models of animal behavior. Data from experimental observations have been analyzed in order to infer the rules that individuals follow in a group [7][8][9][START_REF] Katz | Proceedings of the National Academy of Sciences[END_REF][START_REF] Bialek | [END_REF] and numerous models have been proposed to explain the observed flocking behavior [12][13][14][15][16][17][18]. The great majority of flocking studies [1][2][3][4][5][6][7][8][9][START_REF] Katz | Proceedings of the National Academy of Sciences[END_REF][START_REF] Bialek | [END_REF][12][13][14][15][16][17], except for few exceptions (see [18] references therein), are based on a velocity alignment mechanism that ensures that neighboring individuals move in the same direction. However, the origin of such a velocity alignment, from a cognitive point of view, is not known, and neither its biological function.

The natural mathematical language that we will use here to discuss collective motion is the framework of Multi Agent Reinforcement Learning (MARL) [19,20]. In this scheme, the agents can perform actions in response to external cues that they can sense from the environment as well as from other agents. The goal of each agent is to achieve a given objective. In the case at hand, the agents are individuals who can observe the behavior of their close neighbors and react by steering according to some rule. Since it has been hypothesized that there exist many benefits associated to group-living, such as predator avoidance [21] and collective foraging [22], we assume that the objective of the agents is to increase or maintain the cohesion of the group. The essence of Reinforcement Learning (RL) is that, by repeated trial and error, the agents can learn how to behave in an approximately optimal way so as to achieve their goals [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Here, we show that velocity alignment emerges spontaneously in a RL process from the minimization of the rate of neighbor loss, and represents a optimal strategy to keep group cohesion.

In the following we will consider individual agents that move at constant speed in a two-dimensional box with periodic boundary conditions. The density of agents is kept fixed to ρ = 2 agents/(unit length) 2 . Updates are performed at discrete time steps as follows. For the i-th agent, the position update is:

r t+1 i = r t i + v 0 v t i ∆t , (1)
where r t i and v t i , with ||v t i || = 1, are the position and moving direction, respectively, of the agent at time t; the term v 0 corresponds to the speed, which we fix to v 0 = 0.5, and ∆t = 1. At each time step, each agents makes a decision on whether keeping the current heading direction or performing a turn. The decisionmaking process is based on the sensorial input of the agent, which corresponds to the angular difference between the (normalized) average velocity defined by P i = ( |rj -ri|<R v t j )/n i (with n i the number of neighbors of agent i within its perception range R) and the moving direction of the agent v t i . Below, we take R = 1. We can express the state as

s t i = arg(P i , v t i ) , (2)
where the function arg(P i , v t i ) is defined as arccos

(P i • v t i /||P i ||) for P i •(v t i ) ⊥ > 0 with (v t i )
⊥ obtained by rotating π/2 counter clockwise the unit vector v t i , and minus this quantity otherwise. This means that s t i ∈ [-π, π). For computational simplicity, we discretize s t i by dividing 2π into K s equally spaced elements. In the RL language, the relative angle s t i is the contextual information that defines the current state of the i-th agent. Knowing s t i , the agent updates v t i by turning this vector an angle

a t i v t+1 = R(a t i )v t , (3)
where R(a t i ) is a rotation matrix. Note that there are K a possible turning angles, equally spaced in [-θ max , θ max ]. In the RL jargon, choosing the turning angle a t i represents an "action" performed by the agent. The association of a given state s i t with an action a i t is called a policy. Policy evaluation takes place at each time step as the agent receives a (negative) reinforcement signal in the form of a cost c t+1 i for losing neighbors within its perception range R

c t+1 i = 1, if n t+1 i < n t i 0, otherwise (4)
where n t i is the current number of neighbors. The goal of the learning agent is to find a policy that minimizes the cost. To achieve this goal the agent makes use of a simple learning rule [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Watkins | [END_REF]. The i-th learning agent keeps in memory a table of values Q i (s, a) for each stateaction pair (here a matrix K a × K s ) which is updated at each step of the dynamics -only for the entry that corresponds to the state just visited and the action just taken -according to

Q i (s t i , a t i ) ← Q i (s t i , a t i ) + α[c t+1 i -Q i (s t i , a t i )] . (5)
This update rule effectively constructs an estimator for the expected cost that will be incurred by starting in a given state and performing a given action. The policy at each time-step is based upon the current Q i according to so-called -greedy exploration scheme:

a t i = argmin a Q i (s t i , a ) with prob. 1 -
an action at random with prob.

.

In the simulations we have used α = 0.005 and various different schedules for the exploration probability [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. We start by considering the case when there is a single learning agent in a crowd of N teachers (see Figure 1)

FIG. 1.
Learning to flock within a group of teachers. (A) Scheme of reinforcement learning. (B) Neighbors (black) within the perception range of the learner (red).

who have a hard-wired policy:

a t i (s t i ) =    s t i if |s t i | ≤ θ max θ max if s t i > θ max -θ max if s t i < -θ max . ( 7
)
This decision rule is nothing else but a version of the Vicsek model of flocking with a discrete number of possible moving direction and limited angular speed. Thus, teachers display robust collective motion. The learning agent, on the contrary, does not have a fixed policy, but one that evolves over time as it acquires experience, i.e. by visiting a large number states, and evaluating the outcome of executing its actions. In this case we find that for suitably chosen learning rates α and exploration probability the algorithm approaches an approximately optimal solution to the decision-making problem after some period of training.

In simulations, we break the training session into a number of training episodes of equal prescribed duration of 10 4 time steps. In each episode the teachers start with random initial positions and velocities. After a transient, they form an ordered flock and at this time we introduce the learner and implement the learning algorithm. At the very beginning, the learner starts with a Q-matrix with all zero entries, which in a case of optimistic initialization (the naive learner expects to incur no costs), a choice that is known to favor exploration [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. From one episode to the following, the learner keeps in memory the Q-matrix that it has learned so far. During the training session, we measure the success of the learning process with the average cost that a learner pays per time step, that is the rate at which it is losing contact with the teachers (see Fig. 2A). As the training progresses, the rate at which neighbors are lost starts from an initial value of 0.5, meaning that on average the learner loses contact with some neighbor every other step, to decrease and eventually saturate down to a value around 0.1 meaning that the contact is kept for 90% of the time. In the insets of Fig 2A we show samples of short trajectories of the learner and some teachers at the early and later phase of the training process. We observe that in the early phase of the training, the learner essentially moves at random (see movie1.mp4) and eventually it learns to stay within the flock (see movie2.mp4). In Fig 2C we show that the policy discovered by the learner is identical with the pre-defined policy of the teachers, see Eq. ( 7) and Fig. 2B. It is important to remark that the one and only goal of the learner is to keep contact with its neighboring teachers, not to imitate their behavior. It simply turns out that the best strategy for the learner is in fact the underlying rule that was assigned to teachers. Now, let us move our focus to the situation where there are no teachers, but only N independently learning agents (see Figure 3). A distinctive difficulty of applying reinforcement learning to the multi-agent setting is that all individuals have to concurrently discover the best way to behave and cannot rely upon knowledge previously acquired by their peers. However, we find that N learning agents are able to overcome this hurdle and are actually capable of learning to flock even in the case when all of them start as absolute beginners (all Q-matrices initialized to zero).

To characterize the performance of the learners, we measure the average rate of loss of neighbors. In Fig 4A we show the average cost for various groups sizes and state-action space discretizations {K s , K a }. The cost reaches a small and steady value after few hundreds of episodes. As the group size grows, the performance remains essentially the same. Conversely, refining the dis-cretization allows to further reduce the costs: for 128 relative alignment angles and 28 turning angles the agents do not lose neighbors for about 97% of the time.

The resulting Q-matrix at the end of the training, averaged over all learners, is shown in Fig. 4B. The colors represent the numerical values in the Q-matrix and the discovered policy is shown with white points. We observe that the discovered policy is the same that the one learned by the single agent with N teachers. It is worth stressing that all agents independently learn the same strategy. We have collected the values of the Q-matrix for a given state (s = 0) and different actions, for all agents, at the end of training. The histogram for the frequency of Q values is shown in Fig. 5 where one can observe that there is a clear gap that separates the estimated costs for the optimal turning angle, which lie around 0.1, from the suboptimal actions that have significantly larger costs.

A customary measure of the degree of alignment is the polar order parameter:

ψ(t) = 1 N N i=1 v t i . (8)
If all the agents are oriented randomly then, as N → ∞, ψ → 0 whereas if all the agents are oriented in the same direction then ψ = 1. In Fig 6 we show the evolution of order parameter versus the average cost as the multiagent learning is advancing. We observe that in the early phases of training the rate of loss of neighbors is comparatively high and the direction consensus among the agents is low, in agreement with the notion that the agents are behaving randomly (see movie3.mp4). As the learning progresses, the agents discover how to keep cohesion, and in doing so they achieve a highly ordered state (see movie4.mp4). Therefore we conclude that the obtained results proves that the velocity alignment mechanism of the Vicsek model (see Eq. ( 7)) -based on energy minimization of spin-spin interaction of the XY model -can spontaneously emerge, counterintuitively, from the minimization of neighbor-loss rate, and furthermore represents an optimal strategy to keep group cohesion when the perception is limited to velocity of neighboring agents. In FIG. 5.

All agents independently learn the same optimal strategy. The histogram shows the frequency of a give numerical value of Q(0, a) across all learners, at the end of training. The best action a = 0 always performs better than any other action. The same holds for other states (not shown). Data obtained with N = 100 agents, Ks = 32, Ka = 7.

summary, if the agents want to stay together, they must learn that they have to steer together. FIG. 6. Average polar order parameter versus the rate of loss of neighbors. In the insets we show a few short trajectories of naive and trained agents.

In more general terms, we have shown Multi-Agent Reinforcement Learning can provide an effective way to deal with questions about the emergence of collective behaviors and the driving forces behind them. Our present contribution is just an initial step in this direction and we feel that prospective applications of this approach remain largely unexplored.

For instance, in the present work we have decided at the outset the structure of the perceptual space of the agents, namely the choice of the radius of perception as the relevant parameter and of the relative angle as the relevant state variable. In doing so, we bypassed fundamental questions like: Is the metric distance the most appropriate choice for ranking neighbors ? How should the information given by other individuals be discounted depending on their ranking ? A more ambitious approach would tackle these issues directly through MARL and try to learn from experience what are better choices of the state variable that allow to achieve optimal cohesion.

As another example, here we have tasked our agents with the goal of keeping contact with neighbors, which in itself is understood to be a secondary goal motivated by the primary need of avoiding predators (safety by the numbers) or of increasing the efficiency of foraging. Can one recapitulate the congregation behavior by tasking agents with the primary goal itself ? More explicitly, would agents learn to align themselves by rewarding behaviors that reduce their risk of being predated or increase their chance of getting some food ? Also, in this paper we have considered a group of identical agents. When agents differ for their perceptual abilities or their dexterity in taking the appropriate actions, then competitive behaviors may arise within the group and the problem acquires a new challenging dimension. How much heterogeneity and competition can be tolerated before it starts impacting the benefit of staying in a group ?

These and many other questions lend themselves to be attacked by the techniques of MARL and we believe that the approach that we have delineated here will show its full potential in the near future.

FIG. 2 .

 2 FIG. 2. Single learner results. (A) Performance of the learner as training progresses. The error bars indicate standard deviation in the values in 20 training sessions. In the inset, some short trajectories of the learning agent (red) and teachers (black) at the stages indicated by the arrows. The number of teachers is N = 200. The maximal turning angle is θmax = 3π/16. (B) The Q-matrix, i.e the average cost incurred for a given state-action pair by the teachers. White stars shows the action a taken by the teacher when in state s. (C) Q-matrix of the learner at the end the training session. White stars denote the best estimated action of the learner for each state.

FIG. 3 .

 3 FIG. 3.Multi-agent concurrent learning. (A) Multiple agents with their perception range. (B) Agents interact with short-range, reciprocal interactions.

FIG. 4 .

 4 FIG. 4. Results for the multi-agent concurrent learning. (A) Average performance of learners in groups of different sizes. Black, magenta and blue colors corresponds to state-action spaces of size {Ks, Ka}={32, 7}, {64, 14}, {128, 28} respectively. Error bars indicate standard deviation in the average values for each agent. (B) Average Q-matrix at the end of the training for N = 200 agents with combination of {Ks, Ka}={32, 7}. White points indicate actions with estimated minimum cost for given state. The colors represent values in the Q-matrix.

  M. D. acknowledges fruitful discussion with A. Pezzotta, M. Adorisio and A. Mazzolini. M. D. is grateful for the graduate fellowship by the ICTP and University of Trieste. M. D. and A. C. acknowledge hospitality and support from Laboratoire J. A. Dieudonné, Université Côte d'Azur, France.


Olivier Meste

Stef Zeemering

Joël Karel

Theo Lankveld

Ulrich Schotten

Harry Crijns

Ralf Peeters

Pietro Bonizzi

Body-Surface Atrial Signals Analysis Based on Spatial Frequency Distribution : Comparison Between Different Signal Transformations

In contrast to electrograms, Body-Surface Potential Mapping (BSPM) records the global atrial activity, at the expenses of a lower spatial accuracy. The aim of this study is to investigate whether BSPM recordings can discriminate persistent patients undergoing electrical cardioversion, based on the body-surface normalized AF spatial frequency distribution. High-density BSPMs (120 anterior, 64 posterior electrodes) were recorded in 63 patients with persistent AF. For each patient and electrode recording, the frequency content of AF was analyzed on the raw signal, and also by means of the normalized correlation function, and by Singular Spectrum Analysis (SSA). In order to compare the body-surface spatial distributions of AF frequency in all patients, these distributions were first normalized, before performing statistical analysis.

We found that the distribution of AF frequency on the body-surface, and its interpretation, are strongly dependent on the specific method employed. Moreover, the estimated body-surface AF frequency was greater over the central posterior and the right anterior BSPM locations. Finally, SSA-based decomposition followed by frequency analysis could discriminate AF patients recurring 4 to 6 weeks after electrical cardioversion from those who did not, based on the frequency content in the proximity of V1.

Introduction

Atrial fibrillation (AF) dominant frequency has been shown to be a relevant feature in both determining the stage of AF, and in patient's characterization. As already noticed in [START_REF] Guillem | Noninvasive localization of maximal frequency sites of atrial fribrillation by body surface potential mapping[END_REF], the noninvasive analysis of AF by means of Body-Surface Potential Mapping (BSPM), allows the measurement of the AF spatial frequency distribution with still good accuracy with respect to the same quantity measured invasively. This distribution informs us on the propagation of AF waveforms over the atria with the potential of guiding radiofrequency ablation procedures. Considering a single frequency value for global AF characterization may not suffice for further analysis such as discriminating between patients showing AF recurrence (R) or nonrecurrence (NR) after electrical cardioversion.

It has been shown in [START_REF] Bonizzi | A novel framework for noninvasive analysis of shortterm atrial activity dynamics during persistent atrial fibrillation[END_REF] that multidimensional autocorrelation function enables such discrimination based on short and long term features of the multidimensional AF vector. In addition, this discrimination has also been performed successfully at the level of individual electrodes by using similar method [START_REF] Meste | Use of normalized correlation function to discriminate outcome of persistent patients undergoing electrical cardioversion[END_REF], and showing that spatial dimension plays an important role in this process. In order to test the power of AF frequency distribution to discriminate R from NR patients, the frequency content recorded by each electrodes was processed by using different methods. In particular, the approach proposed in [START_REF] Meste | Use of normalized correlation function to discriminate outcome of persistent patients undergoing electrical cardioversion[END_REF] and was compared to regular spectral analysis based on windowed Fast Fourier Transform (FFT) and Singular Spectrum Analysis (SSA) [START_REF] Vautard | Singular spectrum analysis in nonlinear dynamics, with applications to paleoclimatic time series[END_REF]. Results showed that, thanks also to a patientspecific frequency normalization, specific regions on the torso differ from others in terms of frequency distribution, and that a gradient in magnitude exists instead of a frequency one.

Methods

BSPM data and pre-processing

BSPMs were recorded in 63 patients in persistent AF using a custom configuration of 184 leads with 120 anterior and 64 posterior leads (ActiveTwo BSM Panels Carbon Electrodes, Biosemi B.V., The Netherlands). All pa-tients underwent electrical cardioversion. After a blanking period from 4 to 6 weeks after procedure, 32 patients showed AF recurrence (labeled R) and 31 remained in sinus rhythm (labeled NR). ECGs were sampled at 256Hz. A one-minute segment was selected for each subject, lowquality leads were excluded (low signal-to-noise ratio, poor electrode contact, motion artefacts), and Wilsons Central Terminal was subtracted in line with conventional ECG analysis. After band-pass filtering the signals between 1 and 100Hz (3rd order Chebyshev), QRST cancellation was performed using an adaptive singular value decomposition method, inspired by the approach in [START_REF] Alcaraz | Adaptive singular value cancellation of ventricular activity in single-lead atrial fibrillation electrocardiograms[END_REF], with multiple QRST window templates defined using hierarchical clustering. The extracted atrial signals were postfiltered with a zerophase notch filter at 50Hz to suppress power line noise, and with a 3Hz zero-phase highpass filter (3rd order Chebyshev) to remove low-frequency residuals not related to (persistent) AF.

AF vector Autocorrelation

In a previous study [START_REF] Meste | Use of normalized correlation function to discriminate outcome of persistent patients undergoing electrical cardioversion[END_REF], we showed that the multivariable autocorrelation function stems from the contribution of monodimensional autocorrelation functions computed from individual electrode recordings. From that observation, it comes that the AF frequency content of a patient can be extracted for each electrode by computing the FFT of the corresponding monodimensional autocorrelations (producing the results named "Monocorr" in this study). Furthermore, the value of the first max of the monodimensional autocorrelation function plays also a crucial role in the characterization of the AF [START_REF] Bonizzi | A novel framework for noninvasive analysis of shortterm atrial activity dynamics during persistent atrial fibrillation[END_REF]. Thus, it is quite intuitive to compute the time lag of this value, assumed to be related to the AF cycle length, and calculate a second estimate of the AF frequency for each electrode (named "Monolag").

Singular Spectrum Analysis

The singular spectrum analysis is based on the embedding of the time series x(n), associated to a recorded ECG lead, in a vector of dimension M . In this study,we fixed M to be equal to 150 samples, according to the AF mean short time behavior observed in [START_REF] Bonizzi | A novel framework for noninvasive analysis of shortterm atrial activity dynamics during persistent atrial fibrillation[END_REF]. The embedding procedure forms K = N -M + 1 lagged vectors x i = (x(i), . . . , x(i + M -1)) T . A matrix X is then generated as X = [x 1 , x 2 , . . . , x M ], such that this is a Hankel matrix. The singular value decomposition (SVD) of X is then computed , such that X = U DV T . The matrix X is then reconstructed by the SVD-based rank two approximation

X ≈ σ 1 u 1 v T 1 + σ 2 u 2 v T 2 .
The σ, u and v values stands for the two highest singular values and the corresponding right and left eigenvectors, respectively. The rank two approximation was selected in order to minimize the approximation error, while taking into account the complementarity of sin and cos functions (expected to be reflected in the first two principal components, respectively). The last step of the SSA consists in averaging the diagonal of the reconstructed matrix to provide the approximated time series x(n) (see [START_REF] Bonizzi | Singular spectrum decomposition: A new method for time series decomposition[END_REF] for a complete description of the procedure and possible extensions). Finally, an FFTbased spectral analysis was applied to the reconstructed series.

Results and Analysis

The four frequency analyses were performed on the two groups of subjects, according to the R and NR classification. Thus four estimations (FFT, Monocorr, Monolag, SSA) of the frequency with the highest power were provided for each patient and for each electrodes recording. For a given patient, the dominant frequencies associated to each electrode were normalized in the range [0-1]. This normalization is crucial to account for inter-patient variability when assessing the locations of the highest and lowest frequency value (regardless their absolute values). As it can be noticed from the figures, all maps exhibit similar results, with the highest frequencies localized on the right anterior side of the torso. The corresponding location on the heart surface is the right atrium. In addition, the FFT results seem to be the more scattered, missing the regional homogeneity of frequency content. Thus, an additional criterion is needed to compare the performance of the three methods left. This criterion could be the frequency resolution associated with each method and its influence over the frequency patients and for all electrodes. See text for details map interpretation. This is illustrated with a single NR patient exhibiting a clear AF pattern. In Fig. [START_REF] Bonizzi | Singular spectrum decomposition: A new method for time series decomposition[END_REF], and (8) regions of constant frequencies are clearly identifiable. Such properties have been already observed in [START_REF] Guillem | Noninvasive localization of maximal frequency sites of atrial fribrillation by body surface potential mapping[END_REF], and can be related to dominant frequencies computed over electrograms recorded in various regions of the heart. In contrast, in Fig. [START_REF] Alcaraz | Adaptive singular value cancellation of ventricular activity in single-lead atrial fibrillation electrocardiograms[END_REF], and (7) a frequency gradient is visible and can be explained by the low frequency resolution of the corresponding frequency estimators. It can be easily observed that a gradient in the magnitude of the dominant frequency can be easily converted (and mistaken) into a gradient in frequency because of a limited resolution. This example shows that SSA and FFT seem to provide more accurate estimates, although as shown above the FFT globally per-formed badly in term of AF patients discrimination.

Conclusions

This study compared different (AF) dominant frequency estimation methods for the discrimination of R and NR AF patients. It is shown that this comparison can be performed at the level of each electrodes. As expected the dominant frequency varies along the body surface in accordance to the heart electrophysiology during AF. The dominant AF frequency is on average localized at the right anterior side of the torso corresponding to the right atrium. This results are consistent with clinical outcomes using electro- grams. Finally, it seems that the SSA-based approach allows a plausible interpretation and discrimination between NR and R patients together with the generation of an accurate frequency mapping. These results have been obtained thanks to a proper normalization of the frequency range applied to each patient.

  Figures (1), (2), (3),[START_REF] Vautard | Singular spectrum analysis in nonlinear dynamics, with applications to paleoclimatic time series[END_REF] show the median values of the dominant frequencies of all patients, computed over each electrode. The black circles stand for the electrode locations and the red ones for the lead V1 position. A Man-Whitney U test was performed on the R and NR patients populations for stratification purposes, and applied individually to each electrode. A significant (p < 0.05) difference is highlighted by a bold black circle. It should be noted that no corrections were applied for multiple comparisons.

Figure 1 .

 1 Figure 1. Median normalized dominant frequencies (Monocorr) for all patients and for all electrodes. See text for details.

Figure 2 .

 2 Figure 2. Median normalized dominant frequencies (FFT) for all patients and for all electrodes. See text for details.

Figure 3 .

 3 Figure 3. Median normalized dominant frequencies (Monolag) for all patients and for all electrodes. See text for details.

Figure 4 .

 4 Figure 4. Median normalized dominant frequencies (SSA) for all

Figure 5 .

 5 Figure 5. Normalized dominant frequencies (Monocorr) for a NR patient and for all electrodes.

Figure 6 .

 6 Figure 6. Normalized dominant frequencies (FFT) for a NR patient and for all electrodes.

Figure 7 .

 7 Figure 7. Normalized dominant frequencies (Monolag) for a NR patient and for all electrodes.

Figure 8 .

 8 Figure 8. Normalized dominant frequencies (SSA) for a NR patient and for all electrodes.