Nicolas Fontbonne
email: nicolas.fontbonne@sorbonne-universite.fr

Olivier Dauchot
email: olivier.dauchot@espci.fr

Nicolas Bredeche
email: nicolas.bredeche@sorbonne-universite.fr

Distributed On-line Learning in Swarm Robotics with Limited Communication Bandwidth

Keywords: online distributed learning, swarm robotics, black-box objective function, embodied evolution, social learning, evolutionary robotics

This paper presents a new algorithm for distributed on-line evolutionary learning in swarm robotics. The challenge we address is to cope with the limited computation and communication capabilities of low cost robots, which are often used in swarm robotics. In order to do so, the algorithm decouples computation and communication and ensures learning of efficient control policies even when only a limited amount of information can be exchanged between neighbouring robots. We show experimentally that this algorithm is both remarkably robust with respect to its meta-parameter values, and able to adapt automatically to the available communication bandwidth.

I. INTRODUCTION

Swarm robotics is a sub-domain of collective robotics. This field is characterised by the use of a generally large number of robots with limited communication and computation capabilities, and an objective defined at the level of the whole swarm (e.g. exploring the environment, searching for resources, collectively transporting heavy objects). The challenge is to design the rules of microscopic interactions between the robots in order to achieve a relevant swarm organisation at the macroscopic level [START_REF] Bayindir | A review of swarm robotics tasks[END_REF]- [START_REF] Brambilla | Swarm robotics : A review from the swarm engineering perspective[END_REF], [START_REF] Hamann | Swarm Robotics -A Formal Approach[END_REF].

There are several bio-inspired methods to address this problem of distributed programming, which can be grouped into two main classes: manual programming and automatic design methods. In the first case, the aim is to explicitly reproduce at a more or less abstract level behaviours observed in nature, but whose macroscopic result is sometimes difficult to foresee in advance.

In the second case, the use of evolutionary optimisation algorithms makes it possible to automate the design of individual behaviours according to a global objective function [START_REF] Trianni | Evolution, Selforganization and Swarm Robotics[END_REF]. Indeed, since the structure of the environment is not known in advance, writing the objective function makes it possible to specify the general objective of the task (e.g. maximizing the number of objects picked up for a foraging task), without giving any precision on the structure of the strategy to be deployed. This is a well-known problem framework in evolutionary robotics: the structures of the research space and the fitness landscape have a tenuous relationship, for which stochastic optimisation methods are a good fit.

However, an important assumption that underlies manual or automatic design methods is that behavioural rules are obtained in the laboratory, and then deployed in a real-world situation as is without any further tuning. In other words, the environment and the nature of the task are considered stationary between the conditions known at the time of design and the conditions actually encountered afterwards.

In this paper, we are interested in a different class of problem. We consider that the environment in which the robots will be deployed is not known in advance. Provided a general objective, that can be written down as an objective function defined at the level of the individual (e.g. given a foraging task, each robot should try to get the largest number of items), we implement a distributed on-line learning algorithm to allow the swarm of robots to progressively acquire the necessary skills to perform the target task. The behaviours learned depend on the nature of the task, but also on the particularities of the environment. For example, picking up objects in a foraging task requires different strategies depending on whether the objects are grouped together in a specific area or distributed in the environment.

The class of evolutionary algorithms that addresses this problem is known as either embodied evolution or social learning. Whether biological or cultural evolution is considered, both methods can be seen as instances of designing algorithms inspired by evolutionary dynamics for swarm robotics. These methods stand as on-line distributed learning that takes into account interactions between hardware limited robots distributed over a possibly large space with local communication. On the one hand, these evolutionary dynamics methods are similar to the classical evolutionary robotics approach as the goal is to optimised a black-box objective function whose analytical form is not known. On the other hand, they differ in that the problem of the transition to reality is simply non-existent in evolutionary dynamics methods: the actual learning starts at the time of operational deployment of the robots, and not before it. In other words, the goal is to design robust on-line learning algorithms as much as robust solutions.

To date, several evolutionary dynamics algorithms have been validated on real robots, addressing various scientific and practical issues. However, the number of robots is often in the order of ten(s) due to the hardware and CPU requirements of such robots to implement learning algorithms (e.g. 20 e-Puck robots in [START_REF] Bredeche | Environment-driven Distributed Evolutionary Adaptation in a Population of Autonomous Robotic Agents[END_REF], 6 Thymio-2 robot with a Raspberry board in [START_REF] Heinerman | Evolution, individual learning, and social learning in a swarm of real robots[END_REF]). This is in stark contrast with other works in swarm robotics that do not implement learning capabilities, where the number robots is often counted by the hundreds [START_REF] Carranza | Morphogenesis in robot swarms[END_REF], [START_REF] Valentini | Collective decision with 100 kilobots: Speed versus accuracy in binary discrimination problems[END_REF] and even up to slightly more than one thousand robots [START_REF] Cornejo | Programmable self-assembly in a thousand-robot swarm[END_REF]. The challenge remains open as to implementing online distributed learning on such a large swam of robots.

In this paper, we propose a new distributed on-line learning algorithm inspired by evolutionary dynamics. The originality of this algorithm is to minimise the requirements in terms of memory and communication cost, in order to be deployed on a swarm of very low cost robots with limited computing and communication capabilities (e.g. a Kilobot robot). The proposed algorithm is based on the horizontal gene transfer mechanism observed in bacteria: when two robots interact, a part of the control parameters is transferred to the robot's memory and communication system. Thus, the amount of information transferred can take into account time and bandwidth limitations, regardless of the number of control parameters governing decision making.

Though similar ideas have been explored in evolutionary computation [START_REF] Harvey | The microbial genetic algorithm[END_REF], it has never been employed in the context of distributed on-line learning, where the possibility of modulating the amount of information transmitted between robots makes it possible to take into account hardware and environmental constraints. Obviously, the price to pay for such an algorithm is a reduced convergence speed. But it also makes it possible to adjust the amount of information exchanged to account from practical limitations at hand. Indeed, the quality of the communication bandwidth depends both on technical characteristics and environmental contingencies (number of packet collisions that increase with the number of robots, disturbances due to materials used in the environment, etc.).

The rest of the paper is organised as follow : Section 2 presents the algorithm and evolutionary operators. Section 3 describes the experimental setup. Section 4 presents results on a classic foraging task, first by comparing the proposed algorithm with a state-of-the-art counterpart, and second, by performing a sensitivity analysis of the various hyperparameters of the Algorithm. We then present an extension of the Algorithm to automatically adapt to the available communication bandwidth.

II. ALGORITHM

Both embodied evolution [START_REF] Bredeche | Embodied evolution in collective robotics: A review[END_REF], [START_REF] Watson | Embodied Evolution: Distributing an evolutionary algorithm in a population of robots[END_REF] and social learning [START_REF] Heinerman | Three-fold Adaptivity in Groups of Robots: The Effect of Social Learning[END_REF] implement an evolutionary algorithm scheme, adapted to perform distributed on-line learning as illustrated in Figure 1. Each robot optimises a policy to maximise a score, G, that is locally defined in each robot.

Robots follow a sense-act loop commonly used in robotics for reactive agents. Each robot i runs a deterministic policy π θi . At initialisation, a robot is initialised with a random control parameter set θ i , referred to as active parameters. These parameters are then used by the policy to determine the actions to be taken by the agent in reaction to its observations. The typical algorithm goes as follow : as a robot moves through its environment, the quality of its behaviour is assessed using a score function which depends on the userdefined task to be achieved (e.g. number of items gathered for a foraging task, which is also sometimes referred to as fitness value, reward, performance or utility). When two robots are within communication range, they exchange their active parameters and the (current) assessment of their policies. This operation is called mating. The choice of accepting a partner may be purely based on the environmental contingencies but other consideration may play a role such as performance or similarity. Mating does not automatically imply a change in the active parameters of one agent, but incoming data is generally stored for later use in a reservoir.

The renewal of a robot's active parameters typically occurs after some predefined amount of time. At this point, the robot will use information stored in the reservoir to update its current active parameter set, which imply constructing a new candidate set of parameters from the reservoir using typical selection and variation evolutionary operators [START_REF] Bredeche | Environment-driven Embodied Evolution in a Population of Autonomous Agents[END_REF], [START_REF] Fernandez Pérez | Comparison of Selection Methods in On-line Distributed Evolutionary Robotics[END_REF], [START_REF] Hart | Improving Survivability in Environment-driven Distributed Evolutionary Algorithms through Explicit Relative Fitness and Fitness Proportionate Communication[END_REF], [START_REF] Heinerman | Three-fold Adaptivity in Groups of Robots: The Effect of Social Learning[END_REF], [START_REF] Prieto | Asynchronous Situated Coevolution and Embryonic Reproduction as a Means to Autonomously Coordinate Robot Teams[END_REF].

A. Horizontal Information Transfer

All embodied evolutionary algorithm to date assume that the whole set of active parameters (and current score) can be sent as a single communication packet. This is generally true where the bandwidth is virtually infinite (as in simulation) or where it is order of magnitude larger than the size of messages to be sent (as with Linux-running robots using WIFI exchanging a few hundreds neural network weights) [START_REF] Bredeche | Embodied evolution in collective robotics: A review[END_REF]. Using hardwarelimited robots such as Kilobots [START_REF] Michael Rubenstein | Kilobot: A low cost scalable robot system for collective behaviors[END_REF] then raise the question of limiting the number of control parameters to stay within the limits of the communication bandwidth, which can be due to either technical limitations or environmental contingencies. For example, Kilobots not only use slow IR communication (a few octets per seconds), but also are limited when the number of Kilobots closeby increases due packet collisions.

In order to decouple communication and computation constraints, we introduce the HIT algorithm. HIT stands for Horizontal Information Transfer, and can be seen as an instance of either embodied evolution or social learning algorithms. It uses evolutionary operators in a distributed on-line fashion, and manage communication between robots by exchanging a part, rather than all, of the robot's control parameters. By exchanging only partial information, we expect two benefits:

• the possible recombination of behavioural skills (i.e. efficient subsets of the policy parameters) obtained by separate robots, which would not be possible in a winnertake-all approach. This will be shown is Section IV-B;

• the decoupling of the number of control parameters that can be used for control and sent to other robots. In other words, it makes it possible to exploit the computation and memory capabilities of the robot, without being limited by its communication capability. This will be shown is Section IV-C. In addition, HIT differs from other similar algorithms as it does not require a reservoir to store incoming information. Upon receiving control parameters and current score from a nearby robot, a robot will immediately integrate the new parameter values (i.e. overwriting the current corresponding values) if its interlocutor's score is better.

robot #1 robot #2 robot #N ... t T T T T T Fig. 2.
Communication or mating between the agents. Each robot has a maturation period of time T where communication with neighbours is turned off. A new maturation period is initiated after each modification of the active parameters. During this period, the robot only evaluate it policy Algorithm 1 describes HIT as implemented in each robot. It includes both evolutionary learning and decision-making. It can be decomposed in two cycles:

• lines 4-7, sense-act cycle: The agent retrieves information from the environment via its sensors using the sense() function. It retrieves an observation vector o and a reward scalar r. The reward is stored in a queue, here G[.], of size T for later use. As for the observation vector, it is used by the deterministic policy π θ to compute the action vector a. Finally, the act() function is responsible for transmitting the commands to the actuators.

• lines 9-15, evolutionary cycle: After an evaluation period of fixed size T , the agent can enter the evolutionary cycle. It broadcasts a random subset of its control parameter set to its neighbours with an evaluation of its quality (i.e. the score). The subset size is controlled by the transfer rate α. When it receives a new message (indicated by the new message variable), if the received parameters have a better score, then it replaces its own control parameters by the sender's selected parameters (function TRANSFER). Then it applies a Gaussian mutation of variance σ on all its parameter set (function MUTATION). In the following, we provide more details on the selection and variation operators.

B. Evaluation and Selection

The assessment of individuals depends mainly on the quality of the policy but can be very noisy due to non-stationary stochastic variabilities in the environment and behaviour of other agents. Depending on the definition of the score function and the distribution of the rewards in the environment, some individuals may find themselves naturally favoured by chance and thus spread misguided policies.

HIT uses a sliding time window of size T to assess the robot's performance. The window size depends on the task and the environment at hand, and must be set so that sufficient information is gathered to provide a relevant estimate of the quality of current policy. In order to compare policies in a fair manner, robots can only exchange information after the sliding window has been completely filled. We call this maturation period, which duration corresponds to the time required for a full self-evaluation, i.e. the evaluation time T .

Figure 2 illustrates the dynamics of the algorithm. Whenever two robots meet, the worst-scoring robot replace part of its control parameter with those of the best-scoring robot. Then, the updated robot resets its score and enters a maturation period during which communication is disabled. At each step, the agent receives a reward R t . We call score G t , the cumulated reward obtained by the agent during T steps (sliding window). The evaluation time T , the score G t and reward R t at step t are linked by the relation:

G t = T -1 k=0 R t-k (1)
While a longer evaluation time should allow for more accurate assessment of a given policy's quality, there is of course a cost in terms of convergence speed.

C. Transfer operator

HIT introduces a transfer operator with rate α ∈ [0, 1] that defines the amount of information that will be transferred during communication between two robots. As an example, a transfer rate of α = 0.5 means that half of the control parameters will be randomly selected to be sent. From one interaction to another, a different subset of parameters can be selected for sending, and two interacting robots will send the same quantity of possibly different parameters.

There can be different methods to randomly pick the parameters to be sent. A basic method, which we use afterwards, is to send n parameters stored in 32-bit float, along withan additional n bytes to send the indexes of these parameters. However, several strategies can be used to compress or limit the quantity of information without compromising overall optimisation. It can also possible to send a segment of parameters whose offset changes randomly in-between each new message.

Algorithm 2 details the transfer mechanism during the mating operation. Unlike other embodied evolution algorithms, HIT does not store incoming information in a reservoir. Whenever the score of the sender is greater or equals to the score of the receiver, the received parameter are directly used to overwrite the corresponding local parameter.

D. Mutation operator

HIT implements a classic Gaussian mutation operator. It applies a perturbation centred on the current parameter value, with a variance σ. It is defined as follow, for the m parameters:

θ[i] ← N (θ[i], σ) ∀i ∈ [1, m]
Mutation allows to introduce and maintain some level of diversity during the exploration of the parameters space. While it may not be useful in the first steps of evolution, it eventually maintain some level of diversity for exploring the parameters space as using the transfer operator can only leverage what is already present in the initial population.

III. EXPERIMENTAL SETUP

A. Task and Environment

In order to study the dynamics of HIT, we devise a foraging task, similar to tasks solved by many species of insect collectives. It is also a good abstraction of a search and retrieve robotic task, where an unknown environment must be explored to retrieve specific objects or resources.

The goal here is for each robot to collect as many items as possible. Both robots and items are initially randomly placed in the arena. Whenever a robot picks up an object at iteration t, it gets a reward of r t = 1, and a new item appears at a random location in the arena to maintain a constant number of resources.

B. Simulation environment

We used the Roborobo3 simulator [START_REF] Bredeche | Roborobo! a fast robot simulator for swarm and collective robotics[END_REF], which is a pseudorealistic, light and fast multi-agent simulation environment developed in C++. It provides a pseudo-realistic physics robotic model similar to the seminal Khepera2 and e-Puck 2-wheeled mobile robots while still ensuring fast enough simulation to allow for extensive experimental work involving hundreds of robots.

Robots, objects and the environment are physically represented by bitmap images, which allows roborobo3 to manage collisions at the pixel level, though location, perception and displacement are handled in the continuous domain.

Robots have a size of 5px × 5px . They move in an environment of 1400px × 800px that is uniformly filled with objects.

C. Robot Model

Robots are subject to a kinematic model. It is, therefore, a question of controlling a velocity vector. Thus, the robots have a 2-dimensional action space A where the two dimensions represent speed (a 0 ∈ [-1, 1] for [backward, forward]) and angular speed (a 1 ∈ [-1, 1] for [clockwise, anti-clockwise]).

Their observation space O is composed of 16 range sensors that get information about the surrounding of the agent. They are ray-casting sensors that have a maximum range of three times the robot length (15px). For each of these sensors, four information are extracted: the distance to contact if an object is detected, and three other Boolean information for each sensor to explicit the type of information (object, wall or agent). Thus, we obtain an observation vector of dim(O) = 64 dimensions.

The robot's policy maps observations o ∈ O to actions a ∈ A. For that purpose, we use a multi-layered Perceptron (MLP) as the main policy structure. Figure 4 details the full topology between inputs and outputs, and the number of parameters. This architecture imposes a large number of free control parameters that need to be optimised. In the present case, this means 1074 parameters.

IV. RESULT

In this Section, we conduct an experimental study of the HIT algorithm. Values for all experimental parameters can be found in Table I, including environmental, neural network controller and HIT parameters. The rectangular arena used is represented in Figure 5, with 150 robots and 100 items.

A. Qualitative and Quantitative Evaluation

We analyse the dynamics of HIT while solving the foraging task for a well chosen set of meta-parameters (See next Section for an extensive analysis of the α and σ meta-parameters). To analyse the results, we define two notions: Trajectories of agents at different steps. Robots are initialized with random policies. We can therefore observe a lot of curved trajectories. Robots that achieve high translation will get more rewards and will better propagate their parameters in the population. After convergence to a unique policy, we observe large translation for all agents.

Final value

Characteristic time

• the final score: to assert the quality of a particular algorithm, we measure the score after convergence;

• the characteristic time: to evaluate the speed of convergence, we measure the time at which the average score is half the final score. This is approximately the position of the inflexion point of the data sequence that plots the median score. This was chosen a posteriori as in all the experiments we conducted, we always observed a sigmoid-like increase of the score when switching from the initial low score values to the final score values. Figure 6 compiles the results obtained with 128 replicates of the HIT algorithm, with α = 0.8, σ = 0.001, T = 400.

Starting with low values, the score increases between 40000 and 100000 steps (characteristic time ∼70000), and then converge to a stable value (final score of ∼2.4).

It should also be noted that the variance after convergence is rather small, advocating for the robustness of the algorithm. This is actually confirmed by comparing HIT with a canonical state-of-the-art embodied evolution algorithm, which we refer to as VanillaEE [START_REF] Fernandez Pérez | Comparison of Selection Methods in On-line Distributed Evolutionary Robotics[END_REF], [START_REF] Hart | Improving Survivability in Environment-driven Distributed Evolutionary Algorithms through Explicit Relative Fitness and Fitness Proportionate Communication[END_REF], [START_REF] Montanier | Behavioural Specialization in Embodied Evolutionary Robotics: Why so Difficult[END_REF].

Figure 8 shows the results in term of characteristic times and final scores of two variants of the HIT algorithm (HIT(α = 0.3), which is expected converge slower, and HIT(α = 0.8), as shown before) and of the best-shot of VanillaEE we could find (i.e. σ = 0.001 mutation rate, elitist selection). To account for the implementation difference between HIT and VanillaEE1 , we re-evaluate the final scores by extracting the control parameters from the last generation, and then running these with the learning algorithm deactivated. All claims are backed using Mann-Whitney U Test.

Firstly, the final score is roughly similar for both HIT variants, which is expected. Both display also an advantage over the VanillaEE control algorithm. Secondly, the characteristic time shows, as expected, that HIT(α = 0.3) provides the slowest convergence speed. HIT(α = 0.8) and VanillaEE converge faster, which is actually true on average. However, VanillaEE displays a higher variance both in terms of convergence speed and final scores when compared to HIT(α = 0.8).

Finally, we take a closer look at one typical run of HIT(α = 0.8). randomly initial policy parameters, which produces turning or non-moving behaviours, robots gradually learn to move around to explore their environment, which is both efficient with respect to foraging and mating.

This change in behaviour is also captured by looking at the evolution of the amount of communication during the course of evolution. For this typical run, Figures 10 and9 respectively show the evolution of score and number of messages exchanged through time. Comparing the two figures reveals that the increase in communication between robots actually precedes the increase of score values, and remains stable throughout the experiment, even before the final score value is reached.

B. Tradeoff between Speed and Accuracy

Both the transfer rate α and the mutation size σ can have an impact on learning speed and quality. In this Section, we provide an extensive analysis of these two meta-parameters. We measure the characteristic time and the final score the policies obtained by the HIT algorithm, for a large combination of α and σ values.

We explore σ values from 0.2 to 1.0 with a step size of 0.1, and σ from 10 -2 to 10 -8 , and 0. This represents a total of 18432 independent runs (9 × 8 combinations, 256 replicates per combination) 2 . In order to minimise to computational cost, 0.00 0.04 0.08 0.12 0.16 0.20 steps we use Perceptron with no hidden layer, but with additional sensory inputs that do not bring useful information.

Results are shown in Figures 11 (final score) and 12 (characteristic time). We can observe that the average score plateaus for mutation rates with σ < 10 2 and a transfer rate with α ≤ 0.9. Large mutation rate of σ > 0.01 as well as the maximum value for transfer rate α = 1 are both destructive, whatever the other meta-parameter value. Regarding mutation, it is expected that a too large mutation rate can disrupt selection, and injects noise rather diversity that can be exploited. As for transfer rate, setting α = 1 limits convergence to full control parameter sets present in the initial population only, as it is does not allow to benefit from recombination.

Aside from these extremes, HIT is revealed to be remarkably robust in terms of the final score that is reached. The transfer rate appears as the main parameter to modulate the convergence speed, with mutation being either destructive in the worst case σ ≥ 10 2 , or of limited interest. In terms of efficiency of policies, the algorithm is rather robust with respect to its meta-parameter values, as long as extreme values are avoided (α > 0.9 and σ ≥ 10 2 ).

Final value

Characteristic time

C. Learning the Transfer Rate

As stated at the beginning of this paper, our aim is to provide an algorithm that can cope with limited communication capabilities. However, it is hardly possible to know in advance what are the limits. Technical specifications provide a good start, but the impact of robots density on packet collisions or perturbations from other sources makes it difficult to set an appropriate transfer rate beforehand.

Here, we study the HIT algorithm in a pseudo-realistic context where communication is artificially limited. In this experimental setting, all messages larger than m × 0.6 × size(parameter set), are lost. The optimal transfer rate is therefore α opt = 0.6, but is unknown before deployment 3 .

Resilience to communication failure is addressed by implementing a straight-forward learning method for the transfer rate. The transfer rate value is incorporated in the parameter set of each robot, and initially set to a random value α ∈ [0, 1]. It is then tuned by selection pressure.

We performed 64 experiments, setting a mutation rate to σ = 0.001 only for the parameter α and T = 400 (as earlier). For all experiments, the whole robot swarm systematically converged to an α value close to the optimal transfer rate as shown on Figure 13 (left).

Histogram of average

Canonical run, This is illustrated in Figure 13 (center), which represents the distribution of all α from the 150 robots across time steps for a typical run. As expected, α values are initially uniformly distributed between 0 and 1, and then quickly converge to a value close to α opt .

Robots carrying α values which are too large with respect to the communication constraints simply cannot spread, and can only survive as long as it takes to encounter a robot with better score and capable of transmitting its own parameters. On the other hand, robots with small values of α can spread, but may do so at a smaller frequency than robots with relevant α values. This can be viewed as a lexicographic selection: the better performing robot will always fare better, and will diffuse faster if its α value makes the best of the environment at hand.

From a practical viewpoint, and extrapolating also from results in the previous section, it means that learning the transfer rate is always a good idea as long as it is limited to be strictly inferior to 1.

V. CONCLUSION

In this paper, we introduced a new algorithm called Horizontal Information Transfer, which is at the crossroad of embodied evolution and social learning. We showed that this algorithm is competitive with the state of the art, but is also able to deal with limited communication capabilities that are often met with low-cost robots used in swarm robotics.

The obvious next step is to run full experiments with a swarm of low-cost robots for which preliminary results (not shown here) are still limited but encouraging.

Fig. 1 .

 1 Fig. 1. Principle of embodied evolution algorithms. The robotic agent sends and receives parameters of a policy function, and uses a evolutionary algorithm to improve it own active parameters.

Algorithm 1 : 5 R 6 a

 156 The HIT algorithm (Horizontal Information Transfer) Data: α : transfer rate ∈ [0, 1], T : evaluation time, π : Policy function, θ : Random uniform initialisation of policy parameters, dim(θ) = m, R[T ] : Empty reward buffer of size T , r : Null reward scalar, a : Null action vector, o : Null observation vector 1 [t mod T ] = r = π(o|θ) 7 act(a) 8 if t > T then 9 G = T -1 k=0 R[k] 10 Create the Idx array by drawing randomly αm integers in range [0, m -1] without replacement 11 broadcast(θ[Idx], Idx, G) 12 if new message then 13 θ = TRANSFER(θ, G, Idx message ,θ message , G message )

Algorithm 2 :

 2 Transfer function Data: θ: active parameters, G: current evaluation, Idx message : received parameter indexes, θ message : received parameters, G message : received evaluation 1 begin 2 if G message > G then 3 for i ∈ Idx message do 4 θ[i] = θ message [i]

Fig. 3 .

 3 Fig. 3. Robot model with 16 sensors, a velocity v, a maximal speed vmax, an angular speed ω and a maximal angular speed ωmax. The control variables are then a 0 and a 1 .

::Fig. 4 .

 4 Fig.[START_REF] Bredeche | Embodied evolution in collective robotics: A review[END_REF]. Architecture of the policy function. Each sensor information about the distance to obstacle and the type of object detected, if any. The architecture represented here is a multi-layered Perceptron, used for the experiments. It has 64 dimensions as input, a hidden layer of 16 dimensions plus a bias term for each layer. The action dimension is 2. The total number of parameters is 1074.

Fig. 5 .

 5 Fig. 5. Arena used for the experiments. It features 150 robots (small blue dots) with 16 short-range sensors and 100 items (green dots). Items disappear when caught, to reappear at a new random location. Robots are never relocated, and the HIT algorithm runs as a distributed on-line fashion.

Fig. 6 .

 6 Fig.6. Results with HIT, α = 0.8, σ = 0.001, T = 400. The parameters used for these simulation are described on TableI. The Y-axis represents the distribution of the average score among all 150 agents, over all runs. Results are compiled from 128 independent simulation runs.

  Fig.7. Trajectories of agents at different steps. Robots are initialized with random policies. We can therefore observe a lot of curved trajectories. Robots that achieve high translation will get more rewards and will better propagate their parameters in the population. After convergence to a unique policy, we observe large translation for all agents.

1 stepsFig. 8 .

 18 Fig. 8. Comparison between HIT and VanillaEE all with σ = 10 -3 . The box plots are computed with 80 independent runs. The characteristic time measures the step at which the average score reaches half its final value. The final score is the average score after saturation.

Fig. 9 .

 9 Fig. 9. Evolution of the frequency of messages for one experiment, α = 0.8, σ = 0, T = 400. We count the number of message sent in a constant interval of 400 iterations and report this value as a function of the step number

Fig. 10 .

 10 Fig.10. Evolution of the average score among 150 agents for one experiment of foraging 100 objects, α = 0.8, σ = 0, T = 400. At every encounter, the best agent share half randomly chosen parameters to the other agent. The worst receives these parameters and use them to improve it own policy. This process can only happen after both agent have spend at least 400 steps to evaluate themselves. During the maturation period, the score before modification of the best is reported for the averaging. No mutations are used here.

Fig. 11 .

 11 Fig. 11. Final value of the average score for various mutation size and transfer rate. This value is the average of all score on the last 100 measurements (one measurement per 400 iterations). 256 independent simulation run have been used. Lighter colour means better saturation value.

Fig. 12 .

 12 Fig. 12. Characteristic time for various mutation size and transfer rate. The characteristic time is the step at which the average score reach half of it final value. 256 independent simulation run have been used. Lighter colour means shorter convergence time.

Fig. 13 .

 13 Fig. 13. Left: distribution on 64 independents runs of the average transfer rate α after 5 × 10 5 iterations, with αopt = 0.6. Center and Right: learning the transfer rate α in a canonical run. Center: each line represent the trajectory of the α value for one specific agent. Right: average score for the population until convergence.

The original implementation of VanillaEE is synchronous, meaning that all robots update their policy at the same time, which HIT does not do.

Experiments took 12 days using an Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz

We also tested for α = 0.3 with identical results (not shown here).

ACKNOWLEDGEMENT

This work is funded by the Agence Nationale pour la Recherche under Grant No ANR-18-CE33-0006.


Keywords: Passengering, Subjectivities, Mobilities, Time-spaces, Transport design, Transport planning 1. Geographies

At the intersection of transport geography and mobilities studies, scholars have paid increasing attention to passengering as a key characteristic in transport systems and a window through which one can understand how transport unfolds on the ground. In this editorial to the virtual special issue, we contribute to these debates by thinking through how passengers and passenger groups are deeply heterogeneous in nature, being far from singular or discrete, and escaping easy definitions of what passengering is or does. We discuss such variability in passenger formations and roles in three ways. First, we consider how passengering is a nonreplicable process involving different compositions of people each time. Then, we highlight passengers' inconstant roles and subjectivities while on the move. And finally, we delineate how the act/art of passengering can be extended across multiple timespaces involving and exceeding immediate transport environments. We show that these three prompts have important implications for transport infrastructures and services, as well as for fotransport design and planning. By tackling these ideas, the papers in this issue offer new insights on the spatialities of transport and on the site-specific productions of passengering.

Over the last decade, the subject of passengering has come to the fore of transport studies. While appearing nondescript and mundane, the activity constitutes a significant part of transport experiences for many people, and is an important consideration in the daily practicalities of a person's travels (see, for example, [START_REF] Binnie | Mundane mobilities, banal travels[END_REF]. In recent years, both academics and practitioners have taken this topic into the planning domain, turning micro transport spaces such as stations, platforms and the interiors of transport vehicles-from aircraft to buses-into objects of design and improvement (see [START_REF] Fernandes | Characteristics and human factors of older drivers: improvement opportunities in automotive interior design[END_REF][START_REF] Rui | Themed passenger carriages: promoting commuters' happiness on rapid transit systems through ambient and aesthetic intelligence[END_REF][START_REF] Vink | Aircraft Interior Comfort and Design[END_REF]. With this increasing interest in the minutiae of passengering, transport studies have also gradually opened up a new line of inquiry that used to be on the margins of more conventional concerns such as networks and nodes. Within geography, there has been a growing attention to passengering as well. In transport geography specifically, one notable collection of work is found in Bissell et al.'s (2011) special issue (in this journal) on the topic. Here, the authors lay out a foundational, socio-cultural framework for thinking about and researching the 'figure of the passenger'. This discussion spans four themes. The first pertains to the social nature of passenger experiences; the second focuses on processes and practices that produce the passenger; the third considers the material affordances of transport experiences; and, lastly, the fourth explores the relationship between passengers and transport structures. For the authors, training an analytical lens on the social production of the passenger affords expository insights into how quotidian spaces are experienced and constructed on a daily basis in motion (ibid: 1008). It adds a thick sense of how transport spaces are actually lived and inhabited, while giving salience to the role of institutions and governance regimes in framing transport use [START_REF] Lin | Transport provision and the practice of mobilities production[END_REF].

Relatedly, a second corpus of work more closely allied with mobilities studies can extend Bissell and colleagues' arguments. In this broader literature which charts multiple empirical studies, scholars have variously unpacked 'passengers' for their diverse typologies, as well as the different senses of being 'on the move'. In particular, interrogating the movements of economic streams such as businesspersons [START_REF] Faulconbridge | The executive[END_REF][START_REF] Unger | The business travel experience[END_REF], commuters [START_REF] Bissell | Passenger mobilities: affective atmospheres and the sociality of public transport[END_REF][START_REF] Hislop | Driving, communicating and working: understanding the work-related communication behaviours of business travellers on work-related car journeys[END_REF] and tourists [START_REF] Cohen | Lifestyle mobilities: the crossroads of travel, leisure and migration[END_REF], as well as social types like families [START_REF] Vannini | Mobility, ritual and performance: an ethnography of parents', children's, and youths' ferry boat travel[END_REF], students [START_REF] Symes | Coaching and training: an ethnography of student commuting on Sydney's suburban trains[END_REF] and migrants [START_REF] Abranches | When people stay and things make their way: airports, mobilities and materialities of a transnational landscape[END_REF][START_REF] Sirriyeh | Home journeys: Im/mobilities in young refugee and asylum-seeking women's negotiations of home[END_REF], the corpus has outlined the myriad mobile practices, social experiences and corporeal affects that animate the act/art of passengering in different contexts. Collectively, this research affirms Adey et al.'s (2012: 173) assertion that 'passenger experiences are multiple, heterogenous and are mediated along many different lines'. At the same time, it also attests to the notion that passengering is an important extension of transport systems, capable of affecting journeys and their accomplishment by the things that passengers 'do'.

It is at the intersection of these two bodies of work-transport geography and mobilities studies-that new insights about the spatialities of transport have also begun to develop (see [START_REF] Shaw | Transport, geography and the "new" mobilities[END_REF][START_REF] Shaw | Making links: on (re)engaging with transport and transport geography[END_REF]. On the one hand, there is now a more intimate grasp of the subjective agencies at work in the micro spaces of transport, extending existing understandings on travel time use and transport behaviours [START_REF] Jain | The gift of travel time[END_REF][START_REF] Tranter | Children's travel behaviours in Canberra: car-dependent lifestyles in a low-density city[END_REF][START_REF] Van Acker | When transport geography meets social psychology: toward a conceptual model of travel behaviour[END_REF]. On the other hand, by attending to how different passengers live out their (im)mobilities practically, a focus on passengering has productively laid bare the ways in which transport is a highly unequal resource. Indeed, well-networked transport systems do not guarantee universal passenger access, comfort, or experience, as some may be denied participation and/or channelled to slower lanes (see [START_REF] Fransen | Identifying public transport gaps using time-dependent accessibility levels[END_REF][START_REF] Martin | Desperate passage: violent mobilities and the politics of discomfort[END_REF]. These comprehensions arguably lead to a more fine-toothed understanding of how transport unfolds on the ground-not as abstract models or grant planning analyses, but lived spaces where flesh-and-blood dwells.

Heterogeneous passengers and passengerings

Despite these illuminating insights, we argue that existing research has yet to fully address how passengering is a much more complex and heterogeneous affair than simply having multiple (but disparate) subjectivities and experiences. This is not to downplay the current attention to different subtypes of passengers, but rather to acknowledge that any passenger or passenger group is far from singular or discrete. Indeed, variable axes of difference may overlap in the same setting, appear in the same body, or stretch across multiple contexts at once, resulting in complex situations that escape easy definitions of what passengering is or does. As one of us argues elsewhere with respect to frequent air travellers, passenger personas, practices and behaviours tend to elude easy qualifications, or quick archetypal solutions to subsume them and all they entail [START_REF] Frétigny | L'hypermobilité: une figure introuvable ? SociologieS (2 November)[END_REF]. Admittedly, such a stance complicates theorisations on the subject; but there is also value in uncovering how such unusual configurations may impinge on transport planning decisions in nonconventional ways. Instead of grouping people according to their mode of travel, social role or economic class, this issue emphasises the fluid role that each passenger can, to some extent, play in (re)creating and changing mobile situations on the ground. Building on Bissell et al.'s (2011) arguments, we seek to open the field up to a greater latitude of possibilities and diversities, and allow transport to play out in more flexible and creative ways.

To give more substance to this reorientation, we consider three ways in which such fluidities may arise. The first relates to how, for most transport modes, passengering is a heterogeneous and nonreplicable process, involving different compositions of people each time. Due to heuristic reasons, most studies have tended to approach passengering as generic experiences shared by segments of, if not all, transport users onboard a vehicle. From car rides, to train and bus journeys, to airline jaunts [START_REF] Bissell | Virtual infrastructures of habit: The changing intensities of habit through gracefulness, restlessness and clumsiness[END_REF][START_REF] Edensor | Rhythmanalysing the coach tour: the Ring of Kerry, Ireland[END_REF][START_REF] Jain | The making of mundane bus journeys[END_REF][START_REF] Waitt | Parenting, care and the family car[END_REF], there is a tacit assumption that passengering experiences are largely shared as collective, more-or-less stable undertakings among members of a mobile community. Commonly, these writings allude to senses of national or subcultural belonging (see [START_REF] Burrell | Going steerage on Ryanair: cultures of migrant air travel between Poland and the UK[END_REF][START_REF] Wilson | Passing propinquities in the multicultural city: the everyday encounters of bus passengering[END_REF]; at other times, passengers are divided up into broad groups identified by their socio-economic class or race (see [START_REF] Bissell | Unveiling seductions beyond societies of control: affect, security, and humour in spaces of aeromobility[END_REF][START_REF] Hirsh | Airport Urbanism: Infrastructure and Mobility in Asia[END_REF]. There is of course merit in such generalisations, but using these frames repeatedly can also lead to inadvertent omissions of other markers of difference such as gender, age, disabilities and body weight that often become salient in unexpected ways in a transport situation. Indeed, some studies have interrogated such manifestations of multidimensional assemblages (see [START_REF] Kloppenburg | Confined mobilities: following Indonesian migrant workers on their way home[END_REF][START_REF] Wilson | Passing propinquities in the multicultural city: the everyday encounters of bus passengering[END_REF], but the intersectionality at play in the power geometries of passengering remains largely to be written.

In lieu of a homogenous view of passenger experiences, there can thus be a greater stress on the relational instabilities of transport spaces, as passenger communities of different persuasions form and collide with each other. Not only is this awareness key to recognising the variability in passenger formations, it also alerts one to the material effects that people of varying comports, habits, dispositions, and (non)agencies coming together can have on transport systems, entangling them in friction and (temporary) disruption. Take for example [START_REF] Small | Crying babies on planes: aeromobility and parenting[END_REF] cue on the angst and degradation of travel time when babies cry on airliners; or Bissell's work on the encumbrance that unexpected artefacts like luggage and bicycles can cause other passengers on trains; or Muñoz's (2020) contemplation of the incompatibility of some infrastructure when 'older', wheelchair-bound and 'fat' passengers try to board the bus. Each of these coalescences of bodies and subjects must be recognised as part of a unique actor-network that holds-or fails to hold-transport systems together in that instance. They impel a rethinking of transport design that goes beyond the provision of material infrastructure for one generic group, and a need for more adaptable spaces that can accommodate different eruptions and permutations of heterogeneity in transport. Against the backdrop of increasing expectations to integrate myriad users in transport services (as often seen in public committees and participatory planning), these kaleidoscopic conceptions of passengers' traits have important implications for future transport planning. Although filled with challenge, they help inch transport operators, authorities, civil society and academics alike towards conceiving transport infrastructures that are truly shared, and engaging with what Nikolaeva et al. ( 2019) have termed 'commoning mobility'.

As a second prompt, passengering can also be a varied phenomenon because people's subjectivities are not constant while on the move. Indeed, over the course of a single journey, an individual can take on, and oscillate between, different roles in quick succession, according different qualities and meanings to their movements. It may seem trivial if someone decides to sleep, eat or work during rides [START_REF] Jain | The classy coach commute[END_REF][START_REF] Laurier | Doing office work on the motorway[END_REF], but, a passenger slipping in and out of character has significant consequences for what -and who -exactly constitutes 'being-in-transit'. Consider frequent fliers who vacillate between being elite passengers and aviation enthusiasts seeking to experience airport lounges with no intention to fly [START_REF] Zuskáčová | Elite diversities in practice: the case of frequent flyers in the Czech Republic and Slovakia[END_REF]; or urban commuters who make stops for non-transport reasons, truncating their trips and/or folding in other roles to result in nestled journeys [START_REF] Kitamura | Does a grande latte really stir up gridlock? Stops in commute journeys and incremental travel[END_REF]. These deviations make passengering harder to pin down as a concept, seeing that identities can be fluid and potentially switch at will. In a loopback to an earlier point, they furthermore create additional axes of difference in communities of travellers, where some have more agency to transform their experiences than others. To the extent that these fluctuations have real effects, they also have to be factored into planning decisions. From infrastructural provision to peak travel distribution, passengers' inconstant subjectivities only serve to complicate how the geographies of transport pan out. Their understanding can contribute to orienting policy approaches towards new conceptions of 'flexi-mobility' [START_REF] Chatterton | Flexi-Mobility[END_REF], i.e. promoting forms of transport that are more adept at attending to variations in everyone's uses of travel modes.

In starker cases, the boundary between passengers and drivers-often perceived to fulfil binary functions in transport-can also be blurred because of this oscillation of roles. Adey et al. (2012: 172) allude to this notion when they argue that the dichotomy between the former (as a 'withdrawn body' being transported passively) and the latter (as a 'more powerful and commandeering figure ') is a false one and may not stand to scrutiny in real life. Indeed, passengers do not all suffer from a lack of agency (especially in intimate settings like a car), neither do drivers carry out their responsibilities separately from those they ferry. In fact, it is plausible for both personas to reside in the same body, as a single individual plays the part of both 'passenger' and 'driver' to achieve certain 'mobility possibilities' [START_REF] Dant | The driver and the passenger[END_REF]. Referring to the term 'rider', conventionally used to reflect each of these roles, could help make our imaginations more attuned to this ambiguity. Most clearly seen in the rise of on-demand transport services and autonomous vehicles, such mergers of characters are likely to intensify as technologies advance [START_REF] Herrmann | Autonomous driving: how the driverless revolution will change the world[END_REF][START_REF] Litman | Autonomous Vehicle Implementation Predictions[END_REF]. As subjects glide seamlessly from one role to the other, important questions would need to be raised about the future of transport, as well as networks and nodes increasingly determined by the user.

A third trope we want to pursue pertains to how the meaning of passengering draws from an extended geography of heterogeneous values lying both within and beyond transport systems. Indeed, though a performance that appears to be locked in time and space, the act of passengering practically invokes a wide variety of disconnected terrains, as people tend to bring with them memories, habits, experiences and social identities drawn from non-mobile situations [START_REF] Bissell | Transit Life: How Commuting Is Transforming our Cities[END_REF]. Having these values bleed across different time-spaces not only necessitates an extra-contextual understanding of transport; it also requires an attention to how passengers embody lifeworlds that exceed immediate transport environments. In this sense, the figure of the passenger is not just composed from a blank slate, beginning and ending where the journey does. Instead, it is a simultaneous expression of both the present and a long genealogy of elsewhere and elsewhen.

Such spillovers across geographies and temporalities portend both a politics of, and a potential in, passengering. On the one hand, transport systems can sometimes serve as an offshoot of existing social hierarchies. Expressly, they can play a vital role in reinforcing otherness, not because passengers are innately different, but because wider social prejudices-e.g. of racism, intolerance for foreignersare allowed to transfer and fester within transport's confined spaces [START_REF] Wilson | Passing propinquities in the multicultural city: the everyday encounters of bus passengering[END_REF][START_REF] Lobo | Everyday multiculturalism: catching the bus in Darwin, Australia[END_REF]. On the other hand, the propensity in passengers to import external time-spaces while on the movewhether certain moods or certain personal encounters prior to travel-can also create opportunities for experimentations and the forging of newfound camaraderie with fellow travellers (see [START_REF] Cervero | Mobility niches: jitneys to robo-taxis[END_REF][START_REF] Crang | Transient dwelling: trains as places of identification for the floating population of China[END_REF][START_REF] Higate | Ex-servicemen on the road: travel and homelessness[END_REF]. In this context, transport spaces are like crucibles that fold into themselves social experiences and knowledge from the outside, through the body of the passenger. Insofar as these infusions lead to new heterogeneous reactions and possibilities, they are also poised to enrich the micro geographies of transport.

The papers

The three prompts above seek to diversify current understandings of passengering in transport geography, by further dissecting and nuancing the namesake figure. The papers in this issue are specially curated to reflect on these finer heterogeneities-of communities of travellers sub-divided into more dynamic distinctions than generic modal or socio-economic types; of passengers capable of assuming multiple roles and characters in the course of a journey; and of social subjects prone to connecting mobile experiences with those beyond transport. Each of these folds considers how the spatialities of transport might change with these new travel propensities, and raises important implications for transport planning. The papers in this issue tackle these ideas and questions across land (road and rail), sea and air transport. They further traverse through a spectrum of spatial terrains and realms, drawing insights from the airspaces of Finland, to the southeast coast of Newfoundland in Canada, to urban milieus in the UK, New Zealand, and Japan. Such contrasted geographies contribute to a better understanding of the site-specific productions of passengering.

Exactly cognisant that passengers are not all uniform within their travel group or class, Sysiö (this issue) opens the discussion with an examination of diplomatic air travel. Using the Finnish foreign service as a case study, his work points to the fact that air passengers do not always neatly fit into airline-designated service levels (e.g. ambassador or business class for diplomats); even if they are in the same cabin class, neither are they exactly the same generic type. Specifically, the paper notes that diplomats span across multiple ranks, and, as such, some of them often travel in economy class. Moreover, in a reflection of how multiple subjectivities can reside in a passenger, Sysiö's work also highlights how diplomats often use the aeroplane as a space to engage in work and relaxation, diminishing 'the functional difference' between the office, home spaces and a moving cabin (p. 93). What is even more interesting is the way in which diplomats appear as passive subjects while on the aeroplane, but in fact have played an active role in negotiating international air services agreements that made the service they partake in possible in the first place. These insights represent a departure from purely experiential analyses that have dominated literature on air passengering. In addition, they signal a need to understand the synergistic connections between air transport systems and (some of their) passengers.

Turning to another expression of heterogeneity, this time in urban travel, Doody's (this issue) paper examines the contours of passengering among New Zealander expatriates in London. His work is not only unique in that it interrogates the travels of a niche-and highly skilled-migrant group in the city; it is also distinctive for its use of a biographical approach to trace the way they use urban transport. In particular, his paper highlights how passengering is heavily influenced by specific events and life stages that passengers underwent/undergo prior to travel. He argues that these 'small-scale, ephemeral, subtle and multi-directional processes, events, actions and decisions' have 'cumulative, amplifying and transformative effects' on the ways in which they inhabit transport spaces (p. 2). Specifically, the paper illustrates how the migrants' travel habits are gradually changed through their everyday interactions with support groups in the host society, and the transport systems themselves. Doody's paper offers a clear case of how passengering is a culmination of multiple time-spaces, and not an impromptu act that happens while in transit. It recognises how transport design must increasingly account for such variable customs and habits that exceed the local, in global cities like London and beyond.

Equally sensitive to the diffuse time-spaces in passengering, Negishi and Bissell's (this issue) paper approaches the extra-contextuality of transport through the lens of imagination. Via a study on urban railway travel in Tokyo, their work demonstrates how transit experiences are highly generative of new senses of self, as passengers are suspended in a liminal zone with time to (re)evaluate their subjectivity. The paper begins by looking into literary explorations of urban transport spaces, positing that mobility provides an opportunity for passengers to transcendentally change their outlooks and escape both the physical setting of the railway cabin, and also the socio-cultural conditions in which they are embedded (e.g. long work hours, and the mundaneity of life). Using autoethnographical techniques to decipher how a rail advertisement can evoke memories and recollections in that respect, the paper highlights how imagination constitutes a potentiality in transport spaces that can either align the passenger with the transport provider's will, or alter the mood and feel of being a passenger in more liberating ways. Concerned with these minutiae in travel, the piece presents a micropolitical perspective that stresses how imagination has the power to reshape experiences of passengering in heterogeneous and surprising ways. At the same time, it is a reminder to transport designers to take heed of the material environments that they create, so that transport spaces open up, rather than shut down, the positive potentials of imagination. The final paper by Royal and Roseman (this issue) tackles the question of heterogeneity through a thoughtful contemplation on ferry commuters' gendered relationships while in transit. Drawing on ethnographic fieldwork with residents of ferry-reliant Bell Island, Canada, their work develops the concept of 'co-passengering' to evince how riding the ferry is a rich act in mutual assistance and social support. In particular, they highlight how co-passengering often involves sociable extensions beyond the ferry commute itself, invoking activities such as shared car rides (to/from the pier), collective schedule planning, and even home gatherings and visits among fellow passengers. Such social relationships play out in more fine-grained ways onboard the ferry, as co-passengers engage in mutual care, leisure activities and group emotional support in highly gendered and distinguished ways. Royal and Roseman's work not only underscores again the nonuniformity of passengers in a transport mode. It also critically shows, like the other papers before, the intersecting lines of relations that render each passenger always multiply imbricated in many subjectivities at once, as well as the inter-connections between many (im)mobile others. Such is the messiness of the sociality of transport worlds, requiring our continued attention to their constant (re)configuration and evolution, in order to come to grips with their manifold appearances. Perhaps as a continuing thread in this story, the health crisis of COVID-19 and its profound impacts on transport have already begun another mutating turn in these ever-changing geographies of the passenger.