Guglielmo Fernandez
email: guglielmofernandez@gmail.com

Thomas Corpetti

Marie Nevoux

François Laurent Beaulaton

Garcia Fernandez

Guglielmo

Martignac François

Nevoux Marie

Corpetti Thomas

Garcia Fernandez

AcousticIA, a deep neural network for multi

Keywords: ARIS, artificial intelligence, convolutional neural network, DIDSON, fisheries management, sonar camera

Recent technological innovations largely increase the capacities for scientists to collect at high frequency in situ data. Consequently, ecology, like other branches of the biological and Earth sciences, has entered a world of big data, and is confronting the attendant opportunities and challenges [START_REF] Farley | Situating Ecology as a Big-Data Science : Current Advances, Challenges, and Solutions[END_REF][START_REF] Guo | Big Earth data : A new frontier in Earth and information sciences[END_REF]. To convert high-resolution data into high-value data, the development of optimized methodological approaches to extract relevant information in a timeand cost-effective way is a key step. This main challenge is common to all new numerical devices, in all ecosystems [START_REF] Valdenegro-Toro | Submerged Marine Debris Detection with Autonomous Underwater Vehicles[END_REF]. In aquatic environments, multi-beam high frequency acoustic or sonar cameras belong to the newest generation of hydroacoustic devices used for fish monitoring and species identification thanks to their high frequency and large detection beam [START_REF] Foote | Acoustic Methods : Brief Review and Prospects for Advancing Fisheries Research[END_REF][START_REF] Martignac | The use of acoustic cameras in shallow waters : New hydroacoustic tools for monitoring migratory fish population. A review of DIDSON technology[END_REF]. These sonars record continuously data at very high frequency (e.g. up to 10 frames per second), leading to data storage issues and substantial time dedicated to data analysis [START_REF] Martignac | The use of acoustic cameras in shallow waters : New hydroacoustic tools for monitoring migratory fish population. A review of DIDSON technology[END_REF], but they can overcome many unsolvable operational limits of optical video-counting methods: the presence of a solid structure restricting the fish passages and an imperative light source, consequences of a very limited detection range. Acoustic cameras are relevant alternatives but the nature of the images is remarkably different from common underwater optical imaging [START_REF] Kocak | A Focus on Recent Developments and Trends in Underwater Imaging[END_REF][START_REF] Lu | Underwater Optical Image Processing : A Comprehensive Review[END_REF][START_REF] Wang | An Experimental-Based Review of Image Enhancement and Image Restoration Methods for Underwater Imaging[END_REF]. Although acoustic camera is efficient and high potential device to diverse applications in ecology science, the lack of solution for automatic analysis of the data [START_REF] Helminen | Object and behavior differentiation for improved automated counts of migrating river fish using imaging sonar data[END_REF][START_REF] Shahrestani | Detecting a nearshore fish parade using the adaptive resolution imaging sonar (ARIS) : An automated procedure for data analysis[END_REF], greatly limits its deployment.

Computer Vision (CV) techniques, ranging from classical image analysis and machine learning (as Support Vector Machine, SVM, or Random Forests, RF) to deep learning techniques, have opened the possibility to automatically identify and classify objects of interest, offering tools that have displayed high efficiency and repeatability [START_REF] Lecun | Deep learning[END_REF]. Such methods can answer the challenge of extracting valuables informations from big data also in ecology. Indeed, in recent years there has been a remarkable growth of the using of such methods for image data analysis [START_REF] Christin | Applications for deep learning in ecology[END_REF][START_REF] Kwok | Deep learning powers a motion-tracking revolution[END_REF]: from camera-trap species detection [START_REF] Beery | Recognition in Terra Incognita[END_REF][START_REF] Norouzzadeh | Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning[END_REF][START_REF] Schneider | Deep Learning Object Detection Methods for Ecological Camera Trap Data[END_REF][START_REF] Weinstein | A computer vision for animal ecology[END_REF] and photo-identification of individuals [START_REF] Miele | Revisiting giraffe photo-identification using deep learning and network analysis[END_REF] to video tracking [START_REF] Bonneau | Outdoor animal tracking combining neural network and time-lapse cameras[END_REF], these methods are paving the way to automation also in the wide range of conditions that are of interest in ecology, such as night/day cycle, partial exposition of the object, different meteorological conditions. Most of these methods are developed on optical data [START_REF] Weinstein | A computer vision for animal ecology[END_REF], while sonar imaging possess several different features [START_REF] Lankowicz | Sonar imaging surveys fill data gaps in forage fish populations in shallow estuarine tributaries[END_REF] that makes the direct application of these models impossible and the development of an analysis tool a more challenging task: for examples low signal-to-noise ratio, a large spectrum of noise frequencies and a low resolution of the objects of interest. These characteristics already make sonar imaging challenging for automatic analysis, but each camera model exhibit slightly different features, adding the feat of creating multi-camera models. It has to be stressed that answering to these challenges might also lead to new insights on how to solve similar complex problems in ecology (e.g. noisy data). First experiments to automate the analysis have been conducted in other water-related fields such as AUV guidance systems [START_REF] Zacchini | Forward-Looking Sonar CNN-based Automatic Target Recognition : An experimental campaign with FeelHippo AUV[END_REF] or underwater litter object detection [START_REF] Valdenegro-Toro | Submerged Marine Debris Detection with Autonomous Underwater Vehicles[END_REF]. Concerning fisheries management a few attempts have emerged recently [START_REF] Christensen | Deep Learning based Segmentation of Fish in Noisy Forward Looking MBES Images[END_REF][START_REF] Lee | Deep Learning from Shallow Dives : Sonar Image Generation and Training for Underwater Object Detection[END_REF][START_REF] Tarling | Deep learning with self-supervision and uncertainty regularization to count fish in underwater images[END_REF]. However the latter are limited or to single camera studies, or to synthetic data or to fish counting (without explicit detection): as we will see, this work represents therefore a first multi-camera, multi-species study, based on a large dataset (with a high variability data recorded on a long period of time), that poses a first step towards the development of general methods for acoustic cameras.

Machine learning algorithms require the calculation of some measurable property or characteristic of the images (so called "features") handly designed in advance, making the choice of the "good features" complicated (Jianbo [START_REF] Shi | Good features to track[END_REF][START_REF] Nixon | Feature Extraction and Image Processing for Computer Vision[END_REF]. On the contrary, deep learning algorithms are capable of calculating such "features" without any human intervention. Indeed, in recent years, important gains in performances have appeared in the community of image processing with the use of deep neural networks, whose objective is to construct a neural network composed of a large number of layers composed of elementary functions (neurons that model a simple relation between input/output). The concatenation of layers enables to model potentially very complex relations between inputs (noisy videos in our case) and outputs (detection of fish). Though the idea of combining several layers of neurons is old (H. [START_REF] Wang | On the Origin of Deep Learning[END_REF], the progresses in the recent years come from the fact that we have now enough data and associated computational resources to train such complex networks. In addition, some theoretical progresses on the definition and optimization of such networks have opened a wide range of applications. The reader can find in [START_REF] Lecun | Deep learning[END_REF] a general introduction to deep learning.

As for the processing of spatial data, the state of the art network for assigning a label to an image is the well-known CNN (Convolutional Neural Network). With this family of networks, a series of convolution and pooling layers jointly acts as automatic feature extractors that represent in a very relevant way the information contained in raw data. These features are so informative that complex computer vision tasks (classification, object detection, etc) become simpler. In recent years many architectures, either adapted to assign a label to each pixel (Fully Convolutional Networks and variants [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]), to deal with unstructured data [START_REF] Qi | PointNet : Deep Learning on Point Sets for 3D Classification and Segmentation[END_REF] or to time series [START_REF] Karim | LSTM Fully Convolutional Networks for Time Series Classification[END_REF] for example have been proposed. More recently, the community is also focusing on the fusion of complex data, as for example in the context of satellite images [START_REF] Audebert | Deep Learning for Classification of Hyperspectral Data : A Comparative Review[END_REF].

In the context of acoustic imaging, because of the particular noise, the complexity of such data and their limited spatio-temporal variability, the feature extractor part of deep networks is still limited to distinguish fishes from noise, leading thus to poor performances. As a solution, we propose here to pre-process the data in a specific pipeline, based on Computer Vision, to help the optimization of the network to converge to a better minimum. The preprocessed data are then combined with the original one to provide "multichannel-layers", highlighting the interesting signal. This strategy, that is in general not common in deep learning, has already proven its results on optical underwater imaging [START_REF] Salman | Automatic fish detection in underwater videos by a deep neural network-based hybrid motion learning system[END_REF] and should constitute an original but simple solution to the acoustic imaging problems.

Apart from showing the potential of the developed model, we aim also to provide recommendations for the implementation of this kind of approaches. Particular attention will also be given to the possibility for a technology transfer to users, by testing the model on an ecologically relevant dataset from different types of acoustic cameras. Reaching the goal of automatic data analysis will make possible in the future to count fish passage in real time, which will be an important tool for biodiversity protection and management structures to better understand the interactions between aquatic ecosystems and the human sphere.

. Materials and Methods

1 Deep Learning Approach

We propose a hybrid method that embeds hand-crafted features in addition with raw images, to reduce the signal-to-noise ratio of the sonar data. By doing so, we inform the algorithm on interesting areas and therefore help the optimization on the network. Indeed, as it will be shown in Section 3.1, using raw images only leads to poor results. In practice we compute common CV features used in various domains such as image processing [START_REF] Nixon | Feature Extraction and Image Processing for Computer Vision[END_REF], physics [START_REF] Raissi | Physics-informed neural networks : A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations[END_REF][START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF] or geography [START_REF] Chen | Handbook Of Pattern Recognition And Computer Vision[END_REF][START_REF] Ma | Deep learning in remote sensing applications : A meta-analysis and review[END_REF]. As for the network, we choose an architecture devoted to "object detection", that localises a bounding box and assigns a label to it. In this context the R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], that combines several CNN with selective search, enables to isolate regions of interest. In this work, we used the You Only Look Once (YOLO v3) developed by [START_REF] Redmon | You Only Look Once : Unified, Real-Time Object Detection[END_REF][START_REF] Redmon | YOLOv3 : An Incremental Improvement[END_REF]. While other common architectures use a sliding trained classifier to detect object (as Faster R-CNN [START_REF] Ren | Faster R-CNN : Towards Real-Time Object Detection with Region Proposal Networks[END_REF]), YOLO simultaneously predicts the position, the size and the probability for the object to belong to one class, leading to fast performances without losing accuracy.

Among the objectives of this work, we also aim at testing the feasibility of using well known neural networks, keep the model simple, and help detection by adding information about the image through pretreatment. Reasons why we choose to employ a well-known CNN in our pipeline. Our model should not be understood only as an efficient model for the automation of acoustic camera analysis, but also as an important step before the application of more complex architectures on this type of signal, such as LSTM+CNNs (Long short-term memory) [START_REF] Ning | Spatially supervised recurrent convolutional neural networks for visual object tracking[END_REF][START_REF] Xia | LSTM-CNN Architecture for Human Activity Recognition[END_REF].

Hydroacoustic methods: acoustic cameras

An acoustic camera is a multi-beam high-frequency sonar designed to create high-resolution images. Unlike common hydroacoustic methods, fish's skin and fins are better perceived by acoustic camera's very high frequencies, making centimetric fish measurement possible from the images [START_REF] Martignac | The use of acoustic cameras in shallow waters : New hydroacoustic tools for monitoring migratory fish population. A review of DIDSON technology[END_REF]. According to their intrinsic characteristics (Table S.1), and being the first acoustic cameras available for ecological study purposes, DIDSON (Dual-frequency IDentification SONar; Sound Metrics Corp.) and ARIS (Adaptive Resolution Imaging Sonar; Sound Metrics Corp) are the most commonly used cameras to monitor fish in the wild [START_REF] Cook | Validation of fish length estimations from a high frequency multi-beam sonar (ARIS) and its utilisation as a fieldbased measurement technique[END_REF][START_REF] Helminen | Object and behavior differentiation for improved automated counts of migrating river fish using imaging sonar data[END_REF][START_REF] Keeken | Migration of silver eel, Anguilla anguilla, through three water pumping stations in The Netherlands[END_REF][START_REF] Lagarde | In situ evaluation of European eel counts and length estimates accuracy from an acoustic camera (ARIS)[END_REF][START_REF] Lenihan | Assessment of silver eel (Anguilla anguilla) route selection at a water-regulating weir using an acoustic camera[END_REF][START_REF] Shahrestani | Detecting a nearshore fish parade using the adaptive resolution imaging sonar (ARIS) : An automated procedure for data analysis[END_REF][START_REF] Yang | Fish Monitoring through a Fish Run on the Nakdong River using an Acoustic Camera System[END_REF]. In the context of fish population monitoring in rivers, acoustic camera are horizontally oriented, set on one riverbank looking the other [START_REF] Daum | Use of Fixed-Location, Split-Beam Sonar to Describe Temporal and Spatial Patterns of Adult Fall Chum Salmon Migration in the Chandalar River, Alaska[END_REF][START_REF] Maxwell | The Feasability of Estimating Migrating Salmon Passage Rates in Turbid Rivers using a Dual Frequency Identification Sonar (DIDSON)[END_REF].

The acoustic datasets employed in this study have been recorded on monitoring sites during longterm and continuous surveys. Thus, the datasets include all the aquatic fauna activity in a multispecific population with several species of interest in term of conservation and management of populations, such as diadromous or invasive fish species. DIDSON videos have been recorded on the Sélune River (Normandy, France) and ARIS videos on both Sélune and Touques rivers (Normandy, France). Examples of DIDSON and ARIS images are reported in Figure 1, panel a. The acoustic videos are sequences of contiguous images of more than 15 minutes. For both cameras, fish passages have been recorded at distances ranging from 1 to 17 meters, at a 1.8 MHz frequency (HF for DIDSON; LF for ARIS) at 5 frames per second. The fish species have been identified by an experienced operator on the video datasets, interpreting behavioural and morphological fish characteristics, adding biological expertise such as diadromous fish phenology or trophic behaviours [START_REF] Martignac | The use of acoustic cameras in shallow waters : New hydroacoustic tools for monitoring migratory fish population. A review of DIDSON technology[END_REF]. These videos were used as a basis to elaborate both datasets.

3 Datasets

Two datasets have been used in this study: the "Training Dataset" (TD) to train, evaluate and test the deep learning model on a dataset of images and the "Validation Dataset" (VD), to test the full pipeline on videos with a high abundance of fish passages (more details on the datasets are given in SI).

The TD is composed of 4 817 images, 1 612 from ARIS camera's clips (short sequences where only the images with fishes have been chosen) and 3 205 from DIDSON camera' clips. The clips have been carefully chosen to integrate variability in the TD: different species, thus individual with large ranges of morphology (body shape, undulation, lengths) and behaviour (i.e. swimming pattern), different environmental conditions (i.e. river flow) and extracted from two monitoring sites, different recording window lengths and ranges, different times of year. We ensured that passages of fish species of interest were recorded in the TD: diadromous species, such as the Atlantic salmon (Salmo salar), the European eel (Anguilla anguilla), the sea lamprey (Petromyzon marinus), the Allis shad (Alosa alosa) and an opportunistic predator, the European catfish (Silurus glanis). In the TD, all the images have been annotated with a bounding box and a label corresponding to the species, even if all the labels coalesce during the treatment in a unique "fish" label (see Table S.2 for the distribution of each species in the TD). In order to ensure the accuracy of the model also on sequences where no passage are present, 1 196 images where no fish are present have been added to the test set (428 for the ARIS camera, 768 for the DIDSON camera).

The VD is composed of 184 videos (sequences of more than 15 minutes), 80 for ARIS camera and 104 for DIDSON camera, for a total of around 40 hours of recording. The videos of the VD integrate the same variability as the TD. Each fish larger than 20 cm have been counted and described with the passage hour (20 cm been, according our experience, the minimum size limit to efficiently see and track a fish), the fish size range (four classes: 20-40 cm, 40-60 cm, 60-80 cm and larger than 80 cm), its position in the detection beam, its swimming direction and its fish species if an undoubted identification was possible according its morphology and behaviour (otherwise a "generic fish" label was used), no bounding boxes were saved (see Table S.3 for the distribution of each species in the VD). In 33 clips (6 for ARIS camera and 27 for DIDSON camera), not a single fish passage is present, for a total of around 6 hours. More details on the datasets are given in S.1.

4 Deep Learning Model and Validation

During the training, the whole set of weights was optimised, starting from the pre-trained Darknet's weights of the original YOLO [START_REF] Redmon | You Only Look Once : Unified, Real-Time Object Detection[END_REF]. All training were performed with the same hyperparameters: learning rate of 0.001, decay of 0.0005 and momentum of 0.9. The network resolution was set to 608 x 608 pixels. For more details on the optimization of neural networks, we refer the reader to (R. [START_REF] Sun | Optimization for deep learning : Theory and algorithms[END_REF].

We defined two levels of validation: a "model validation", with a set of metrics commonly used to evaluate CNN models and calculated on the TD dataset, and an "ecological validation", in which the CNN model is tested on the VD dataset to assess the efficiency of the model in the frame of ecological studies and to conclude about the real-applicability on monitoring sites data. positives and TN true negative. The confusion matrix is normalised by column, so TP+FN=1 and FP+TN=1. We expect a confusion matrix that presents at least a TP rate greater than 70%-75% and a FP rate as low as possible, to avoid the operator spending too much time verifying the detections found. As mentioned in Section 2.2, a large part of the video stream consists of images with no fish (TN). Apart from the confusion matrix, to evaluate the equilibrium between TN and TP we have included in the metrics the Cohen's kappa coefficient, κ [START_REF] Cohen | A Coefficient of Agreement for Nominal Scales[END_REF][START_REF] Landis | The Measurement of Observer Agreement for Categorical Data[END_REF]. The latter already provides an interpretative framework, with a value considered "good" starting in general from 0.60-0.80.

Concerning the "ecological validation", the model is applied on the entire VD to assess its efficiency to detect fish larger than 20 cm. The entire reading of the VD videos' by an operator are compared to the model outputs to validate if the model detects efficiently the passage of one fish described by an operator. Even if the classifier is single-class, the results have been evaluated in terms of all species of interest. The comparison aims to assess the possible effects of biological and environmental variables on the model efficiency, such as the fish species (by their morphological and behavioural differences), the fish size range, distance to the camera, or the seasonal environmental conditions. When evaluating the VD, a filter has been added at the end of the pipeline, to avoid flash detections: all detections that do not have a subsequent detection in the next image with an overlapping bounding box are automatically rejected. Since this validation deals with sequences, we defined as a TP each passage of a fish for which the neural network have found the fish on at least two consecutive images, to avoid counting flash detections (detection on a single image). Indeed a fish passage can be identified by the detection on only a few consecutive images, so TP and FN can be easily defined in function of detected fishes. Such reasoning cannot be applied for the calculation of TN, expressed in number of images where no fish is found compared to the total number of images without any fish. This limitation of our approach makes impossible to write a confusion matrix where all elements are expressed in function of the same quantity. To evaluate the TN%, we took into account only the 33 clips without any individual present. Finally the FP are defined as each false detection of a fish consecutive on at least two images. We compared the FP number with the entire number of passages in the ground truth on the same clip. Such estimation gives an idea of how many detections an operator have to manually verify for each clip and consequently appear as a proxy of the operator time spending. In general, we aimed to maximize the TN% (to sort the empty sequences), but also to minimize the undetected fish passages, meaning a high TP% (larger than ~75%) and a rate between FP/TP as low as possible.

5 Data preprocessing

The first preprocessing step was the background segmentation of each clip, using the Gaussian Mixture-based Background/Foreground Segmentation Algorithm as implemented by Zivkovic et al. [START_REF] Zivkovic | Efficient adaptive density estimation per image pixel for the task of background subtraction[END_REF], to generate the "background masks", "b". The variance threshold for the pixel-model match depends on the camera type and were set to 130 for DIDSON clips and 10 for ARIS clips. Due to the noisy nature of the signal, the parameters were chosen to optimise noise reduction limiting meaningful loss of the signals. In order to further reduce the saltand-pepper noise in the "b" masks, each one has been smoothed using a median filter with a 3x3 aperture [START_REF] Huang | A fast two-dimensional median filtering algorithm[END_REF]. The resulting images were further processed with an opening operator [START_REF] Dougherty | Hands-on Morphological Image Processing[END_REF], with a structuring element of dimensions 3x3 and a cross-hair shape.

The resulting masks are named "background filtered masks", "bf" (see Figure 1). The two masks and the raw images (called "r") are used to compose the processed RGB images "rbbf". All the information "r" images are in the blue channel, therefore only this channel was used. The green channel was assigned to the "b" mask and the red channel to the "bf" masks. To demonstrate the influence of each mask, we tested also the 2-channel images "rb" and "rbf". During in the training, for all the models (with exception of the raw's one), colour data augmentation was disabled. The choice of these masks was driven by a "best performance" principle: several other sets of masks have been tested, ranging from temporal median filters to optical flow maps, with some of them converging to the same type of results (not presented). However a specific discussion of these sets of masks is beyond the scope of this paper. As this is the first work datasets, multiple cameras and approach, it was necessary to assess the Deep Learning architecture on the different level of preprocessing. Amongst the different performance measures, we choose to compare the models using the AP@0.50 metric evaluated on the "val" set (see SI for details on the dataset splitting). As shown in Figure 2, the models trained on the unprocessed images and in combination of a single mask present an oscillating behaviour of the AP@0.50, suggesting an overfitting, and stabilise only at the end of the training. The model with unprocessed images ("r") reaches an AP@0.50 of 53%: this data constitutes, so, the "null model" for the others. Both the "rb" images and "rbf" images exhibit a similar behaviour, showing an improvement in the evaluation metric that reaches 69% for the "rb" model. This data suggest that both the "b" mask or the "bf", separately, improve the performance of the detection, but also that these masks individually are not a viable alternative to the original images (see Figure S.1). Finally the model trained on the "rbb f " images shows stable convergence and is able to reach 72% of AP@0.50. In order to verify which model is able to well generalise (being able to classify data of the same kind of images as the training ones, but that the CNN has never seen before), we compared the evaluation metrics (precision, recall, F 1 -score and AP@0.50) for the different models on the "test" set of the TD dataset (see Table 1 and Figures S.1 and S.2). The AP@0.50 presents lower values for all the models, as expected, but surprisingly the "rb" model exhibits metrics similar to the "rbbf", thus suggesting that in the latter the "b" feature is the most important one. The highest F 1score is found for the "rbbf" model: this, added to the well-behaved training showed in Figure 2, leaded to the choice of the "rbbf" model (a scheme of this model is given in Figure 3). 1. Validation metrics for the various models calculated on the "test" set of the TD dataset. 13

. Results

Influence of the data treatment

Model validation

As mentioned in Section 2.4, the focus of this work is not only on the detection of TP (of which some examples are showed in Figure 4), but also to reach good performances on TN/FN detections. The Cohen's kappa coefficient, κ, was calculated for the model trained on the "rbbf" images on the "test" set of the TD dataset, leading to a κ = 0.73: for similar tasks [START_REF] Fleiss | Large sample standard errors of kappa and weighted kappa[END_REF] such score is largely above the threshold of the "good" score. This suggest that the detector is able to discriminate between TP and TN. To confirm this, we calculated the confusion matrix: as showed in Table 2, the detector is performing very well (87%) on the TN, while keeping good results (61%) for TP. Such result suggests that our detector is able to sort out efficiently all the sequences without any passage of fish, while maintaining a high probability to identify the sequences with the presence of an individual, even looking at the single image and not to a sequence. The confusion matrices of each camera (Table S.5) show different performances: while for DIDSON we reach optimal scores (larger than ~70%) for both TP% and TN%, for ARIS the TP% is too low, at least for such test based on single images.

Ground truth

Fish No Fish Predicted Fish 0.61 0.13 No Fish 0.39 0.87 Table 2. Confusion matrix for the "rbbf" model, normalised by column, calculated on the "test" set of the TD dataset.

Ecological validation

As shown in Table 3, the total recall, or TP%, for the DIDSON camera exhibit an increasing trend with the fish size, with the best results starting from the 40-60 cm class and reaching more than 90% for large size fish. The same trend is present for the ARIS camera, but with lower results, with a maximum around 80% for large size fish. For small and medium-sized fish and for fish detected on the ARIS camera, the results are below the threshold of ~70-75%, for which the signal can be confused with the noise of sonar signal. Even if these results outline the general efficiency of our method, a detailed analysis of the main species of interest (Atlantic salmon and European eel) reveals some differences in the performances. For the Atlantic salmon, the results are remarkably high with both cameras and for all fish sizes, even if for the ARIS camera no large dataset is available. On the contrary, for the European eel the DIDSON camera offers satisfactory results

(with TP larger than ~70%) only for 60-80 cm size. For the ARIS camera, none of the values exceed 75%. The FN% is simply 1-TP%, therefore the same observations apply: higher values of FN% are concentrated on small sizes, meaning that small individuals will not be efficiently detected by our automatic analysis.

Recall (%TP) in function of fish size For both cameras TN%, calculated on a set of empty long clips as explained in 2.4, is very high (around 99%), hinting that the method is capable to identify the sequences where an individual is probably present, discarding those where only sonar noise is present.

The average FP% on all the long clips is 0.67% for the DIDSON camera and 0.48% for the ARIS camera, meaning that an operator can expect a small rate of detections that are not fish. A more detailed analysis of the FP% reveals a large variance, especially for the DIDSON camera: the value, when calculated on the individual long clip, oscillates between 0.30 and 30, depending on the environmental conditions and/or the state of cleanliness of the lenses (on which the noise level depends). A more interesting indicator is therefore the median of the FP/TP rate. For the DIDSON data this metric is 0.83, while it is 0.40 for ARIS data (see Figure S.2 and S.3 for the distributions of the FT/TP rate). In both cases the median is below 1, suggesting that for most of the clips the number of FP will be smaller than the TP: with our model it is necessary a check of the detections by an operator, but in many cases it is a trivial operation.

. Discussion

In a context of big data acquirement in ecological studies, such as camera trap data, the automation of the data extraction for analysis is one main issue that tends to be solved by the Convolutional Neural Networks (CNN) [START_REF] Valdenegro-Toro | Submerged Marine Debris Detection with Autonomous Underwater Vehicles[END_REF]. Indeed, those models are seeing a growing interest for their capacity to calculate the "good features" without any pre-treatment, but in case of complex data, such as sonar imaging, it yields to poor performances. A common route to solve this is to design or to modify an existing CNN architecture, for example adding more deep layers that are able to extract the peculiarity of the data as for noise treatment. However, this increases the number of parameters to optimize and yields to minimization difficulties. As a less common process, we suggest to pre-treat the data in order to elaborate an image composed by different masks, enhancing the signal. The advantages of this route are that it is easier to compare performances with existing ones developed in other fields, it can be based on the parameters and weights of other models even in complex situations and it is conceptually straightforward (opposed to over-complexed neural networks [START_REF] Belkin | Reconciling modern machine-learning practice and the classical bias-variance trade-off[END_REF]).

The protocol of validation must take into account the metrics used in deep learning to assess the performances of the algorithm. However real life application may deal with an amount of data with high levels of variability that might not be fully taken into account by standard deep learning validation protocols. Increasing the training dataset does not necessarily guarantee better results and requires considerable operator effort [START_REF] Cho | How much data is needed to train a medical image deep learning system to achieve necessary high accuracy[END_REF][START_REF] Sun | Revisiting Unreasonable Effectiveness of Data in Deep Learning Era[END_REF]: the two-step validation that we proposed here can be considered as a viable strategy in order to limit the effort of labelling and validate the method on real-life scenarios. When the above conditions are met, we can therefore recommend trying the method outlined here at least as a first approach to exploring the limits of the most current CNNs.

The presented model itself exhibit interesting results. Looking at the TP% in function of fish size, we can affirm that the method is more than satisfactory (with values larger than ~75%) for all fish with size > 40 cm moving on DIDSON camera videos and > 60 cm for fish on ARIS. Despite the poor resolution of the acoustic camera images resolution face to optical images [START_REF] Martignac | The use of acoustic cameras in shallow waters : New hydroacoustic tools for monitoring migratory fish population. A review of DIDSON technology[END_REF], our method offers the possibility to optimize the detection of fish corresponding to these size thresholds without interfering with their behaviour with a transposable device on until 15meters wide river sections, a clear advantage compared to the common monitoring devices such as video-counting.

Since we expect to apply such model to monitor fish populations in rivers where most of the recorded images will be of True Negative type (images where no a single fish is present) to reduce drastically the time an operator spend to search for a fish passage. With this in mind, our method proved to be remarkably efficient with both acoustic cameras, despite the differences in image quality and resolution. In parallel, we desired also a model able to minimize the FP rate, so that less time possible is spent in checking the correctness of the detections. Overall, the average FP% is low enough to reach our operational objectives, when compared to the total number of fish passages.

However some flaws are identified. The model show some difference in performances between DIDSON and ARIS cameras: for example, the ARIS TP% are always lower than DIDSON's (between ~5% and ~25% lower, see Table 3), for all fish sizes and species. This might be due a number of different aspects that differentiate the ARIS camera: a lower noise but higher image resolution, deeper record window resulting in smaller objects but also fewer images in the training set. A first possibility for improvement may be to increase the amount of ARIS data in the TD dataset, before moving on to develop two different models for the cameras. A second flaw is given by the different performances in function of the species of interest as, for example, for the European eel: the TP% is not optimal (below ~70% for 40-60 cm) neither with the DIDSON camera, nor with the ARIS (none of the values exceeds 70%). Indeed, the eels have a particular body shape and swimming behaviour, with a serpentine undulation [START_REF] Webb | Locomotor Patterns in the Evolution of Actinopterygian Fishes[END_REF] that make them less visible on the sonar images than the other fish species [START_REF] Lagarde | In situ evaluation of European eel counts and length estimates accuracy from an acoustic camera (ARIS)[END_REF]. Furthermore, European eel adults can be shorter than adults from another serpentine shape aquatic species like the Sea lamprey. Solving this requires to take into account explicitly the temporal correlation between images in which the same species is present. For example developing a LSTM-CNN based video object trackers as the ROLO architecture [START_REF] Ning | Spatially supervised recurrent convolutional neural networks for visual object tracking[END_REF]. Such kind of neural networks might help also to improve another characteristic: the low rate detections on small sized fishes, lower than 40 cm. The difficulty of some neural networks in detecting small objects is a well-known problem [START_REF] Tong | Recent advances in small object detection based on deep learning : A review[END_REF]: an alternative method can be to test a CNN designed specifically for this kind of problem.

Beyond the performances, our model is characterized to have only one class "fish". This feature should not be considered a pitfall if the goal is a global monitoring of fish biological activity. Even if the datasets are elaborated from videos that were recorded on two sites, it has not to be excluded a certain degree of transferability of the model. In order to expand the method to more specifics uses (such as study of a single species), one of the most important steps in the future is, indeed, to improve this first version to increase the general efficiency and add multi-class identification.

To summarise, we developed a multi-camera and multi-species CNN-based neural network pipeline to automate fish recognition in a sonar video flux. We tried also to propose some check the metrics during the optimisation and "test" data to calculate the metrics once the optimisation is over.

In the VD dataset, the total dataset correspond to around 660 000 images. FN is the total number of False Negative. R is a measure of how many relevant data are found, compared to the whole relevant data.

F1-score

F 1 =2 PxR P+ R
F1 is a mean between P and R. Its are between 0 and 1 (perfect recall and precision).

Intersection over Union (IoU)

IoU= area( bb p ∩bb g -t ) area( bb p ∪bb g-t )

bbp is a predicted bounding box and bbg-t is a ground-truth box. The IoU, used in the calculation of the AP@0.50, is a measure of the spatial agreement between the ground-truth and predicted boxes.

Average Precision (AP@0.50)

AP= ∑ i n-1 (r i +1 -r i ) p (r i+1 )
ri are the recall levels and p(ri) is the interpolated precision. See PASCAL development kit for more details. Only the detections with an IoU ≥ 0.50 are counted in the calculation of the AP. This metric, commonly used in deep learning, is the average precision between different recall levels.

Cohen's kappa (κ) κ= p 0 -p e 1-p e κ is a measure of the inter-rater reliability. In our case the first rater is the neural network and the second is the operator that annotated the ground-truth. p0 is the observed agreement between raters. Pe is the probability of a random agreement.

Table S.4: Metrics used for the model validation.

Figure 1 .

 1 Figure 1. Examples of images (ARIS on top, DIDSON on bottom); raw images (a), background mask (b) and background filtered mask (c); the red arrow points to the fish position.

Equation 1 .Confusion

 1 Confusion matrix; TP is the number of true positives, FP of false positives, FN of False positives and TN of True NegativeConcerning the model validation, we adopted six metrics: precision, recall, F 1 -score, AP@0.50, Cohen's kappa coefficient (explained in TableS.4) and confusion matrix. The latter is defined in Equation1, where TP is the number of true positives images, FP false positives, FN False 9

Figure 2 .

 2 Figure 2. Loss (dotted line) and AP (solid line) for different models: r (blue), rb (green), rbf (violet) and rbbf (red).

Figure 3 .

 3 Figure 3. Scheme of the "rbbf" model's pipeline, from raw data ("r") to the detections, through the background segmentation ("b" mask), the denoising step ("b f " mask) and the CNN network.

Figure 4 :

 4 Figure 4: example of detections on the rbbf images from the test set of the TD dataset; ARIS images on top (a) with the detection of (from left to right) an Atlantic Salmon, a Sea Lamprey and a Giant catfish; DIDSON images on bottom (b) with the detection of (from left to right) a Sea Lamprey, Atlantic Salmon and a European catfish.

  recommendations in order to put to good use our experience such as the importance of preprocessing to keep models simple, the importance of a double validation or how to tackle cases of abundant TN data. Our model satisfy most of the required features, but this study only started exploring the potential of deep learning on such complex imaging and many challenges are still present. In a more general manner, the possibility to extract an accurate monitoring of fish populations from video-like analysis might answer to long-standing questions in aquatic ecology and open new ones, not only for researchers but also for biodiversity protection and management structures. G.F.G. developed the model, based on ideas developed with T.C.. M.M., F.M. and L.B. guided the development of the model concerning the ecological applications. F.M. collected the data and created the datasets along with G.F.G.. G.F.G. and F.M. wrote the first drafts and all authors discussed the results, implications and edited the manuscript.

Figure

  Figure S.1: training run with only the 'b' mask (violet) and only the 'b f ' mask (green). In dark colors the loss function and in light colors the AP@0.50 as a validation metric.

Figure S. 2 :

 2 Figure S.2 : FT/TP histogram for ARIS camera on the VD dataset ; to highlight the data, the scale has increments of 0.1 between 0.0 and 1.0, 1.5 between 1.0 and 10, 5 between 10 and 100.

Figure

  Figure S.3 : FT/TP histogram for DIDSON camera on the VD dataset ; to highlight the data, the scale has increments of 0.1 between 0.0 and 1.0, 1.5 between 1.0 and 10, 5 between 10 and 50.

Table of Content

 of

	Introduction
	Materials and Methods
	Deep Learning Approach
	Hydroacoustic methods: acoustic cameras
	Datasets
	Deep Learning Model and Validation
	Data preprocessing
	Results
	Influence of the data treatment
	Model validation
	Ecological validation
	Discussion 1. Introduction

Table 3 .

 3 Recall (%TP) for the rbb f model, calculated on the VD dataset (dataset of videos).

	Species	Camera	20-40	40-60	60-80	> 80	Total
	All species	DIDSON	57.62	76.22	90.00	93.33	67.91
		ARIS	52.87	55.39	83.72	88.89	55.13
	Atlantic salmon	DIDSON ARIS	NA NA	92.86 100.00	96.97 100,00	100.00 NA	96.23 100.00
	European eel	DIDSON	NA	57.14	93.33	66.67	75.00
		ARIS	7.69	25.00	66.67	50.00	30.49
	Other species of interest	DIDSON ARIS	0 NA	100.00 NA	100.00 NA	83.33 71.43	94.44 71.43

Table S .

 S 2 : distribution of each species of interest in the 50 clips of the TD. Each clip may contain more than one species and more than one passage. Table S.3: Number of clips for the species of interest by size in the VD dataset. TP is the total number of True Positive and FP of False Positive. P is a measure of how many relevant data are found, compared to the whole found data.

	Recall (R)		R=	TP TP +FN
	Species	Total clips	Total	ARIS clips	ARIS	DIDSON	DIDSON
			annotations			annotations	clips	annotations
	Atlantic salmon	20	1117		6	268	14	849
	European eel	12	1161		8	568	4	593
	Sea lamprey	9	324		1	22	8	324
	Allis shad	11	718		1	290	10	428
	Giant catfish	10	633		7	418	3	152
	Total	62	3953		23	1566	39	2346
						Fish size
	Species	Camera	20-40	40-60	60-80	> 80	Total
			Fish species of main interest
	Atlantic salmon	DIDSON ARIS	0 0		14 2	33 10	6 0	53 12
	European eel	DIDSON	0		14	15	3	32
		ARIS	13		52	15	2	82
			Other fish species, included
	Generic Fish (unidentified)	DIDSON ARIS	315 248		505 419	30 18	2 0	852 685
	Sea lamprey	DIDSON	0		0	2	2	4
		ARIS	0		0	0	1	1
	Allis shad	DIDSON	0		1	0	0	1
		ARIS	0		0	0	0	0
	European catfish	DIDSON ARIS	0 0		0 0	0 0	2 6	2 6
	Metric		Equation		Notes
	Precision (P)		P=	TP TP +FP

Acknowledgements Authors' contributions Data Availability References

Acknowledgements

We gratefully thank the OFB (Office Français de la Biodiversité -French Office for Biodiversity) for the foundings.

Authors' contributions

Data Availability

The TD and VD datasets are available, respectively, at doi:10.5281/zenodo.5046708 and doi:10.5281/zenodo.5092010. The code is available at the following GitLab: https://forgemia.inra.fr/pole-migrateurs/Acoustic/acoustique.

In the TD dataset, all the images are extracted from 23 ARIS clips and 39 DIDSON clips. The TD is divided into a training set (60% of the images), an evaluation set (19%) and a test set (21%): in this scheme, typical in deep learning, "train" data is used to optimise the CNN parameters, "val" data to


Evelyne Benoit

Michel De Waard

Rémy Béroud

Michel Partiseti

Denis Servent

Multi-scale evaluation of spider toxins as potential anti-nociceptive agents: example of Cyriotoxin-1a

Aim

Our expertise is the identification and structural and functional characterization of original natural toxins targeting receptors and ion channels, and which may have applications in human health. In this context, and in collaboration with the Institut du Thorax, Smartox Biotechnology and Sanofi R & D, we are interested in the multi-scale evaluation (from the cell in vitro to the organism in vivo) of spider toxins as potential anti-nociceptive agents, as illustrated by the example of Cyriotoxin-1a (CyrTx-1a). Na V 1.7 channel subtype is highly expressed in dorsal root ganglia (DRG) of the sensory nervous system and plays a central role in the pain signaling process. We investigated a library prepared from original venoms of 117 different animals to identify new selective inhibitors of this target.

Methods

We used high-throughput screening of the venom library, using automated patch-clamp experiments on human voltage-gated sodium channel subtypes, and then in vitro and in vivo electrophysiological experiments to characterize the active peptides that have been purified, sequenced and chemically synthesized. Analgesic effects were evaluated in mice in vivo.

Results

We identified and further characterized CyrTx-1a, a novel peptide isolated from Cyriopagopus schioedtei spider venom. This 33 amino acids toxin belongs to the inhibitor cystine knot structural family and inhibits hNa V 1.1-1.3 and 1.6-1.7 in the low nanomolar range, compared to the micromolar range for hNa V 1.4-1.5 and 1.8. CyrTx-1a was 920 times more efficient at inhibiting tetrodotoxin (TTX)-sensitive than TTX-resistant sodium currents recorded from adult mouse DRG neurons in vitro, and approximately 170 times less efficient than huwentoxin-IV at altering mouse skeletal neuromuscular excitability in vivo. CyrTx-1a exhibited an analgesic effect in mice by significantly increasing reaction time in the hot-plate assay.

Main Conclusions

The pharmacological profile of CyrTx-1a paves the way for further engineering studies aimed to optimize the potential anti-nociceptive properties of this peptide.

1

  evaluation of spider toxins as potential anti-nociceptive agents: Example of Cyriotoxin-1a 1 Benoit, E., 2 De Waard, M., 3 Béroud, R., 4 Partiseti, M., 1 Servent, D. Service d'Ingénierie Moléculaire pour la Santé, ERL CNRS/CEA 9004, Gif sur Yvette, France 2 Institut du Thorax, Inserm UMR 1087 / CNRS UMR 6291, Nantes, France 3 Smartox Biotechnology, Saint-Egrève, France 4 Sanofi R & D, Integrated Drug Discovery -High Content Biology, Vitry-sur-Seine, France

th -16 th September 2021 www.euven-congress.eu


Nicolas Fontbonne
email: nicolas.fontbonne@sorbonne-universite.fr

Nicolas Maudet
email: nicolas.maudet@lip6.fr

Nicolas Bredeche
email: nicolas.bredeche@sorbonne-universite.fr

Adaptive Team Cooperative Co-Evolution for a Multi-Rover Distribution Problem

Keywords: Computing methodologies → Evolutionary robotics;, Theory of computation multi-robots, multi-agent systems, evolutionary robotics, multiagent reinforcement learning, multi-armed bandits, marginal contribution, fitness evaluation, cooperative coevolution EA

This paper deals with policy learning for a team of heterogeneous robotic agents when the whole team shares a single reward. We address the problem of providing an accurate estimation of the contribution of each agent in tasks where coordination between agents requires joint policy updates of two (or more) agents. This is typically the case when two agents must simultaneously modify their behaviors to perform a joint action that leads to a performance gain for the whole team. We propose a cooperative co-evolutionary algorithm extended with a multi-armed bandit algorithm that dynamically adjusts the number of agents that should update their policies simultaneously, aiming both for performance and learning speed. We use a realistic robotic multi-rover task where agents must physically distribute themselves on points of interest of different natures to complete the task. Results show that the algorithm is able to select the best group size for policy updates that reflects the task's coordination requirements. Surprisingly, we also reveal that coupling between agents' actions in a realistic setup can also emerge from interactions at the phenotypical level, hinting at subtle interactions during learning between the control parameter space and the behavioral space.

INTRODUCTION

Multi-robot systems can be used to tackle tasks where many smaller subtasks have to be addressed, sometimes simultaneously. This means that robots often have to coordinate their policies so as to perform complementary actions. This is the case when all robots are part of the same team, where everyone works towards the same goal. However, the problem of designing efficient individual robot's policies is challenging when the whole team shares a single reward signal, similar to a football team where all players work together to increase the team's score. In this setup, there is a credit assignment problem as information is not directly accessible to assess a single individual's contribution to the team's performance.

, , .

Credit assignments in teams of agents have been explored in both theoretical setups (e.g., N-person coordination games) and experimental setups (e.g., Robocup soccer). Early results in theoretical setups showed that solving an exact formulation is intractable in general as each agent's contribution would require averaging the counterfactual marginal contribution on all possible coalitions [START_REF] Lloyd | A Value for n-person Games[END_REF]. However, many practical implementations have been proposed throughout the years to estimate the contribution of each of the agents through approximation [START_REF] Chen | L-shapley and c-shapley: Efficient model interpretation for structured data[END_REF][START_REF] Frye | Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability[END_REF][START_REF] Heskes | Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models[END_REF] or using various kinds of simplifying assumptions [START_REF] Foerster | Counterfactual Multi-Agent Policy Gradients[END_REF][START_REF] Perolat | Actor-critic fictitious play in simultaneous move multistage games[END_REF][START_REF] Rahmattalabi | D++: Structural credit assignment in tightly coupled multiagent domains[END_REF][START_REF] Son | Qtran: Learning to factorize with transformation for cooperative multiagent reinforcement learning[END_REF][START_REF] Stone | Ad Hoc Autonomous Agent Teams: collaboration without pre-coordination[END_REF][START_REF] Sunehag | Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward[END_REF][START_REF] Wolpert | An introduction to collective intelligence[END_REF] Learning in Multi-agent systems offers a promising avenue to address coordination problems in multi-robot systems, especially when the state and action spaces are continuous, and only partial observations are available for each robot. In the last few years, multi-robot systems have indeed drawn a lot of attention from both Multi-Agent Deep Reinforcement Learning (MADRL) [START_REF] Hernandez-Leal | A survey and critique of multiagent deep reinforcement learning[END_REF] and Evolutionary Collective Robotics [START_REF] Doncieux | Evolutionary Robotics: What, Why, and Where to[END_REF][START_REF] Trianni | Evolution, Self-organization and Swarm Robotics[END_REF]. On the one side, state-of-the-art MARL approaches such as QMIX [START_REF] Rashid | Weighted QMIX: Expanding monotonic value function factorisation for deep multiagent reinforcement learning[END_REF][START_REF] Rashid | QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning[END_REF][START_REF] Son | Qtran: Learning to factorize with transformation for cooperative multiagent reinforcement learning[END_REF] or MAVEN [START_REF] Mahajan | MAVEN: Multi-Agent Variational Exploration[END_REF] make use of a combination of individual policy learning agents and shared critics to favour coordination towards better performance.

On the other side, Evolutionary Robotics have pursued different research tracks on whether teams of agents are considered homogeneous (i.e. teams of clones) or heterogeneous. Homogeneous teams are naturally scalable [START_REF] Baray | Evolving cooperation via communication in homogeneous multi-agent systems[END_REF][START_REF] Trianni | Self-organisation and communication in groups of simulated and physical robots[END_REF] and fast to evaluate [START_REF] Luke | Co-evolving soccer softbot team coordination with genetic programming[END_REF], and they have been shown to display behavioural specialization by exploiting environmental cues, at least to some extent [START_REF] Bobby | Neuroevolution for adaptive teams[END_REF][START_REF] Ferrante | Evolution of Self-Organized Task Specialization in Robot Swarms[END_REF]. When further specialization is required, heterogeneous teams nonetheless allow for the necessary flexibility in terms of role specialization [START_REF] Baldassarre | Evolving mobile robots able to display collective behaviors[END_REF][START_REF] Bernard | To cooperate or not to cooperate: why behavioural mechanisms matter[END_REF][START_REF] Josh | Investigating morphological symmetry and locomotive efficiency using virtual embodied evolution[END_REF][START_REF] Quinn | Evolving formation movement for a homogeneous multi-robot system: Teamwork and roleallocation with real robots[END_REF][START_REF] Waibel | Genetic team composition and level of selection in the evolution of cooperation[END_REF].

However, learning methods for multi-robot systems with a single team-level reward are faced with a dilemma caused by two contradictory issues:

• the problem of over-generalization [START_REF] Wei | Lenient learning in independent-learner stochastic cooperative games[END_REF][START_REF] Paul | An Analysis of Cooperative Coevolutionary Algorithms[END_REF], which a team faces when the task requires extensive coordination between agents. In this case, joint policy updates may be necessary to discover efficient synergies between agents. This can be solved by updating all policies simultaneously to allow for the discovery of new synergies; • the problem of exploratory action noise [START_REF] Jen | When less is more: Reducing agent noise with probabilistically learning agents[END_REF][START_REF] Holmesparker | Clean rewards to improve coordination by removing exploratory action noise[END_REF][START_REF] Matignon | Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems[END_REF], which occurs when an increase in team performance is observed that follows multiple agents updating their policy, without any means to identify who is responsible for such an increase. The actual decrease in the contribution of one particular agent can be shadowed (and sometimes caused) by other suddenly more successful agents. This can be solved by updating only one policy at a time to estimate the benefit (or loss) provided by the agent executing this policy.

Our hypothesis is that the optimal balance between updating a single policy or all policies concurrently is contingent upon the specific problem and may change throughout the learning process. The contribution of the work presented here takes the form of a multiagent evolutionary learning algorithm where the number of agents that undergoes policy updates can be changed to match the problem.

A classical robotic domain task is used for comparison and analysis that involves a team of robots, each following a deterministic learned policy that maps partial observation as continuous states to continuous actions. The tasks used involve multiple rovers trying to observe a set of points of interest, some of which require close coordination between sub-part of the team. Most importantly, these robotic tasks reveal how deceptive a seemingly simple robotic problem can be: the emergence of coordination may indeed result from joint policy updates and subtle behavioural interactions.

We propose combining a Cooperative Coevolution Evolutionary Algorithm with a multi-armed Bandits Algorithm in order to automatically adjust the amplitude of policy updates during the course of learning, to guess which best trade-off should be operated to maximize both the discovery of synergies between agents and the reliability of credit assignment for the agents' contribution to the performance of the team.

In the rest of the paper, previous works are presented that propose solutions for various formulations of the problem to provide an accurate estimation of the contribution of each member of a team, using more or less strong assumptions on the a priori knowledge on the task to be done. We then describe two algorithms for the general case and the multi-robot tasks used for evaluation. Finally, results are shown with both algorithms, emphasizing the particularities arising when agents' trajectories are coupled with one another, and the benefits of automatically tuning the number of agents that should simultaneously update their policies.

RELATED WORK

Learning complex coordination in a team of agents is a challenging problem, which several methods have tried to address. These approaches can be classified depending on the amount of background knowledge available prior to learning. For example, the D++ algorithm [START_REF] Rahmattalabi | D++: Structural credit assignment in tightly coupled multiagent domains[END_REF] makes strong assumptions about the problem to be solved as requirements for coordination are supposed to be known beforehand (e.g. the number of robots required at a specific location). D++ considers the presence of counterfactual agents at specific times and locations to compute the reward of agents with respect to their possible (rather than actual) contribution to the team's performance. Counterfactual agents are virtual agents used to compute the reward of actual agents as if they were not alone. This is particularly relevant for tasks that require coordination among agents who pursue a similar goal (e.g. observing a particular point of interest). However, its requirements for prior knowledge limit its applicability.

Other approaches aim at marginalizing or decomposing the global reward in order to estimate the contribution of each individual to the team performance [START_REF] Sunehag | Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward[END_REF]. The difference reward algorithm [START_REF] Agogino | Efficient Evaluation Functions for Multi-rover Systems[END_REF] is a translation in terms of reward shaping [START_REF] Ng | Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping[END_REF] of the Wonderful Life Utility [START_REF] Wolpert | Optimal Wonderful Life Utility Functions in Multi-Agent Systems[END_REF]. The goal is here to extract one agent's marginal contribution by subtracting a counterfactual reward, as if the agent performed a default action, from the gained reward. It can be formalized as:

𝑑 (𝑖 ) (𝒂) = 𝑟 (𝒂) -𝑟 (𝒂 𝑎 (𝑖 ) ←𝑎 (𝑖 ) ′ ) (1)
With 𝒂 the joint actions of all agents, and 𝑟 (𝒂) is its corresponding global reward. 𝑟 (𝒂 𝑎 (𝑖 ) ←𝑎 (𝑖 ) ′ ) is the global reward when agent 𝑖 perform a default action 𝑎 (𝑖 ) ′ instead of the action its policy dictates. 𝑑 (𝑖 ) is then, for agent 𝑖 the advantage of using its policy instead of a default action, freed from the impact of other agents. This method has been successfully applied in several domains including air traffic control [START_REF] Cruciol | Reward functions for learning to control in air traffic flow management[END_REF], rover navigation [START_REF] Agogino | Analyzing and visualizing multiagent rewards in dynamic and stochastic domains[END_REF][START_REF] Knudson | Coevolution of heterogeneous multi-robot teams[END_REF], satellite coordination [START_REF] Agogino | Evolving distributed resource sharing for cubesat constellations[END_REF], distributed sensor networks [START_REF] Holmesparker | Exploiting structure and utilizing agent-centric rewards to promote coordination in large multiagent systems[END_REF][START_REF] Turner | Designing agent utilities for coordinated, scalable and robust multi-agent systems[END_REF], communication system optimization [START_REF] Agogino | Evolving large scale UAV communication system[END_REF], and robot swarm coordination [START_REF] Douchan | Swarms can be rational[END_REF].

Multi-Agent Deep Reinforcement Learning combines the decentralized execution of agents' actions and a centralized critic used to approximate the agents' contributions. For example, the QMIX algorithm [START_REF] Rashid | Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning[END_REF][START_REF] Rashid | Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning[END_REF] uses a neural network to map a joint value function to individual value functions. Assuming a counterfactual action is available, the Counterfactual Multi-Agent Policy Gradients algorithm (COMA) [START_REF] Foerster | Counterfactual Multi-Agent Policy Gradients[END_REF] estimates the contribution of any single agent by comparing team performance with each agent playing its true action versus the same agent playing the counterfactual action baseline.

Reward shaping can also be used to compute counterfactual rewards for actions that were not taken by agents. This is achieved by using a model of the reward function and the CLEAN shaping structure [START_REF] Holmesparker | Clean rewards to improve coordination by removing exploratory action noise[END_REF]. One approach is to compare the score of the current action with a new exploratory counterfactual action 𝑎 (𝑖 ) ′ :

𝑐1 (𝑖 ) (𝒂) = 𝑟 𝒂 𝑎 (𝑖 ) ←𝑎 (𝑖 ) ′ -𝑟 (𝒂) (2)
This results in the opposite of the difference reward, which was calculated by comparing the reward of the current action to a default counterfactual. Combining these two approaches, a second version of the CLEAN reward can be obtained. This consists of comparing an exploratory action 𝑎 (𝑖 ) ′ to a default action 𝑎 (𝑖 ) ′′ :

𝑐2 (𝑖 ) (𝒂) = 𝑟 𝒂 𝑎 (𝑖 ) ←𝑎 (𝑖 ) ′ -𝑟 𝒂 𝑎 (𝑖 ) ←𝑎 (𝑖 ) ′′ (3)
This calculation is used to update the agent's policy. Only the non-exploratory action is visible to other agents, which promotes coordination among them as individual exploration does not impact others. Additionally, limiting the number of concurrently learning agents across each epoch can be beneficial for reducing agent noise [START_REF] Jen | When less is more: Reducing agent noise with probabilistically learning agents[END_REF]. In this case, agents update their policies depending on a probability that depends on their predicted impact.

In the next Section, we elaborate on works presented here to propose a method that requires neither a priori knowledge of the task nor the definition of a default counterfactual action baseline. Similar to the CLEAN reward, we compare different strategies for the team. However, we also enable the modulation of the number of agents simultaneously undergoing a policy update to reduce the exploratory action noise.

ALGORITHM

To address the compromise between over-generalization and exploratory action noise, we propose an extension of the Cooperative Co-Evolution Algorithm (CCEA) [START_REF] Ma | A Survey on Cooperative Co-Evolutionary Algorithms[END_REF][START_REF] Mitchell | A cooperative coevolutionary approach to function optimization[END_REF][START_REF] Mitchell | Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents[END_REF] where the number of agents that are updated at each generation is modulated. We propose two algorithms: (1) the CC-(1+1)-ES 𝑘 fixed algorithm updates the policy of a fixed number of agents per learning iteration, and (2) the CC-(1+1)-ES 𝑘 adaptive algorithm dynamically adapts the number of agent's policies updated per learning step. The two algorithms are described in Figure 1, and both are extensions of our previous work [START_REF] Fontbonne | Cooperative co-evolution and adaptive team composition for a multi-rover resource allocation problem[END_REF] to the domain of the continuous state, action and parameter spaces for robotics problems.

CC-(1+1)-ES 𝑘 fixed with Fixed Policy Updates

CC-(1+1)-ES 𝑘 fixed runs a collection of (1+1)-ES algorithms in parallel. Each (1+1)-ES algorithm 𝑖 provides the policy parameters 𝜃 (𝑖 ) for its corresponding agent 𝑖. The whole team is evaluated together, and the score is provided for the team. If 𝐹 is the policy evaluation function, we note 𝑓 = 𝐹 (𝜃 (0) , . . . , 𝜃 (𝑁 -1) ) the score of a team composed of 𝑁 agents.

Each (1+1)-ES algorithm maintains a population of two individuals [START_REF] Beyer | Evolution strategies -A comprehensive introduction[END_REF], a parent 𝜃 (𝑖 ) parent and a child 𝜃 (𝑖 ) child . Both are candidate policy parameters for agent 𝑖. The parent is replaced when the child outperforms it, using the possibly noisy team-level score obtained during evaluation. A new child is created by applying mutation to the new parent. If the child does not outperform its parent, it is replaced by a new child mutated from the current parent.

The mutation operator depends on the problem. In the application we tackle, the mutation operator chooses with probability 𝑝 = 0.2 between a Gaussian mutation operator on all parameters of an agent, and with probability 1 -𝑝 = 0.8 a uniform mutation operator of a subset of its parameters. The Gaussian mutation operator applies a perturbation centred on the current parameter value, with a variance 𝜎: 𝜃 ← N (𝜃, 𝜎 )

Figure 1: The CC-1+1-ES algorithm. Each square represents some policy parameters 𝜃 (𝑖 ) . In this example, 𝑘 = 2. Agents 0 and 2 are randomly selected and then mutated to produce new children. These children are evaluated alongside the parents of agents 1 and 3 to produce the team score 𝑓 (0,2) child . If it is greater or equal to the previously evaluated team score (𝑓 parent ), the children are kept, or else they are discarded. The top red part represents the choice of 𝑘 in the 𝑘 adaptive version of the algorithm.

And the uniform mutation operator on range R of a subset 𝑉 𝑚 of 𝑚 randomly selected parameters is defined as:

𝜃 [𝑖 ] ← U ( R ) ∀𝑖 ∈ 𝑉 𝑚
At each new generation, 𝑘 agents among 𝑁 are drawn and randomly changed in the team, with 0 < 𝑘 ≤ 𝑁 . The 𝑘 new children are kept only if they increase the team score. Therefore, the expert's challenge is finding the most efficient size 𝑘 of agents to change in each generation before learning starts.

CC-(1+1)-ES 𝑘 adaptive with Adaptive Updates

So far, 𝑘 is pre-determined by the user and may benefit from prior knowledge about the task regarding possible necessary coupling between agents' policy updates. However, such prior knowledge may not be available. In particular, a relevant value of 𝑘 depends on the problem and on the current state of the optimization (e.g., broad initial search steps vs. refined tuning near the optimal solution).

To address this, the CC-(1+1)-ES 𝑘 adaptive allows for adapting 𝑘 during evolution. The value of 𝑘 is modified at each generation using the multiarmed bandit learning algorithm EXP3 (Exponential-weight algorithm for Exploration and Exploitation [START_REF] Auer | The Nonstochastic Multiarmed Bandit Problem[END_REF]). This method allows choosing among a set of possible values for 𝑘, according to the performance gains they allowed. It uses an egalitarianism factor 𝛾 , which allows modulating the exploration or the exploitation. A 𝛾 closer to 0 indicates more exploitation of the best-performing group size 𝑘, while a 𝛾 closer to 1 will allow more exploration and 𝑘 will be drawn almost uniformly. The full algorithm is shown in Fig. 1 with the multi-armed bandit for dynamic selection in red. As using 𝛾 = 0.1, 0.5 or 0.9 yielded similar results, we only show results obtained with 𝛾 = 0.1.

EXPERIMENTAL SETUP 4.1 Task and Environment

To study the dynamics of these algorithms, we devise a rover exploration task inspired by [START_REF] Agogino | Efficient evaluation functions for multi-rover systems[END_REF] (and used in further research [START_REF] Dixit | Behavior Exploration and Team Balancing for Heterogeneous Multiagent Coordination[END_REF][START_REF] Dixit | Dirichlet-Multinomial Counterfactual Rewards for Heterogeneous Multiagent Systems[END_REF][START_REF] Zerbel | The Power of Suggestion[END_REF]). It is a good abstraction of a search and retrieve robotic task, where an unknown environment must be explored to retrieve specific objects or resources. The goal here is for a team of rovers to reach some points of interest (POI) that would give the most rewards.

Four rovers start in a small aggregate in the center of an area with some random variation of coordinates and orientation. Each rover senses both its teammates and points of interest (POI) if close enough. The team is expected to coordinate to observe POIs to maximize the score. It depends on the nature of the POIs and the number of rovers that end up observing each POI. Table 1 provides details on the location and orientation of the robots, which are also represented by small coloured squares in Figure 2.

Parameter

Value Agent 0 x,y,orient.

𝑥 ∈ [0, 0.5], 𝑦 ∈ [0, 0.5], 𝛼 ∈ [0, 𝜋 2 ] Agent 1 x,y,orient.

𝑥 ∈ [ -0.5, 0], 𝑦 ∈ [0, 0.5], 𝛼 ∈ [ 𝜋 2 , 𝜋 ] Agent 2 x,y,orient.

𝑥 ∈ [ -0.5, 0], 𝑦 ∈ [ -0.5, 0], 𝛼 ∈ [𝜋, 3𝜋 4 ] Agent 3 x,y,orient.

𝑥 ∈ [0, 0.5], 𝑦 ∈ [ -0. Ten POIs placed in a circle around the robots' starting position. The agents have 𝑇 steps to reach a POI, after which the team reward is computed. Each POI delivers some reward if the 𝑛 agents that visit it are in their radius of observation = 2. Each POI can be of either of two types:

• solo-POI:

𝑟 = 1 if 𝑛 ≥ 1; • duo-POI: 𝑟 = 3 if 𝑛 ≥ 2.
The team thus obtains a global reward 𝑅 equal to the sum of all rewards delivered by the POI, 𝑅 = POI 𝑟 . We devise two different setups, which are illustrated in Figure 2 and characterized as follows:

• Setup 1: 10 solo-POI are placed in a circle around the center of the arena. The maximal score is attained if each agent moves to a unique POI. The number of POIs is larger than the number of agents, and the task requires a low level of coordination between agents. • Setup 2: 8 solo-POI and 2 duo-POI are placed in a circle around the center of the arena. The 2 duo-POI are aligned on the horizontal axis. There are three equilibria in which teams may end up, i.e. situations from which the behavioural change of a single agent will either decrease or leave an unchanged score. The first two equilibria are suboptimal: (a) agents may spread among solo-POIs, for a score of 4, and (b) agents may split among 2 solo-POIs and 1 duo-POIS for a score of 5. The single optimal equilibrium is reached when agents pair together two by two and spread on the two duo-POIs for a score of 6. The task is assumed to require a high level of coordination between agents.

Figure 2:

The two setups explored in this study. Setup 1 has 10 solo-POI organized in a circle around the center. We refer to this setup as the 10 solo-POI setup. Setup 2 has the same circular configuration, but the POI on the horizontal axis are duo-POI. We refer to this setup as the 8 solo 2 duo-POI setup. In both figures, the small coloured square in the middle represents the initialization area of the agents described in Table 1. Blue represents the initialization area of agent 1; orange represents agent 2; green represents agent 3; and red represents agent 4.

Robot model

Agents in the experiment represent robots subject to a kinematic model, with an action space 𝐴 and observation space 𝑂. Concerning the action space, it is a question of controlling a velocity vector. Thus, the agents have a 2-dimensional action space 𝐴 where the two dimensions represent speed (𝑎 0 ∈ [0, 1] for [stop, maximum speed]) and angular speed (

𝑎 1 ∈ [ -1, 1] for [clockwise, anti-clockwise]).
Their observation space 𝑂 is composed of :

• 3 sensors per POI that get information about their distance, angle and type; • 2 sensors per rover that get information about their distance and angle. Thus, for our setups with 10 POI and 4 rovers, we obtain an observation vector of dim(𝑂 ) = 38 dimensions.

Policy

The agent's policy 𝜋 maps observations o ∈ 𝑂 to actions a ∈ 𝐴. For that purpose, we use a discrete artificial neural network with one hidden layer as the main policy structure. Figure 3 details the full topology between inputs and outputs, and the number of parameters. This architecture imposes a relatively large number of free control parameters 𝜃 that need to be optimized. In the present case, this means 𝑑𝑖𝑚 (𝜃 ) = 658 parameters.

RESULTS

This section describes the results of applying the CC-1+1-ES algorithms to the two setups described earlier. We first evaluate the performance of the CC-(1+1)-ES 𝑘 fixed algorithm with different values of 𝑘 to assert the ... relevance of tuning the number of joint policy updates depending on the problem at hand. We also provide further experimental studies to uncover the dynamics of coordination between robots. Secondly, we conduct an experimental study of the CC-(1+1)-ES 𝑘 adaptive algorithm in order to show that the value of 𝑘 can indeed be dynamically tuned over time to match the requirement for the problem at hand. The environment is stochastic, as the same policy parameters can lead to different trajectories and score outcomes due to the random variation in initial conditions. In order to provide a fair estimation of team performance, we compute and use the expected return as a policy evaluation function 𝐹 (𝜽 ) of the team parameters 𝜽 .

𝐹 (𝜽 ) = E 𝜏 ∼𝝅 𝜽 [𝑅 (𝜏 ) ] (4)
In practice, this means the return (or fitness) of a single trajectory 𝑅 (𝜏 ) is the sum of the rewards given by the POIs for this particular evaluation as described in Section 4.1. During all the following experiments, we use the term score (as it is more general than the term fitness) to refer to the experimental approximation of 𝐹 (𝜽 ) by sampling multiple trajectories from different initial conditions (see Section 4.1). More precisely, each score represents the average return of the trajectories from 16 different initial conditions that are kept constant from one generation to another. Using as many different repetitions for evaluation ensures that agents learn policies that generalize behaviour regardless of initial conditions. Table 2 lists parameters and values used in all experiments. All code is available on a repository at removed for review. Each experiment is performed with 16 independent runs to account for algorithmic stochasticity.

Setup 1: Coordination via Specialization

In the first setup, each of the 10 points of interest provides a reward of 𝑟 = 1 if at least one agent comes close enough. The maximum team score of 4 is reached if agents spread out on different points of interest. In this setup, agents are not required to perform complex coordination except for avoiding each other to reach separate POI.

With respect to the number of simultaneous policy updates, it is not clear a priori which value of 𝑘 should allow the fastest convergence and best performance. A value of 𝑘 = 1 is less destructive as it ensures that a policy is updated only if it is better. However, using 𝑘 > 1 allows for a more exploratory learning strategy, which may not be a problem as optimal policies are largely independent: save for avoiding each other, the task does not require robots to perform complex coordination behaviours.

In order to provide a fair comparison, we use two control experiments built from classic models featured in the literature:

• heterogeneous team with simultaneous policy updates. This corresponds to using 𝑘 = 4, where all policies are updated simultaneously aiming for maximal exploration (see [START_REF] Waibel | Genetic team composition and level of selection in the evolution of cooperation[END_REF] for a comprehensive overview of genetic composition); • homogeneous team. In this scheme, all individuals are clones, meaning they use the same control parameters. This scheme is particularly well-fitted when the score is assigned to the whole team, as the trick is to consider the team as a single agent, rather than a composite. A successful policy update is then guaranteed to be beneficial for the whole team at once. This approach has been shown by others to provide some level of behavioural specialization [START_REF] Ferrante | Evolution of Self-Organized Task Specialization in Robot Swarms[END_REF].

Figure 4 shows the evaluation for group sizes of 𝑘 = 1, 𝑘 = 2, 𝑘 = 4 (control 1) and a ℎ𝑜𝑚𝑜𝑔𝑒𝑛𝑒𝑜𝑢𝑠 team (control 2). All versions of the algorithm could reach optimal performance in at least one run out of 16, but displayed very different dynamics and overall performance both in speed and quality. 𝑘 = 1 is the fastest to converge, followed by 𝑘 = 2, which is twice slower when the median value is considered. Both control experiments 𝑘 = 4 and ℎ𝑜𝑚𝑜𝑔𝑒𝑛𝑒𝑜𝑢𝑠 are significantly slower, and their medians never get even close to the maximum score. They struggle in terms of the number of runs that discover the optimal strategy. Indeed, there is a single run for each control that reaches the optimal score while using 𝑘 = 1 and 𝑘 = 2 lead to all runs converging towards teams with optimal performance.

The first observation from these results is that, as postulated in the introduction of this paper, balancing the number of policy updates is critical to getting the best team performance. Larger values of 𝑘 experience degradation in performance due to concurrent exploration by multiple agents. This tends to add noise during the learning process since the evaluation variation will depend on the modification of several agents. These are similar to the exploratory action noise highlighted by [START_REF] Jen | When less is more: Reducing agent noise with probabilistically learning agents[END_REF][START_REF] Holmesparker | Clean rewards to improve coordination by removing exploratory action noise[END_REF], which is completely avoided for the lower value of 𝑘 = 1, and near-completely with 𝑘 = 2.

The second observation is that the canonical homogeneous team composition approach falls largely behind using 𝑘 = 1 and 2 in the heterogeneous team composition. While the need for coordination between agents might seem limited at first, agents still have to observe different points of interests. With homogeneous team composition, this requires learning a policy that can both target a single point of interest and avoid any point of interest already being observed. This can be an issue with agents sharing the same policy as coordination may be more challenging to achieve, a problem that will be explored further in the following Sections.

Setup 2: Coordination via Limited Synergies

The second setup also features 10 point of interest, with 2 special points of interest (the duo-POIs) that must be observed by 2 agents to provide a reward of 𝑟 = 3 points. As in the previous setup, agents may split among the normal points of interest (the 8 solo-POIs) for a sub-optimal maximal score of 4. In order to obtain the best score, the team must split into two groups, one for each duo-POIs, for a maximum score of 6. The obvious pitfall is that observing solo-POIs is a local optimum from which improvement can be attained only if two agents simultaneously change their behaviours to converge towards the same unoccupied duo-POI (see Section 4.1).

Figure 5 shows the score for group sizes of 𝑘 = 1, 𝑘 = 2, 𝑘 = 4 (control 1) and a homogeneous team (control 2). Figure 5-top shows that median values for the four methods converge to a final score of 5, with 𝑘 = 4 ending slightly below this score. The homogeneous team composition is the fastest to converge, followed by 𝑘 = 2 and 𝑘 = 1, which is counter-intuitive as one The four figures represent the learning curves of the sixteen independent replications of the four cases studied. The vertical dotted lines indicate when a specific number of independent runs were able to reach a score 𝑓 ≥ 5: 1/4 of the runs (thin), 1/2 (bold) and 3/4 (thin).

would have expected both the ℎ𝑜𝑚𝑜𝑔𝑒𝑛𝑒𝑜𝑢𝑠 and 𝑘 = 1 to fail at capturing even one duo-POI. Moreover, all methods converge to suboptimal team performance, with a score of 5 corresponding to the team observing 1 duo-POIs and 2 solo-POIs. 𝑘 = 2 (as well as 𝑘 = 4, at least to some limits) could have been expected to learn successful teams able to capture both duo-POIs by jointly updating policies of more than one agent. This does not seem to be the case at least when median values are considered.

The picture is very different when we look at independent runs for each experiment (the four graphs in Fig 5 -bottom). 𝑘 = 1 and 𝑘 = 2 outperform the two control experiments by both succeeding at reaching the optimal score of 6 (2 runs out of 16 for each method). This means that teams can capture both duo-POIs, a feat that is never achieved with either 𝑘 = 4 or the homogeneous team. Moreover, 𝑘 = 2 is faster to converge than 𝑘 = 1 in general (median values and proportion of runs than reach the threshold score of 5).

These results confirm the conclusion obtained with the first setup regarding the benefit of limiting the number of policy updates when confronted with a task that requires agents to coordinate. However, it is surprising that the level of coordination obtained with 𝑘 = 1 is sufficient to reach optimal team performance. By construction, the task was meant to require that two agents change their behaviours simultaneously, abandoning a suboptimal yet stable equilibrium (which is here illustrated by team scores of either 4 or 5, which are visible in the curves) in order to reach the optimal equilibrium (𝑠𝑐𝑜𝑟𝑒 = 6). The next Section provides an analysis as to why 𝑘 = 1 is able to learn optimal policies even if strong coordination is required to solve the task. modification of multiple agent trajectories can arise even if a single agent updates its policy. Rather than occurring purely during joint policy updates, coordination may arise between agents solely due to one agent influencing another. This is observed in the second setup when learning with 𝑘 = 1.

Coordination in the Phenotypical Space

Figure 6 shows examples of trajectories for all agents with 𝑘 = 1, extracted from a run where the team reaches the optimal score. The two graphs on Fig. 6-top show performance during learning and the distribution of agents among the points of interest starting from one initial condition used during evaluation (taken randomly out of the 16 initial conditions). It confirms the switch from one equilibrium to a better one as two agents jointly stop observing distinct solo-POIs and start observing the same duo-POI.

Figure 6-bottom features trajectories of agents for all 16 initial conditions for this particular run before and after two events: (a) when the team observes its first duo-POI and (b) when the team reaches optimal distribution over the two duo-POIs. Fig. 6-bottom-(a) show how the policy update of a single agent (here, agent no.2) changes the behaviour of other agents whether this change is slight (agents no.0 and no.3 marginally changes trajectories but still observe the same POI) or with significant impact on the score (agent no.1 switches and now observe a duo-POI, jointly with agent no.2). Similarly, Fig. 6-bottom-(b) show the same team transitioning to the optimal team coordination behaviour.

It is interesting to note that the right panel of Fig. 6-bottom-(a) and the left panel of Fig. 6-bottom-(b) show different team trajectories while they come from the same run and corresponds to a similar outcome (𝑠𝑐𝑜𝑟𝑒 = 5). This is due to the selection operator used in our evolutionary algorithm, which conserves a policy update as long as it yields a better or similar score than previously. This "soft" selection operator enables genetic drift, that is, the possibility to explore different behavioural strategies with a similar score.

Figure 7 shows results obtained using either the default "soft" selection operator (ie. a strategy change is kept if it does not worsen team performance) or a "strict" operator (ie. a strategy change is kept only if it improves team performance). Using a "strict" selection operator, that is, when policy updates are kept only if the team score improves, turns out to be detrimental for 𝑘 = 1. It significantly reduces the overall performance for 𝑘 = 1 as the median remains stuck to a score of 4, which corresponds to the worse sub-optimal equilibrium. 𝑘 = 2 moderately suffers from using a strict selection operator, as performance increase is only delayed. Nevertheless, the algorithm using 𝑘 = 2 can still learn optimal team strategies. At the same time, 𝑘 = 1 now fails at completing this task (only one run reaches the sub-optimal score of 5).

Due to the complex and counter-intuitive effect of the genotype-phenotype mapping at work in a robotic task where robots interact, it is no surprise that choosing a priori the number of policy updates to perform per learning step is challenging. In the next Section, we present results using an adaptive method for dynamically choosing the 𝑘 value of our algorithm.

Adaptive group size

An important lesson from the previous results is that different setups require different values for the number of policies to be updated simultaneously. In the general case, it can be challenging to guess to which value 𝑘 should be set, and it is natural to aim for an adaptive method that would automatically tune the value of 𝑘 to match the problem at hand.

In this Section, we evaluate the CC-1+1-ES 𝑘 adaptive algorithm presented in Section 3. CC-1+1-ES 𝑘 adaptive extends the algorithm used so far by encapsulating the EXP3 Multi-Armed Bandit algorithm in order to dynamically tune the number of policies that should be updated, based on results obtained so far.

Figure 8 shows the median on 16 independent runs of the score for the CC-1+1-ES 𝑘 adaptive algorithm, along with results shown previously with the original CC-1+1-ES 𝑘fixed algorithm with 𝑘 = 1, 2 and 4 for reference (see Fig. 4 and5, here with transparent curves).

Results show that the adaptive method closely matches the best-performing algorithm with fixed 𝑘 (median values, first row), though it is a bit slower to reach the best value. This is expected as the cost of exploration cannot compete with an algorithm initialized with the best value for 𝑘. This is confirmed by looking at the independent runs (second row): in both setups, the adaptive version is shown to be able to reach the optimal team behaviours on par with the non-adaptive method using the optimal value for 𝑘, and without the burden to guess this value prior to learning.

CONCLUSION

In this paper, we addressed the problem of estimating the contribution of robotic agents when performance is measured at the level of the team. When several agents update their policy simultaneously, it is difficult to identify a single agent's contribution to the team, even though joint updates are necessary to find synergies when a high level of coordination is required. We proposed an algorithm that automatically balances the number of agents' policies that should be updated at each learning step in order to discover the degree of coordination required, without requiring prior assumptions on the task at hand.

1 1 Figure 3 :

 113 Figure3: Architecture of the policy function. Each sensor gives information about the distance to the obstacle and the type of POI detected if within range. The policy is a multilayered neural network, with 38 dimensions as input, a hidden layer of 16 dimensions plus a bias term for each layer. Hyperbolic tangent (tanh) is used as an activation function. The action dimension is 2. The number of parameters is 658.

Figure 4 :

 4 Figure 4: Performance of the algorithm with fixed 𝑘 = 1, 2, 4 and the homogeneous team on Setup 1. Top: each curve represents the median on 16 independent replications. Bottom: The four figures represent the learning curves of the sixteen independent replications of the four cases studied.

Figure 5 :

 5 Figure 5: Performance of the algorithm with fixed 𝑘 = 1, 2, 4 and the homogeneous team on Setup 2. Top: each curve represents the median on 16 independent replications. Bottom: The four figures represent the learning curves of the sixteen independent replications of the four cases studied. The vertical dotted lines indicate when a specific number of independent runs were able to reach a score 𝑓 ≥ 5: 1/4 of the runs (thin), 1/2 (bold) and 3/4 (thin).

Figure 6 :Figure 7 :

 67 Figure6: Impact of single mutations on the agents' behaviours in Setup 2 for 𝑘 = 1. Top: results from a single run (run #6) where the score reaches the maximal value of 6, along with the distribution of the POI throughout learning for one of the initial conditions out of 16. Red and blue lines indicate that respectively one and two agents occupy these POI. Bottom: Trajectories of agents from the 16 initial conditions before and after a single agent's policy update yields a significant score increase. The team switches (a) from 4 solo-POIs observed to 2 solo-POIs and 1 duo-POIs observed and (b) from 2 solo-POIs and 1 duo-POIs to 2 duo-POIs.Each agent perceives the others during the experiment and may either ignore or be influenced by their teammates. Consequently, simultaneous

Figure 8 :

 8 Figure 8: Performance of the CC-1+1-ES 𝑘 adaptive algorithm 𝑘 in the two setups. Top: median scores. Previous results shown with transparent curves. Bottom: scores for each of the 16 replications. The vertical dotted lines indicate when a specific number of independent runs are able to reach a score 𝑓 ≥ 5: 1/4 of the runs (thin), 1/2 (bold) and 3/4 (thin).

Table 1 :

 1 Initial range values for locations of rovers and POI.

	5, 0], 𝛼 ∈ [ 3𝜋 4 , 2𝜋 ]

Table 2 :

 2 Experimental parameters

ACKNOWLEDGMENTS

This work is funded by ANR grant ANR-18-CE33-0006.


Abderezak Aouali

Thomas Lafargue-Tallet

Stéphane Chevalier

Alain Sommier

Raymond Peiffer

Maximilian Taillandier

Jean-Christophe Batsale

High power density laser estimation using quantitative thermal imaging method

teaching and research institutions in France or abroad, or from public or private research centers.