Luisa Werner

Nabil Layaïda

Pierre Genevès

Jérôme Euzenat
email: jerome.euzenat@inria.fr

Damien Graux
email: grauxd@tcd.ie

Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods

Reproducibility is a desirable property of scientific research. On the one hand, it increases confidence in results. On the other hand, reproducible results can be extended on a solid basis. In rapidly developing fields such as machine learning, the latter is particularly important to ensure the reliability of research. In this paper, we present a systematic approach to reproducing (using the available implementation), replicating (using an alternative implementation) and reevaluating (using different datasets) state-of-the-art experiments. This approach enables the early detection and correction of deficiencies and thus the development of more robust and transparent machine learning methods. We detail the independent reproduction, replication, and reevaluation of the initially published experiments with a method that we want to extend. For each step, we identify issues and draw lessons learned. We further discuss solutions that have proven effective in overcoming the encountered problems. This work can serve as a guide for further reproducibility studies and generally improve reproducibility in machine learning.

Introduction

Experiment reproducibility is a cornerstone of scientific research. It is so for the experimenters, because it allows them to be more confident in the results they claim. It is so for the research community, because it allows other researchers to stand on a solid ground in developing their own work. This leads to more reliable and superior research results, which is beneficial for the society as a whole.

In the machine learning community, strong incentives emerge to provide the necessary information to reproduce results [START_REF] Pineau | Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)[END_REF]. More generally, this is a basic open science requirement to increase accountability and reproducibility. Reproducibility guidelines and checklists focus on encouraging researchers to provide more detailed documentation of their published work (IJCAI 2022;ICML 2023;AAAI 2023). However, it is not obvious that such guidelines are sufficient to independently reproduce experiments. Indeed, random operations may be introduced in several places [START_REF] Chen | Towards Training Reproducible Deep Learning Models[END_REF], hyperparameters may change the behavior of models [START_REF] Henderson | Deep Reinforcement Learning That Matters[END_REF][START_REF] Lucic | Are GANs Created Equal? A Large-Scale Study[END_REF], components may exchange information in imprecise ways, the data processing pipeline may be overlooked, etc.

Same software Different software

Same datasets

Reproduce Replicate

Different datasets Reevaluate

Table 1: Overview of the three steps. [START_REF] Lones | How to avoid machine learning pitfalls: a guide for academic researchers[END_REF][START_REF] Forde | The Scientific Method in the Science of Machine Learning[END_REF]. All these factors can compromise the reproducibility of experiments.

Various terminologies have been considered for reproducibility in computer science [START_REF] Drummond | Replicability is not reproducibility: nor is it good science[END_REF][START_REF] Goodman | What does research reproducibility mean?[END_REF][START_REF] Rougier | Sustainable computational science: the ReScience initiative[END_REF][START_REF] Plesser | Reproducibility vs. replicability: a brief history of a confused terminology[END_REF][START_REF] Gundersen | On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications[END_REF]. We use and enrich the terminology adopted by the ACM (ACM 2016), as summarized in Table 1: • 'repeat' means reexecuting an experiment with the same code, the same parameters, the same data by the same experimenter; • 'reproduce' means performing an experiment with the same software, the same parameters and the same data but by a different experimenter; • 'replicate' means performing an experiment independently on the same data but using different software and by a different experimenter; • to these, we add 'reevaluate' which means performing an experiment independently on different data. When necessary, 'reproducibility' is also used as a term covering all such activities.

This work is motivated by our objective to extend the Knowledge Enhanced Neural Networks (KENN) [START_REF] Daniele | Knowledge Enhanced Neural Networks for Relational Domains[END_REF] to knowledge graphs. To that extent, we need to reimplement the system, available in Tensorflow [START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems[END_REF], in the graph-specific library Py-Torch Geometric [START_REF] Fey | Fast Graph Representation Learning with PyTorch Geometric[END_REF]. This is a common use case as, beyond the motivation to extend, there may be various other reasons to reimplement a system [START_REF] Tian | ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero[END_REF][START_REF] Alahmari | Challenges for the Repeatability of Deep Learning Models[END_REF]. For that purpose, we provide a progressive approach for extending a machine learning method by first going through reproducing, replicating and reevaluating its results. This approach aims at ensuring that the reimplementation is reliable. It is tempting to skip these steps and jump directly to method extension. On the contrary, we argue that this approach allows re-searchers to better understand potential problems at the specific stage where they occur. Further, we identify the obstacles encountered when conducting these steps and how they may be overcome. The particular case on which we detail and demonstrate our approach allows us to derive general lessons learned. Although we do not claim that this work can solve all reproducibility problems, it contributes to improve reproducibility in the field of machine learning.

Related Work

Given the high degree of non-determinism of machine learning techniques in general, the problem of reproducing results has gained importance with the increase of interest in the field. Recent surveys [START_REF] Gundersen | State of the Art: Reproducibility in Artificial Intelligence[END_REF] show that even papers published at prestigious conferences are not sufficiently documented and report on a reproducibility crisis [START_REF] Gundersen | The Reproducibility Crisis Is Real[END_REF]). An empirical study [START_REF] Raff | A Step Toward Quantifying Independently Reproducible Machine Learning Research[END_REF]) reveals difficulties in independently replicating published papers. Due to increased awareness, several conferences, including AAAI, promote reproducibility by encouraging authors to provide source code, experimental descriptions and datasets of their papers through reproducibility guidelines and checklists [START_REF] Pineau | Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)[END_REF]IJCAI 2022;AAAI 2023).

Further, there has been a surge in documented reproduction and replication attempts, particularly encouraged through reproducibility challenges [START_REF] Pineau | Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)[END_REF][START_REF] Sinha | ML Reproducibility Challenge 2021[END_REF][START_REF] Manninen | Challenges in Reproducibility, Replicability, and Comparability of Computational Models and Tools for Neuronal and Glial Networks, Cells, and Subcellular Structures[END_REF]. Such challenges ask interested individuals to replicate results from recent papers. Difficulties such as the lack of detailed documentation, e.g. hyperparameters (Ankit et al. 2022) or even failure to reproduce [START_REF] Blanco | Non-replicability circumstances in a neural network model with Hodgkin-Huxley-type neurons[END_REF]) are frequently reported. Their motivation differs from ours, as our goal is not to achieve reproducibility for its own sake. We reproduce experiments to ensure that the system we reimplement retains the function of the initial implementation.

Knowledge Enhanced Neural Networks

To overcome the limitations of deep learning such as datahungriness or black-box properties, the research field of neuro-symbolic integration aims at combining neural methods with symbolic AI [START_REF] Garnelo | Reconciling deep learning with symbolic artificial intelligence: representing objects and relations[END_REF]. KENN is a neuro-symbolic method that integrates a base neural network with knowledge enhancement layers in an end-to-end differentiable way. The knowledge enhancement layers encode prior knowledge in form of first-order logic clauses and revise the predictions of the neural network with respect to the satisfaction of the prior knowledge. Fuzzy logic is used to represent first-order logic clauses in the real-valued domain. A knowledge enhancement layer contains clause weights as trainable parameters which are jointly optimized with the parameters of the base neural network and quantify the importance of a clause. By introducing binary predicates in its logic language, KENN (Daniele and Serafini 2023) can model graph data and has been applied to a node classification task on the Citeseer dataset [START_REF] Lu | Link-Based Classification[END_REF]. The Citeseer dataset consists of scientific papers belonging to one of the six computer science research classes and citations among them. The dataset can be modelled as Because we aim at extending KENN with more graph learning capabilities, we need to be sure that our work stands on firm ground. For that purpose, we reproduce the experiments performed by the authors.

In [START_REF] Daniele | Knowledge Enhanced Neural Networks for Relational Domains[END_REF], the authors report the performance of KENN for experiments on multiple training set sizes (10%, 25%, 50%, 75%, 90%). For each size, 100 independent runs are conducted. KENN is compared to the standalone base neural network and to Relational Neural Machines (RNM) [START_REF] Marra | Relational Neural Machines[END_REF] and Semantic-Based Regularization (SBR) [START_REF] Diligenti | Semantic-based regularization for learning and inference[END_REF]. The experiments support the following hypotheses: H1 KENN consistently outperforms the base neural network for all training set sizes. H2 The performance gain due to the knowledge enhancement is larger when training data is limited. H3 KENN leads to similar or superior results compared to RNM and SBR.

Methodology

We extend KENN in the framework PyTorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF] in conjunction with the library PyTorch Geometric. To this end, a comprehensive reproducibility study is the best way to ensure that KENN is extended on a reliable basis. With this objective in mind, as illustrated in Fig. 1, we conduct the following steps: 1. We reproduce the results obtained with the initial implementation; 2. We reimplement the initial experiments and replicate them; 3. We reevaluate the experiments on additional datasets.

For each step, we detail the required information, the obstacles we had to overcome and how we managed to overcome them. Further, we evaluate whether the KENN results are confirmed and report the lessons learned.

We refer to the following publicly available material. The reproduction and replication is based on the results reported in the paper (Daniele and Serafini 2023) (that we call the initial paper) and on the reported experiments1 (that we call the initial experiments). The experiments make use of the Python package KENN22,3 . In addition, we also extract information from other related papers (Daniele andSerafini 2019, 2020). We preserve, as far as possible, the input (dataset) and output format (results) of the initial experiment. In this work, we focus on the transductive part of the initial experiments with KENN. In the following, we refer to our implementation as reimplementation.

All experiments in this work are conducted on a machine running an Ubuntu 20.4 equipped with an Intel(R) Xeon(R) Silver 4114 CPU 2.20GHz processor, 192G of RAM and one GPU Nvidia Quadro P5000.

Evaluation Criteria

In order to judge whether we have successfully reproduced, the question of evaluating reproducibility arises. For this purpose we define two reproducibility targets.

As common in natural sciences, qualitative reproducibility refers to checking whether experimental results support the hypotheses of the initial paper and come to the same conclusions. Coming rather from an engineering perspective, quantitative reproducibility aims at obtaining the same or close results as the initial experiment. In Fig. 1, they are respectively noted as > and =. In the context of the experiments with KENN, qualitative reproducibility refers to checking if the confirmed hypotheses in the initial paper are still supported by the reimplementation.

Regarding qualitative reproducibility, we focus on the hypotheses H1 and H2. The verification of H3 would involve reproducing the results of the baselines SBR and RNM which is beyond the scope of this work. To test H1 and H2, we adopt the procedures used in the initial paper. For H1, a one-sided independent Student t-test with significance threshold 0.01 is employed to assess the superiority of the mean accuracy of KENN over the base neural network. For H2, which establishes a relation between the training set size and the delta of base neural network and KENN, no precise evaluation procedure is mentioned. However, the absolute difference between the mean test accuracies for the experiments is observed.

Regarding the quantitative reproducibility, absolute equality of the results is hard to be expected, since even reexecuting the initial implementation does not lead to absolutely equal numbers. This can be due to differences in software or hardware or the absence of fixed seeds [START_REF] Chen | Towards Training Reproducible Deep Learning Models[END_REF][START_REF] Pham | Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance[END_REF][START_REF] Pytorch | Reproducibility[END_REF]

Reproduction

As a first step, we reproduce the initial experiments with KENN. Therefore, we need (a) the executable, (b) the dataset, (c) the instructions and environment information to run the executable, (d) the procedure to collect and interpret the results. Given these elements, it is possible to process the instructions to reproduce the experiment.

Pitfalls and Workarounds

The paper (Daniele and Serafini 2020) provides the KENN2 2 package. It contains the implementation of the knowledge enhancement layers in Tensorflow but not the experiments. Through an exchange with the authors, we gained access to the source code and data of the experiments which were made publicly available 1 . They also pointed to another paper (Daniele and Serafini 2023) which further documents the experiments. The hyperparameters in the public implementation correspond to the results reported in [START_REF] Daniele | Knowledge Enhanced Neural Networks for Relational Domains[END_REF] but not to the results found in [START_REF] Daniele | Neural Networks Enhancement through Prior Logical Knowledge[END_REF].

The repository provides (a) a reference to the KENN2 package, (b) a link to the dataset, (c) a README file with instructions to run the code, and (d) a Jupyter notebook to analyze the results. The initial experiments use a dataset that is included in the repository instead of referring to an external, widely available version of the dataset. A link to the data source was provided, but it was inaccessible. Neither the origin of the data nor any filtering or preprocessing steps were documented. The instructions to run the experiments are reasonably complete, including a description of the required Python modules as a requirements file and the full command line to be run. However, the requirements only contain a list of packages without their version number. This can be a critical concern as Python packages evolve frequently, sometimes compromising compatibility. The Python version is not defined either which can be crucial for the successful execution of the experiments. We inferred it from the descrip-tion of the KENN2 package. In both papers (Daniele andSerafini 2020, 2023) experimental results are reported in numeric tables. Despite seeded random operations, we observe subtle variations in the results. We note a non-deterministic set operation which may introduce variation. As a result, exact quantitative reproducibility cannot be guaranteed.

Results

After achieving a fully functional environment setup, we successfully run the experiments. The obtained results are reported in Table 2. In comparison to the results initially provided [START_REF] Daniele | Knowledge Enhanced Neural Networks for Relational Domains[END_REF], we add information such as the standard deviation of the test accuracies over all runs and the p-values for a t-test.

Qualitative Reproducibility For H1, KENN significantly outperforms the base neural network for all training set sizes (p ≪ 0.01) which is consistent with the authors' claims. Regarding H2, we observe that the reported difference (column delta) in mean accuracy between KENN and the base neural network is larger for smaller training set sizes. Note that, since the initial experiments use paired samples, the means of the deltas can be reported, though we can only report the difference of the means.

Quantitative Reproducibility

The reported and reproduced results cannot be compared statistically, since the full sample of the reported results is not available.

Lessons Learned

L1: Provide a Precise Identification of the Software Environment, Source Code and Dataset A first lesson learned is that the complete description of the software environment including version numbers of all modules should be provided. Furthermore, the datasets used should be described precisely, including information on their origin and on any applied preprocessing. Both are critical information to reproduction.

L2: Automate the Steps for Reproduction A second lesson learned is that the reproduction should be made as automatic as possible by the initial authors. Such an automation would even be useful for the initial authors to be able to repeat their experiments routinely.

L3: Provide Contact Information and Be Reachable

The reachability and assistance of the authors was essential to clarify open questions and to get access to key components for reproducibility such as the experiment source code.

L4: Distinguish Different Experiments We missed a unique reference to the experiments. In fact, the authors published a first version of a paper [START_REF] Daniele | Knowledge Enhanced Neural Networks[END_REF] and then several others [START_REF] Daniele | Knowledge Enhanced Neural Networks for Relational Domains[END_REF], 2020[START_REF] Lones | How to avoid machine learning pitfalls: a guide for academic researchers[END_REF] which contain improvements and additional experimental settings. Still, all papers point to the same repository. This complicates the reproduction of the experiments of each paper. It would be a good practice to specify per paper the version of the repository or to use another repository.

Replication

In this section, we refer to the reimplementation in PyTorch and the replication of the experiments. Again, we compare the obtained results to the results of the published and reproduced experiments. To reimplement the system, we first identify the main components of the method by examining, on the one hand, the concepts of KENN introduced in the initial paper. On the other hand, we examine the code of the initial implementation to recover necessary information that is underspecified or not explicitly mentioned in the paper.

As main components we identify (a) the data preprocessing, (b) the model definition, (c) the training loop, and (d) the hyperparameter definition. We reimplement these components as follows.

Data Preprocessing. We use the same Citeseer dataset and prior knowledge as provided for the initial experiments. We preserve the data splitting procedure, as well as the rebalancing of the prediction classes.

Model Definition. The KENN model consists of two stacked components: The base neural network and the knowledge enhancement layers. In the initial implementation, the base neural network is implemented with Keras [START_REF] Chollet | Keras[END_REF] and the knowledge enhancement layers are imported from the KENN2 2 package written in Tensorflow. We reimplement both the base neural network and use the knowledge enhancement layers from the KENN2 package 3 .

Training Loop. In the training loop, we replace the optimizer and the loss function implemented in Tensorflow by their PyTorch equivalent.

Hyperparameter Definition. We identify the hyperparameters in the experiments and their values. To keep track of hyperparameters in a clean manner, we connect the reimplementation to an experiment tracking tool [START_REF] Biewald | Experiment Tracking with Weights and Biases[END_REF].

Pitfalls and Workarounds

At first sight, the reimplementation in PyTorch seems straightforward. However, we struggle in identifying the hyperparameters used in the initial implementation. The relevant information has to be gathered from various sources. The set of revealed hyperparameters, their assignments and how we recovered them is summarized in and the binary preactivation value. An additional subset of parameters is defined in a user modifiable script in the repository of the initial implementation. These script contains information on early stopping (and related parameters), the number of epochs, as well as the size of the validation set. Additional parameter values are required that are not mentioned in the paper, nor in the code documentation. By reviewing the source code, the number of knowledge enhancement layers and the batch size, for example. An additional subset of hyperparameters is implicitly introduced and defined by framework-specific functions and their default assignments. These default values are found in the Tensorflow software documentation. From there, we recover the weight initialization of the dense layers in the base neural network and the optimizer-specific parameters.

Determining the values of some additional parameters turns out to be even more challenging. To get useful insights, we examine the isolated behavior of each component to ensure that they produce the same output, given some input. While randomness is essential to training neural networks, it complicates the analysis of the algorithm behavior. For the inspection of some components, we temporarily simplify operations and replace random numbers by fixed values. This allows to identify that linear layers are by default differently initialized in Tensorflow and in PyTorch. In Tensorflow, they are initialized randomly following a Glorot uniform distribution and the bias is initialized with zeroes. However, in PyTorch the weights and biases are randomly initialized following the uniform distribution.

Results

After recovering the full set of required parameter values and completing the reimplementation of KENN, we now assess the replicability of experiments. The results of the replication are summarized in Table 3.

Qualitative Reproducibility We first check whether H1 and H2 hold in the replicated experiment. We observe that the p-values of the t-test comparing the test accuracies for all training set sizes fall below the significance threshold in favor of H1 which is thus supported by the replication. For H2, considering the deltas of the mean test accuracies across the training set sizes, no clear relationship between training set size and its effect on the knowledge enhancement is apparent. This may be due to a reproducibility failure or to the lack of an explicit procedure to test H2 in the initial work. In conclusion, we are able to confirm H1, but we are not able to replicate the results concerning H2.

Quantitative Reproducibility

The average accuracies of both implementations are relatively close (the highest observed difference is 0.02). However, their distributions are not identical. The similarity of the test accuracy distributions is computed against the test accuracy of the reproduced results, since we do not have the initial results from the authors of KENN. The p-values of the KS-test are listed in the right column. For the 10% training size, a significant (p < 0.01) difference between the distributions of the reproduced and replicated results is detected. For the remaining training set sizes, no difference appears significant. Hence, we consider our results as reasonably similar to those of the initial implementation, at least for large sample sizes. The accuracy obtained with 10% training set size appears less variable than the one on larger training sets.

Lessons Learned

L5: Document Hyperparameters Exhaustively The identification of all hyperparameters together with their assignments is critical for replication as they can considerably affect the results and thus the conclusions drawn from them. While some hyperparameters such as learning rate and batch size are standard in deep learning methods, custom models often define their own hyperparameters, such as KENN's clause weights. Since they affect different components, hyperparameters are often declared at different stages in the experiment, which complicates their identification. In particular, some hyperparameters are implicitly defined as default parameters in the used library and are easily overlooked. A complete configuration file with the exhaustive list of hyperparameters, their description, and their value is key to replicability.

L6: Provide Clear Procedures to Check Claims To evaluate qualitative reproducibility, we rely on the authors' procedures for verifying their hypotheses. With respect to H2, the lack of a precise procedure complicates the evaluation of the replicability. Clear procedures for the verification of claims are essential to ensure their replicability.

L7: Define Standards to Evaluate Reproducibility

In order to evaluate quantitative reproducibility, we use the KStest that can only give statistical evidence that two distributions are not equal. Other metrics are proposed in (Piantadosi, Marrone, and Sansone 2019). To the best of our knowledge, no community standard on the evaluation of quantitative reproducibility exists. A community-agreed standard is needed to determine when two distributions of results can be considered equivalent.

Reevaluation

Having reproduced and replicated the experiments with KENN, we now reevaluate the method on two more datasets Cora and Pubmed [START_REF] Yang | Revisiting Semi-Supervised Learning with Graph Embeddings[END_REF].

Results

The results for Cora and Pubmed are shown in Table 5 and6.

When applying KENN to Pubmed, we encounter a performance issue with the data preprocessing in the initial implementation. We modify the preprocessing to improve its scalability while maintaining its functionality.

Conclusion and Perspective

In this article, we first progressively reproduced, replicated and reevaluated the experiments with KENN before extending it. In terms of qualitative reproducibility, in most experiments the hypothesis that KENN outperforms the base neural network (H1) is supported. The relationship between training set size and accuracy (H2) is observed for the reproduced experiments but less clear for the replicated and reevaluated experiments. We summarize all the lessons learned in Table 7.

In a broader context, the progressive approach in this work can be incorporated in a general workflow of extending a related method in machine learning while considering reproducibility steps, as illustrated in Fig. 2. While it is tempting to directly jump from a task to the development of a new method, the inclusion of the reproduce, replicate and reevaluate steps can increase the trustworthiness in experimental results. To the three steps conducted in this work, record and repeat can be added to better integrate the new (extended) method in the SOTA. Recording provides a sufficiently detailed documentation to make the new method accessible and reproducible to the community. Repeating the experiment can serve as a self-check of documentation and/or automation. tension is flawless. In reality, each step may fail and lead the design back to the previous action. However, it helps to isolate the cause of failure at the earliest possible stage. Further, we note that KENN predominantly satisfies common reproducibility guidelines. However, we encounter difficulties in reproducing, replicating and reevaluating this method. This shows that even though guidelines impose an additional burden for developers, easily and clearly reproducible work reduces the effort required for extending previous work and therefore accelerates advancements. In this sense, reproducibility guidelines should be continuously evaluated and modified (if needed) to ensure that they effectively serve their purpose. From our point of view in the context of this work, reproducibility checklists could improve by integrating some of our lessons learned.

Finally, common reproducibility guidelines are mainly oriented towards reproduction, which ensures that published results correspond to those obtained from the code. Given strong automation, this may only require to clone and run. Even if replication and reevaluation outweigh the effort of reproduction, in return, they provide more insight into the details of an implementation or a method. Moreover, reimplementations make methods more available to the community. In this sense, efforts towards replication and reevaluation should continue to be encouraged, e.g. through reproducibility tracks or challenges [START_REF] Neurips | NeurIPS 2019 Reproducibility Challenge[END_REF]ICLR 2019).

Experiment Material

The reimplementation of the experiments is available at a dedicated Git repository4 under the hash a894cae297b47f6d1acfa8e8dab99603f7b5e996. It contains the following elements:

• the source code,

• the execution instructions,

• the software requirements,

• the result evaluation script,

• the raw files of the experiment results. Further, the adaptions of the initial implementation to Pubmed and Cora is included.

Fig. 2

 2 displays an ideal situation in which the path to ex-Reproduction L1 Provide a precise identification of the software environment, source code and dataset L2 Automate experiment execution L3 Provide contact information and be reachable L4 Distinguish different experiments Replication L5 Document hyperparameters exhaustively L6 Provide clear procedures to check claims L7 Define standards to evaluate reproducibility Reevaluation L8 Conduct hyperparameter search L9 Verify results on other datasets

  . Therefore,

		Reported results		Initial Implementation
	train	NN	KENN Delta	NN	KENN	Delta
	10% 0.544 0.652 0.108 0.540 (0.067) 0.651 (0.017) 0.110 (0.067)
	25% 0.629 0.702 0.073 0.630 (0.018) 0.702 (0.012) 0.072 (0.012)
	50% 0.680 0.744 0.065 0.681 (0.187) 0.745 (0.012) 0.064 (0.021)
	75% 0.733 0.788 0.055 0.733 (0.025) 0.791 (0.016) 0.058 (0.026)
	90% 0.759 0.808 0.049 0.758 (0.027) 0.807 (0.022) 0.049 (0.028)

Table 2 :

 2 Reproduction results.we evaluate the steps of replication and reevaluation by examining the distributions of the reported test results and in particular their similarity. In order to test their equality, we compute the two-sided Kolmogorov-Smirnov goodness-offit test (KS-test)[START_REF] Massey | The Kolmogorov-Smirnov test for goodness of fit[END_REF], which checks whether two samples are drawn from the same distribution. If the p-value is below the significance threshold, there is evidence that the results of the experiments are not drawn from the same distribution. It has to be mentioned that the KS-test can only provide significant results in measuring inequality.

Table 3 :

 3 Replication results.

			Reimplementation
	train	NN	KENN	Delta of means	p-values (KS Test)
	10%	0.550 (0.042)	0.629 (0.076)	0.079	0.001
	25%	0.632 (0.016)	0.676 (0.088)	0.044	0.024
	50%	0.681 (0.014)	0.741 (0.028)	0.060	0.583
	75%	0.729 (0.022)	0.785 (0.039)	0.056	0.054
	90%	0.748 (0.026)	0.806 (0.020)	0.058	0.702

Table 4 .

 4 A first subset of hyperparameter values is explicitly named in the initial paper including the architecture of the base neural network, the initialization of the clause weights

	Parameter	Value	In Paper	In Code	Defaults
	NN -Number of Hidden Layers	3	✓	✓
	NN -Number of Hidden Neurons	50	✓	✓
	NN -Hidden Layer Activation	ReLu	✓	✓
	NN -Output Layer Activation	Linear	✓	✓
	KENN -Clause Weight Initialization	Constant, 0.5	✓	✓
	KENN -Binary Preactivations	500	✓	✓
	KENN -Number of KE Layers	3		✓
	KENN -Range Constraint	[0.0, 500.0]		✓
	Epochs	300		✓
	Batch Size	Full-batch		✓
	Loss function	Categorical Cross-entropy		✓
	Early Stopping -Patience	10		✓
	Early Stopping -Min Delta	0.001		✓
	Learning Rate	0.001			✓
	NN -Weight Initialization	Random, Glorot uniform			✓
	NN -Bias Initialization	Constant, Zeroes			✓
	Optimizer	Adam (Kingma and Ba 2015), β1 = 0.9, β2 = 0.99, ϵ = 10 -7			✓
	Dropout Rate	0.0, no dropout

Table 4 :

 4 The set of hyperparameters used in the initial experiments.

Table 5 :

 5 Reevaluation results on the Cora dataset.

Table 6 :

 6 The goal of this reevaluation is (1) to check if the implementations behave robustly in the same way on different datasets and (2) to evaluate whether the hypotheses for KENN are valid on other datasets. These datasets are also citation graphs. While Cora and Citeseer have a similar size, Pubmed is significantly larger. Similarly to the experiments on Citeseer, the prior knowledge encodes a relation of paper category and citations. To avoid variations in the results due to hyperparameters, we use the set identified in Table4for all datasets. In general, hyperparameters should be determined separately for each dataset in order to obtain models with the best possible prediction quality. However, we are mainly concerned with obtaining the same results instead of the best Reevaluation results on the Pubmed dataset.

		Initial Implementation	Reimplementation
	train NN	KENN Delta	NN	KENN	Delta of means	p-values (KS-Test)
	10%	0.333 (0.096)	0.405 (0.002)	0.071 (0.096)	0.326 (0.098)	0.404 (0.003)	0.077	3.9 10 -1
	25%	0.341 (0.108)	0.416 (0.002)	0.075 (0.108)	0.380 (0.080)	0.414 (0.004)	0.034	7.1 10 -2
	50%	0.409 (0.095)	0.443 (0.004)	0.033 (0.096)	0.364 (0.133)	0.441 (0.005)	0.077	1.3 10 -1
	75%	0.447 (0.143)	0.498 (0.004)	0.051 (0.143)	0.414 (0.176)	0.495 (0.007)	0.081	1.3 10 -1
	90%	0.505 (0.011)	0.510 (0.008)	0.004 (0.011)	0.504 (0.015)	0.504 (0.015)	-0.0001 5.9 10 -1
	possible results.

  Integrate new work in the state-of-the-artFigure 2: Extending machine learning methods with reproducibility steps.L9: Verify Results on Other Datasets Overall, the reevaluation experiments show that KENN improves the accuracy on the Cora dataset, but not on the Pubmed dataset. Furthermore, by reevaluation on PubMed, we can detect performance issues and address them, which makes the code applicable to larger datasets in general. As a lesson learned, it can be derived that the application of a model to various datasets can either strengthen result by showing robustness or reveal weaknesses of methods with respect to specific datasets.

	SOTA	method choose	reproduce	replicate reevaluate	method extend	record	repeat	method new
				Qualitative Reproducibility H1 is supported in both im-
				plementations (p ≪ 0.01) for Cora. Concerning H2, the
				deltas in the initial implementation indicate a relationship
				between decreasing training set size and knowledge en-
				hancement. However, this difference is rather uncertain for
				the reimplementation. For H1 on PubMed, the p-values of
				the t-tests are not below the significance threshold across
				all training set sizes, for the initial implementation only for
				training set sizes 10% and 25% and for the reimplementation
				for training set sizes 10%, 50%, 75% and 90%. Hence, no
				significant performance is gained with KENN. Furthermore,
				no monotonically increasing benefit of knowledge enhance-
				ment for smaller training set sizes can be observed with both
				implementations, therefore H2 is not supported by Pubmed.
				Quantitative Reproducibility Comparing the distribu-
				tion of the test accuracies in both experiments, the p-values
				of the KS-tests are below the significance threshold and thus
				suggest rejection for all training dimensions except 50%.
				Therefore, we have statistical evidence for inequality and
				thus have not reached qualitative reproducibility on Cora.
				However, for Pubmed, the behavior in both implementations
				is aligned. The KS-test suggests no significant difference be-
				tween both implementations. In conclusion, quantitative re-
				producibility is considered achieved on Pubmed.
				Lessons Learned
				L8: Conduct Hyperparameter Search As expected, with
				the hyperparameter values considered earlier KENN does
				not achieve competitive results on Cora and Pubmed in com-
				parison to the SOTA (PapersWithCode 2023). Hyperparam-
				eter tuning significantly affects the results of the experiment.
				With an appropriate set of hyperparameters for Cora and
				Pubmed, better results with KENN may be obtained.

Table 7 :

 7 Summary of the lessons learned.

https://github.com/rmazzier/KENN-Citeseer-Experiments

https://github.com/DanieleAlessandro/KENN2

https://github.com/HEmile/KENN-PyTorch

https://gitlab.inria.fr/tyrex-public/reproducibility-aaai24

Acknowledgements

We are grateful to the authors of KENN for their help. This research has been partially supported by the ANR project GraphRec (ANR-23-CE23-0010) and by the MIAI Knowledge communication and evolution chair (ANR-19-P3IA-0003).


Julie Albagnac

Pierre Léard

Dominique Anne-Archard

Pierre Brancher

Johan Pinaud

Sébastien Cazin

Zeinab Rida

Dynamics of an inclined Vortex Ring interacting with a density stratification

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.


Mathilde Theunissen

Hugo Pousseur

Pedro Castillo

Alessandro Corrêa

Cooperative Architecture Using Air and Ground Vehicles for the Search and Recognition of Targets

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Cooperative architecture using air and ground vehicles for the search and recognition of targets M. Theunissen, H. Pousseur, P. Castillo, A. Correa Victorino,

Abstract-A cooperative navigation architecture for the search and recognition of targets using aerial and ground vehicles is proposed in this paper. The architecture allows to manage aerial and ground vehicles to autonomously perform different tasks in an independent or cooperative way. For our application, two main tasks are conceived; aerial monitoring of a surface to search for targets, and target ground recognition. In the target aerial detection task, the aerial drone tracks autonomously a trajectory, computed to cover all the surface to monitoring, to search for targets using vision algorithms. Once one of them is detected its relative position is sent to the cooperative architecture. After the aerial drone has covered the entire area, the architecture computes and assigns to each ground vehicle the closest target found. Then, each ground vehicle navigates autonomously avoiding obstacles (if presents) to its assigned target. For verifying the success of the mission, the aerial vehicle flies following the dynamic center of mass of the ground vehicles. Real-time experiments are carried out to validate the proposed architecture. Main results, depicted in some graphs, corroborate the good performance in closed loop.

I. INTRODUCTION

In recent years, works on cooperation between heterogeneous multi-robot systems have multiplied. By sharing their capacities, such systems aim to gain in flexibility, efficiency and robustness and tend to achieve total autonomy in dynamic environments. Some frameworks for cooperative air and ground vehicles have been mostly used for collaborative path-planning and obstacle avoidance [START_REF] Kandath | Autonomous Navigation and Sensorless Obstacle Avoidance for UGV with Environment Information from UAV[END_REF]- [START_REF] Chen | Motion Planning for Heterogeneous Unmanned Systems under Partial Observation from UAV[END_REF], target tracking [START_REF] Su | A Framework of Cooperative UAV-UGV System for Target Tracking[END_REF], exploration and mapping missions [START_REF] Qin | Autonomous Exploration and Mapping System Using Heterogeneous UAVs and UGVs in GPS-denied Environments[END_REF] and surveillance applications [START_REF] Reardon | Air-ground robot team surveillance of complex 3D environments[END_REF].

A cooperative robot system for target searching and recognition in an unknown environment is an open challenge that requires a close cooperation between robots. Some works to address this problem use a preliminary exploration step to map the environment. For example, in [START_REF] Lakas | A Framework for a Cooperative UAV-UGV System for Path Discovery and Planning[END_REF], the authors proposed to take advantage of the UAV (unmanned aerial vehicle) aerial view to detect, by image processing, obstacles that are mapped. A path planning algorithm then determines a clear path that will be follow by a UGV (unmanned ground vehicle) to reach a target point. In this work, the navigation of the ground robot depends entirely on the created map. It is also assumed that the UAV can see the environment entirely which is difficult to do in real scenarios.

Other works, such as [START_REF] Qin | Autonomous Exploration and Mapping System Using Heterogeneous UAVs and UGVs in GPS-denied Environments[END_REF] or [START_REF] Wang | A Collaborative Aerial-Ground Robotic System for Fast Exploration[END_REF], suggest a cooperative environment exploration. In [START_REF] Qin | Autonomous Exploration and Mapping System Using Heterogeneous UAVs and UGVs in GPS-denied Environments[END_REF], the authors proposed a team of UAV and UGV with LiDARs performing active exploration to build a 3D map of an unknown environment. However, the proposed scenario is performed sequentially thus weakening the cooperation: a first map is constructed by a UGV that is refined by the UAV. [START_REF] Wang | A Collaborative Aerial-Ground Robotic System for Fast Exploration[END_REF] improved this concept by proposing a cooperative framework to gain in efficiency and robustness, reducing drastically the robots' movements. The environmental scan merges sensors from different natures: a stereo camera and a LiDAR that benefit the mapping.

In [START_REF] Miller | Stronger Together: Air-Ground Robotic Collaboration Using Semantics[END_REF], the authors proposed to make ground robots navigation in a GPS denied environment thanks to a semantic map built by a UAV. This work highlights the importance of the UGVs' local planner to avoid obstacles in real experiments. Otherwise, the success of the mission cannot be guaranteed. This point raises the question of the necessary accuracy of the map built during the exploration and the long time allocated to this step. In [START_REF] Su | A Framework of Cooperative UAV-UGV System for Target Tracking[END_REF], a target tracking framework is proposed. A UAV is used to detect and track a target, and send the ground vehicle to the detected location. In this work, communication has been reduced to a minimum. Obstacles avoidance is only performed by a local planner without the help of the aerial robot. In [START_REF] Shen | Collaborative Air-Ground Target Searching in Complex Environments[END_REF], the authors have developed a cooperative navigation between one UGV and one UAV for targets search in an unknown environment with obstacles. The UAV first scans the environment with a constant altitude to find all targets that are sent to the ground robot. The UGV then reaches the targets on a first-come-first-served basis thanks to an online trajectory generator to avoid obstacles. This work has the advantages of reducing communication between robots. However, the proposed scenario is not optimized in terms of navigation time and is not compatible with multiple ground robots to speed up the mission. This work focuses on air-ground cooperative systems for targets search and recognition. The main interest comes from the great complementarity between the capacities (payload capacity, speed, stability, detection, communication, etc.) of UAVs and UGVs which make them powerful to complete complex tasks. Nevertheless, perception and control issues are still open for both systems (aerial and ground) to have a good performance and guarantee the success of the mission. In addition, in this work, we propose a cooperative architecture to ensure the coherence of the robots' activities to optimize the mission. Our architecture involves an unmanned aerial drone and several independent unmanned ground robots. In addition, this platform has been designed to make common navigation possible in an unmapped environment. In our work, we make sure to cover, in an aerial mode, all the surface to be monitored in order to find all the unknown targets. Moreover, the found targets location is sent to the cooperative architecture to assign tasks to each ground vehicle to reach the closest target. Our cooperative architecture is validated on a physical twin in realtime experiments.

The paper is organized as follows: the problem statement and main dynamics equations for aerial and ground vehicles are presented in section II. In section III, the cooperative navigation architecture is explained in details. The experimental setup and the obtained results are shown in section IV. Finally, conclusions and some future works are given in section V.

II. PROBLEM STATEMENT AND PRELIMINARIES

For success of some civil missions, the cooperation between aerial and ground vehicles is necessary. For example, to find and track targets using homogeneous robots (for example, humans lost in the forest or in sand dunes), the mission success is sometimes hard (or takes a lot of time) or never achieved. Therefore the challenge to use an heterogeneous robots cooperation scheme is to avoid targets' occlusions (produced by obstacles) and develop the mission in a short time. Some works in literature propose solution for this kind of civil applications using only one kind of robots, others include both configurations but some solutions are only validated numerically.

Our solution proposes to use heterogeneous robots (aerial and ground) to perform monitoring tasks for the search and recognition of targets. The validation scenario is presented in Figure 1, where an aerial vehicle must cover an area to find specific targets (whose position is unknown), later, the architecture computes closest target position from each ground vehicle presented in the scenario awarding the task of going to recognize the objective. The monitoring, searching and recognition tasks are done by the robots in autonomous mode. Communication between ground and aerial vehicles is necessary for the success of the mission. In this work, three robots are used; one aerial vehicle with a four rotors configuration and two ground vehicles as shown in Figure 2. The dynamic or kinematic models of each vehicle will be introduced in the following.

A. Aerial dynamic model

For the aerial mission, a quadcopter vehicle is considered. This vehicle can be represented as a rigid body evolving in three dimensions thanks to external actions: a main thrust Fig. 2: Ground and aerial robots used in the cooperation system.

u q and three torques τ q = [τ ψ , τ θ , τ φ ] [10]. Its mathematical equations can written using the quaternion representation as

m q ξq = q ⊗ F th ⊗ q * + m q   0 0 g   + ζ p (1)
J Ω = τ q -Ω × J Ω + ζ Ω (2)
where

F th :=   0 0 u q   (3)
is the vehicle's thrust force vector in the body frame; its magnitude is often considered as a control input (u q ), however, note its effect in the inertial frame directly depends on the system's attitude. ξ q = [x q , y q , z q ] T denotes the aerial vehicle position in the inertial frame I, m q the mass of the aerial vehicle and g the gravity force. The term q ∈ H is the system's orientation in unit quaternion form and q * denotes its conjugate term. J is the inertia matrix, Ω denotes the angular velocity and the terms ζ p and ζ Ω are the external disturbances.

B. Ground kinematic model

Ground unicycle-type robots are used for the mission. Each ground vehicle i is represented using the following kinematic equations and assuming that their altitude z gi is not changing.

ξgi ψgi =   cos ψ gi 0 sin ψ gi 0 0 1   U (4)
where ξ gi = [x gi , y gi ] T denotes the position of the i th ground robot in I and ψ gi its orientation. The linear and angular velocities are considered as control inputs of the vehicle, i.e., U = [v gi , ω gi ] T .

The wheel speeds (φ g1 i and φ g2 i ) of the ground robot i are related with the linear and angular vehicle velocities using the following expression

φg1 i φg2 i = 1 r 1 L 1 -L U ( 5
)
with r is the wheels' radius and L the center distance between the wheels. We consider two grounds vehicles with the same structure and dimensions. In addition, we assume that all robots know their position in a shared global reference frame and that all robots are connected to the same functional network in order to communicate.

III. COOPERATIVE NAVIGATION ARCHITECTURE

Our cooperative navigation architecture is composed of two main parts, one dedicated to aerial monitoring for targets' search and an other one for targets' reaching. Each part is further divided into subroutines aimed at achieving the final goal of their main part. They are described in the following.

A. Aerial monitoring

Three main tasks compose this part and they are performed only by the aerial drone; trajectory generation for scanning all the surface, the trajectory tracking and the target detection. The two last one are done at the same time. In this work, targets are represented by specific markers scattered in the environment.

1) Trajectory generation:

To scan the area to be monitored, a trajectory must first be designed using the shape and measurements of the surface, in order to cover the entire search area. To simplify further computations and streamline its practical validation, we have chosen a rectangular area of size L x × L y centered in p c = c x c y , see Figure 3.

Nevertheless, other types of surfaces can also be used for trajectory design. Without loss of generality, the area covered by the camera can be modeled as a rectangle of size l x × l y where l x = z q .r x f.D x l y = z q .r y f.D y with f represents the focal length, (r x , r y ) the camera's resolution and D x and D y are the number of pixels per unit length along respectively x and y axis, in the camera frame. We consider that the drone flies at a desired altitude z q d , i.e. z q → z q d . Then, the waypoints (meeting points ξ mi ) for the drone can be computed as

ξ mi =          pc -1 2 . Lx Ly + 1 2 . lx(1 + i) ly -1 2 . δr.i 0 if i is even pc -1 2 . Lx -Ly + 1 2 . lx.i -ly -1 2 .
δr.(i -1) 0 else for i : 0 : n with n is the number of waypoints required to scan the entire zone and δ r > 0 a small overlapping constant to make possible targets detection close to the border of the vision area. These points compose the desired trajectory to be tracked by the aerial drone.

2) Trajectory tracking: Once the meeting points ξ mi are computed, trapezoidal speed profile trajectories are generated to connect them. These trajectories are defined by desired points ξ q d (ξ mi , t) = [x d (t), y d (t), z d ] and velocities ξq d that will be used for the aerial vehicle controller.

The aerial drone's controller is given by

F u = m q g -K dt ς β d ξe -K pt ς βp K dt ξ e + ξe τ q = Ω × J Ω -K d ϑ Ω e -K p ϑ (2 ln q e ) (6) where K pt , K dt , K p ϑ , K d ϑ ∈ R 3×3 + are constant gain matrices, g = 0 0 g T . In addition, F u → F th when q → q d .
Moreover ς β (•) defines a saturation function defined as ς β (•) : R n → R n for a vector λ ∈ R n and a positive constant scalar β ∈ R + as

ς β λ :=    λ , λ < β β sgn λ , λ ≥ β (7)
where the sign function sgn (•) : R n → R n is defined element-wise. The stability analysis of this controller was demonstrated in [START_REF] Carino | A Fully-Actuated Quadcopter Representation using Quaternions[END_REF]. This control algorithm will assure the position errors converge to zero, i.e. ξ e → 0 and which implies that ξ q → ξ q d . Moreover, we will bound the desired velocity, | ξq d |≤ with > 0 small assuring that z q = z q d and (x q , y q ) (x, y).

During the trajectory tracking the target detection algorithm is being executed online and the position of the found targets is estimated ξTj .

3) Targets' detection: ArUco markers are square-based fiducial markers with binary codes. We chose these markers as targets for easy practical implementation and because this method is robust against false detections.

We have improved work in [START_REF] Garrido-Jurado | Automatic generation and detection of highly reliable fiducial markers under occlusion[END_REF] by proposing a robust candidate detection method, which takes into account the specific features of aerial images. In the cited method, a good separation of the marker from its environment is required and a dark object, even of small size, present on the margin of the marker prevents the detection. In our method, once the length of the marker edges, d, is known and with the fact that the controller ensures that the aerial drone is parallel to the ground, i.e., (x q , y q ) (x, y), then the marker to be detected in the image frame will be a square of size a = f. d zq , in pixels. Our algorithm, see Figure 4, for contour extraction and filtering can be summarized as follows:

1) Binarization of the image using a local adaptive threshold. 2) Image segmentation by connected component labeling.

This step is necessary to avoid the fusion of connected components during the morphological operations that follows. 3) Filtering to remove too small connected components. 4) Morphological operations taking into account the marker's size : these operations are a morphological closing operation with a disk shape structuring element of diameter 2.a b , followed by an opening operation with a disk shape structuring element of diameter a b being b the size of the marker in bits. Morphological operations are processed on each segmented component taken separately to prevent their fusion. The closure ensures that the integrity of the marker is maintained during the morphological opening operation which removes small objects present on the margin of the marker. 5) Contour extraction and polynomial approximation. 6) Filtering to get only square marker candidates of size a.

The advantage to use this algorithm is that it is possible to detect markers even in presence of an object of size smaller than a b on the white margin of the marker. The four corners' positions of the detected marker in the image frame [u i , v i ] T , i : 1 : 4 are used as inputs for the pose estimation algorithm. The relation between a 3D corner pose in the camera frame, C ξ T cj , and its 2D projection in the image frame is given by :

  u i v i 1   = A.Π C R m C ξTj [0] 3×1 1 m ξ T ci 1 ( 8
)
where C R m and C ξTj represent the unknown rotation and translation vectors from the marker to the camera frame. A is the camera intrinsic parameters matrix and Π the perspective projection model. From those four points and using equation ( 8), the optimization function solvePnP [START_REF] Marchand | Pose Estimation for Augmented Reality: A Hands-On Survey[END_REF] is able to estimate C R m and C ξTj that minimize the retro-projection error.

The computed position is then projected in the inertial frame, I, using the following relation

I ξTj 1 = I T B . B T C C ξTj 1 (9)
Observe that the precision of ξTj in I is related with the camera parameters (intrinsic and extrinsic) and the detection quality (related with the camera's resolution, the marker size and the sharpness of the image).

The position of ξTj is then sent from the aerial drone to the ground station via a wifi protocol. This information is stored for beginning the allocation task.

B. Targets' reaching

Once the aerial vehicle has covered all the surface, it goes to the center of mass of the ground vehicles, for activating the targets' reaching task.

1) Allocation task:

We have inspired our task allocation algorithm from the Auction Based Algorithm [START_REF] Lozenguez | Punctual versus continuous auction coordination for multi-robot and multi-task topological navigation[END_REF]. The advantage of our algorithm is that it is computed on-line and on-board of the aerial vehicle. It computes the task for each ground robot considering distance between each one of them to the nearest target. The allocation method is summarized in Algorithm 1.

2) Ground and aerial vehicles navigation: From Algorithm 1, D is the distance matrix of size n × m and is defined as

D = (d i,j ) 1≤i≤n, 1≤j≤m
with d i,j = ξ gi -ξTj representing the Euclidean distance between the robot i and the target j, n the number of robots and m the number of tasks.

Notice that this allocation algorithm is iterative and linked to the robots' position and reached tasks (targets). Therefore, it is necessary to update the whole allocation each time a robot reaches an assigned target.

Ground robots are placed in random positions and they do not know the environment (i.e., there is no map or knowledge about the area). Nevertheless, they known their position. We propose a reactive local planner with an online collision strategy to reach the target points. The method used here is inspired by the Dynamic Window Approach (DWA) algorithm [START_REF] Fox | The dynamic window approach to collision avoidance[END_REF]. Our main idea is to reduce the complexity of the original DWA algorithm, which will decrease of the execution time. Therefore, the objective function G(v, ω) has been modified. In the DWA, the goal is to choose at each time step a pair of angular and linear velocity among the admissible velocities that maximizes G(v, ω) defined in the following equation

G(v, ω) = α.heading(v, ω) + δ.dist(v, ω) +γ.velocity(v, ω) (10)
The function heading quantifies whether the robot will be close to the goal or not, the function dist quantifies whether the robot will be far from obstacles or not and the function velocity quantifies whether the robot is close to the desired speed. The algorithm's complexity is O(n × m) with n and m the size of the search space of admissible velocities. This leads to find a compromise between the discretization step of the velocities and the execution time.

We have modified equation ( 10) by changing the functions heading(v, ω), dist(v, ω) and velocity(v, ω) to convex loss function [START_REF] Pousseur | Gradient Descent Dynamic Window Approach to the Mobile Robot Autonomous Navigation[END_REF]. The new objective function L(v, ω) defined in equation ( 11) is as well a convex function that must be minimize. The couple of linear and angular velocity that minimize L(v, ω) can be obtained by applying a gradient descent on L(v, ω).

L(v, ω) = α.heading loss (v, ω) + δ.dist loss (v, ω) +γ.velocity loss (v, ω) (11)
The original concept of search space is not modified. The gradient descent is limited in the search space. This modification enable us to reduce by four approximately the number of calculation.

IV. EXPERIMENTAL RESULTS

A. Experimental platform

The experimental setup consists of two ground robots Turtlebot3 equipped with a LiDAR to detect obstacles. They are programmed using ROS. The aerial drone is a quadcopter Parrot AR.Drone 2.0 equipped with a downward-facing camera running at 15 fps with a 320x240 pixels resolution. It has also an internal Inertial Measurement Unit (IMU) and an ultrasonic sensor. Its firmware has been modified to work with the open source software FL-AIR (Framework Libre Air) [START_REF] Sanahuja | Fl-AIR -Framework libre AIR[END_REF], used for its programming. The UAV communicates with the ground robots via a TCP socket. An OptiTrack motion capture system is used to estimate the robots' position at 100Hz with a precision of 1 mm. The robots' velocities are calculated by differentiating the position and filtering.

The platform includes also a remote computer called Ground Control Station (GCS) used as human-uav interface. The GCS is connected to the Optitrack software and communicates robots' position and flight parameters to the UAV. It is also used to graph in real time the states of the UAV for analysis purposes. This remote computer is also responsible for processing the drone images that are sent via TCP sockets. Figure 5 summarizes the proposed framework.

B. Results

Experimental tests in real time were conducted to validate the proposed architecture. Firstly, the perception part (for detecting markers) was carried out for validating the good markers' detection. Secondly, the drone control was tested for tracking the desired trajectory and verifying that it covers the whole surface to be monitored while detecting the targets. Finally, the whole system was validated including ground vehicles and obstacle avoidance navigation for reaching targets, see Figure 6. In the following the results when testing the proposed control architecture.

1) Aerial monitoring and targets' searching: In this part, the goal was to verify the well performance of the control and vision algorithm. For the vision algorithm, we use markers with a blank margin (Figure 7) and we add black splines in the markers's margin for perturbing the algorithm. In Table I, the accuracy of the marker position estimation algorithm is presented. We note that all the 5 markers were detected by using our method. When using the original detection algorithm, in [START_REF] Garrido-Jurado | Automatic generation and detection of highly reliable fiducial markers under occlusion[END_REF], we observed that the markers detection was impossible. During the aerial monitoring, the drone follows, with pretty good precision, the desired trajectory scanning the entire area, see Figure 8 and 10. At the end of the aerial monitoring task, all markers are detected with an estimation error smaller than 10 cm. This error is mainly due to the estimation of the camera's intrinsic parameters. In fact, we observed that the most accurate markers' placement are those positioned close to the scan path. We have also observed that the height of the drone and the camera resolution also impacts the results: for z > 2m, the marker is no longer readable. The accuracy of the pose estimation is not impacted by the presence of splines on the markers' margin, which are removed by morphological operations. 2) Targets' reaching: As explained before, to verify the target locations, two ground vehicles must navigate in autonomous mode to their nearest target avoiding obstacles if they are present. During this ground navigation, the aerial vehicle monitors the ground mission performance by flying in the center of mass of both vehicles.

In Figure 9 robots' trajectories during the targets-reaching step are shown. As expected, ground robots reach targets in a straight line. However, their trajectories become curved to avoid obstacles. In this figure, we can also observe that the drone tracks the motion of the robots well. When the ground vehicle is in the vicinity of the target (previously defined), the drone considers that the target is reached and the tasks are then immediately updated.

V. CONCLUSION AND FUTURE WORKS

In this paper, a centralized cooperative architecture for searching and reaching unknown targets was proposed. The whole robotic system is composed of one aerial vehicle and two ground vehicles. An aerial target detection algorithm was first developed to estimate the relative position of the unknown targets. This information was then used to the cooperative architecture to assign tasks to the ground vehicles in order to reach their closest target. The robots navigation was done in autonomous mode and the ground vehicle control was capable to avoid obstacles. Real-time experiments corroborated the good performance of the whole system. This solution can potentially be used as a basis for target recognition applications. Further works will take into account communication and localization hazards to increase robustness. Possible extensions in dynamic environments will be studied and will require improving the cooperation between robots to make the system more flexible and adaptable.

Fig. 1 :

 1 Fig. 1: Search and recognition of targets using heterogeneous vehicles.

Fig. 3 :

 3 Fig. 3: Path-planning proposed to monitor the entire environment in a constant altitude.

Fig. 4 :

 4 Fig. 4: Markers detection flowchart

Algorithm 1 Fig. 5 :

 15 Fig. 5: Framework of the cooperative system. A module UGV is on-boarded on each ground robot.

Figure 6

 6 Figure 6 illustrates the experimental environment. Five ArUco markers of size 10 cm × 10 cm have been printed on sheets of paper and placed randomly in the search area. Each marker encodes a unique identifier facilitating the position estimation merging. The final estimation of the j th marker's position ξTj is the mean of the position estimations ξTj . Unmapped obstacles have been placed between targets, across the way of the ground vehicles.

Fig. 6 :

 6 Fig. 6: Experimental environment.

Fig. 7 :

 7 Fig. 7: (Above) Example of markers with black splines in their margin used for the tests. (Below) Pose estimation results.

  Presence of an object in the markers' marginMarkers detectedMean pose estimation error (mm) Markers' detection and pose estimation results during one scan of the search area.

Fig. 8 :

 8 Fig. 8: Real-time results when the UAV follows a desired trajectory during the searching mission for finding targets.

Fig. 9 :

 9 Fig. 9: Robots trajectories during the targets reaching task.

Fig. 10 :

 10 Fig. 10: UAV trajectory performance along x, y and z axis