Miguel Da Silva

Remi Regnier

Maria Makarov

Guillaume Avrin

Didier Dumur

Evaluation of intelligent collaborative robots: a review

Intelligent collaborative robots or smart cobots can achieve high levels of flexibility by combining the human ability to adapt to new tasks with the performance of automated robots (precision, repeatability, etc.). This is a major innovation for Industry 4.0. Nevertheless, at present, cobots are not widely deployed in industry because they are difficult to evaluate and current standards for them are limited. The evaluation of collaborative tasks is difficult due to their specificities such as the variability of human behavior, artificial intelligence systems (e.g., smart cameras), and advanced control laws. This paper is a short review that aims to identify the methods and material resources needed to address this need for evaluation of intelligent cobots to foster their development and acceptability.

I. INTRODUCTION

Market research [START_REF] Callet | Le marché mondial de la robotique industrielle[END_REF] shows that global sales of collaborative robots (or cobots) nearly doubled between 2017 and 2019. However, they are still a minority compared to conventional industrial robots, which accounted for 95% of global sales of industrial robot in 2019. The lack of standards to assess cobots is holding back their deployment. This statement is confirmed in the paper [START_REF] Quispe | A taxonomy of benchmark tasks for robot manipulation[END_REF], it explains that robot manipulators are difficult to assess because of the "lack of guidelines for a realistic experimental setup". The evolution of the number of research papers available on the Science Direct database that mentions the terms "Human Robot Collaboration" and "Machine learning" in their title, abstract or keywords per year from 1996 to 2020 shows that the interest for intelligent collaborative robots is growing exponentially [START_REF] Mukherjee | A Surv. of Robot Learn. Strategies for Human-Robot Collaboration in Ind. Settings[END_REF]. Nevertheless, it is important to be aware that the term "collaborative robot" can have different meanings. Most papers about cobots give their own definition of the word "collaborative" in the context of human-robot collaboration, sometimes through examples or sometimes more clearly, but there is no consensus on the definition of this term. This problem is also identified in [START_REF] Knudsen | Collaborative Robots: Frontiers of Current Literature[END_REF]: "Several definitions exist in literature distinguishing collaboration from cooperation or interaction. For example, [START_REF] Bendel | Co-robots from an Ethical Perspective[END_REF] notes that cooperation robots work with people step by step for a common goal, while 1 Miguel Da Silva is with Laboratoire National de métrologie et d'Essais (LNE), 78197 Trappes, France & CentraleSupélec, Laboratoire des signaux et systèmes (L2S), 91190, Gif-sur-Yvette, France miguel.dasilva@centralesupelec.fr 2 Remi Regnier is with the Laboratoire National de métrologie et d'Essais (LNE), 78197 Trappes, France remi.regnier@lne.fr 3 didier.dumur@centralesupelec.fr collaborative robots work with people hand in hand on a common task. With a more lenient definition corresponding to the definition of cobot manufacturers, any robot operating alongside a human without a fence can be characterized as a collaborative robot [START_REF] Zaatari | Cobot programming for collaborative industrial tasks: An overview[END_REF]". Several papers try to tackle this issue by proposing a global definition of "collaborative robots" or related concepts such as human-robot interaction (HRI). For example, [START_REF] Mukherjee | A Surv. of Robot Learn. Strategies for Human-Robot Collaboration in Ind. Settings[END_REF] classifies robots according to their level of interaction. It proposes six levels of interaction by making an analogy with the standard levels of autonomy of autonomous vehicles created by the Society of Automotive Engineers (SAE): fully programmed, co-existence, assistance, co-operation, collaboration and fully autonomous.

In this case, collaborative robots are those who belong to the level of interaction number 4 called "collaboration". It means that a collaborative robot shares its workspace and works simultaneously with the human to achieve a common goal. Therefore, according to this definition, a robot that shares its workspace with a human and works on the same part at different times (e.g. not simultaneously) is not a collaborative robot but a cooperative robot. Mukherjee's definition is incompatible with the one given by the ISO/TS 15066:2016 standard which does not make any difference between collaborative robots and cooperative robots. In this paper, the definition of cobot that is used, is the one the proposed by the ISO/TS 15066:2016 standard. It includes a wide variety of robots including automated guided vehicles (AGV) designed to perform fetching tasks [START_REF] Inoue | High-Precision Mobile Robot. Manipulator for Reconfigurable Manuf[END_REF], drones that are controlled by humans without any remote [START_REF] Medeiros | 3D pointing gestures as target selection tools: guiding monocular UAVs during window selection in an outdoor environment[END_REF], exoskeletons [START_REF] Brahmi | Impedance learning control for physical human-robot cooperative interaction[END_REF] and robotic arms that work alongside humans [START_REF] Fortineau | Interactive robotics for human impedance estimation in a rhythmic task[END_REF]. We propose that the words collaboration and cooperation have the same meaning and we chose two terms to distinguish the different types of collaboration between humans and cobots: direct collaboration and indirect collaboration. Both of these terms are explained in the following section, which presents a taxonomy of cobots. The third section is about the technical requirements for collaborative robots that are considered in the selected research papers. To verify that these requirements are satisfied, the authors have applied evaluation methods that are discussed in section 4. To contribute to the creation of new standards, we compare a wide variety of methods used by different researchers to identify hindrances and promising methods. In the last section, thoughts on the prospects for the evolution of testing tools for intelligent cobot that comes from the literature analysis are presented.

Robot

II. TAXONOMY OF INTELLIGENT COBOTS

The purpose of the following taxonomy (Figure 1, 2, 3) is to define the terms used to characterize intelligent collaborative robotic systems. The words chosen for this taxonomy can have different meanings. To avoid ambiguities we propose the following glossary that details as much as possible the ambiguous terms of the taxonomy.

A. Glossary

AI type: type of AI algorithm that is caracterised by its learning method (supervised, unsupervised, etc.) or by its architecture (symbolic, etc.). AI application: task performed by the AI system. Collaboration type: how the human collaborates with the robot to perform a collaborative task. Direct collaboration: collaboration where the human and the robot interact on the same part at different times. Indirect collaboration: collaboration where the human and the robot interact on the same part at the same time. Task: common goal that human and robot try to reach in collaboration with each other. Co-carrying: the robot and the human are carrying the same object at the same time. Gesture assistance: hand-guided tasks where the robot reduces the effort required by the user to perform a task or constraints the user's movements to help the user maintain a proper trajectory to perform the task. Handover: the robot places an object in the human's hand or holds it while the human interacts with it. Subsystem: system that can complete a subtask that contributes to a collaborative task (ex: perception of obstacles with a 3D camera [START_REF] Flacco | A depth space approach to human-robot collision avoidance[END_REF]). System: hardware component that is necessary to complete a collaborative task: robot, camera or computer. that can be described by several other concepts such as fluency, helpfulness, trust, flexibility, ergonomics, etc.

III. TECHNICAL REQUIREMENTS FOR INTELLIGENT

COBOTS

In most of research papers about intelligent cobots, the requirements are implicit and there are no clear specifications. For example, [START_REF] Brito | A Mach. Learn. Approach for Collaborative Robot Smart Manuf. Inspection for Quality Control Syst[END_REF], [START_REF] Dimitropoulos | Seamless Human-Robot Collaborative Assem[END_REF] and [START_REF] Kragic | Interactive, Collaborative Robots: Challenges and Opportunities[END_REF] aim to demonstrate that their system or algorithm work in a controlled environment. This approach highlights the relevance of using a new technology for a given collaborative robotics application, but it does not give enough guarantees on its performance to allow its commercialization. However, by observing what is tested during the evaluation, several high level requirements can be deduced. They can be classified in two main categories: safety and performance. The safety requirements concern the ability of the cobot to operate without causing any damage to the human coworker, its working environment or itself. Hundreds of papers have addressed safety issues with collaborative robots including [START_REF] Aliev | Proposal of a Monitoring System for Collaborative Robots to Predict Outages and to Assess Reliability Factors Exploiting Mach[END_REF], [START_REF] Bi | Saf. assurance mechanisms of collaborative robotic systems in manufacturing[END_REF]. Therefore, in the present paper, we have chosen to focus on the performance requirements that are less discussed in the literature. We divided these performance requirements in five subcategories: accuracy, robustness, resilience, rapidity and quality of HRI.

A. Accuracy

Most of the reviewed documents use the accuracy metric to evaluate their collaborative systems. In this paper, the definition of the term "accuracy" is close to the one used on AI domain, which means that it is not related to position or trajectory errors. It mainly focuses on the success of the achievement of a collaborative task. It is a general term and therefore encompasses many other requirements. It only provides a very high level information on the performance of the system. For example, [START_REF] Kolosowski | Collaborative Robot System for Playing Chess[END_REF], a collaborative robot capable of playing chess with a human or with another cobot is presented and the main requirement of this system is accuracy in the sense of "success of the task". In order for a cobot to be able to perform such a difficult task, the author divided it into three subtasks: the vision algorithm, the decision making algorithm, and the manipulator control system. The first and second subtasks involve AI algorithms and the third one is about control algorithms. None of these subtasks are evaluated with complex metrics, the authors focused on the success of the task or the subtask that can be called "accuracy". In this context, the accuracy of the visual motion detection is the ability of the robotic system to locate the pieces on the board and successfully detect the movements of each player. The success of all subtasks and their association is demonstrated by experimentation but this method is limited. Nevertheless, several papers such as [START_REF] Mangat | Interactive Process Automat. based on lightweight object detection in manufacturing processes[END_REF] define precise metrics to explicit and measure the accuracy of their system. Their paper presents an image recognition AI algorithm that can recognize and locate an object on the cobot workspace regardless of how it is placed. This algorithm is designed to be integrated into collaborative production tasks in which the human deposits parts randomly in the robot's workspace. To evaluate the accuracy of the AI algorithm, they use an accuracy score metric, calculated as the ratio of the number of error-free detections for each image to the total number of images. Other classical metrics in the evaluation of AI algorithms, such as F1-score, Jaccard and precision are also used in this paper. These metrics only guarantee the accuracy of the AI algorithm under certain conditions (parameters such as lighting) but are not sufficient to ensure the accuracy of the entire collaborative system. Evaluating the accuracy of several subtasks does not guarantee the accuracy of the whole system [START_REF] Quispe | A taxonomy of benchmark tasks for robot manipulation[END_REF]. Moreover, making various tests varying the influence factor is necessary to guarantee the accuracy of the system in the largest number of use cases. This is the notion of robustness.

B. Robustness and resilience

Evaluating the robustness of an intelligent collaborative robotic system allows guaranteeing its performance on a large operating domain that takes into account the possible variations of the environment and system parameters. The robustness of all the cobot subsystems does not guarantee the robustness of the whole cobot. It still allows to identify influence factors that can disturb the robot and to make sure that critical subsystems are able to work under certain constraints. Consequently, there is a need to evaluate both the robustness of the subsystems and the robustness of the complete cobot. Nevertheless, in the literature, most papers in which collaborative robotic systems or subsystems are tested do not test their robustness. For example, [START_REF] Galin | Math. Modelling and Simul. of Human-Robot Collaboration[END_REF] present a collaborative robot that sorts different objects based on their color and shape is tested, but the parameters of the test such as lighting conditions are fixed in advance and do not vary. Thus, if one those parameters vary, the behavior of the system is unpredictable. Some robots are designed to operate in controlled environments where most of the influence factors are controlled. For these robots, the required level of robustness is lower than the one required for autonomous robots that operates in unknown environments. Testing the resilience of a system takes the evaluation of the system one step further than a simple robustness test. It is the test of the system's ability to continue operating outside its operating range (ex: in case of a component failure or a cyber-attack) [START_REF] Wybo | The Role of Simul. Exercises in the Assessment of Robustness and Resilience of Private or Public Org[END_REF]. Although robustness and resilience are not often evaluated in the case of collaborative robots, they are evaluated a lot in the field of autonomous vehicles, which today is one of the most advanced fields in terms of evaluation techniques for complex and intelligent systems. That is why we believe that these requirements will be evaluated more and more for collaborative robots. Moreover, new evaluation tools and methods are being developed for collaborative robots, they are presented in the next section.

C. Rapidity

Rapidity is another important requirement. [START_REF] Malik | Digit. twins for collaborative robots: A case study in human-robot interaction[END_REF], [START_REF] Sadrfaridpour | Collaborative Assem. in Hybrid Manuf. Cells: An Integrated Framework for Human-Robot Interact[END_REF] measure task completion time to see if human-robot collaboration has an impact on production rate. We noticed that this requirement concerns mainly pick and place collaborative tasks. Currently, for safety reasons, the velocity of cobots is limited by the ISO/TS 15066:2016 standard. This standard links the rapidity to safety requirements. However, safety requirements are currently very restrictive because of the lack of standards to test robustness of the systems. Thus, guaranteeing the robustness of a cobot can make the speed limitations imposed by the safety standard less severe and it could lead to an increase of the rapidity of the cobot. This statement shows once again the importance of the evaluation of the cobot's robustness.

D. Quality of HRI

The quality of HRI is an essential yet difficult-to-assess requirement because it covers various concepts that are often subjective. Several papers try to evaluate this interaction in an explicit way [START_REF] Marchi | Integration of an Artif. Intelligence Based 3D Perception Device into a Human-Robot Collaborative Workstation for Learn. Factories[END_REF], [START_REF] Sadrfaridpour | Collaborative Assem. in Hybrid Manuf. Cells: An Integrated Framework for Human-Robot Interact[END_REF]. Human-robot interaction is a concept that cannot be measured directly but these papers have tried to demystify this concept and to quantify it. For example, [START_REF] Marchi | Integration of an Artif. Intelligence Based 3D Perception Device into a Human-Robot Collaborative Workstation for Learn. Factories[END_REF] define human-robot interaction throughout several related concepts like robot's trust, system usability, frustration, pleasure, satisfaction, and perceived physical workload. Moreover, [START_REF] Marchi | Integration of an Artif. Intelligence Based 3D Perception Device into a Human-Robot Collaborative Workstation for Learn. Factories[END_REF], [START_REF] Metzner | A system for human-in-the-loop simulation of industrial collaborative robot applications*[END_REF] take into account the quality of HRI through the ergonomics of the selected collaborative tasks. They are measuring the human health benefit of the collaborative robot to see if it improves human posture or reduces the loads carried by the human. Other similar concepts such as fluidity of human-robot interaction have been developed to describe HRI [START_REF] Hoffman | Evaluating Fluency in Human-Robot Collaboration[END_REF]. Thus, the demystification of the concept of quality of HRI made possible its evaluation even though it cannot be perfect because the related concepts are often subjective and the reproducibility of experiments is limited by the human factor.

IV. EVALUATION METHODS

In the scientific literature, most of the papers about the evaluation of cobots focus on the safety of cobots or on the HRI. However, there are also a few papers that use evaluation methods to test the whole intelligent collaborative robotic system or the subsystem.

A. Evaluation of safety of cobots

The safety of collaborative robots is one of the main barriers to the deployment of collaborative robots. This is the reason why the researchers are focusing mainly on this aspect. [START_REF] Shin | A Real-Time Human-Robot Collision Saf. Eval. Method for Collaborative Robot[END_REF], [START_REF] Bi | Saf. assurance mechanisms of collaborative robotic systems in manufacturing[END_REF] refer to the only safety standard that was designed specifically for cobots, ISO/TS 15066:2016, in order to guarantee the safety of their system. They perform physical tests to make sure that the system is compliant with this standard (i.e. that respects the constraints of speed and pressure on the human body stated by this standard). These constraints are expressed in the form of maximum values not to be exceeded according to the mass of the robot and the potential human/robot contact zones. However, as shown by [START_REF] Mukherjee | A Surv. of Robot Learn. Strategies for Human-Robot Collaboration in Ind. Settings[END_REF], this standard is limited to robot related constraints but does not take into account the tool of the robot that can be dangerous (eg. soldering iron) and the human related factors. Moreover, it does not take into account the danger related to the AI systems. For example, [START_REF] Zhang | Recurrent neural network for motion trajectory prediction in human-robot collaborative assembly[END_REF] present a collaborative robot that offers tools to a human to help with an assembly task and if an error in the AI system that plans the robot's handover motion occurs then the human can be injured by a screw driver. That is why the evaluation of AI systems is essential to guarantee the safety of an intelligent cobot. Methods for evaluating the performance of intelligent cobots can be used to evaluate several aspects of cobot safety, but not necessarily in accordance with ISO/TS 15066:2016.

B. Evaluation of HRI

HRI is one of the most complex aspects of cobot evaluation due to its subjectivity. Nelles et al. (2019) selected 30 relevant papers on HRI evaluation and described their evaluation method. These methods can be distinguished into two categories: subjective and objective.

1) Subjective evaluation methods: Subjective methods do not rely on measured data. They mainly rely on surveys that are answered by groups of users of the cobot or by experts who watch and interpret videos of users using the cobot. Besides, the result of these questionnaires are scores that correspond to the evaluation metrics. The scoring method depend on the type of questionnaire that is used, it is either personal with for example, scores from 1 to 5 for each question [START_REF] Charalambous | The development of a scale to evaluate trust in industrial human-robot collaboration[END_REF] or standard such as the NASA-TLX which aims a evaluating the perceived workload for the cobot user [START_REF] Bortot | Directly or on detours? How should industrial robots approximate humans?[END_REF], [START_REF] Rahman | Trust-based optimal subtask allocation and model predictive control for human-robot collaborative assembly in manufacturing[END_REF].

2) Objective evaluation methods: Contrary to subjective evaluation methods, objective methods rely on physical data that is provided by sensors or measuring instruments. This data is mainly physiological, for example: the heart rate, the skin temperature, human energy consumption (kcal) etc. Such measures are often used to build metrics that quantify subcategories of HRI like anxiety [START_REF] Daniel | Simplified Human-Robot Interaction: Modeling and Evaluation[END_REF], [START_REF] Maurtua | FourByThree: Imagine humans and robots working hand in hand[END_REF]. Instead of using scores from questionnaire results to evaluate the quality of HRI, they use physical measurements and try to correlate these measures with concepts related to HRI. For instance, [START_REF] Novak | Psychophysiological responses to different levels of cognitive and physical workload in haptic interaction[END_REF] measured the trust of the human coworker in the cobot through the heart rate because they stated that there is a correlation between the stress level of the human and the heart rate. This kind of correlation allows to evaluate HRI but they are not exact and do not take into account several influencing factors and biases. Consequently, objective and subjective methods are not opposed but rather complemen-tary. Thus several papers such as [START_REF] Weistroffer | Assessing the acceptability of human-robot copresence on assembly lines: A comparison between actual situations and their virtual reality counterparts[END_REF] use both approaches mixing physical measures with scores from questionnaires to evaluate HRI.

C. Evaluation of cobot's performances

The cobot performance is the ability of the robot and the human to perform a given collaborative task according to a set of specifications. If the specifications are guaranteed every time the robot and the human work collaboratively then the performance of the cobot is acceptable. This definition of performance includes safety and HRI, but as they were already discussed in the previous sections, this section focuses on task-related performance evaluation. There are various existing methods to assess the ability of a cobot and a coworker to achieve a common goal. These methods are sorted in four main categories: analytical methods, virtual tests, mixed tests and physical tests.

1) Analytical methods: In general, analytical methods do not concern the test of the entire cobot because modeling the behavior of such a complex system with all the constraints associated to the task that it is supposed to accomplish is impossible without the use of numerical methods. However, analytical methods are useful to evaluate elementary functionalities of cobots using simplified models and to prove the stability of the cobot in a given test configuration. Thus, it allows the cobot to pass a first validation step. For example, [START_REF] Brahmi | Impedance learning control for physical human-robot cooperative interaction[END_REF] proved the Lyapunov stability of the impedance control law under certain conditions. Nevertheless, when a cobot is equipped with an AI system, this approach is limited because most of them act like black boxes whose behavior cannot be predicted analytically.

2) Virtual tests: In contrast to analytical methods, virtual tests allows the prediction of cobot behavior in complex scenarios. They allow to numerically solve problems that are analytically difficult or even impossible to solve including those that use artificial intelligence. Thus, there are two types of virtual tests: a first one, closer to the analytical method, which consists in solving equations numerically without including a 3D model of the robot [START_REF] Dimeas | Online Stability in Human-Robot Cooperation with Admittance Control[END_REF], and a second one which includes a 3D model of the robot and which aims at reproducing the scene of the cobot performing its task in a virtual environment [START_REF] Parsa | Human-robot collaboration disassembly planning for end-of-life product disassembly process[END_REF]). Simulations without computeraided design (CAD) models are used to assess specific cobot functionalities but they are not relevant to assess a cobot in complex scenarios. Nevertheless, they allow to evaluate several complex functionalities that involves AI such as task planning [START_REF] Yu | Estimation of human impedance and motion intention for constrained human-robot interaction[END_REF].

Virtual tests with CAD models allow to simulate realistic use cases of the cobots thanks to simulation softwares like Gazebo [START_REF] Galin | Math. Modelling and Simul. of Human-Robot Collaboration[END_REF], RobWorkStudio [START_REF] Kolosowski | Collaborative Robot System for Playing Chess[END_REF], Tecnomatix [START_REF] Metzner | A system for human-in-the-loop simulation of industrial collaborative robot applications*[END_REF], etc. This type of simulation is often called "virtual commissioning". Currently, it is mainly used to validate prototypes but it can be unrepresentative of the reality, especially if the task performed by the human in the simulation is complex. Modeling humans for simulation purposes is still an active research area. However, several papers such as [START_REF] Galin | Math. Modelling and Simul. of Human-Robot Collaboration[END_REF] show that while modeling and simulating a human interacting with a cobot is possible, it is still difficult to obtain reliable results with such models if the level of collaboration between the human and the robot is high. This puts into question the reliability of the simulation. Consequently, physical tests are sometimes required in addition to simulation in order to validate the system. The fact that the collaborative robotic system reaches the expected performance level in a simulation does not always mean that it will reach the same performances in the real life. However, if the system does not achieve the expected level of performance in the simulation, it is unlikely to achieve that level of performance in a physical test. In short, virtual testing is a powerful tool that can be used to test almost any cobot functionality and that is very efficient for prototyping but to validate an intelligent cobot in the case where the collaboration between the cobot and the human is direct and complex, this approach is limited. Its lack of realism is its main flaw.

3) Physical tests: There are two categories of physical testing: those performed in a laboratory in a controlled environment and those performed directly in the final use environment. Both have the advantage of being more realistic than simulation. Nevertheless, the number of use scenarios that can be assessed is limited because of the costs, the test time and the safety problems of several use cases. Actually, safety issues are particularly present in the physical evaluation of collaborative robots because their direct interaction with humans increases the risk associated with the cobot task. Therefore, evaluation of cobots in dangerous situations or near misses is difficult to implement. Consequently, several use cases cannot be evaluated properly with physical tests and that is why in the majority of cases, the robustness cannot be evaluated with physical tests. To overcome that issue, researchers often combine several testing methods. This allows to take advantage of the benefits of each method. Among the papers we analysed, 80% of those which use physical methods, such as [START_REF] Brahmi | Impedance learning control for physical human-robot cooperative interaction[END_REF], also use analytical or simulation to evaluate their system.

4) Mixed tests: Mixed tests are the least used among the sample of papers we analysed. They are still recent and sometimes difficult to implement. However, they have the advantage of offering a good compromise between the number of applicable test scenarios and the realism of the results. There are three main type of mixed tests for cobots: hardware in the loop (HIL), robot in the loop (RIL) and human in the loop (HuIL).

Hardware in the loop: [START_REF] Tongtong | Develop. of Hardware-in-Loop Simul. Platform for Collaborative Robots Based on LinuxCNC and V-rep[END_REF] presents a simulation of a cobot arm close to the ones presented in the "Virtual tests" section but instead of being only virtual, it integrates the real single-board computer module of the cobot that is used to process the control algorithm. This approach takes into account all the errors linked to the single-board computer module and makes the simulation much more realistic than a traditional simulation. It allows to test real algorithms in various use cases by taking into account the potential errors or disturbances related to the properties of the hardware that will run it.

Robot in the loop: RIL testing is still rare in the field of intelligent collaborative robotic systems. Nevertheless, this type of test is developing in the field of autonomous vehicles. The latter is a critical application field in which the slightest error can lead to dramatic consequences with fatal accidents. Thus, for such a technology to be available one day on the general public market, its safety must be guaranteed and for that, it is necessary to demonstrate the robustness and resilience of these systems. This is often done through virtual or HIL tests, but these lack realism, so RIL or VIL (vehicle in the loop) tests have been developed. The principle is to immerse the real vehicle or robot in a virtual environment. It is the same principle as virtual reality for humans but applied to robots or autonomous vehicles. Only the operating environment is simulated and the robot or vehicle interacts with it via gateways that link its sensors to the virtual environment. In the case of the autonomous vehicle, screens are placed in front of the vehicle's cameras, a computer that broadcasts the images of the virtual environment on these screens is placed in the trunk of the vehicle and by making the car drive in a large empty space, one can test the behavior of the vehicle on any virtually generated road (intersections, traffic circles, etc.). This is a promising evaluation method but it is still underdeveloped in the field of collaborative robots and might be unaffordable for most evaluators.

Human in the loop: During the HuIL assessment, the human is immersed in a simulated virtual environment using a virtual reality headset, a human motion detection system and sometimes wearing haptic devices. The human can interact with the virtual environment, for example pick a part and place it on a table near the cobot in the virtual world. HuIL evaluations overcome one of the main obstacles to the simulation of collaborative tasks, which is the modeling of human behavior. A significant advantage over physical testing is the ability to test dangerous or near miss use cases and its reproducibility is better than 100% virtual simulations. This is the evaluation method that was selected in [START_REF] Metzner | A system for human-in-the-loop simulation of industrial collaborative robot applications*[END_REF] to test different safety devices to protect the human of a cobot in two different cases. The first one concerns a mechanical parts cleaning task in which the human's role is to replenish the cobot's cleaning tool, which is disposable. The second one concerns a printed circuit board (PCB) handling task in which the human has no role but for which the robot has to stop when he approaches to avoid injuring him with sharp PCBs. In both cases, dangerous situations could be tested and new sources of danger could be discovered without the human taking any risk. Thanks to the information provided by this simulation on the robot's behavior, the algorithms that define the robot's behavior in the presence of a human could be adjusted to make the HRI safer. However, even if this method does not model the human's behavior, it still requires modeling the interaction between the human and the cobot in the virtual environment. This means that for cases where the collaborative task is complex, the simulation is likely to be unrealistic. This method is therefore optimal for evaluating systems where the human behavior is not predictable and difficult to model and where the human-robot interaction is not complex. This is often the case for autonomous vehicles, the equivalent in this domain is the driving simulator, which allows to acquire data on human and to adapt the vehicle behavior to each driver.

V. CONCLUSIONS

In this paper, a taxonomy was developed to clarify the notion of collaborative robotics and to introduce different notions related to evaluation. Through this litterature review, we have identified several gaps and areas of progress for the evaluation of cobots. We noticed that robustness and resilience of a full collaborative robotic system are poorly evaluated. This is partly due to the fact that most of the cobots mentioned in the literature are proofs of concepts or prototypes that are not intented to be used in an industrial setting. It is therefore an interesting research area to explore. The progress of simulation software and the computing power of computers allows to set up numerical models which are more and more faithful to reality and which can provide information on the robustness of the intelligent cobot. Moreover, they give rise to new types of tests that we have called "mixed tests". These tests are still at an early stage development and are sometimes difficult to implement. They require a lot of equipment as well as direct access to the sensors of the cobots which are not always possible. However, we believe that they represent the future of simulation because they have the advantage of having the best compromise between exhaustiveness and realism of test scenarios. Through our analysis, we also identified the lack of linkages between the different evaluation methods. We believe that the different tests (physical, virtual, etc.) can be correlated. For example, we can imagine an approach where a collaborative task would be evaluated through a mixed test and then reproduced in a real environment in order to validate the mixed testbed which could then be used without physical testing afterwards.

Fig. 1 .

 1 Fig. 1. Taxonomy of intelligent cobots

Fig. 2 .

 2 Fig. 2. Taxonomy of intelligent cobots

Fig. 3 .

 3 Fig. 3. Taxonomy of intelligent cobots


Catherine BAROIN PREMIERES ENQUETES CHEZ LES RWA 1 (D'après le journal de mission)

En raison de l'effervescence religieuse qui régnait chez les Rwa lors de mon arrivée en Tanzanie, début octobre 1992, l'enquête ethnographique débuta non pas sur les verts flancs sudest du mont Méru en plein coeur du pays rwa, où se pratique une polyculture intensive, mais dans la zone agropastorale un peu plus sèche d'Engare Nanyuki, au nord-est, dont les habitants étaient restés en dehors du conflit.

  Après avoir traversé à gué la rivière Waato, je rejoins P. dans son village où l'accueil est des plus aimables. Enquête sur les tours d'eau avec le "Chairman of water", personnage dont le niveau de vie est au demeurant très modeste. En chemin, visite à une accouchée, recluse comme il convient pour plus d'un mois dans sa chambre. La maison est luxueuse : le sol est cimenté, les murs et les portes sont peints et les fenêtres sont même munies de vitres. Comble du chic, devant la maison s'étend un petit jardin d'agrément, avec pelouse à l'anglaise et fleurs magnifiques. Mais ce qui est exceptionnel ici l'est beaucoup moins, comme je le verrai plus tard, en plein coeur du pays rwa sur la montagne, car les animaux en stabulation ne risquent pas d'y brouter les plantes d'ornement. Quant au luxe de la demeure, il s'explique par la très bonne situation du mari de cette dame, qui travaille comme beaucoup d'autres assez loin d'ici, et ne revient que de temps en temps.Grâce à l'introduction de P., enfant du pays, chacun dans ce village tient à me recevoir. Le terme de village est d'ailleurs fallacieux, car l'habitat comme partout ailleurs en pays rwa est très dispersé. On m'offre de l'ugali, boule de farine de maïs accompagnée d'une sauce, qui a le statut de plat national en Tanzanie : c'est le repas moderne de ceux qui se veulent ouverts sur le monde extérieur. Ailleurs c'est le nswa, gruau typiquement rwa à base de maïs, de banane et de lait, ou encore le ndolo, préparation plus solide à base de maïs et de haricot, souvent agrémentée d'un peu de tomate et de viande. Toilette complète le soir, en dépit de la fraîcheur qui vient vite, grâce au seau que m'apporte ma voisine. Le robinet qui alimente les maisons du voisinage est à plusieurs centaines de mètres de là. Plaisir d'être propre.Nouvelle journée très bien remplie. Photos de labours et d'habitat. Enquête sur les cultures puis visite à Mr. George (75 ans), l'homme riche de ce secteur. Il possède de grandes plantations de café et des bananeraies dans trois endroits différents. Fait très exceptionnel, il a cinq femmes. Chauffeur dans sa jeunesse, il a ensuite travaillé pour un planteur blanc qui lui a laissé toutes ses terres en partant. Sa maison de ciment est spacieuse, avec une large véranda, mais elle n'est pas terminée car la scolarité de ses trente enfants lui coûte cher. Comment faire face à la dispendieuse scolarité des enfants ? Tel est un des sujets de préoccupation récurrents chez les pères de famille rwa. J'enquête avec Mr. George sur la propriété (héritage et répartition des biens dans une famille polygame, telle que la sienne), les marques de clan, le foncier. Mr. George est trapu, les lobes de ses oreilles sont distendus par de larges trous à la manière maasaï comme c'est le cas ici de beaucoup d'hommes de son âge, que les plus jeunes n'imitent pas. Mon hôte est si parfaitement intégré dans la vie sociale locale que c'est par hasard, beaucoup plus tard, que j'apprendrai qu'il est d'origine arusha.En sortant de chez lui je tombe à l'improviste sur une petite fête : les paroissiens sont venus rendre visite à une jeune maman, dont le nourrisson a trois mois. Ils sont installés au dehors sous une bâche et chantent en choeur, hommes et femmes ensemble avec beaucoup d'entrain, un grand nombre de cantiques. Cette séance musicale est prolongée par l'écoute d'autres chants religieux sur radio-cassette. On sert à cette compagnie l'habituel thé au lait (un seau de 20 litres est nécessaire) et l'obligatoire uru, bouillie de millet et de lait dont sont nourries les accouchées et régalés tous leurs visiteurs.Retour à la tombée de la nuit, à travers la plaine où passent parfois les buffles le soir, car nous sommes à proximité du Parc National d'Arusha. Il n'y a rien à craindre d'un troupeau de buffles, me rassure P. Seul un animal isolé peut être dangereux. S'il en vient un, il faut grimper à un arbre ou se coucher par terre pour éviter d'être vu. Mais alors attention : s'il vous voit il peut vous piétiner, et un coup de sa langue puissante suffit à vous arracher la peau. Ces accidents cependant ne sont pas si fréquents : le dernier remonte au mois de juin dernier. Heureusement pour moi, les buffles ne sont pas au rendez-vous ce soir.

	Il s'agit de quelques textes et proverbes que P. aura plus tard le plus grand mal à traduire. Il faut dire que, peut-être sous l'influence de sa longue scolarité (en swahili dans le premier cycle, puis en anglais dans le second), il possède un vocabulaire plus étendu en swahili que dans sa langue maternelle. La journée se poursuit avec des compléments d'enquête sur le vocabulaire de parenté, avec l'aide d'une vieille femme car les jeunes ne connaissent plus ces mots, puis j'amorce deux autres chapitres Mercredi 21 octobre 1992

Lundi 19 octobre 1992

Comme presque chaque matin dans cette zone agropastorale qui s'étend à 1600 m d'altitude, sur les flancs orientaux du mont Méru, les nuages se sont accumulés en haut de la montagne à l'ouest, avant que la chaleur de la journée ne les dissipe. De l'autre côté, le Kilimandjaro, noyé dans les nuages, n'est que rarement visible. En compagnie de mon jeune interprète P. venu du village voisin, je rends visite comme prévu au pasteur luthérien local qui me vend un vieux livre de ki-rwa édité en 1931 par la Mission Evangélique Luthérienne de Leipzig. importants de la vie des Rwa : les clans et la circoncision.

Mardi 20 octobre 1992

Dans la petite maison de torchis, au toit de tôle ondulée et au sol de terre battue que je loue dans ce village et dont le propriétaire, un jeune homme non marié, est allé se faire héberger ailleurs pour la durée de mon séjour, j'ai reçu une visite nocturne : une souris est venue se promener à mon chevet, et s'est même autorisée à manger une de mes tomates. Petite pluie fine le matin.

Catherine BAROIN, 1994, Nairobi, La lettre d'information de l'IFRA, vol.

2, n° 2, pp. 8-9.