Ege Gursoy

Benjamin Navarro

Akansel Cosgun

Dana Kulić

Andrea Cherubini

Towards vision-based dual arm robotic fruit harvesting

Interest in agricultural robotics has increased considerably in recent years due to benefits such as improvement in productivity and labor reduction. However, current problems associated with unstructured environments make the development of robotic harvesters challenging. Most research in agricultural robotics focuses on single arm manipulation. Here, we propose a dual-arm approach. We present a dual-arm fruit harvesting robot equipped with a RGB-D camera, cutting and collecting tools. We exploit the cooperative task description to maximize the capabilities of the dual-arm robot. We designed a Hierarchical Quadratic Programming based control strategy to fulfill the set of hard constrains related to the robot and environment: robot joint limits, robot self-collisions, robotfruit and robot-tree collisions. We combine deep learning and standard image processing algorithms to detect and track fruits as well as the tree trunk in the scene. We validate our perception methods on real-world RGB-D images and our control method on simulated experiments.

I. INTRODUCTION

As one of the most important processes in agriculture, manual harvesting remains challenging due to its physically demanding nature. With the expected global labor shortage [START_REF] Karan | Resilience of food, energy, and water systems to a sudden labor shortage[END_REF], robotic harvesting becomes an important research area to explore. Robotic harvesting can improve productivity by: reducing labor costs, increasing yield quality and enabling better control of environmental implications [START_REF] Kootstra | Selective harvesting robotics: current research, trends, and future directions[END_REF]. The key challenges in robotic harvesting are:

1) Fruit localization: Determining the location of each fruit wrt the robot. 2) Scene understanding: Localizing potential obstacles in the scene such as branches, tree trunk, leaves etc. 3) Motion control: Reaching the fruit while avoiding collisions, to detach and collect the fruit. In this work, we address fruit localization by detecting fruits in RGB-D images using machine learning and tracking them by assigning the same ID over successive frames. Scene understanding is necessary to avoid robot collisions while reaching the fruit. On each image, we segment tree trunk and branches by combining a mask in Hue Saturation Value (HSV) color space with one in the depth image. We assume that tree branches are symmetric around their trunk. Then, we define a cylinder around the trunk, which the robot will avoid during operation. For motion control, two arms (cutting and collecting) move towards the closest fruit, according to the cooperative task representation [START_REF] Chiacchio | Direct and Inverse Kinematics for Coordinated Motion Tasks of a Two-Manipulator System[END_REF] while avoiding the trunk and other fruit. We are using a Hierarchical Quadratic Programming (HQP) based control algorithm to fulfill the constraints related to the robot and environment such as robot joint limits, robot self-collisions and robot-tree collisions. In 1 LIRMM, Univ. Montpellier, CNRS, Montpellier, France 2 Monash University, Australia 3 Deakin University, Australia Fig. 1. Dual arm robot BAZAR [START_REF] Cherubini | A collaborative robot for the factory of the future: Bazar[END_REF] with cutting and collecting arms.

the last phase, the cutting arm moves to the stalk bearing of the fruit (peduncle) while the collecting arm moves below the fruit. The same cycle continues until there are no more reachable fruits left in the scene. An overview of our system is shown in Figure 2 A dual-arm robot compared to a single-arm, can perform more complex and delicate movements, ensuring careful handling of fruits, particularly those that are fragile or require precise detachment. The flexibility of the dual-arm increases the adaptability to different fruit types with the same set of end-effector tools. Moreover, the use of dualarm robots enhances safety by minimizing damage to fruits and surrounding plant structures through increased precision.

The main contribution of this study is the preliminary validation of a dual-arm harvesting robot with vision and control, including : (i) real-time detection and tracking of fruit using deep learning methods (ii) dual-arm control in cooperative-task space based on Hierarchical Quadratic Programming (iii) collision avoidance of fruits and tree trunk by 2D RGB images and depth information (iv) generalization to round fruits grow on trees.

The remainder of the paper is structured as follows. A review of the state of art in autonomous fruit harvesting is given in Section II. The perception of our robot is presented in Section III. The motion control and picking steps are explained in Section IV. Finally, results of the experiments are presented in Section V, followed by a discussion and propositions for future work (Sec. VI).

II. RELATED WORK

Robotic fruit harvesting has been demonstrated for various fruits such as apples [START_REF] Davidson | Dual robot coordination for apple harvesting[END_REF], aubergines [START_REF] Sepúlveda | Robotic aubergine harvesting using dual-arm manipulation[END_REF], citrus [START_REF] Mehta | Vision-based control of robotic manipulator for citrus harvesting[END_REF], cucumbers [START_REF] Van Henten | An autonomous robot for harvesting cucumbers in greenhouses[END_REF], tomatoes [START_REF] Ling | Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision[END_REF], strawberries [START_REF] Hayashi | Evaluation of a strawberry-harvesting robot in a field test[END_REF] and sweet peppers [START_REF] Lehnert | Autonomous sweet pepper harvesting for protected cropping systems[END_REF]. Apart from [START_REF] Davidson | Dual robot coordination for apple harvesting[END_REF][6] [START_REF] Ling | Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision[END_REF], the cited systems only use one robotic arm, whose end-effector is responsible for both grasping and detaching the fruit, using rigidly linked gripper and cutter. This rigid link is specific for each fruit, limiting versatility. To address the broad diversity of fruit shapes and sizes, we propose the use of two arms.

Detecting fruits and locating them is the first challenge for the robot. Deep learning detectors are popular among agricultural robotics researchers. In [START_REF] Santos | Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association[END_REF], Santos et al. introduce an interactive instance mask generation on RGB images based on graph matching and compare the performance of learning based object detection algorithms on grapes. Kang et al. [START_REF] Kang | Fast implementation of real-time fruit detection in apple orchards using deep learning[END_REF] introduce an automatic label generation module for the detection of apples in RGB images of orchards. In [START_REF] Koirala | Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of 'mangoyolo[END_REF], Koirala et al. investigate popular object detection frameworks for mango detection and design a custom detection network. Liu et al. [START_REF] Liu | A visual system of citrus picking robot using convolutional neural networks[END_REF] follow a multi-class approach in the detection of citrus by splitting their samples into classes. Kirk et al. [START_REF] Kirk | L* a* b* fruits: A rapid and robust outdoor fruit detection system combining bio-inspired features with one-stage deep learning networks[END_REF] use a custom neural network which takes a 6-channel RGB + CIELAB representation as input. In [START_REF] Ganesh | Deep orange: Mask r-cnn based orange detection and segmentation[END_REF] Ganesh et al. train a neural network with RGB and HSV 6-channel input to detect oranges. However, the majority of these approaches have complex pipelines which are difficult to train and hard to generalize to other crop types.

Fruit tracking is another key requirement for autonomous fruit harvesting. Roy et al. [START_REF] Roy | Surveying apple orchards with a monocular vision system[END_REF] use apple contours in an Structure from Motion (SfM) pipeline, to generate dense matches and reconstruct the apples in 3D. However, SfM may have difficulty distinguishing between fruit and non-fruit objects that have similar contours. Das et al. [START_REF] Das | Devices, systems, and methods for automated monitoring enabling precision agriculture[END_REF] use a Support Vector Machine to detect fruits, optical flow to associate the fruits between frames. However, optical flow may suffer in case of occlusions. Liu et al. present a convolutional network to separate fruit and non-fruit pixels and use a Kalman Filter based Hungarian algorithm to correct Kanade-Lucas-Tomasi tracker which is computationally expensive.

Motion control is an equally important part of the harvesting robot. Ling et al. [START_REF] Ling | Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision[END_REF] propose a dual-arm robot for harvesting tomatoes. The vacuum cup-type tool attached to one of the end-effectors grasps the target fruit, while the other arm (equipped with a cutter) cuts the stem. However, the vacuum cup-type tool may not work well with fruits that have softer or more delicate skin, which may get damaged during the harvesting process. Sepulveda et al. [START_REF] Sepúlveda | Robotic aubergine harvesting using dual-arm manipulation[END_REF] introduce a dual-arm aubergine harvesting robot configured similarly to humans. The planning algorithm decides the movement modes which are the simultaneous harvesting of two fruits or harvesting a single fruit with one arm, depending on the fruit location. Davidson et al. [START_REF] Davidson | Dual robot coordination for apple harvesting[END_REF] propose a dual-arm coordination strategy for fruit collection to reduce the harvesting cycle time. Rather than displacing the fruit to a bin, the picking arm drops the apple into a robotic collector near the point of fruit detachment. Their approach is hard to generalize to different fruits. Additionally, the dropping of the fruit may result in damage or bruising, which can reduce the quality and value of the harvested fruit.

III. PERCEPTION

Here, we present our perception system for detecting, tracking and localizing fruits, the fruit selection strategy, and the tree location estimation methodology.

In this work, we assume that fruits are ellipsoids (see Figure 3), which appear in a 2D image as ellipses with axes aligned with the image height and width. The ellipsoid surrounding the fruit has a circular symmetry i.e. it has two equal semi-diameters along the x and z axes in the camera frame. Since most round fruits (e.g., apples, lemons) have this property, we can assume that our approach is valid in those cases.

A. Fruit Localization

Fruit detection consists in determining the pixel coordinates of the corners of the rectangle which encloses the fruits in the image. The specifications of each detected fruit are fed to the tracking algorithm, which assigns a unique ID to each fruit and correlates the same fruit data in different instances to determine whether or not new detected fruits can be associated with the existing IDs. Finally, fruit localization refers to calculating the 3D spatial position of the fruits the in camera frame shown in Figure 4.

We use a deep learning object detection algorithm, YOLOv5 [START_REF] Jocher | ultralytics/yolov5: v6.2 -YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations[END_REF] because of its speed and accuracy. We conduct domain adaptation and transfer learning of YOLOv5. We initialize the network with pre-trained weights on COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] to benefit from their feature extractors and use 300 orange images from AgriPlant Dataset [START_REF] Pawara | Comparing local descriptors and bags of visual words to deep convolutional neural networks for plant recognition[END_REF] to train. We split the dataset in 70:30 ratio for training and validation. The fruit tracking task relies on DeepSORT [START_REF] Wojke | Simple online and realtime tracking with a deep association metric[END_REF], an improved version of SORT [START_REF] Bewley | Simple online and realtime tracking[END_REF], based on frame-to-frame association with a deep association metric. With the detected fruits in bounding boxes, fruit 3D positions are computed by incorporating the depth information from the RGB-D camera. We denote (u T L , v T L ) and (u BR , v BR ) the top-left and bottom-right pixel coordinates of the bounding box detected by YOLO. Fruit center in pixels is calculated as:

(u C , v C ) = u BR -u T L 2 , v BR -v T L 2 (1)
Under this hypothesis, we compute the horizontal and vertical semi-axes of the ellipse surrounding the fruit

r h = u BR -u C and r v = v BR -v C
We denote the projection of a pixel in color information (u, v) to a 3D position in camera frame (x, y, z) as follows:

x, y, z = proj(u, v) (2)
For each detected fruit, we compute its center P f = x f , y f , z f and distance to the peduncle from the center d v . Under the ellipsoidal fruit assumption, the point visible at (u C , v C ) is the nearest to the camera optical center. We denote its depth z min . the fruit depth is calculated by:

z f = z min + dist[ proj(u C , v C ), proj(u C + r h , v C ) ] (3)
with dist as the distance between two points in 3D Cartesian space.

The distance to the peduncle from the center is considered to be equal to the 3D transformation of the vertical semi-axis of the ellipse surrounding the fruit:

d v = dist[ proj(u C , v C ), proj(u C , v C + r v ) ] (4)
These results will be used in the following to determine the cutting point.

B. Fruit Selection

Fruit selection describes the process of determining the best candidate fruit to harvest. We compute the distance of the fruits in the scene to the optical center of the camera using the color and depth information as described in Section III-A. For each image, these distances with their associated IDs are stored in a list. We update the list if the position error between the previous state of a fruit ID is smaller than a threshold, otherwise the previous state is kept. This ensures a robust approach to counter temporary ID errors caused by occlusions. We delete IDs if they are not detected in the scene for 100 image frames. Finally, we select the ID with smallest distance for harvesting.

C. Tree Localization

We must estimate the tree position to avoid collisions between the robot and trunk to prevent damaging the plant or the robot while reaching for a fruit. Instead of machine learning, we use a simpler, computationally efficient method, based only on color and depth information.

A HSV color space segmentation mask and a depth mask separate tree branches and trunk from the rest of the scene while assuming that they are not occluded in the scene. We convert the RGB information into the more consistent HSV space. The HSV space is chosen since its dependence upon hue is assigned to a single value (H) rather than across all three RGB values.

On top of the HSV mask, we create a second mask using the depth information to filter out pixels that exceed a certain depth threshold. The secondary mask ensures pixels in the HSV range are not too far away and do not correspond to 3D points which do not belong to the tree.

The mean of the (u, v) pixel coordinates in the RGB image and the depth value of the remaining pixels give the position of the tree. We denote the estimation of 3D coordinates of the tree in camera frame as P t = x t , y t , z t . The mean depth of the remaining pixels after the mask gives us z t , while x t and y t are calculated using (2).

IV. MOTION CONTROL

For a dual-arm fruit harvesting robot, both manipulator arms need to approach the fruits located at different positions with high accuracy and flexibility, while safely manipulating the object. To achieve this, we design a dual-arm control strategy paired with a cooperative representation method.

A. Dual-arm cooperative control

We describe our fruit picking task using the cooperative task representation [START_REF] Adorno | Dual position control strategies using the cooperative dual task-space framework[END_REF], which fully characterizes the operational space for dual-arm control. This representation allows us to specify the operations in terms of an absolute task with pose attached to a fixed world frame and a relative task attached to the absolute task, as shown in Figure 4. The main reasons for using this representation are the ability to adapt control algorithms originally designed for single manipulators (e.g., quadratic programming based approaches) to dualarm robots, and to perform safe dual-arm object manipulation without hitting the obstacles nor damaging the fruits. Our tasks include a cutting phase to detach the fruit from its stem and a collecting phase to catch the falling fruit. We believe that this representation is a proper fit to our tasks since the cutting is best to describe in a fixed frame while catching is more appropriately described in the relative frame.

We define fruit cutting as the absolute task which controls the motion of the robot in the workspace and fruit collecting as the relative task, which regulates the relative motion between the two arms.

Let us now formally define these absolute and relative tasks. We define a desired pose of the absolute task x a = [x a , y a , z a , ϕ a , θ a , ψ a ] for the cutting (left) arm in a fixed frame, (in our case, the camera frame) and a desired pose of the relative task x r = [x r , y r , z r , ϕ r , θ r , ψ r ] for the collecting (right) arm w.r.t. the frame attached to the cutting (left) arm during the harvesting phase. Once we select the fruit to harvest in Section III-B, both arms should go towards that fruit. While harvesting, the tip of the cutting tool should be placed at the peduncle to detach the fruit while the center of the collecting tool is slightly below the fruit to catch it with the basket when it falls. We compute the peduncle location using the vertical semi-axis d v and fruit position P f as described in Section III-A. As for the orientations, we do not impose any hard constraint on θ a to maximize the reachable points in the workspace since cutting only requires the tip of the cutting tool to reach the peduncle. θ r is constrained to π/2 radian, resulting in π/2 radian angle between two arms at all times, to prevent collisions between the arms. Other orientations are set to stay unchanged from the natural pose of the robot while its arms are in front. The desired poses to reach a fruit relative to the camera frame are described by:

x a = [x f , y f -d v -d s , z f , 0, * , π] x r = [0, -d v -d s , 0, 0, π/2, π] (5)
with d s a security distance while cutting the fruit, and * indicating unconstrained task component.

In the cooperative dual-arm task representation, both the relative and absolute tasks have to be realized at the same time. However, a relevant choice is to prioritize the absolute task over the relative one. Indeed, it is crucial to fulfill the absolute task since the cutting arm position must be more precise than the collecting arm.

Thus, we use Hierarchical Quadratic Programming (HQP) as described in [START_REF] Tarbouriech | An admittance based hierarchical control framework for dual-arm cobots[END_REF]. It consists in solving a sequence of quadratic programming problems for which the solution space of a given task is restricted to the null space of higher priority tasks. This ensures that low priority tasks to do not interfere with the others. The relationship between joint velocities q and task space velocities ẋ is given by ẋ = J q with J the Jacobian matrix.

The Inverse Kinematics module based on HQP outputs the joint velocity vector which will be sent to the robot. HQP includes a set of hard constraints to be fulfilled at all times, defined both at joint space level (joint limits) and task space level (collision avoidance). We define the set of constraints under the standard form of linear inequality equations:

A q ≤ b ( 6
)
where A is the linear coefficients matrix and b the constant vector in the inequality constraints. Therefore, in the cooperative dual-arm task space and taking into account constraints, the highest and lowest priority tasks are solved through the following optimization problems:

q * a = arg min q ||J a qẋa || 2 s.t. A q ≤ b q * a,r = arg min q ||J r qẋr || 2 (7)

s.t. A q ≤ b J a q = J a q * a
where the a, r subscripts define the high priority (absolute) and low priority (relative) tasks and q refers to the vector concatenating the joint velocities of the two arms.

B. Obstacle Avoidance

To prevent collisions with the fruit and branches which can damage the robot and the plant, we approach and leave the fruit from the front before and after the harvesting phase. We define additional tasks for before and after harvesting, named approach and leave tasks. The desired poses for the approach and leaving task are as ( 5) except z a = z f -d z with d z as a parameter defining the distance to the fruit before passing to the harvesting phase.

To prevent any dangerous situation, we define the branches and trunk of the tree as an obstacle to avoid. In a real-world scenario, the branches of a tree have very complex shapes. Hence, we opt for a computationally efficient obstacle which covers the tree branches and trunk. We consider an infinite height cylinder collision object around the trunk of the tree with a variable radius to prevent the robot from entering the area between the selected fruit and the trunk. The cylinder radius depends on the fruit position. Variable radius provides us a safe approach distance towards the fruit which prevents the robot from entering the area behind the fruit. Let r obs be the radius of the cylinder. We place the center of the cylinder at P t . The height of the cylinder lays on the y axis in the camera frame. The radius is defined as r obs = dist[P f , P t ]-d h with dist the distance between two points in 3D Cartesian space. For each new fruit, r obs is updated.

Since we model each fruit as an ellipsoid with circular section in the xz plane, the fruit is characterized by: its position P f and semi-axis lengths d h and d v .

The tree and fruit obstacles are shown in Figure 5. It is important to note that since the fruit and tree positions are calculated in real-time, the location and size of the obstacles is also updated accordingly.

After modeling the obstacles, a collision avoidance constraint is generated for all pairs of potentially colliding parts which are separated by a distance d ≤ d i . The obstacle avoidance strategy is expressed as linear constraints using Velocity Damper [START_REF] Faverjon | A local based approach for path planning of manipulators with a high number of degrees of freedom[END_REF]. The definition of the Velocity Damper is given by:

ḋ * = -ξ d -d s d i -d s , d ≤ d i (8)
Fig. 6. Simulation with scanned tree, detected oranges and BAZAR.

with d i the activation distance, d s the safety distance, d the current distance between bodies, and ξ the convergence speed parameter. The derived inequality constraint is given by ḋ ≥ ḋ * . When two bodies are separated by a distance smaller than d i , their relative speed becomes bounded. The motion is slowed down until d s . If d becomes larger than 0, the only possible motion between the bodies is to maintain or increase their relative distance. The relative distance can be expressed as:

ḋ = nJ p qd (9)
with n the normalized vector joining the points on the objects and J p the Jacobian matrix computed at one object and takes as reference the other one. Using ( 8) and ( 9), we can specify a constraint in function of joint velocities under the standard form of linear inequalities [START_REF] Sepúlveda | Robotic aubergine harvesting using dual-arm manipulation[END_REF] with A = nJ p and b = ḋ * .

C. Detaching and Collecting

Although we do not cite any specific tools, they can be imagined as shears to detach the fruit and a basket to catch them while falling. Once the desired poses are reached by both arms, the cutting tool is triggered and the collecting arm waits the falling fruit. Then, we select a new fruit with the method explained in Section III-B and repeat the process.

V. EXPERIMENTS

We conduct a case study to validate our methodology on artificial orange trees in our laboratory. The fruit detection and tracking strategies are tested on images coming from the RGB-D camera, in real-time. We create a simulation scene with the 3D model of our orange tree as well as the dual arm robot kinematic model to validate our motion control strategy. The video of the experiments attached to the paper is also available online at: https://youtu. be/ObCR5biqlOM.

A. Platform Design

The setup consists of dual-arm mobile cobot BAZAR [START_REF] Cherubini | A collaborative robot for the factory of the future: Bazar[END_REF] shown in Figure 1 equipped with two 7-DOF Kuka LWR4 arms and an Intel Realsense D435 RGB-D camera. We implement the robot controller with the Robot Kinematic Control Library (RKCL). Hyper-parameters for YOLO are set to default, for DeepSORT model type: OSnet x0.25 [START_REF] Zhou | Omni-scale feature learning for person re-identification[END_REF], cosine distance: 0.2, overlap distance: 0.5, age: 300, n init:

B. Autonomous Harvesting

Fruit detection and tracking are realized using the camera on BAZAR. Then, to simulate robot actions, we created a CoppeliaSim scene shown in Figure 6. The scene includes a functional BAZAR robot and a 3D scan of the artificial orange tree. We detect the oranges in our lab using the RGB-D camera and add them to the simulation according to their ID, positions and dimensions wrt the camera frame with a sampling frequency of 15 Hz. These fruits are considered as harvesting targets for the robot.

In the first experiment, the camera detects 5 oranges in the scene as shown in Figure 7. There is 1 undetected orange which is heavily occluded by leaves. We intentionally create obstacles and move fruits with our hands to test the performance of the fruit tracking. We validate our detection and tracking method on 1935 images. The fruit ID assignment is correct in 83% of the 266 occluded images and it is 100% accurate on the rest. Picking and collecting tools are simulated by a black cylinder and a yellow disk in CoppeliaSim. We assume that if both of the end-effectors reach their goal pose for the target fruit, the harvest is successful. The norms of the position error of the absolute and relative tasks in Figure 8 show that the robot successfully harvests all the detected fruits in the scene. The experiment is finished in 32 seconds.

In the second experiment, the camera detects all 5 oranges in the scene and track them accurately at all times. The robot failed to harvest one orange due to its unreachable position. The rest of the oranges are successfully harvested. In both experiments, we do not observe any collision between the robot and the tree trunk or oranges.

VI. CONCLUSION

This paper describes a dual-arm autonomous harvesting system that uses state-of-the-art detection, tracking and control methods, which validated in an orange case study. Fruit detection shows promising results since all fruits except a heavily occluded one are detected correctly. Tracking results are positive despite the occlusions in the scene. Detected fruits in real world are successfully harvested in simulation. However, further experiments with the real robot and different fruits are necessary for the validation of our methods. During our real-robot experiments, we have encountered occlusions caused by robot arms which we could not test in our simulation. Also, since we do not consider branches as obstacles, interactions with them cause fruit displacements.

In future, a more dynamic scene is needed to test the robustness of the tracking algorithm. In some real-world cases, fruits are not always facing downwards. To face this issue, a more precise detection strategy for the peduncle can be explored. A planning algorithm should efficiently determine harvesting path which can also reduce harvest time. Cutting and collecting strategies with tools are to be determined after real-robot experiments due to the difficulties on modeling physical interactions in robot simulators.

The methods presented in this paper provide a step towards the goal of fully autonomous fruit harvesting systems. In this context, one arm solutions have already shown promising results. We believe that using dual-arm robots will improve harvesting performance and efficiency.

Fig. 2 .

 2 Fig. 2. Overview of our robotic harvesting system. Perception and Motion Control blocs are detailed in Section III and IV, respectively.

Fig. 3 .

 3 Fig. 3. Fruit ellipsoid assumption (left) and its ellipse projection (right).

Fig. 4 .

 4 Fig. 4. Coordinate frames definitions. Absolute Frame links to fixed World Frame, while Relative Frame links to moving Absolute Frame.

Fig. 5 .

 5 Fig. 5. Trunk obstacle cylinder in gray and fruit obstacle in orange.

Fig. 7 .

 7 Fig. 7. RGB image with the fruit detection and depth information are shown in top-left and top-right. Bottom-left and bottom-right images show the HSV and depth masks. 10, nn budget: 30. Intel realsense SDK is used for the image and camera parameters acquisition and the 2D to 3D projection functions. We use OpenCV 1 for image processing operations. HSV values used for the segmentation mask are H: 10-20, S: 70-130, V: 80-140 according to OpenCV's HSV conventions. Finally, CoppeliaSim 2 is used to simulate the robot in real-time.

Fig. 8 .

 8 Fig. 8. Results of the harvesting experiment. Top: norm of the position error in meters. Bottom: norm of the orientation error in radian.

https://opencv.org

https://www.coppeliarobotics.com


Célia Saghour

Mathieu Célérier

Philippe Fraisse

Andrea Cherubini

Visual servoing for dual arm shaping of soft objects in 3D

I. INTRODUCTION

Despite the presence of non-rigid objects everywhere in our daily life and the increasing need for automating their manipulation, to date there is no generally applicable and easily implementable method to shape such objects. The high number of Degrees of Freedom (DOF) of soft objects makes it difficult to control their deformation during manipulation.

Manipulating objects such as clothes [START_REF] Schlechter | Manipulating deformable linear objects: Manipulation skill for active damping of oscillations[END_REF], wires [START_REF] Li | Vision-Based Robotic Grasping and Manipulation of USB Wires[END_REF] or tying suture on biological surfaces [START_REF] Shademan | Supervised autonomous robotic soft tissue surgery[END_REF], [START_REF] Zhong | Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing[END_REF] constitute important and open challenges in robotics and related areas. Folding clothes [START_REF] Garcia-Camacho | Benchmarking bimanual cloth manipulation[END_REF] or picking fruits and/or vegetables [START_REF] Seplveda | Robotic aubergine harvesting using dual-arm manipulation[END_REF] are simple tasks for humans, which prove difficult for robots. Such tasks often involve two important aspects: the use of two coordinated arms, and the handling of non-rigid objects.

The works [START_REF] Jagersand | Experimental evaluation of uncalibrated visual servoing for precision manipulation[END_REF] and [START_REF] Shademan | Robust Jacobian estimation for uncalibrated visual servoing[END_REF] have developed uncalibrated visual servoing, a method where the model mapping visual data to robot control is estimated online. Yet, these works focus only on rigid objects. In humanoid robotics, visual servoing for eye-to-hand coordination is largely used for tasks such as reaching and grasping [START_REF] Vahrenkamp | Visual servoing for dual arm motions on a humanoid robot[END_REF] or manipulating rigid objects [START_REF] Claudio | Vision-based manipulation with the humanoid robot Romeo[END_REF] sometimes in interaction with a human [START_REF] Agravante | Collaborative human-humanoid carrying using vision and haptic sensing[END_REF]. More recently, researchers have designed visual shape servoing controllers, relying on model estimation [START_REF] Jin | Robust deformation model approximation for robotic cable manipulation[END_REF], neural networks [START_REF] Nguyen | V2cnet: A deep learning framework to translate videos to commands for robotic manipulation[END_REF] or reinforcement learning [START_REF] Wu | Learning to manipulate deformable objects without demonstrations[END_REF] to compute the command. Although these methods present good results, the first one requires a prior knowledge of the object (shape, material parameters) which limits its range of application, while the others require huge datasets and training time. It should also be noted that most of those methods rely only on single arm manipulation.

*This work was supported by European Union Horizon 2020 Research and Innovation Programme as part of the project SOPHIA under grant agreement No 871237.

Fig. 1: The goal of our work is to control a dual arm robot so that the initial contour (blue) reaches the target contour (red) in three dimensions.

In this paper, we propose a real-time, dual-arm visionbased shape servoing controller. Our method does not require any prior knowledge of the object, apart from its color, needed for visual extraction. It applies to soft objects of different shapes and materials, with very short initialization time. In addition, our task representation allows a cooperative control of the two arms, to command 3D deformations and/or 3D motions of the tracked part of the object.

II. RELATED WORK

Few works present research on soft object manipulation using a dual arm robot; most works on multi-robots manipulation of soft objects are compiled in survey [START_REF] Herguedas | Survey on multi-robot manipulation of deformable objects[END_REF].

In recent years, research for soft object manipulation has mainly focused on learning methods, and none of them, to the best of the authors' knowledge, consider cooperative task representation. The authors of [START_REF] Tsurumine | Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation[END_REF] use Deep Reinforcement Learning for clothing manipulation. To alleviate the need for heavy data-sets imposed by regular reinforcement learning methods, the authors propose two different algorithms that allow the learning process to be more efficient. However, samples (80 to 300 in their experiments) need to be generated by a human operator for prior policy initialization. Similarly, in [START_REF] Hu | 3-D deformable object manipulation using deep neural networks[END_REF], the authors use both Fast Point Feature Histogram and Principal Component Analysis (PCA) to obtain a feature vector, and Deep Neural Networks (DNN) to learn the deformation function between the object and the end-effectors. The network is trained online, but the DNN model is initialized with 100 data beforehand. The need for numerous training data is something we aim to remedy. Some works focus on specific types of objects, such as Deformable Linear Objects (DLO). Paper [START_REF] Lagneau | Automatic shape control of deformable wires based on model-free visual servoing[END_REF] presents a model-free servoing method for shaping deformable wires.

The approach tracks the deformations, to extract a visual feature based on a geometric B-spline model, and to estimate iteratively the deformation Jacobian for control. In [START_REF] Tang | A framework for manipulating deformable linear objects by coherent point drift[END_REF], Coherent Point Drift is used for state estimation, relating the point cloud of the DLO between two iterations. The authors however use learning by demonstration for task and trajectory planning, needing teaching by human operators. While these methods show good results with two endeffectors and in 3D, the use of B-spline modeling can only be used to represent one-dimensional shapes for one, while the second requires demonstrations for control.

In [START_REF] Qi | Contour moments based manipulation of composite rigid-deformable objects with finite time model estimation and shape/position control[END_REF], the authors use contour moments as a state representation of the manipulated object. Their method is applicable for elastic, composite and rigid objects, but is limited to the 2D manipulation space. Similarly, the authors of [START_REF] Zhu | Visionbased manipulation of deformable and rigid objects using subspace projections of 2D contours[END_REF] obtain a good encoding of the object shape, via PCA. Their method shows very promising results for moving and deforming 2D contours with a single arm moving in the plane (3 DOF). It can also be applied on both rigid and soft objects, allowing a large range of applications.

We build on top of that work, going farther and performing deformations in SE (3) with two cooperating robot arms, bearing in mind applications such as "shape and place". Our contributions are the following:

• While [START_REF] Zhu | Visionbased manipulation of deformable and rigid objects using subspace projections of 2D contours[END_REF] considered 3 DOF planar motion (two translations and one rotation) of a single arm, we increase the robot operational space dimension to 12 DOF (6 per arm), hence extending the range of applications. • We apply the cooperative task representation [?] to control the deformation and/or the pose of the object separately. We explore how both tasks affect the global deformation and the reaching of the target. • We validate our controller in 3 dimensions (including orientation) in a series of experiments, with different objects and targets.

III. PROBLEM STATEMENT

We consider a dual-arm robot with both end-effectors holding an object, rigid or deformable. The dimension of the robot operational space is 12 (i.e., each end-effector is free to move in SE(3)). The robot observes the object with a fixed RGB-D camera. The goal is to modify the position and/or shape of the object, to see it in a given fashion (a target image), shown in Fig. 1.

In some cases, the task may consist only in giving the object the desired shape, without caring about its position and orientation. For such a task the robot is redundant: it only requires 6 DOF, and can use the other 6 for other purposes. Yet, doing this is not trivial if each end-effector is modeled and controlled separately, as in [START_REF] Lagneau | Automatic shape control of deformable wires based on model-free visual servoing[END_REF]. A solution is to use the cooperative task representation introduced in [?] and outlined in Fig. 2, to describe the task as the combination of an absolute task and/or a relative task. The absolute task frame F abs ctrl is attached to one of the arms chosen arbitrarily (the left one in our case) by a virtual link (dashed orange in the figure) and it is described in the absolute reference frame F abs ref , which is fixed in the world. The relative task describes the pose of one end-effector F rel ctrl in the frame of the other one F rel ref .

This representation allows to consider as relative task the object deformation, and as absolute task, the object's pose. Then, for operations which solely imply deformation or shaping, only the relative task matters, "freeing" the absolute task's 6 DOF. On the contrary, for rigid object manipulation consisting only in translating or orienting, the relative task's 6 DOF are "freed". More formally, two alternatives are possible (k = 6 or k = 12 DOF), as one can either control only one of the poses or both the relative and absolute poses:

r = r ref ∈ R 6 or r = r abs r rel ∈ R 12 . (1)
with ref = {abs, rel}. To represent orientations, we use angle-axis vectors q r = [x r y r z r ], so r ref is defined as:

r ref = x ref y ref z ref x ref r y ref r z ref r ∈ R 6 . ( 2
)
Another important aspect is the perception of the object. In our work, the robot has no knowledge of the object's material or characteristics, except for its color (needed for visual extraction) and current shape. At each iteration, to represent the object shape, we use the contour seen by the RGB-D camera. We extract the target contour (to be reached) from one of few previously saved images of the same object, held by the robot in various positions/shapes.

It is worth discussing the limits of the contour representation. Using 3D contours as representation will make it impossible to apply any deformation out of the image plane or manage self-occlusions. Furthermore, the target contour is reachable, e.g., it should not require changing from convex to concave bending. We show a case of unreachable target in our experiments video, linked in Section V.

We denote the visible contour, composed of the 3D metric coordinates of p points, c ∈ R 3p . Working with a task of such high dimension (300 if p = 100 points) makes it complex to control the largely lower number of DOF (k = {6, 12}) of our robot. To solve this underactuated problem, we encode the contour c into a smaller feature vector s ∈ R k , with k the dimension of the required robot pose. Furthermore, we consider that the mapping between feature variation δs and robot pose variation δr is linear, through what we refer to as the interaction matrix L ∈ R k×k :

δs = Lδr (3)
In the rest of the paper, we assume the following:

• The object is already grasped by the two robot endeffectors.

• The tracked part of the object stays entirely visible throughout the manipulation, and can be represented by closed contours c. • The object target shape and (when applicable) pose are reachable, i.e. the tracked part of the object can physically be deformed to (and placed at) the target. Our framework (outlined in Fig. 3) operates as follows. First, we control the robot end-effectors (initially in open loop), to collect a sequence of images, with the corresponding extracted contours c and robot poses r. These are stored in matrices C and ∆R, respectively. Then, we perform a PCA (Principal Component Analysis) on the sequence of contours C, to encode the current contour c into a smaller kdimensional feature vector s, via projection operator U. This reduces the task shape dimension to the number of required DOF. We also use these sequences of object contours and corresponding robot poses, to estimate the inverse of the interaction matrix L, which maps feature variations to robot pose variations according to (3)). Since linear mapping (3) is only valid locally, we must limit the motions to small displacements and cannot drive the manipulators to the final target right away. Therefore, at every iteration i, we compute a local target c * i , via a linear interpolation between current and target contours (c and c * , respectively). We encode this c * i into a feature vector as well, denoted s * i . We use the inverse of the interaction matrix L -1 to compute the robot pose variations δr, required to drive the object to the local target, s * i . We then send δr to our dual-arm controller [START_REF] Tarbouriech | Dual-Arm control strategy in industrial environments[END_REF], which relies on hierarchical inverse kinematics, to compute the robot joint commands. The whole process is repeated, by continuously updating data matrices C and ∆R at each iteration i, until the object reaches the target shape and/or position. While previous works made use of these different techniques (i.e. cooperative task representations, image Jacobian estimation), our work presents a promising approach to control deformable objects in SE(3), without considering specific actions or objects (e.g., cables or clothes).

IV. OUR METHOD

In this Section, we detail each of the modules which compose our framework (again, refer to Fig. 3).

A. Image processing for object contour extraction

We use an Intel Realsense D435 camera, which looks at the robot end-effectors and at the object from a fixed (in the Fig. 3: Overview of our framework. The robot poses and object contours are stored in matrices ∆R and C, respectively. At each iteration, PCA yields the projection matrix U, for extracting the feature vector s from contour c, and the local target s * i from contour c * i , obtained by linear interpolation between c and c * . We also project the contour variation matrix C to estimate the inverse interaction matrix L -1 . We use L -1 to compute the robot desired pose variations δr needed to drive s to s * i . These δr are finally sent to the dualarm controller, which computes the robot joint commands θ.

Fig. 4: Steps of the image processing for object contour extraction (here, a rectangular sponge). The 2D RGB image of the camera is used to extract a mask of the object, which in turn lets us obtain its contour. This contour is sampled into a constant number of pixels, which are consistently spaced and ordered. These pixels are then projected as 3D points using their depth and the camera intrinsic parameters. world) position. Both the RGB and depth images are used to obtain a sampled, ordered contour of the object in 3D at each iteration. The different steps of the contour extraction are presented in Fig. 4.

To simplify the image processing algorithm -which is not the main scope of our work -we have only considered blue objects or blue parts of the objects. After thresholding in the HSV space, we obtain the points defining the contour of the object. The contour is then sampled into a given number of points p; it is essential for the contour points to be uniformly spaced and, more importantly, always ordered the same way from one contour to the next. Namely, the indexes of the points in c must correspond in c i and c i+1 , for the PCA to extract the accurate contour variations. We chose to order the contours in a clockwise direction starting from the left end-effector. Once the contour is sampled and ordered, it is projected in 3D metric coordinates using the pixel depth and the camera intrinsic parameters.

The output of this module, at each iteration i, is vector:

c i = [x 1 i , • • • , x p i , y 1 i , • • • , y p i , z 1 i , • • • , z p i ] ∈ R 3p . (4) B.

Generating a sequence of contours and robot poses

The next step consists in generating a sequence of contours and corresponding robot poses. These are respectively the contour variation matrix C and the robot pose variation matrix ∆R, which are needed for two purposes: to extract the contour's principal components, and to estimate the inverse interaction matrix. Matrices C and ∆R are built on a sliding window containing the most recent data: at each iteration, the oldest data in C and ∆R are removed and replaced by the current data.

To build matrices C and ∆R, we collect the last M + 1 samples of contours c and robot poses δr (so as to get M variations) while the robot moves. While after M + 1 iterations > M + 1) the robot autonomously moves using the designed control law, for the first M + 1 iterations we let it move in open-loop. We do this by executing a series of small motions to deform the object, exciting one by one each considered DOF as shown in Fig. 5. The magnitude of these motions is chosen arbitrarily so as to sufficiently vary the contour; we set ±0.02 m for the translations and ± π 6 rad for the rotations. The relative motions should be reduced for stiffer or more fragile objects, to avoid breaking them. We decided to move along each robot DOF in both directions (±) just once yielding M = 2k. M should not be too small, to ensure that L -1 is not under-determined, and not too large, so that the model estimation is local and initialized quickly.

To build matrix C, we stack all the c i :

C = [c 0 , • • • , c M ] ∈ R 3p×M +1 . ( 5
)
Building ∆R is a bit more cumbersome. We must obtain the robot pose variation δr i between consecutive iterations i -1 and i. For the translations, we simply subtract the components. For the orientations, we transform rotation Fig. 5: Initialization motions of the object (here, a rectangular sponge) for the absolute (left) and relative (right) task. Here, each of the k = 12 DOF are stimulated: 3 translations and 3 rotations around the initial position (black) for each task. vectors q ref ri and q ref ri-1 to quaternions, apply the quaternion difference and then transform back to the rotation vector representation.

Finally we stack all δr i , to compose robot pose variation matrix:

∆R = δr 1 . . . δr M ∈ R k×M . (6)

C. Principal Component Analysis

Even for a small dataset and small number of contour samples, the dimension of matrix C is too high compared to the robot DOF (for M = 24 and p = 100 points: C ∈ R 300×25 ). Hence, we perform a PCA on C, to reduce its dimension. First, we shift each column of C by the mean of all columns, c = cj M +1 , j = 0, ..., M , to obtain:

C m = [c 0 -c, • • • , c M -c] ∈ R 3p×M +1 . ( 7
)
Then, we compute Q, the covariance matrix of C m . We obtain the eigenvector matrix U ∈ R 3p×3p by performing a Singular Value Decomposition (SVD) on Q. We select the first k columns of U to be able to control k DOF of our system (no redundant visual data, [START_REF] Chaumette | Visual servoing. In Robot Manipulators: Modeling, Performance Analysis and Control[END_REF]). They define the projection matrix U k ∈ R 3p×k . These columns correspond to the k principal components (with highest variances) in the dataset, and therefore determine the directions of highest variability in the data.

At each iteration i, the reduced feature vector is then:

s i = U k (c i -c) ∈ R k . (8)

D. Estimation of the Inverse Interaction Matrix

The next step consists in estimating the inverse interaction matrix L -1 needed for control. Recall that the interaction matrix is the linear mapping between feature variation δs and robot pose variation δr (see [START_REF] Shademan | Supervised autonomous robotic soft tissue surgery[END_REF]). This matrix is unknown for a non-rigid object, and it should be inverted to control the robot pose.

To this end, we project the contour matrix C into the reduced space as follows:

S = U k C m ∈ R k×M +1 , (9)
and then derive the features variation over the Mdimensional window:

∆S = S 1 -S 0 . . . S M -S M -1 ∈ R k×M . ( 10
)
Finally, the inverse interaction matrix is given by:

L -1 = ∆R∆S + ∈ R k×k (11)
With ∆S + the pseudo-inverse of ∆S.

E. Controlling the robot pose

Linear approximation (3) is only valid locally. Therefore, a control law based on L -1 cannot guarantee convergence if the initial and final contours are too far. To solve this issue, we design a local target contour c * i at each iteration, via linear interpolation:

c * i = c * -c i n (12)
with n big enough to ensure small displacements. Correspondingly, we can project c * i into the feature vector space to obtain the local target feature vector:

s * i = U k (c * i -c) ∈ R k (13)
The desired robot pose variations are then computed via:

δr i = ΛL -1 i (s * i -s i ) ∈ R k (14)
with Λ ∈ R k×k a diagonal matrix of control gains. This feedback controller guarantees asymptotic convergence of s i to s * i , in the ideal case that L -1 is perfectly estimated, as proved using the Lyapunov criterion in [START_REF] Zhu | Visionbased manipulation of deformable and rigid objects using subspace projections of 2D contours[END_REF].

F. Controlling the robot joints

To map the desired robot pose variations δr to the robot joint velocities θ, we use the cooperative task representation, outlined in Sect. III, and recalled here (for further details, refer to [START_REF] Tarbouriech | Dual-Arm control strategy in industrial environments[END_REF], [START_REF] Jamisola | A more compact expression of relative jacobian based on individual manipulator jacobians[END_REF]). We consider δr ∈ R 12 as the variation of r defined in [START_REF] Schlechter | Manipulating deformable linear objects: Manipulation skill for active damping of oscillations[END_REF]. From J lef and J rig (the Jacobian matrices of the left and right arm in F abs ref ), we derive the cooperative task Jacobian matrices:

J abs = 1 2 J lef 1 2 J rig J rel = -ΨΩJ lef 1 2 ΩJ rig , (15)
where

Ψ = I -Υ lef 0 I , Ω = Φ lef 0 0 Φ lef , (16)
with Υ lef the skew-symmetric matrix of the position of the left arm in the relative task frame, and Φ lef the rotation matrix of the left arm in F abs ref .

If both the relative and absolute tasks have to be satisfied at the same time (k = 12), a relevant choice is to prioritize the relative task, to avoid undesired internal stress which may damage the object and/or the robot. Taking into account joint (position, velocity and acceleration) limits, the highest priority task is solved through: θ1 ∈ min θ ||J rel θ -δr rel || 2 subject to: joint limits. [START_REF] Tsurumine | Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation[END_REF] The obtained solution vector θ1 provides a null-space condition to the second task, so the final joint velocities to be sent to the robot are: θ ∈ min θ ||J abs θ -δr abs || 2 subject to: joint limits,

J rel θ = J rel θ1 . ( 18
)
We add the relative task solution as a constraint to the absolute task to avoid interference. Using this formulation, the absolute task error is minimized as long as the resulting joint velocity vector provides the best solution for [START_REF] Tsurumine | Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation[END_REF].

If only the relative task has to be satisfied to deform the object, and we "free" the object pose, k = 6 and we can simply apply [START_REF] Tsurumine | Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation[END_REF] and set θ = θ1 .

V. RESULTS

To validate the method, we conducted many experiments, during which we tested different initial and target shapes, as well as different objects. A video of experiments and the source code are accessible at https://gite.lirmm. fr/csaghour/rkcl-bazar-flex-app.git.

At the beginning of each experiment, M = 24 predefined motions are executed. We set the number of contour points to p = 100 and the number of intermediate targets to n = 20. The control gain matrix was tuned experimentally to Λ = 0.07 I k , to obtain motions neither too fast nor too slow.

The experiments using a rectangular sponge demonstrate successful convergence to the different final targets (in red in the figure), involving the deformation, as well as change in the orientation and position of the object (see Fig. 6).The evolution of the cooperative task frames is shown in the third column of Fig. 6.

We define the task error as the difference between the current contour and the final target contour:

= c * -c i (19)
This metric decreases until reaching an acceptable threshold (see the last column of Fig. 6).

We also performed experiments with the other objects, soft (a differently shaped sponge, a plastic glove) and rigid (a cardboard box). The method proves to be effective with those as well, as shown in Fig. 6, despite the rough contours for an object of more complex shape like the glove. For this experiment, noise was detected in the images, explaining the wide oscillations of the error observed in Fig. 6. Nevertheless, the controller converged to the final target shape, proving the robustness of the method. Again, tasks like bending, compressing, rotating and translating were successfully carried out, proving the large spectrum of applications of our controller.

Regarding the manipulation of the cardboard box, which we considered as rigid, the experiment consists only in orienting and translating the object. Through this experiment, we show that our framework can be used for manipulating a wider spectrum of objects, including rigid ones. In this case, only k = 6 DOF are needed for the absolute task, and the 6 other DOF are freed. These could be used, for example, for obstacle avoidance.

We can, on the contrary, choose to only deform the object and not move its position and orientation in the space. Then, only the relative task is needed, therefore reducing the number of DOF to k = 6. Such an experiment is presented in Fig. 7. To achieve this, the target is aligned so as to match with the fixed end-effector during the experiment. This allows us to shape the object with no concern on the orientation and position. The error is then computed between the aligned contour and the current contour.

VI. CONCLUSIONS

This paper presents a complete method for dual-arm shaping of soft objects in 3D and in real time. We use a Fig. 6: The two first columns are the robot camera view of seven experiments. Starting from different initial configurations (in blue on the first column), we reach the final targets (in red). The third column shows the initial (transparent) and final poses of the absolute frame F abs ctrl and relative frame F rel ctrl . The last column shows the evolution of the shape difference defined in [START_REF] Lagneau | Automatic shape control of deformable wires based on model-free visual servoing[END_REF] according to the iteration step. The experiments include, from top to bottom: four different shaping of a rectangular sponge, then shaping of different objects: a crown-shaped sponge, a plastic glove, and a cardboard box. For the cardboard box, the relative frame F rel ctrl is freed, so only the absolute task F abs ctrl evolves. Noise in the background during the experiment with the glove causes the shape difference to oscillate greatly, but the error still decreases. few initializing motions to both execute a PCA to reduce the dimensions of the visual data and to compute an interaction matrix, and iteratively obtain robot control inputs to reach a final target shape. The framework is able to handle objects of different geometry and materials, soft and rigid alike without any prior knowledge, although we experimented on a limited variety of objects. Nevertheless, the 3D target shapes could be successfully reached in these experiments.

From a theoretical viewpoint, our work raised questions on the representations, while considering the numerous DOFs needed to shape and move the objects. We believe that using a relative task to describe and control exclusively deformations is a promising approach. Our work addresses different challenges regarding 3D motion representation, while considering the limitations of the objects' geometrical representation. In an industrial context, we aim at proposing solutions for tasks such as "deform and place".

One of the main aspects to improve is vision. Contours are not well suited to describe volumetric objects and limit the deformations to those of the visible surface. Yet, a 3D volumetric sampling might be challenging to achieve, since the success of our method largely depends on the consistency of the points and on their order from one iteration to the other, which is not assured with simple point clouds. Occlusions could also be difficult to handle.

As future work, the addition of an adaptive deformation model would be interesting, including material parameters estimation. Such a hybrid controller would be able, for instance, to avoid non recoverable deformation of the object and to ensure target reachability.

Fig. 2 :

 2 Fig. 2: Schema of the task representation. The relative task (green) describes the motion of F rel ctrl in relation to the relative frame F rel ref , i.e. the right end-effector in relation to the left one. The absolute task (red) describes the motion of F abs ctrl , in relation to the absolute frame F abs ref . The dashed line is the virtual link between the end-effector and F abs ctrl .

Fig. 7 :

 7 Fig. 7: Experiment with the rectangular sponge, reaching the target configuration from the initial position (in blue, left) while only controlling the relative task. The target (in red, top row) is aligned with the relative reference frame F rel ref (the left end-effector) so that the aligned target (in red, bottom row) can be achieved with only the relative task being controlled. On the bottom row are shown the evolution of the cooperative task frames (left) and of the error (right).


Guest Editorial Special Issue on Active Perception for Industrial Intelligence

Huaping Liu1 , Nathan Lepora2 , Andrea Cherubini3

Information technologies are permeating all aspects of manufacturing systems as well as other fields, expediting the generation of industrial big data. Traditionally, the devices collected the sensor data from various sources, and information fusion was then performed. This incurs higher burden of time and storage cost. Recently, more and more intelligent devices are equipped in the industrial environment. This provides more opportunities for better data collection and processing for industrial intelligence. Active perception technology, which performs control strategies on the data acquisition process, enables the devices to seamlessly integrate the perception and action to reach high-level goals rather than to accomplish low-level commands. It helps to select more useful information and may save the life of the sensors. However, there exist many unsolved challenging problems since the feedback is performed on complex processed sensory data, i.e., various extracted features.

In this special issue, we mainly focus on active perception technology for industrial intelligence, addressing both original algorithmic development and new applications of active perception. We received 24 high-quality submissions and only 16 ones are finally accepted for publication after several rounds of peer review. The accepted contributions can be divided into the following three parts.

The first part solves some basic problems for active percep-tion tasks. In Unsupervised machine learning based scalable fusion for active perception, M. Jayaratne, D. Silva, and D. Alahakoon propose a scalable self-organizing neural archi-tecture for environmental perception based on multimodal fusion using unsupervised machine learning. In Metric learn-ing for patch-based 3D image registration, S. Li, T. Zhang, D. Zhang, Y. Nie, and J. Wang propose learning a simi-larity metric of patches from reference and target images such that the pairwise patches with small projection error receive high similarity scores. In Active perception for foreground segmentation: An RGB-D data-based background modeling method, Y. Sun, M. Liu, and M. Meng develop an innovative background modeling method by using both the color and depth information from an RGB-D camera. In Adaptive control and optimization of mobile manipulation subject to input saturation and switching constraints, T. Zhao, Y. Liu, Z. Li, C. Su, and Y. Feng develop hierarchical hybrid motion/force control architecture for the manipulation and grasping of mobile manipulators. All of the work provides strong support for the active perception technology in industrial environments.

In the second part, various challenging active perception problems are tackled. In Coarse-to-fine UAV target tracking with deep reinforcement learning, W. Zhang, K. Song, X. Rong, and Y. Li propose a coarse-to-fine deep scheme to address the aspect ratio variation in UAV tracking. In Haptics electromyogrphy perception and learning enhanced intelli-gence for teleoperated robot, C. Yang, J. Luo, C. Liu, M. Li, and S. Dai propose a new perception mechanism by integrating a novel learning method to operate the robots in the distance. In Learning physical humanrobot interaction with coupled cooperative primitives for a lower exoskeleton, R. Huang, H. Cheng, J. Qiu, and J. Zhang present a novel coupled cooperative primitive strategy, which aims at modeling the motion trajectories online. In Electrotactile feedback in a virtual hand rehabilitation platform: Evaluation and implemen-tation, K. Li, P. Boyd, Y. Zhou, Z. Ju, and H. Liu perform the implementation and evaluation of the electrotactile feed-back enhanced rehabilitation system. In Autonomous indoor exploration via polygon map construction and graph-based SLAM using directional endpoint features, H. Gao, X. Zhang, J. Wen, J. Yuan, and Y. Fang develop a novel 2-D laser-based autonomous exploration approach for mobile robots. In SalientDSO: Bringing attention to direct sparse odometry, H. Liang, N. Sanket, C. Fermller, and Y. Aloimonos present a preprocessing method to incorporate semantic information in the form of visual saliency to direct sparse odometry. In Development of a humanrobot hybrid intelligent sys-tem based on brain teleoperation and deep learning SLAM, J. Li, Z. Li, Y. Feng, Y. Liu, and G. Shi develop a novel humanrobot hybrid system incorporating a motor-imagery-based brain teleoperation control. In Active learning-based grasp for accurate industrial manipulation, X. Fu, Y. Liu, and Z. Wang propose an active learningbased grasp method for accurate industrial manipulation that combines the high accuracy of geometrically driven grasp methods and the gen-eralization ability of data-driven grasp methods.

The third part presents diversified industrial applications of active perception technology. In A bionic robot navigation algorithm based on cognitive mechanism of hippocampus, N. Yu, Y. Zhai, Y. Yuan, and Z. Wang propose a navigation algorithm for constructing an accurate environmental cognitive map. In Finite-time PLOS-based integral sliding-mode adaptive neural path following for unmanned surface vessels with unknown dynamics and disturbances, Y. Yu, C. Guo, and H. Yu propose a finite-time predictor-line-of-sight-based integral sliding-mode adaptive neural scheme for the path following for USVs in the presence of unknown dynamics and external disturbances. In Contact force/torque control based on viscoelastic model for stable bipedal walking on indefinite uneven terrain, Q. Li, Z. Yu, X. Chen, Q. Zhou, W. Zhang, L. Meng, and Q. Huang adopt active contact perception instead of terrain perception and propose a contact force/torque control method based on the viscoelastic model. In Reinforced nonlinear model with update-driven for gas component perception, P. Zhang, H. Cao, Y. Zhang, and Y. Wang propose a new reinforced nonlinear model with update driven for gas component perception.

We are glad to see that the papers included in this special issue cover different aspects of the theoretical algorithm, technology, and applications. In particular, we can find lots of promising results using different perception modalities including vision, haptics, and so on. We have to acknowledge that many submissions to this special issue exhibit high quality but we were not able to accept them due to the space limitation. During the review process, reviewers contributed a lot to the quality of this spe-cial issue by thoroughly evaluating the papers and providing a great deal of constructive comments. Last but not least, the completion of this special issue cannot be separated from the strong support from the Editor-in-Chief, Prof. M. Y. Wang. We would like to thank him for his great support.

Tsinghua University Beijing 100084, China hpliu@tsinghua.edu.cn

Department of Engineering Maths University of Bristol, Bristol BS8 1TH, U.K. n.lepora@bristol.ac.uk

Laboratory for Computer Science, Microelectronics and Robotics LIRMM -Université de Montpellier CNRS, 161 Rue Ada, 34000 Montpellier, France. E-mail: cherubini@lirmm.fr