Ege Gursoy

Sonny Tarbouriech

Andrea Cherubini

Can robots mold soft plastic materials by shaping depth images?

Keywords: Robotic Manipulation, Vision-Based Control, RGB-D Image Processing, Machine Learning

Can robots mold soft plastic materials by shaping depth images? The short answer is no: current day robots can't. In this article, we address the problem of shaping plastic material with an anthropomorphic arm/hand robot, which observes the material with a fixed depth camera. Robots capable of molding could assist humans in many tasks, such as cooking, scooping or gardening. Yet, the problem is complex, due to its highdimensionality at both perception and control levels. To address it, we design three alternative data-based methods for predicting the effect of robot actions on the material. Then, the robot can plan the sequence of actions and their positions, to mold the material into a desired shape. To make the prediction problem tractable, we rely on two original ideas. First, we prove that under reasonable assumptions, the shaping problem can be mapped from point cloud to depth image space, with many benefits (simpler processing, no need for registration, lower computation time and memory requirements). Second, we design a novel, simple metric for quickly measuring the distance between two depth images. The metric is based on the inherent point cloud representation of depth images, which enables direct and consistent comparison of image pairs through a non-uniform scaling approach, and therefore opens promising perspectives for designing depth image -based robot controllers. We assess our approach in a series of unprecedented experiments, where a robotic arm/hand molds flour from initial to final shapes, either with its own dataset, or by transfer learning from a human dataset. We conclude the article by discussing the limitations of our framework and those of current day hardware, which make human-like robot molding a challenging open research problem.

I. INTRODUCTION

T HROUGHOUT evolution, humans have assimilated amazing manipulation skills. An example is sculpting, which has progressed from prehistoric cave handprints to Renaissance masterpieces (see Fig. 1 1 ). Achieving these results requires many capabilities. First, the sculptor must conceive a rough representation of the intermediate shapes, leading from initial to desired state of the molding material. Then, s/he must choose and use the most appropriate actions and tools for obtaining each of these intermediate shapes. While sculpting, s/he relies on perception (mostly visual and tactile) of the material properties and shape, to continuously update the process. Similar, albeit simpler, skills are needed for everyday tasks, such as gardening, cooking, massaging or manipulating cloak.

Recently, [START_REF] Duenser | Robocut: Hotwire cutting with robot-controlled flexible rods[END_REF], [START_REF] Ma | Stylized robotic clay sculpting[END_REF] have presented impressive robotic sculpting, achieved with a dual arm manipulator cutting clay via a hot wire. Other pioneer works on robotic sculpture include [START_REF] Xuejuan | Robot 3D sculpturing based on extracted nurbs[END_REF],

All authors are with LIRMM, Univ. Montpellier, CNRS, Montpellier, France. firstname.lastname@lirmm.fr Fig. 1: It took over 20'000 years for human sculpture to evolve from the prehistoric cave finger flutings of Gargas in France (left) to Michelangelo's Moses (right). [START_REF] Pagliarini | The development of robot art[END_REF]. While their approaches are very original, the authors of these works focus on a specific task (sculpting), and rely on tools (e.g., hot wire) customized for their application and matter (e.g., clay). In the authors' view, molding in humanlike manner soft materials would enable robots to perform a plethora of new operations and assist humans in many daily chores, well beyond robotic sculpture [START_REF] Duenser | Robocut: Hotwire cutting with robot-controlled flexible rods[END_REF]- [START_REF] Pagliarini | The development of robot art[END_REF]. However, to date, roboticists have mainly focused on rigid object manipulation. The reason is that manipulating soft materials requires perceiving and modeling deformations at a high frame rate. While visual features can be consistently detected and tracked on rigid objects, they change over time on deformable materials, misleading both model-based and feature-based visual trackers. Tactile and force feedback could also be beneficial. These senses could complement the 3D geometry measured by vision, to infer the material's mechanical properties. Yet, complicated contact models are necessary to map force/tactile signals to the corresponding object surface displacements. Furthermore, the force/tactile measures should cover, with sufficiently high resolution, the whole robot skin. To the best of our knowledge, to date there exists no machine with such actuation and sensing capabilities.

In our previous work [START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF], we have addressed kinetic sand molding with 2D vision. Yet, without depth information, the set of controllable actions/effects is reduced. Typically, in [START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF], we could only modify the material's visible contours. That experience convinced us to opt, here, for depth vision, which -compared to 2D vision -enables shaping along all three directions. Besides, depth cameras, such as the Microsoft Kinect and the Intel RealSense, have recently gained a huge success in robotics.

Therefore, here, we explore the possibility of deforming soft plastic materials 2 by relying solely on depth images (i.e., with neither a model of the material, nor tactile feedback, nor registered point clouds). We propose the direct use of raw depth images instead of point clouds, to avoid cumbersome processing involving registration, and requiring high computational times and memory usage. Our rationale is similar to the one which motivated visual servoing directly in the image, rather than in the 3D space [START_REF] Chaumette | Visual servo control, Part I: Basic approaches[END_REF].

A. Related Work

Researchers have used point clouds to detect objects in indoor scenes [START_REF] Anand | Contextually guided semantic labeling and search for three-dimensional point clouds[END_REF], and to estimate their 6-DOF pose using 3D Geometric and Photometric feature descriptors [START_REF] Hwang | Robust descriptors for 3d point clouds using geometric and photometric local feature[END_REF]. A related application is semantic object labelling, realized in [START_REF] Engelmann | Know what your neighbors do: 3d semantic segmentation of point clouds[END_REF] with deep learning. These techniques require rich datasets, such as the Yale-CMU-Berkeley real-life objects set [START_REF] Calli | Yale-cmu-berkeley dataset for robotic manipulation research[END_REF], which is specifically designed for benchmarking manipulation research. Crowd-sourcing can be useful to build and label point cloud datasets, as shown in [START_REF] Sung | Robobarista: Object part based transfer of manipulation trajectories from crowd-sourcing in 3d pointclouds[END_REF]. Another application of point clouds is robot grasping, both with hands and suction cups. Pas and Platt [START_REF] Pas | Using Geometry to Detect Grasp Poses in 3D Point Clouds[END_REF] propose to detect grasp points on novel objects presented in clutter, while accounting for the robot hand geometry. The authors of [START_REF] Mahler | Dexnet 3.0: Computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning[END_REF] have introduced Dex-Net 3.0, a dataset of 2.8 million point clouds, suction grasps, and grasp robustness labels, which computes the quality of the seal between suction cup and local object surface. Point clouds are studied well beyond robotics, with applications including plant phenotyping [START_REF] Ziamtsov | Machine learning approaches to improve three basic plant phenotyping tasks using three-dimensional point clouds[END_REF].

Despite this success, point cloud dimensions make processing cumbersome: even most current day implementations of the Iterative Closest Point (ICP) algorithm [START_REF] Zhang | Iterative point matching for registration of free-form curves and surfaces[END_REF] for aligning two point clouds are too slow for robot feedback control. ICP has been originally designed and applied to rigid scenes [START_REF] Fioraio | Realtime visual and point cloud slam nicola fioraio willow garage[END_REF], with improved versions, as in [START_REF] Liu | Efficient global point cloud registration by matching rotation invariant features through translation search[END_REF], where translation and rotation are decoupled, and a fast Bound and Branch algorithm globally optimizes the 3D translation parameter first. Recently, [START_REF] Yang | TEASER: Fast and certifiable point cloud registration[END_REF] has proposed a fast (milliseconds) and certifiable algorithm for the registration of two point clouds in the presence of numerous outlier correspondences. The authors of [START_REF] Ma | Non-rigid point set registration with robust transformation learning under manifold regularization[END_REF] have extended point registration to non-rigid clouds, via a robust transformation learning scheme. The principle is to iteratively establish point correspondences and learn the nonrigid transformation between two given sets of points. To this end, they cast the point set registration into a semisupervised learning problem.

At a higher level, many efforts have been put into non-rigid object tracking. DynamicFusion [START_REF] Newcombe | DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time[END_REF] is a dense simultaneous localization and mapping framework capable of reconstructing non-rigidly deforming scenes in real-time, by fusing RGB-D images. Since it does not require a template or other prior scene model, the approach is applicable to a wide range of moving objects and scenes. Other model-free approaches are: [START_REF] Khalil | Visual monitoring of surface deformations on objects manipulated with a robotic hand[END_REF], which tracks -using touch and vision -the surface of a 2 Plastic materials remain deformed after the external force is removed. deformable object to be grasped by a robot, and [START_REF] Staffa | Segmentation performance in tracking deformable objects via WNNs[END_REF], where a neural network can visually segment a pizza dough. An elegant alternative [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF] consists in including -if they are known -the physical properties of the object, to track it from a sequence of point clouds, with a probabilistic generative model.

When it comes down to shaping the object, the robot should be capable of predicting the next state given the current state and a proposed action. To this end, [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF] use some defined primitive tool action for rearranging dirt and [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF] presents a Convolutional Neural Network for scooping and dumping granular materials. The authors of [START_REF] Li | Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids[END_REF] learn a particle-based simulator for complex control tasks; this simulator adapts to new environments or to unknown dynamics within few observations.

B. Contributions

In this paper, we consider a manipulator equipped with an anthropomorphic hand and capable of applying three different actions at various positions of some plastic material, while observing it through a fixed depth camera. The robot must mold the material into a desired shape, by finding and applying the correct sequence of actions at the right positions. To find the correct sequence, the robot should predict the effect of each action on the environment, which is represented by a depth image. We rely on some realistic hypotheses, to make the prediction problem tractable, so that it breaks down to replacing local rectangular patches within the depth image. We generate the patches consequent to action execution, with three alternative data-based methods, two with encoder-decoder image generator network, and a simpler one with difference compensation. We train all three prediction methods with a robot dataset and with a human dataset, and we design a novel scalar metric for assessing the methods' precision, in terms of distance between point clouds. We assess our method in a series of experiments, where a robot molds desired shapes, either with its own dataset, or by transfer learning from the human dataset. Finally, we show that our method for finding the correct sequence of actions could generalize across actions, materials, and geometry.

The contributions of our paper are the following: 1) We propose an original framework for depth imagebased robot shaping of un-modeled plastic material.

Given an initial and a desired point cloud of the material, the robot plans and executes a sequence of different actions at various positions to mold the material. Due to its high dimensionality, the problem of predicting posterior (consequent to robot actions) point clouds, without a model of the material, would be intractable without an original idea: we show that, under reasonable assumptions, the shaping problem can be mapped from point cloud to depth image space, with the following benefits.

• standard image processing techniques as well as deep learning can be applied directly to the depth images. 2) To attain depth image -based shaping, we define a scalar metric for measuring the distance between two point clouds, in depth image space. The proposed distance metric is based on the observation that depth images inherently encode a point cloud, allowing for the direct comparison of image pairs in a consistent manner. In our work, we use this metric: (a) as the loss function of the generator which models changes produced by a robot action, (b) as a measure for planning the sequence of actions leading from initial to final point clouds, and (c) to assess the success of a point cloud shaping task.

Since it is the weighted mean value of the difference between two 2D images, this metric can be computed very quickly, and with low memory needs, opening promising perspectives for designing depth imagebased robot controllers. 3) We assess the framework in a series of unprecedented experiments with a robotic arm + hand, molding flour from initial to final shapes, by relying on both a robot and a human dataset. By exploiting transfer learning from the human, we do not need numerous robot experiments as in [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF]. Furthermore, in contrast with similar works [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF], [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF], [START_REF] Li | Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids[END_REF], by using a anthropomorphic robot, we push robotics one step closer to humanlike molding. Yet, robotic molding remains an open problem, due to the many limitations of current day robots, emphasized in our experiments.

II. MODELING DEPTH IMAGES AND POINT CLOUDS

In this section, we recall the fundamental notions of depth image theory, required in our work.

The luminance of each pixel of a depth camera of resolution w × h and quantification b bits is:

L uv ∈ L = 0, 1, . . . , 2 b -1 ⊂ N (1)
with (u, v) ∈ I = {1, . . . , w} × {1, . . . , h} ⊂ N 2 the pixel coordinates. The entire depth image is the matrix:

L =    L 11 • • • L 1w . . . . . . . . . L h1 • • • L hw    ∈ L w×h . (2)
In the following, we refer to depth images simply as images.

We express point coordinates in meters in the camera frame (center at the camera optical center, X increasing with the pixel columns, Y increasing with the pixel rows, Z coincident with the optical axis, and positive for points in front of the camera). Assuming a linear depth model, for any point p = (XY Z) seen in the depth image at (u, v), luminance L uv can be converted to Z:

Z (L uv ) = Z min -Z max 2 b -1 L uv + Z max ∈ [Z min , Z max ] . (3)
This requires knowing the camera depth range [Z min , Z max ] ⊂ R + . Note that this model clips at Z max (respectively, Z min ) the depths of all points which have depth greater (smaller) than Z max (Z min ); the corresponding pixels are black (white), with L uv = 0 (L uv = 2 b -1). We consider un-distorted perspective projection (i.e., a pinhole camera model). Then, the two other camera frame coordinates of p can be derived from Z as:

X (u) = u -u 0 f x Z and Y (v) = v -v 0 f y Z (4)
with Let us define m the vector function which combines (3) and ( 4) to map a pixel and its luminance to the camera frame coordinates of the corresponding point:

(u 0 , v 0 ) ∈ [1, w]×[1, h] ⊂ R
m : I × L → F   u v L uv   →   X (u) Y (v) Z (L uv )   (5)
Note that this mapping is bijective:

m -1 : F → I × L   X Y Z   →   f x X/Z + u 0 f y Y /Z + v 0 2 b -1 Z-Zmax Zmin-Zmax   . (6)
This is not the case of standard cameras, which cannot measure the point depth. Definition: visible point cloud. Let us now represent the depth image as a set D of w × h triplets, containing the coordinates of each pixel of L along with its luminance:

D = (u i , v i , L uivi ) , i = 1, . . . , w × h ⊂ I × L. (7)
The visible point cloud P = {p 1 , . . . p w×h } ⊂ F corresponding to L is the image (in mathematical terms) of D under m:

m : D → P (8)
Note that dim P = w × h. Note also that not all points of P exist physically. In particular, points corresponding to L uv = 0 and to L uv = 2 b -1 are artefacts due to the limited camera range. Furthermore, two points in P cannot be on the same optical ray (or else, one will occlude the other):

∀ (p i , p j ) ∈ P : k ∈ R : p i = kp j . (9)
In this paper, we will also use normalized pixel coordinates:

x = u -u 0 f x and y = v -v 0 f y , (10)
to rewrite equation (4) as:

X = xZ and Y = yZ. ( 11
)

III. PROBLEM STATEMENT A. The point cloud shaping task

Consider a robot, observing via a fixed depth camera the environment. The robot can execute -on the point cloud P representing the environment -an action a.

Definition: Robot action a. An action is a set of joint space trajectories, which makes the robot modify P. Each action a i is applicable at a position t = (XY Z) in the camera frame, within a finite set of possible (action-dependent) positions, t j , j = 1, . . . , N . We name A = a ij the set of all actions over all positions.

In this work, we have selected 5 actions (grasp, knock, press, pinch and poke) after having inspected the database of human sculpting images presented in [START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF]. Table I shows the characteristics of each of these actions: name, description, illustration (when applied by humans), effect on a depth image L, and "size" (parameters ∆X, ∆Y , ∆Z, which we will detail hereby).

We model the robot and environment (i.e., the point cloud) as a discrete-time system. At each iteration k ∈ [1, K] ⊂ N , the robot executes an action a k ∈ A on point cloud P k . The joint space trajectories of the action are completed within the iteration, and the point cloud is changed to a new point cloud

P k+1 = f (P k , a k ) , (12)
with f (.) an appropriate mathematical function modeling the point cloud changes.

Hypotheses We rely on the following assumptions:

1) The process is Markov; therefore, we can remove the dependency from k and rewrite [START_REF] Pas | Using Geometry to Detect Grasp Poses in 3D Point Clouds[END_REF] as:

P a = f (P, a) . (13)
This assumption also implies that: 1/ no deforming phenomena, aside from those caused by a, act on the point cloud, and 2/ the deforming phenomena caused by a are constant.

2) The effect of each action on the point cloud is bounded in the 3D space, and contained within a box, denoted B ij for action a ij . We have designed the actions in A, so that all boxes B ij are entirely visible (i.e., B ij ⊂ F). We formalize this hypothesis as follows: the points which differ between the prior and posterior point clouds are:

p ∈ P ∪ P aij \ P ∩ P aij =⇒ p ∈ B ij = [X -; X + ] × [Y -; Y + ] × [Z -; Z + ] , (14)
with (X ± Y ± Z ± ) = t ± (∆X∆Y ∆Z) . The box is centered at the action's position t j , and its size depends on the action type (e.g., an action may affect a bigger volume than another, see Table I).

3) The positions of all actions in A have the same Z coordinate, and the depth of B ij is negligible with respect to the action depth (∆Z << Z), so Z -≈ Z + ≈ Z. 4) The effects of the actions are position invariant: given i, the effect of a i on the points contained in B ij is the same for all t j , j = 1, . . . , N . Definition: point cloud shaping task Given a final desired point cloud P * , the point cloud shaping task consists in shaping point cloud P into P * . We define d k (P k , P * ) ∈ R + as a scalar function, which measures the distance between P k and P * , so that d k = 0 ⇐⇒ P k = P * . Then, the task of shaping a point cloud to P * with an accuracy d ≥ 0, consists in applying a finite sequence of actions within A: s * = {a 1 , . . . , a K } so that after K iterations: d K ≤ d. We tolerate such an upper bound on the accuracy, since even a human would be incapable of perfectly reproducing (d K = 0) a given point cloud. Yet, the sequence of actions should make d K decrease. More formally, it must be possible to apply, at each iteration k, an action a k to reduce d k :

d k+1 < d k . ( 15
)
The above equation assumes that the task can be solved using a greedy algorithm. Greedy heuristics are known to We show (for the white pixel) that the depth offset |Z p -Z q | (green) differs from the point distance (purple) by scalar factor R uv . This factor is constant on ellipses centred at the camera principal point, and increases with the sizes of the ellipses. In the top figure, we outline these elipses, which characterize image R. We also show (in blue) the camera and image frames, as well as α, the angle between the pixel projecting ray and the optical axis.

produce suboptimal results on many problems, and longer term planning should be preferred. Yet, since the focus of this paper is not planning, we have posed this assumption to simplify the point cloud shaping implementation. Below, we explain how we have defined d.

B. Distance between two Visible Point Clouds

Consider two point clouds P and Q, visible by the depth camera in images L P and L Q . In this Section, we define the metric d (P, Q) ∈ R + used in our work to measure the distance between P and Q.

A common way of defining d is to set it equal to the mean of the distances between all the pairs (P, Q) of points matched in the two point clouds:

d (P, Q) = 1 dim M (p,q)∈M p -q 2 . ( 16
)
Alternative approaches are the Chamfer distance or the Earth Mover's Distance [START_REF] Fan | A point set generation network for 3d object reconstruction from a single image[END_REF]. However, all these metrics require computing M, the set denoting the association between the point clouds, which is defined as:

M = (p l , q l ) : ∀p l ∈ P, q l = argmin m p l -q m 2 ∈ Q . ( 17
)
This matching operation can be solved by the well known point-to-point ICP algorithm [START_REF] Zhang | Iterative point matching for registration of free-form curves and surfaces[END_REF]. Although it is one of the widely used algorithms for aligning three dimensional models given an initial guess of the required rigid transformation, ICP's complexity (quadratic with the point cloud dimension) makes it very slow.

In this pilot study, the task's complexity is reduced, since the robot can realize a limited number of actions. For humanlike molding -which requires many more action types and positions -the distance computation time becomes crucial, at both learning and planning steps. In this perspective, a metric based on depth images will scale up much better than one based on point clouds. Furthermore, a point cloud uses approximately %60 more memory than the equivalent depth image. Hence, using point clouds will substantially increase memory usage and may not be suitable for deep learning on consumer GPUs.

For all these reasons, we propose to match points from P and Q which project to the same pixel in images L P and L Q (see Fig. 2). Naming

p uv = m (u, v, L p uv ) q uv = m (u, v, L q uv ) (18)
with L p uv (respectively, L q uv ) the luminance of pixel (u, v) in depth image L P (L Q ), we propose to use:

M = {p uv , q uv , ∀ (u, v) ∈ I} . ( 19
)
Note that since their pixel projection (u, v) is the same, p uv and q uv have identical x and y. Applying first [START_REF] Sung | Robobarista: Object part based transfer of manipulation trajectories from crowd-sourcing in 3d pointclouds[END_REF] and then (3), we can express their relative distance as:

p -q 2 = (xZ p -xZ q ) 2 + (yZ p -yZ q ) 2 + (Z p -Z q ) 2 = 1 + x 2 + y 2 |Z p -Z q | = R uv Z max -Z min 2 b -1 |L p uv -L q uv |. (20)
In this equation we have noted:

Z p = Z (L p uv ), Z q = Z (L q uv )
and

R uv = 1 + u-u0 fx 2 + v-v0 fy 2
. The value R uv is independent from the point clouds, is constant on ellipses centered at the camera principal point (where it is 1), and it increases with the sizes of the ellipses. A geometric interpretation when f x = f y is R uv = 1/cosα, with α the angle between the pixel projecting ray and the optical axis (see Fig. 2). For the Intel RealSense D435, with nominal parameters3 u 0 = 424, v 0 = 240, f x = 415, f y = 373, R uv varies between R = 1 (for the principal point) and R = 1.568 (for the four corner pixels).

Injecting [START_REF] Newcombe | DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time[END_REF] in ( 16) for all the pixels yields:

d (P, Q) = Z max -Z min wh (2 b -1) (u,v)∈I R uv |L p uv -L q uv |. ( 21
)
Since Z max and Z min are the same for all pairs of images (L P , L Q ), we can remove them from the equation. We can also divide by R, to redefine the distance on unit interval U:

d (P, Q) = (u,v)∈I R uv |L p uv -L q uv | wh (2 b -1) R ∈ U = [0, 1] . ( 22
)
Let us now define matrix shown in Fig. 2 for the Intel RealSense D435. Then, naming

R = 1 (2 b -1) R    R 11 • • • R 1w . . . . . . . . . R h1 • • • R hw    (23)
I = R • L ∈ U w×h , (24)
with • the Hadamard product, d can be expressed as a function of these new normalized images I:

d (P, Q) = d (I p , I q ) = 1 wh (u,v)∈I |I p uv -I q uv |. ( 25
)
Therefore, in the rest of the paper (except when specified) we work with images I, rather than L.

We have analytically proved that our distance is equivalent to the weighted pixelwise difference between the two depth images. To illustrate the difference between our distance and an unweighted version, refer to the example of Fig. 3. The same two point clouds, red and orange, each composed of three points, are seen with different camera orientations in Fig. 3(left) and Fig. 3(right). With our technique, the distance between the point clouds is the same in Fig. 3(left) and Fig. 3(right), as it should be (i.e., viewpoint invariant). Instead, with the unweighted version, which merely uses the depth differences, the distance in Fig. 3(right) will be much smaller than in Fig. 3(left). To summarize, our method calculates the geometric distance between two points instead of its projection along the camera axis (i.e., instead of the difference between the two points' depths). Hence, we think that our metric is faster and at least as easy to interpret than other similar metrics, such as Chamfer distance or the Earth Mover's Distance. Thus, it can also be useful for planning algorithms.

(P, Q) is a sum of absolute values, if it is null, I p uv = I q uv ∀ (u, v) ∈ I. Since each I p uv (respectively, I q uv ) is proportional to L p uv (respectively, L q uv ) by the corresponding element of R, if d (P, Q) = 0 then L p uv = L q uv .
Then, mapping the two images with m will yield identical point clouds, so if d (P, Q) = 0 then P = Q. Conversely: if the point clouds are identical (P = Q), applying m -1 to each one will yield identical depth images (L p = L q ). Then, applying [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF] followed by [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF] leads to d (P, Q) = 0.

IV. PREDICTING THE ACTION EFFECT ON A POINT CLOUD

In this section, we explain how to predict the effect of an action on a point cloud.

TABLE II: Pixel coordinates of the top left (u, v) and bottom right (u, v) pixel of the rectangle embedding the image of B.

X - u ≥ 0 max 1, min fxX -/Z + + u 0 , w < 0 max 1, min fxX -/Z -+ u 0 , w Y - v ≥ 0 max 1, min fyY -/Z + + v 0 , h < 0 max 1, min fyY -/Z -+ v 0 , h X + u ≥ 0 max 1, min fxX + /Z -+ u 0 , w < 0 max 1, min fxX + /Z + + u 0 , w Y + v ≥ 0 max 1, min fyY + /Z -+ v 0 , h < 0 max 1, min fyY + /Z + + v 0 , h

A. Outline

At each iteration, action a changes the current point cloud P, to produce a new point cloud P a , according to [START_REF] Mahler | Dexnet 3.0: Computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning[END_REF]. Since a can be any action within A = a ij , we must rewrite [START_REF] Mahler | Dexnet 3.0: Computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning[END_REF] for each possible a ij . We can do this by embedding the effect of a ij within the expression of f :

P a ij = f ij (P) , (26)
with:

f ij : R 3×w×h → R 3×w×h . (27)
To predict the effect of each possible action a ij , we need to know each f ij . Both the domain and codomain of these functions have very high dimension (3×w×h), and we need dim A of such functions! This clearly makes the prediction problem intractable, unless we exploit the other work assumptions to simplify it.

B. Simplifications

First, since both the prior and posterior point clouds are visible, we can project them on the corresponding images, by applying (6) followed by [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF]. Then, we can rewrite [START_REF] Fan | A point set generation network for 3d object reconstruction from a single image[END_REF] as:

f ij : U w×h → U w×h I → I a ij . (28)
By operating on images instead of point clouds, we have reduced the dimensions of the functions' domain and codomain from 3 × w × h to w × h. We can further simplify the problem, by considering Hypothesis 2: the effect of actions is bounded. The projection of box B ij in the depth image can be embedded within a rectangular region of interest (ROI). Table II gives the coordinates of the top left (u, v) and bottom right (u, v) pixel of this ROI, depending on the signs of X ± and Y ± (Z ± > 0 for all points of a visible point cloud). We derived the pixel coordinates in the second column of the table by applying (6), rounding to the closest integer (with ) and then clamping the result in the field of view. To illustrate this result, we show in Fig. 4 two examples: a red bounding box with all four values (X + , X -, Y + , Y -) negative and a green bounding box where all four values are positive.

Note that the clamping can be removed, since under Hypothesis 2, all B ij are entirely visible. This hypothesis (equivalent to u, u ∈ [1, w] and v, v ∈ [1, h]) can be guaranteed by setting the action positions t such that:

(1-u0)Z + fx + ∆X ≤ X ≤ (w-u0)Z - fx -∆X (1-v0)Z + fy + ∆Y ≤ Y ≤ (h-v0)Z - fy -∆Y . ( 29
)
Under Hypothesis 3 (i.e., Z -≈ Z + ≈ Z) the ROI top left and bottom right pixels are always:

(u, v) = (f x X -/Z + u 0 , f y Y -/Z + v 0 ) (u, v) = (f x X + /Z + u 0 , f y Y + /Z + v 0 ) . ( 30
)
Note that since all actions have the same Z, the ROI width and height, w = uu = 2f x ∆X/Z and h = vv = 2f y ∆Y /Z, are unique for a given a i (i.e., they do not depend on t).

Let us denote

I =    I u v • • • I uv . . . . . . . . . I uv • • • I u v    ∈ U w×h (31)
the submatrix of I, containing the values of the pixels which could change after an action in B ij . Action a ij will transform I into a new image:

I a ij =     I ij u v • • • I ij uv . . . . . . . . . I ij uv • • • I ij u v     ∈ U w×h . ( 32
)
By focusing on a region of the image, we have further reduced the dimensions of the domain and codomain, from w × h to w × h:

f ij : U w×h → U w×h I → I a ij . (33)
Yet, we still need to know the function f ij for each combination of action i and position j.

To further simplify the problem, we exploit Hypothesis 4: actions are position invariant.

This last simplification reduces the number of functions required: instead of dim A functions, it is sufficient to know the function of each a i :

f i : U w×h → U w×h I → I a i . (34)
In the next subsection, we explain how we determine the function f i predicting the effect of action a i on a region of interest I.

C. Predicting the action effect on a region of interest

To predict the effect of a i (noted a for simplicity) on I, we have tested three different functions f . To design them, we exploit -for each a -a prerecorded dataset of pairs of regions of interest prior/posterior to the action:

G = (I, I a ) 1 , . . . , (I, I a ) dim G ( 35
)
The three designs are: difference compensation, refined generator, and difference compensation with refined generator (denoted respectively f D , f CR , and f DCR ). The designs are outlined in Fig. 5; we will detail them in this section, after having explained how to build the dataset (35) of prior/posterior pairs. 1) Building a dataset of prior/posterior image ROI: We record a sequence of depth images L, while repeatedly applying action a. We use [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF] to convert each image L into the corresponding I.

We remove all images where the environment is occluded (typically, during the action), and extract pairs of prior/posterior images, which represent the image before/after the action. Then, we manually annotate, in these images, the ROI which contains a "visible" (by human standards) difference in depth 4 . Afterwards, we adjust the width w and height h of all the ROI to the mean of all the annotated ones, and we fix their positions so that they are identical in both images of the pair. These ROI will constitute [START_REF] Cherubini | A collaborative robot for the factory of the future: Bazar[END_REF].

2) Difference compensation: A first simple image processing operation consists in approximating the effect of a on I, by the average (over the dataset [START_REF] Cherubini | A collaborative robot for the factory of the future: Bazar[END_REF]) of the ROI differences

∆I = 1 dim G G (I a -I) . (36)
This image average is added to I, to obtain a rough prediction of the ROI, after having applied a:

f D : U w×h → U w×h I → I + ∆I. (37)
Although this operation may lead some pixels to saturate (to 0 or 1), this occurs very rarely in practice.

3) Refined Generator: Our generator is a type of artificial neural network which is constituted by two main parts. The encoder maps the input into feature representations at multiple levels and the decoder maps the features onto the pixel space. We design and train (with the prior/posterior images from G) a generator to generate a ROI from an input one:

f C : U w×h → U w×h I → f C (I) . (38)
Fig. 5: The three prediction functions f D , f DCR , and f CR , illustrated (in red) for the poke action. For the purpose of illustration, in the image ∆I we have set all negative pixels to white (i.e., to unitary value).

We have implemented f C as a modified U-net image generator [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], similar to the one used by the pix2pix generative adversarial network [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], with the following specifications (illustrated in Fig. 6). For the encoder, the size of the input layer varies according to the ROI dimensions (w, h). The input is followed by 4 downscale layers, each consisting of two 3x3 convolution layers followed by a 2x2 maxpooling layer. Filter sizes of the convolution layers are 32, 64, 128 and 256, respectively. The decoder has 5 upscale layers and 1 output layer. Upscale layers consist of two 3x3 convolution layers, followed by a 2x2 upsampling layer. Filter size of convolution layers are 512, 256, 128, 64 and 32, respectively. Outputs of the second convolution layer from downscale layers are concatenated with the output of the symmetrical upsampling layer, before being fed to the following upscale layers. The output layer is a 1x1 convolution layer with tanh activation function. All the strides are set to 1 and rectified linear unit activation is used in all the convolution layers, except for the output layer. We use as loss function between the output of f C and the corresponding posterior I a (ground truth or expected output, taken from G), the distance d defined in [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF]:

d (I a , f C (I)) = |I a -f C (I)| . (39)
Note that this loss function resembles the mean absolute error, commonly used in Convolutional Neural Networks 5 , with the addition of the R uv /R weights.

We have realized experimentally that f C is not very accurate in predicting changes in the areas of I the least affected by the action. In the areas the most affected by the action, the output of f C is accurate, apart from a slight difference in its mean pixel value. To solve both issues, we have added a refining step (described below) to the output of the generator f C .

We start by building a boolean matrix B by applying Otsu thresholding [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF] followed by morphological erosion, to the dataset average image difference, ∆I defined in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. Then:

• pixels which are the least modified by the action are null in B; we set their values to those of the corresponding pixels in image I + ∆I. • pixels which are the most modified by the action are unitary in B; we shift their values, output by f C , by an offset µ C , such that their mean value is identical to that of the same pixels in I + ∆I. In summary:

f CR : U w×h → U w×h I → B • (f C (I) + µ C J) + + (J -B) • (I + ∆I) , (40)
with J a (w x h) matrix of ones, and µ C ∈ [-1, 1] the mean of the non-zero pixels of B • (I + ∆If C (I)). 4) Difference compensation with Refined Generator: The difference compensation function f D has the advantage of not requiring a training process. A drawback is that adding an image average introduces noise (blur) on I. The authors of [START_REF] Jain | Natural image denoising with convolutional networks[END_REF] have shown that to denoise images, convolutional autoencoders, which are simpler architectures than our generator, outperform wavelets and Markov random fields on small samples of training images. Therefore, our third approach consists in passing I + ∆I to a new denoising generator, denoted f DC , and then refining the output, as in (40). We have designed f DC with the same architecture as f C . Yet, the decoder in f DC should map the latent space to a denoised version of its input. Therefore, we train this network with the prior images of G with ∆I added to them. The loss function of f DC is the distance between its output and the corresponding posterior ROI, I a (from G):

d (I a , f DC (I + ∆I)) = |I a -f DC (Ia + ∆I)| . (41)
The images output by f DC have the same issues seen for f C . Therefore, we apply a similar refinement step with boolean matrix B, and design f DCR as:

f DCR : U w×h → U w×h I → B • (f DC (I + ∆I) + µ D J) + + (J -B) • (I + ∆I) , (42)
with µ D ∈ [-1, 1] the mean of the non-zero pixels of B • (I + ∆If DC (I + ∆I)).

We have presented three alternative functions f for predicting the effect of an action on a region of interest I: f D , f CR , and f DCR . Having chosen one of the three (either (37), ( 40) or ( 42)), we can replace the resulting (predicted) submatrix in the original image I, to obtain:

I a ij =        I 11 • • • • • • • • • • • • • • • I 1w • • • • • • I i u v • • • I i uv • • • • • • . . . . . . . . . . . . . . . . . . . . . • • • • • • I i uv • • • I i u v • • • • • • I h1 • • • • • • • • • • • • • • • I hw        . (43)
Note that we have removed from the second term the j dependency (pedix), which appeared in [START_REF] Moll | Path planning for deformable linear objects[END_REF] and which is irrelevant since actions are position invariant.

V. FINDING THE BEST SEQUENCE OF ACTIONS FOR POINT CLOUD SHAPING

In this section, we explain how to compute the sequence of actions s * = {a 1 , . . . , a K } for point cloud shaping, i.e., for modifying the environment from initial point cloud P 0 to final desired point cloud P * .

By mapping point clouds to images with ( 6), we can instantiate this path planning problem in image space, i.e., find s which modifies the environment from L 0 to L * (i.e., from the depth image of P 0 to that of P * ). Furthermore, to express distance d according to ( 25), we will work with weighed normalized images I 0 and I * instead of L 0 and L * by applying [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF]. Within a set of candidate sequences of actions S each leading from I 0 to a different image I |s , we will choose the one leading to the image that is the closest to I * :

s * = argmin s∈S d I |s , I * . ( 44
)
This sequence corresponds to the shortest path from I 0 to I * , in the tree (shown in Fig. 7 with an example of forward search) of all possible images generated by all possible actions in A. Finding the shortest path in a tree is a classic motion planning problem, which can be solved using many state of art algorithms (A * , RRT, etc) out of scope here. We leave to path planning experts the pleasure of proposing the best among such algorithms. Since the focus of our paper is on how to build the tree of posterior images, we propose the following forward search algorithm (refer to Algorithm 1):

1) at each step k, generate a tree from I k to all possible (i.e., obtained after executing any action in A) posterior images I k+1 ; 2) update the sequence of actions with the action that yields the image (among all I k+1 ) which is the closest to I * and set this image as the I k for the following step; 3) loop until none of the posterior images I k+1 is closer to I * than I k -in other words, loop while (15) is verified; 4) output the action sequence s * . The most challenging part of the algorithm is the prediction of the effect of action a i on the image (lines 6 to 8). This requires an appropriate choice of f , among the three implementations proposed in Sec. IV.

A. Reinsertion of the ROI

Interested readers may find more advanced methods (often based on motion planning or on machine learning) in the literature on automatic action selection. A planning method for finding intermediate states to shape a flexible wire is presented for each action a ij ∈ A do 6:

… … … … s ⇤ = a g2 , a k1 , a p2 d 3 = 1.045
extract submatrix I from I k|s via (

using [START_REF] Laskey | Learning robust bed making using deep imitation learning with DART[END_REF] predict effect of a ij on I: I a ij 8:

with (43

) inject I a ij in I k|s to get I k+1|{s,a ij } 9: compute d ij = d I k+1|{s,a ij } , I * with (25) 10:
end for

11: if ∃ d ij < d k then 12:
find best action a k = argmin

a ij ∈A d ij 13:
update sequence s = {s, a k } end if 19: end loop in [START_REF] Moll | Path planning for deformable linear objects[END_REF]. The authors of [START_REF] Dulac-Arnold | Deep reinforcement learning in large discrete action spaces[END_REF] use deep reinforcement learning to choose within a finite number of actions, whereas [START_REF] Laskey | Learning robust bed making using deep imitation learning with DART[END_REF] relies on imitation learning to choose actions for cloth manipulating.

VI. EXPERIMENTS

In this section, we present the experiments that we did to assess our framework. The software and a video of the experiments are available online at: https://github.com/gursoyege/ lirmm-farine. The video is also attached to the paper and available at https://youtu.be/fLGf7c3cbKk.

A. Setup

Figure 8 shows our experimental setup. We focus on flour manipulation, with the three actions grasp, knock and poke shown -along with their parameters -in Table I. We did not apply actions push and pinch, since the robot actuation restrains it from realizing these actions. The three actions can be applied in N = 15 positions; hence, for these experiments, the total number of actions is dim A = 3N = 45. Flour is placed within a custom-made wooden box of size 400 x 500 mm. We use the right arm (KUKA LWR 4+) and Shadow Dexterous Hand of the BAZAR robot [START_REF] Cherubini | A collaborative robot for the factory of the future: Bazar[END_REF] with an Intel Realsense D435 depth camera rigidly linked to BAZAR's right wrist. The camera parameters obtained after calibration are u 0 = 421, v 0 = 243, f x = 433, f y = 433. Although the image resolution is 848 × 480, to calculate d we have focused on the region of interest containing the box and flour, which has size 480 × 395.

The actions are realized using the RKCL framework 6 . This C++ library allows to quickly set up a kinematic controller for any kind of single and multi robot structures, as long as the resulting system has a kinematic tree structure. With RKCL, the BAZAR components used in this experiment (one KUKA LWR 4+ arm and the attached Shadow hand) can be controlled in the operational or in the joint space as a unique entity. To this end, RKCL integrates a Quadratic Programming controller for tracking a desired pose with the robot tool, while respecting some constraints (e.g., joint position/velocity/acceleration limits, restrictions in the Cartesian space to avoid collisions, tool velocity/acceleration).

The grasping action is depicted in Fig. 9: the robot performs six successive operations to accomplish the whole action. A similar approach has been taken for the two other actions. To safely manage contact with the environment, we monitor the KUKA LWR's joint torques and switch to the next operation if they pass a given threshold. After each action, the arm returns to a fixed pose such that the camera is perpendicular to the workspace. There, the Realsense captures a depth image, at a distance of approximately 450 mm from the flour. This value corresponds to the depth of all actions' positions: Z = 450 mm. Since it is much greater than the depth of the three bounding boxes, ∆Z = 50 mm (see Table I), Hypothesis 3 is valid.

We record sequences of images L of the flour, while the actions are applied by either BAZAR or a human. Images are acquired by Intel RealSense SDK 2.0, with temporal filtering and hole filling enabled 7 . We remove all images where either the robot or human are occluding the environment (typically, while the robot/human is moving to execute the action). We apply [START_REF] Elliott | Robotic cleaning through dirt rearrangement planning with learned transition models[END_REF] to convert each image L into I. We divide the recorded images into 6 datasets (one for each of the 3 actions, applied by either robot or human). All datasets are publicly available at the following link: https://seafile.lirmm. fr/d/bc9b413e83184505aa7e/. Each dataset contains 100 pairs of prior/posterior images, representing the flour before/after the action. Then, we manually annotate, in all these images, the ROI which contains a "visible" (by human standards) difference in depth 8 . For each action, we fix the same ROI sizes for the human and robot, so that the generators can be trained/tested across agents. The sizes (w, h) of the grasp, 7 See post-processing filters in Intel RealSense SDK 2.0: https://github.com/ IntelRealSense/librealsense 8 The annotation could be automated using standard image processing techniques, similar to the ones we developed in [START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF]. I.

To implement both generators f C and f DC , we use Tensorflow 2 9 on a Dell Precision 7550 laptop (Intel Xeon W-10885M CPU, 64GB RAM, NVIDIA RTX 5000). We train both networks with the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], setting α = 0.0002, β 1 = 0.5, β 2 = 0.999 and = 10 -8 .

B. Results of Image Prediction

First, to assess our distance d, we have compared the time needed to compute the distance between a pair of depth images, using: our distance, Chamfer distance and Earth Mover's Distance (EMD). The results (on Dell Precision 7550 mentioned above) are: our distance takes 17 ms, Chamfer 153 ms and EMD 157 ms 10 . These results confirm our choice. It should also be noted that both Chamfer and EMD rely on point clouds, with the drawbacks on memory usage mentioned above.

Then, we realized a series of experiments to assess the functions f D , f CR and f DCR . We split each of the 6 datasets into training+validation (75%) and testing (25%) subsets. For cross-validation, the partitions are done four times, each time selecting randomly -as test subset -a different 25% of the whole dataset. The labeled (training+validation) set G is used to: train generators f C and f DC , find the average difference ∆I, boolean matrix B, and mean offsets µ C and µ D . We train a different generator: for each action (three), agent (two: human or robot), task (two: predicting with f C or denoising with f DC ), and partition (four) adding up to a total 48 networks. For each network, we set the training duration to 500 epochs (approximately 1 hour).

Figure 10 shows the distances (in mm) between the predicted and true ROI (mean and standard deviation of the 4 tests) for each action and for each choice of f (either f D , f CR or f DCR ). The distances are obtained by expressing distance d according to [START_REF] Schenck | Learning robotic manipulation of granular media[END_REF], and then converting it to mm. We repeat the measures four times (graphs a-d) by training and testing on either human or robot.

We can draw various conclusions from these graphs. First, testing after having trained with the same agent (Figures 10a and10d) provides lower error than transferring from one agent to the other (Figures 10b and10c); on average, 1.75±0.42 mm versus 2.53 ± 0.46 mm. This is because the images of human and robot actions are quite different, also due to the hand sizes. Yet, the transfer learning error is low enough to encourage some of the real robot tests, which we will present in the next Section. Second, training and testing on robot (1.59 ± 0.41 mm) outperforms training and testing on human (1.92 ± 0.43 mm), likely because of the machine's lower variability. Third, independently from the agent and prediction strategy f , the error (both mean and standard deviation) tends to increase from poke to knock to grasp. This is because of their difference in terms of repeatability (e.g., the effect of a poke, which Fig. 9: Six snapshots taken during the grasp action. Left to right: the arm moves above the desired location; the arm goes down, while the hand opens; the hand closes to grasp some flour; the hand opens to release the flour; the arm goes up; the arm returns to the starting position, to free the camera field of view. involves only one finger, varies less than that of a grasp).

Last, but not least, there is no relevant difference between the mean and standard deviation of the three prediction functions, f D (orange), f CR (green) or f DCR (blue). Overall, the three functions perform similarly: f D with error 2.09 ± 0.44 mm, f CR with error 2.13 ± 0.44 mm, f DCR with 2.21 ± 0.44 mm.

To better discriminate between the three prediction functions, we have run tests with training+validation subsets of 5, 10, 25, 50 and 75 images, while keeping the same testing subset of 25 images. We consider robot data for both training and testing and average over all three actions. The resultsplotted in Fig. 11 -show that as the dataset increases, the deep learning methods f DCR and f CR decrease the error faster than the more naïve f D . This is likely because, with more data, the image output by f D tends to blur, while the generator embedded in f DCR and f CR succeed in capturing the depth image details.

Finally, we have assessed the importance of the refining step, by comparing the prediction error with and without refinement. More specifically, we have compared the errors of f C , f CR , f DC and f DCR in Table III. The results clearly show the usefulness of the refinement step (values in the third and fifth column are all lower than those in the second and fourth column).

C. Robotic point cloud shaping

Since among the three prediction functions, f CR appears to be the most promising as data increases (see Fig. 11), we have used it in the following experiments to predict the effect of an action on a region of interest.

We have run two series of experiments: the first four with robot-generated images and the next two with humangenerated images. In both series, for each of the three actions, we take the best network f C among the four dataset partitions, and then proceed as follows. Given a pair of images (L 0 of the initial point cloud P 0 , and L * of the final desired point cloud P * ), we run Algorithm 1 (with f CR applied at line 7) to obtain the sequence of actions s * for shaping the point cloud from P 0 , to P * . Then, we control BAZAR robot, so that it sequentially executes the actions in s * , starting from point cloud P 0 . The final desired depth images L * have been shaped by the robot in the first series of experiments, and by the human in the second series.

The results are shown in Fig. 12. For each of the six experiments, we show (from left to right) the initial image L 0 , the final desired image L * , the image predicted by Algorithm 1, and the image obtained by the robot; we also indicate the computed sequence s * and the evolution of d (in mm) over the experiments. In all six experiments, Algorithm 1 retrieves the actions which were applied by the agent to obtain L * , although not necessarily in the same order. As the robot replays the sequence, the distance diminishes at each step in all experiments (see blue curves). As expected, the final distance is higher in the transfer learning experiments (both dataset and desired image are human-made) than in the entirely robotic experiments. This is due to the difference in shape and size of the human and robot actions, which is visible by comparing the fourth image to the second and third images, in the last two experiments.

D. Generalizing across actions, materials, and geometry

Given the limitations of the robot hardware, we decided to run another series of experiments on human data, to validate our methodology in more general situations. In particular, we assessed the following aspects:

• applying any among all five actions of Table I (including pinch and press), • shaping new -non granular -material with the functions f D , f CR trained on flour; to this end, we used kinetic sand, a plastic toy material that mimics the physical properties of wet sand, • experimenting action superposition (i.e., situations where multiple actions affected the same region), • shaping non-flat surfaces (both irregular and of constant, non-null curvature).

order. The only exception is the third experiment. Here, our method predicts two knock and two poke actions at the same position (see s * at the third line of Fig. 13), whereas the applied sequence was: s * = a r12 , a k5 , a p18 . When applying knock and poke to generate L * , the human has penetrated the material more than usual. On the human dataset, the deepest points generated by knock and poke have depths respectively 6.80 mm and 7.05 mm. Instead, to generate L * , the human applied a knock and a poke of maximal depths 19.60 mm and 17.05 mm, respectively. Therefore, the solution proposed by Algorithm 1 is coherent with the task. It should also be noted that in all five experiments, the distance diminishes at each step (see blue curves). Finally, we repeated the five experiments using the naïve f D function to predict action effect. The results were much less satisfying in terms of both final image quality and distance error. In none of the experiments, Algorithm 1 managed to retrieve the type, position and number of actions which the human applied to obtain L * . This result leads us to believe that the molding problem cannot be solved by such a simple approach, and that particularly in complex scenarios such as the ones studied here, deep learning can provide a much better solution to the action prediction problem.

VII. CONCLUSIONS AND FUTURE WORK

Although our results are encouraging, they are preliminary -to say the least. The path towards robotic molding is still long and paved with obstacles.

First, the current hardware substantially limits the range of robot operation. Even an extraordinary robot like the Shadow Dexterous Hand, used here, is designed for conventional rigid object manipulation, and can by no means reproduce the versatility and adaptability of the human hand. It is rigid, has non-reproducible behavior (due to cable actuation), is weak (e.g., not strong enough to penetrate compressed flour) and tends to break easily. This is also why if was difficult to actuate the fingers, and in two out of three robot actions (all except grasp) we maintained a constant hand posture while in contact with the material. We had to dedicate most of our time to find the appropriate material and actions, in order to record relevant data without breaking the robot! Also in terms of perception, the gap between technology and nature is critical.

Present-day force sensing is far from providing good accuracy at the low ranges required by our application, and it could not be used to close the actions' feedback loop (e.g., to decide when or how to grasp the flour). Our Shadow Hand is equipped with five (one per fingertip) SynTouch LLC Biotac tactile sensors. Yet, these sensors are uncalibrated and only provide information if the fingertip touches the environment, which is rarely the case when molding. Furthermore, to shield the robot mechanics and the Biotac themselves from flour, the hand wears a glove, which biases the Biotac readings. The ideal sensor for molding should cover palm and fingers, and be dirt and waterproof. For all these reasons, although we acknowledge the importance of force sensing, we have only focused on visual feedback.

In the future, solutions to these actuation and perception issues may come from soft [START_REF] Shintake | Soft robotic grippers[END_REF] and underactuated [START_REF] Della Santina | Dexterity augmentation on a synergistic hand: The pisa/iit softhand+[END_REF] robots or from dense tactile skins [START_REF] Kappassov | Tactile sensing in dexterous robot hands --review[END_REF]. Another worthy aspect is that humans generally rely on both hands for molding; yet, this introduces other difficulties, beyond the scope of this paper A second limitation comes from the absence of a physical model, which led us to use deep machine learning. We could improve our deep learning methods by adding data from multiple humans (not just one, like here), relying on better computational resources, or automating image annotation as in [START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF]. Yet, our results show that deep learning outperforms image difference in generalizing across actions, materials and geometry (e.g., irregular surfaces). One of the limitations of the neural networks developed here is that they are not scalable in terms of image size, and therefore lack robustness with respect to variations of the action depth coordinate. Due to their synthetic nature, the predicted images may contain some artifacts and discontinuities. Yet, despite these artifacts, our framework could compute the correct sequence of actions.

A third issue, outlined above, is that our framework relies on too many assumptions, which should gradually be relaxed, to make it applicable in real scenarios. For instance, we cannot afford variations in the relative pose between the camera and workspace, unless a rigorous calibration phase is carried out systematically, to guarantee consistency between the robot operational space and the camera frame. Besides, our action space is limited to predetermined actions. This raises the question of whether our method could provide a reasonable prediction for unknown actions. Furthermore, the action sequence search problem requires the actions to be applied at a limited and known set of positions, making it easily intractable in ambitious scenarios. To generalize to continuous action positions, one could use random sampling, Monte Carlo tree search or particle swarm optimization, to select the best sequence of actions/positions. These approaches are more general than greedy ones, since they explore a larger space of possible actions sequences. Yet, they can be computationally expensive, especially if the search space is large and the prediction model complex (as is the case here).

Despite the above issues, there is still room for hope. For instance, we believe that our approach -transforming the prediction problem and the distance metric from 3D point cloud to depth image space -is a promising avenue of research for soft plastic object manipulation. First, it can be enriched by 2D image processing techniques (e.g., convolution, mutual information). Second, in contrast with state-of-art methods (such as Iterative Closest Point), it guarantees the fast computation and low memory usage required for robot molding -at both learning and planning steps. Third, it could encompass a coarse mechanical model of the object. Generally, in the authors' view, the future of soft plastic object manipulation lies in the successful integration of a coarse physical model, refined with data-based techniques.

Fig. 2 :

 2 Fig.2: Geometric interpretation of R uv . Perspective (top) and lateral (bottom) view of the pinhole projection of point clouds Q (orange) and P (red). We show (for the white pixel) that the depth offset |Z p -Z q | (green) differs from the point distance (purple) by scalar factor R uv . This factor is constant on ellipses centred at the camera principal point, and increases with the sizes of the ellipses. In the top figure, we outline these elipses, which characterize image R. We also show (in blue) the camera and image frames, as well as α, the angle between the pixel projecting ray and the optical axis.

Fig. 3 :

 3 Fig. 3: The same two point clouds, red and orange, each composed of three points, seen with different camera orientations (left) and (right).

Proposition 1 .

 1 If both P and Q are visible point clouds, d (P, Q) = 0 if and only if P = Q. Therefore, d is an appropriate metric for the Point Cloud Shaping Task. Proof: Since d

Fig. 4 :

 4 Fig. 4: Two examples of bounding boxes (a red one with all four values X + , X -, Y + , Y -negative and a green one, where all four values are positive) with the corresponding rectangular regions of interest (ROI) in image plane. The ROI are characterized by coordinates of the top left (u, v) and bottom right (u, v) pixel. Note that these pixels are the projections of different corners of the two bounding boxes.

Fig. 6 :

 6 Fig. 6: Architecture of the Generator Networks f C and f DC .

Fig. 7 :Algorithm 1

 71 Fig. 7: Forward search in the tree of posterior images, based on Algorithm 1. Given initial image I 0 and desired image I * , at each step the sequence s is updated with the action which yields the image (green) that is the closest (has smallest distance d) to I * . In this example, the search stops after 3 steps with the sequence s * in blue, since any fourth action would increase the distance (d 4 > d 3 ). Note the similarity between the resulting image (I 3 in blue) and I * . For the sake of clarity, at each step we only show 6 of the total dim A posterior images. Distances are expressed in mm. Algorithm 1 Action sequence planner Input: Initial image L 0 and final desired image L * . Output: Sequence of actions s * = {a 1 , . . . , a K }. 1: initialize k = 0, s = ∅ 2: apply (24) to obtain I 0|∅ from L 0 and I * from L * 3: compute d 0 = d I 0|∅ , I * with (25) 4: loop 5:

Fig. 8 :

 8 Fig. 8: Experimental Setup for robotic flour shaping.

  knock and poke ROI are respectively: (183, 183), (112, 60), (40, 45); examples of these ROI are also shown in Table

  (a) Train on R and test on R. (b) Train on R and test on H. (c) Train on H and test on R. (d) Train on H and test on H.

Fig. 10 :

 10 Fig. 10: Mean and standard deviation of the distances (in mm) between the predicted and true ROI, when using f D (orange), f CR (green) or f DCR (blue). The four graphs are obtained by training and testing on either human or robot data (denoted respectively H and R).

Fig. 11 :

 11 Fig. 11: Mean error (in mm) of f D (orange), f CR (green) or f DCR (blue) over the three actions as the training+validation dataset increases: 5, 10, 25, 50 and 75 images.

  2 the camera principal point coordinates in the image plane and (f x , f y ) ∈ R 2 the camera focal lengths expressed in pixels. All points visible in the depth image belong to the camera viewing frustum F ⊂ R 3 , which is a truncated pyramid lying between Z min and Z max .

TABLE I :

 I Actions used in this work, with their characteristics.

	Action a i	Description	Illustration	Effect on L	∆X (mm)	∆Y (mm)	∆Z (mm)
	Grasp a g	The hand initially in claw shape is closed.			218	195	50
	Knock a k	The closed fist pushes downward.			84	75	50
	Press a r	The fist bottom pushes downward.			46	103	50
	Pinch a n	The index and middle fingers close.			42	85	50
	Poke a p	The index finger pushes downward.			22	18	50

TABLE III :

 III Mean error (in mm) of f C , f CR , f DC and f DCR .

	Datasets	f C	f CR	f DC	f DCR
	Train on R and test on R	1.65	1.56	1.73	1.60
	Train on R and test on H	2.68	2.51	2.80	2.55
	Train on H and test on R	2.70	2.55	2.95	2.71
	Train on H and test on H	1.96	1.90	2.11	1.97

This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.IEEE TRANSACTIONS ON ROBOTICS

See https://www.intelrealsense.com/depth-camera-d435 This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

The annotation could be automated using standard image processing techniques, similar to the ones we developed in[START_REF] Cherubini | Modelfree vision-based shaping of deformable plastic materials[END_REF]. This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

See for instance the MeanAbsoluteError class in Keras: https://keras.io/ api/losses/regression losses

https://gite.lirmm.fr/rkcl/rkcl-core This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

https://www.tensorflow.org

We used the following implementations: https://www.tensorflow. org/graphics/api docs/python/tfg/nn/loss/chamfer distance and https: //en.wikipedia.org/wiki/Earth mover's distance#PyTorch Implementation, for Chamfer and EMD (respectively). This article has been accepted for publication in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for in IEEE Transactions on Robotics. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TRO.2023.3288836 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

ACKNOWLEDGMENTS

The authors would like to thank Benjamin Navarro for his help on the robot programming and André Crosnier, Fabrice Caussidery, Phlippe Fraisse, Hervé Louche, Serge Mora, Jean-Michel Muracciole for their suggestions on the material to be used in the experiments.

s ⇤ = a g8 , a g9 , a f 13 , a f 15 , a p3 , a p2 , a f 14 , a p4 , a p1

3 g 0 T 8 w H 8 8 l 8 / r a a x q J m H / 0 J 0 / 0 C 0 n n Q i w = = < / l a t e x i t > s ⇤ = a g8 , a g9 , a f 13 , a p3 , a p2 , a p1 , a p4 , a f 14 , a We have run five experiments on kinetic sand, shown in Fig. 13, with f CR as prediction function. The actions can be applied in N = 21 hence, for these experiments, the total number of actions is dim A = 5N = 105. In all experiments and for each of the five actions, we take the best network f C among the four human dataset partitions, and then proceed as follows. Given a pair of images (L 0 of the initial point cloud P 0 , and L * of the final desired point cloud P * ), we run Algorithm 1 (with f CR applied at line 7) to obtain the sequence of actions s * for shaping the point cloud from P 0 , to P * . Then, we verify if the predicted image qualitatively resembles the final one, and whether Algorithm 1 predicts a sequence of actions similar to the actually applied one.

For each of the 5 experiments, we show (from left to right) the initial image L 0 , the final desired image L * and the image predicted by Algorithm 1; we also indicate the computed sequence s * and the evolution of d (in mm) over the experiments.

In the first experiment, the initial shape has a constant curvature of approximately 15 cm. For the four other experiments, we start with very irregular non-flat shapes (with standard variation of the depth from its mean ±17 mm -see first column of Fig. 13). In the fourth and fifth experiments, we obtained the desired image by apply overlapping actions.

In four of the five experiments, Algorithm 1 retrieves the same type, position and number of actions which the human applied to obtain L * , although not necessarily in the same