T Dellarovere
email: toma.dellarovere@insa-strasbourg.fr

M Koehl

J Rodrigues
email: johanna.rodrigues@insa-strasbourg.fr

STUDY OF DIFFERENT APPROACHES TO PHOTOREALISTIC MODELLING AND RENDERING OF COMPLEX SURFACES

Keywords: Heritage, Castle, 3D modelling, Historical constraints, Extraction, Realistic rendering

Schwarzenburg Castle is one of the few Alsatian castles that can be dated with precision. The Schwarzenbourg family, taking advantage of a favourable historical and political context, built the castle in 1261 on the municipal boundary of the town of Munster. Initially intended for residential use, the castle passed through the centuries from family to family, and shortly becoming a ruin. The first major renovation was realized in the 15 th century, but for military purposes rather than for housing. This was the only one, as the castle was quickly abandoned at the end of the same century. It was only during the First World War that the castle was used by the German army, which built a bunker in the upper courtyard. Schwarzenbourg castle is, therefore, an original in Alsace, both in terms of its history and its characteristics, which have no regional equal. The objective of this modelling project is twofold. On the one hand, to model the current state of the castle, i.e., an advanced state of ruin for certain elements, and, on the other hand, to model the reconstruction of the castle as it was when it was built in 1261. This double objective comes up against several problems, mainly technical. Indeed, how to move from a point cloud of a certain density and quality to a surface model allowing a realistic rendering? How to limit the influence of the noise on the quality of the surface model? How to remain faithful to the cloud while simplifying the geometric primitives to make the rendering more efficient? Or what technique should be used to reconstitute the castle as it was when it was built? The objective here was to realize the transformation from a consolidated and geo-referenced point cloud to a surface model allowing for a photo-realistic rendering.

INTRODUCTION 1.1 Historical context

Schwarzenburg Castle, or also called Schwarzenberg Castle, is one of the few castles that can be accurately dated [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF]. Indeed, the construction of this castle, located in Munster, France, began in the spring of 1261 and ended a year later, which is a certain feat for the time and characteristics of the castle. The first owner and builder was Walter von Geroldseck of the Schwarzenburg family, then at the top of the local political spectrum. The Middle Ages being a time when periods of stability were rare, Schwarzenburg castle did not escape the situation and found itself tossed between several families. Indeed, in 1271 Walter von Geroldseck renounced the castle in exchange for a payment and, at same time, the abbey of Munster became part of the Bishopric of Basel. It was then that the castle became the possession of the Münch von Landskron family between 1341 and 1396 [START_REF] Metz | Enquête : Le château de Schwarzenberg -Histoire et architecture[END_REF]. The castle was mortgaged many times and slowly began to fall into ruin. It was not until 1401 that the Beger de Geispolsheim family recovered the property in exchange for a restoration [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF]. The castle was then modernized by Jacob Beger at the end of the XVth century. However, it fell into ruin again at the beginning of the Modern Era and lost all its interest and military value, as it was declared uninhabitable in 1646 [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF]. It can therefore be estimated that the castle must have been in an advanced state of ruin at this time. In 1800, the castle was sold as national property to a private owner in Munster and the German army built a bunker there during the First World War, which is still visible today. The castle now belongs to the city of Munster.

Current status

All preserved parts of the castle were built exclusively with clear granite found on the hill. These stones remained raw and were not cut, which complicated the distinction of the different phases of construction. In addition, the poor quality of the mortar used was partly responsible for the various collapses over the centuries [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF]. This information is crucial for texturing the ruins as well as reconstruction. Figure 1 reflects the state of the castle in 1995, it turns out that the state of the ruins remained almost identical to that of 26 years ago. Indeed, by making a comparison with the point clouds acquired at TLS over the last three years, very few differences were found. All the hatched elements of the high court are still visible today.

Figure 1: Technical drawing representing the different phases of the castle as well as the remarkable elements available for the reconstruction [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF] and superposition of the DTM and point clouds.

The farmyard sits on a narrow terrace below the high courtyard. Today, there is almost nothing left of it. It was estimated that the farmyard was built in a second time, and it may be in the XIII century.

3D MODELLING OF COMPLEX SURFACE

Heritage modelling, and particularly modelling of ruined heritage from point clouds, required different modelling approaches and methods compared to modelling heritage in better or even newer condition. Here we had to deal with complex surfaces for our castle, which were more difficult to model than simple surfaces.

The problem of complex surfaces

There exist several possible definitions of a complex surface.

From a mathematical point of view, a complex surface can be considered as a complex two-dimensional whole. From a more pragmatic point of view, a complex surface is a surface that is difficult to parameterize using the equations of simple geometric primitives. Complex surfaces are usually surfaces with too much detail or variations in relief. It was then necessary to make a choice between a precise and detailed modelling of the initial object or the simplification of its representation by sampling the initial point cloud.

The associated problems were then of two kinds: first, the pure modelling and second, the appearance of the model. The problems related to modelling concern the editing and cleaning steps of the point cloud as well as the mesh from the original data [START_REF] Remondino | Heritage recording and 3D modeling with photogrammetry and 3D scanning[END_REF]. Care must then be taken to:

-choose the right software and algorithms for efficient processing of dense and sometimes heavy point clouds, -find the best approach according to the level of complexity of the object to be modelled, -use robust 3D mesh algorithms, -choose export formats that limit the loss of information.

Interest of 3D modelling

This modelling is necessary for the conservation of heritage, and particularly of ruined heritage that is doomed to disappear over time. The objective was to be able to keep a trace, but also to be able to study the heritage, whether to make in-depth studies or to renovate it. Heritage modelling can be used in some cases to revive heritage that has been forgotten or of which there remains no trace today. This was particularly what has been done locally in Alsace with the modelling of the disappeared castle of Württemberg in Horbourg-Wihr [START_REF] Cartier | Modélisation 3D du château disparu des Wurtemberg à Horbourg-Wihr et exploitation de la réalité augmentée pour une mise en valeur dans la trame urbaine contemporaine[END_REF]) [START_REF] Koehl | When roman antiquity and renaissance came together in virtual 3D environment: 3D modelling considerations[END_REF]. In any case, although complex surfaces were a hindrance to 3D heritage modelling, it was not impossible and was even necessary from a cultural point of view.

Surfaces used in modelling

Mesh or polygonal surfaces: Mesh surfaces, also called polygonal or mesh, are the most easily achievable surfaces that make it possible to create objects consisting of vertices, edges and faces, all organized in polygons. Meshing consists of connecting, using edges, the different nodes of a point cloud to create faces. Each face consists of 3 nodes and 3 edges. There occur also methods of refining the mesh, via the creation of new points by interpolation between the created mesh and the starting point cloud (mesh in 2 steps).

NURBS surfaces: NURBS or Non-Uniform Rational Basis Splines are a precise mathematical representation of curves and surfaces whose shape is defined by control points [START_REF] Barazzetti | Creation of Parametric BIM Objects from Point Clouds Using Nurbs[END_REF]. The use of these points allowed a simplified modification of the shape and allowed a great interoperability between the different 3D modelling software. NURBS surfaces are the final step for object representation. They are based on NURBS curves. These surfaces can be estimated from one, two or more curves that act as a geometric constraint when interpolating surfaces. It is also possible to create surfaces by direct interpolation of the point cloud, but this method turns out to be less accurate than passing through curves. Indeed, by using the direct passage, the edges of the surface will not be well defined and will be of poor quality [START_REF] Barazzetti | Creation of Parametric BIM Objects from Point Clouds Using Nurbs[END_REF]. SUBD surfaces: Subdivision surfaces, known as SUBD surfaces, are a hybrid way of creating smooth surfaces based on the structure of NURBS curves and the detail of a mesh. This type of surface makes it possible to obtain a smoother rendering with a limited number of points. SUBDs tend to replace NURBS surfaces in modeling software, as they allow for faster and more accurate modelling. Subdivision surfaces are created using a recursive algorithm applied to a starting mesh. This algorithm works by subdividing the mesh, by creating new points and faces from the position of the old points. The resulting mesh will then have many more faces than the old mesh. It will then be possible to continue the recursive algorithm to obtain an even finer mesh. There are two types of subdivision schemes: interpolating schemes and approximating schemes [START_REF] Habib | Edge and vertex insertion for a class of c1 subdivision surfaces[END_REF].

Table 1. Comparison of the different types of surfaces.

Surface modelling approaches

Unlike simple surfaces, complex surfaces are unique and therefore difficult to create by following conventional processes. Indeed, it is difficult, if not impossible, to look for the most likely equation of plans representing the ruins of a castle. There are several approaches to adapt to different situations:

-A classic 3D mesh approach, which makes it easy to obtain a 3D representation of an object, -An approach resulting from the creation of HBIM and using the Scan-to-BIM method via NURBS, -An approach derived from the use of DSM creation algorithms.

The 3D mesh approach is the simplest approach from a computational point of view, because it is based on the creation of a set of triangles all around the object to be modeled.

The HBIM approach has been used in several studies [START_REF] Diara | Building archaeology documentation and analysis through open source HBIM solutions via nurbs modelling[END_REF], [START_REF] Barazzetti | Creation of Parametric BIM Objects from Point Clouds Using Nurbs[END_REF][START_REF] Barazzetti | Parametric as-built model generation of complex shapes from point clouds[END_REF]. The creation of surfaces following an HBIM approach allows the modelling of surfaces via a NURBS creation process. Although this approach is mostly applied to point clouds representing simple geometric primitives, it turns out that it is also possible to apply them to more complex surfaces, subject to sufficient computing power.

The DSM-derived approach is an interesting approach for modeling objects whose point cloud is not dense enough or which has too much variation in relief. This approach is to be used when 3D meshes and HBIM approaches do not give conclusive results. The only negative point of this approach is the topology of the resulting mesh. Indeed, the algorithms for creating DSM will try to plate the edges of the mesh on a plane. It will then be necessary to leave more margin, during segmentation, to be able to subsequently rework the edge or sew it with other meshes. It is also necessary to perform a coordinate system transformation, the DSM using normals according to the z axis.

Reconstitution approach

The objectives of the reconstitution were identical to those of the modelling of complex surfaces, except for one detail, because here we seek to model a building as it was when it was built and not as it is today. In any case, the goal was to create a 3D model of the building that will then make it possible to make a realistic rendering thanks to the application of physical texture. This reconstruction was based exclusively on historical and architectural knowledge at a given moment. It was therefore important to have solid archives or hypotheses allowing modelling. In addition, as knowledge evolves over time, it was important to create a model that can be easily reused or exchanged between different software.

If complete archives are available, it's important to synthesize all the architectural and historical constraints. The objective of this synthesis is to be able to facilitate the analysis and interpretation of the data. This step is essential for the reconstruction of the various elements of a building [START_REF] Cartier | Modélisation 3D du château disparu des Wurtemberg à Horbourg-Wihr et exploitation de la réalité augmentée pour une mise en valeur dans la trame urbaine contemporaine[END_REF]. If there is little or no archive available on the building to be reconstituted, the process is more complex. It is obviously impossible to envisage modelling without a minimum of information. In this case, it is necessary to go through the implementation of reconstruction hypotheses based on possible archaeological excavations, or on the architecture of comparable buildings. The support of archaeologists and historians is then essential.

To follow up on the analysis of the data, it was necessary to create the database that will then be used to model the building 3D. This base was a plan which represents the footprint, certain or extrapolated, of foundations, walls, etc. It was important that this plan was in digital format so that it can be used as a basis in 3D modelling software. In some cases, the archives already offer a hypothetical plan in paper format. In this case, if the latter includes a scale, it is easily possible to scan it. The modelling step was divided into two phases. The first was to roughly model the building by extrusion of the basic plan. This extrusion made it possible to give volume to our plan and consequently to create an overall shape of the building. The second step was to refine the 3D model by improving the quality of the details. This refining was done locally according to the constraints highlighted.

Realistic rendering: the main interests of 3D modelling are to be able to preserve the heritage, but also to represent it for study or tourist purposes. This representation goes through a photorealistic rendering, which proves to be easier to access with the software on the market. To achieve a rendering, it was important to use surfaces that allow the application of texture. These textures should be created from field information or archives. Textures were not the only components that make it possible to render successfully. Indeed, it was important to set up the scene to model, the position of the object, the environment, the position, and angle of the camera as well as the type and position of the lights. All these parameters cannot be neglected, and it was highly recommended to carry out this implementation from the beginning of the modelling.

3D MODELLING OF THE RUINS OF SCHWARZENBURG CASTLE

Point cloud preparing

The point cloud of the project (Figure 2), containing the scans of all the ruins of the castle as well as its close environment, was a point cloud that was considered raw, because it has not been cleaned, segmented, or sampled, etc. It was therefore necessary, at first and before moving on to 3D modelling, to prepare this point cloud by performing the previous actions.

Data characteristics

The data acquisition was carried out in several measurement campaigns. Conventional processing results in a fully consolidated and georeferenced cloud in RGF93-CC48 (EPSG:3948). A study on the quality of the dataset was carried out for its validation. The cleaning step, which includes the cleaning itself, as well as the de-noise of the cloud, was an important and necessary step for the smooth running of the 3D modelling. Indeed, the Schwarzenburg Castle was a castle that has been in ruins for several centuries. Vegetation has therefore taken over for many years, although the site has been maintained since the beginning of the twentieth century.

It was therefore necessary to eliminate from the cloud, the trunks of trees cut or not, the vegetation present on the top of the ruins, the elements too complex to model and off-site or the coarse noise. Once the coarse cleaning step has been completed, it was necessary to carry out a more meticulous cleaning through filtering (3DReshaper). Although this tool is a "black box" tool, we have chosen to use it because the extent of our data was small. It was therefore possible to easily visualize the result proposed by the tool before validating it. Finally, as cloud cleaning was a time-consuming and meticulous operation, it was advisable to carry out a first segmentation to separate the ruins from the ground. The Figure 3 shows the whole project with a separation between DTM of environment and dense point cloud of near environment and ruins of castle. In the courtyard a bunker built by German army was also finely recorded.

Figure 3: Ruins of castle and environment (DTM).

First approach: 3D mesh in 2 steps

The 3D mesh tool (3Dreshaper) can be used in one or two steps. The first version consists of making a regular mesh all around the object. Creation parameters such as the average length of the sides of the triangles, holes detection, creation of closed or open mesh, etc. must be set out. This is a "gray box" tool because only certain parameters can be modified. The second version of the tool, which is the one used throughout this study, is a two-step mesh generation, i.e., a coarse regular mesh and then a refinement of the mesh by interpolation. For keeping the detail of a point cloud in the resulting mesh, go through the first version of the tool. However, for simplifying the surfaces, the second version was totally suitable. Therefore, we used the second one, to obtain simpler and smoother surfaces and so to be able to achieve a possible realistic rendering later. Refining was carried out by resampling the vertices, which reintroduces noise due to the use of interpolation. As with the starting point cloud, the different meshes obtained were not perfect because of the complex surfaces they represent. It was therefore necessary to clean the obtained surface, which involves eliminating artifacts (purely visual), managing holes (calculating edge adjustment using interpolation based on a buffer distance from the initial edge), smoothing (using a neighborhood topology) and reducing the number of triangles. The 3D mesh approach is an approach that can be considered easy to access. Indeed, no matter what complex surface to be modelled, we will always be able to have a result by playing with the modeling parameters.

Despite the advantages of this approach, it can be quickly realized that in cases where the cloud was not dense enough, complete, or even representing too much detail or variation in relief, the resulting mesh was not of good quality and requires post-production work that can be time-consuming. In addition, despite all the cleanings and improvements that can be envisaged, it happened that the mesh still had errors such as holes, double layers, non-manifold edges. In this case, there existed several solutions: -Simplify the surface as much as possible, but in return, loose a lot of detail.

-Segment subsets into smaller subsets representing only the different faces of the same object. This approach improved 3D meshes, but in return it was necessary to deal with topology and face join problems to export the mesh in one piece.

-Use the approach derived from DSM.

Second approach: DSM

Complex surfaces that cannot be well modelled with the previous approach can be considered as complex relief with either little or too much detail. Using this hypothesis, it was possible to reason by analogy using the algorithms for creating DSM on our complex surfaces. The first step needed the rotation of the point cloud so that its faces were perpendicular to the z axis. Indeed, if this transformation was not performed, the DSM creation algorithm would use the upper faces, if there were any, or would not create any mesh if all the faces were almost on the same plane. It was important to note the coordinates of the origin point of the rotation as well as the axis of reference to return to the initial georeferencing after processing. Although this DSM approach made it possible to obtain closed and error-free meshes, there were still other problems to be solved. First, the edges of the mesh that were plated on the datum plane. It was then necessary to provide a wider edge during segmentation to be able to rework this edge afterwards before sewing the different meshes between them.

Then, the holes that did not need to be filled were filled using this algorithm, it was necessary to recreate these holes manually by editing the mesh. To help with the reconstruction, it was possible to display the point cloud and the mesh at the same time which made it possible to better delimit the working area.

It was also necessary to rework the edge as for the 3D mesh. This approach derived from DSM allowed to overcome some of the problems encountered by 3D meshing. However, no approach was perfect, and each was accompanied by its own problems. Here for the DSM, despite an error-free mesh, the rework of the edges and holes of the model was required. The solution would then be to use a hybrid approach using 3D mesh for complex surfaces that accepts this method and the DSM approach for others. In any case, there was no single and perfect approach to correctly model any type of complex surface, because a generalist model cannot emerge. Figures 4 show the different results obtained after application of the different algorithms.

Results

All the results of the following comparisons derived from the CloudCompare software and more particularly from the C2M (Cloud-to-Mesh Distance) algorithm. This algorithm can be used both to calculate the offset between a point cloud and a mesh, but also to calculate the distance between two meshes. It was preferred to use this open-source software to more easily access the operation of the algorithms used. This C2M tool calculated the true distance between two entities, one of which will be the reference and the other will be the version compared.

When the two entities were meshes, one could choose the role of each. However, when comparing a cloud and a mesh, the second one will be considered as a reference. To calculate the true distance, the algorithm used the vertices of the comparative entity and calculate the distance from the faces of the reference entity. If the orthogonal projection of the point was inside the face, the orthogonal distance will be calculated, otherwise if the projection falls outside, the distance from the nearest edge will be taken.

The relative processing time per approach is considered on a scale of 1 to 10 where 1 is fast and 10 is very long. The operator's level of intervention is considered on a scale of * to *** where * represents little manipulation and *** a lot of manipulations.

Table 2: Absolute comparison of meshes versus unsampled source point cloud

An absolute comparison between meshes and a reference point cloud can be like an analysis of true deviations. Here, the standard deviation denomination was used and therefore performed a residue analysis, because despite all the care taken to clean the point cloud, it still contained noise, especially since part of the ruins has been overgrown with vegetation.

The analysis of this result table (Table 2) permitted to draw the following conclusions:

-The 3D mesh approach provided more accurate results than the DSM method. This conclusion resulted from the fact that the DSM mesh was carried out in relation to a plane. There was therefore interpolation of the data, while the 3D mesh approach remained more faithful to the starting point cloud.

-The 3D mesh method had less systematism than DSM modeling compared to the starting point cloud. Indeed, we noticed in the first case that the values of standard deviations and RMS were very close. Slight deviations may be due to low systematism and therefore a slight correlation. On the other hand, the two values for the DSM approach were more distant, which reflected the presence of a systematism and therefore a correlation.

-For the same object, sampling slightly modified the file size in the case of a 3D mesh. This was directly related to the number of faces in the mesh. However, for the DSM approach, the file size was multiplied by five each time the spatial resolution was halved.

-The relative processing time increased linearly in relation to the chosen sampling value. However, in the case of the DSM approach, it was multiplied by four each time the resolution was halved.

-Finally, the DSM approach required less operator intervention than the 3D mesh approach. Indeed, DSM meshes were cleaner than 3D meshes, because they were meshed relative to a reference plane (the rotated virtual ground). 3D meshes usually had more artifacts and especially when the density increased because of noise and strong variations in relief. This comparison allowed us to confirm previous conclusions that indicated that the best approach was a hybrid mesh using the 3D mesh approach and derived from DSM. Indeed, the 3D mesh, with good sampling, supplemented by the DSM approach made it possible to have a compromise in terms of detail, weight, processing time and level of operator intervention. In addition, in cases where the first approach gave unsatisfactory results or if we wanted to have much more detail locally, we could use the DSM approach.

In conclusion, this comparison showed that it was not relevant to compare meshes from two different approaches, because the construction methods, one in 3D space while the other in relation to the plan, made that there will necessarily deviations. This comparison also showed sampling limitations. Indeed, in the 3D approach, keep the starting resolution or the modifying slightly was the same statistically as doing a slightly larger sampling. For example, the mean values and standard deviations of the sampled meshes at 10 and 15 cm were almost identical. It would therefore be necessary to proceed in this way to limit the weight of the mesh while maintaining a good level of detail. This reasoning also applied to the DSM approach, where was no need to keep the initial sampling, as the weight and level of detail would be too great. In addition, to achieve a realistic rendering, it was necessary to obtain as few triangles as possible so as not to not overpower rendering engines.

After making a comparison of the point clouds with each other under the same conditions, following the same approach and the same mesh parameters, it was interesting to compare the 3D mesh and SUBD approach. As a basis for this comparison a part of the southern towers divided into two pieces and then brought together by sewing process was used. The objective, in addition to comparing the approaches with each other, was to define the relevance of such a division of faces in the modelling process, but also to see if the sewing process was relevant.

The results of both modelling approaches were quite close, and even equivalent. Of course, these results depended on the algorithm of calculation of deviations used, but CloudCompare was considered reliable. SUBD mesh being obtained by individual mesh of two faces and then by sewing, it could be concluded that such a process was relevant because deviations from the overall 3D mesh were acceptable. Moreover, by this approach, it was possible to model a complex surface more easily. The file size of the SUBD mesh was much larger than that of the 3D mesh, which was due to the subdivision of the surfaces in the case of the SUBD to smooth it. With a view to realistic rendering, the SUBD mesh was preferable, which will facilitate the application of texture. On the other hand, it will be necessary to find a medium value in the intensity of subdivision to limit the number of faces to improve the rendering performance.

MODELLING PROFILE

During the different modeling tests on the ruins of the castle, we were able to obtain for the same face different levels of detail depending on the process and the approach chosen. To synthesize the different processes, but also to give the level of detail that we obtained in output, we propose a modelling profile, which is like a multi-choice protocol, but also to create a new definition of the level of detail called Level of Modelling.

Based on the levels of detail (LoD) that have become widespread in BIM protocols, but also in GIS, the idea was of creating modelling levels (LoM) to generalize the types of models that were found at the output of the different processes.

After an initial analysis, five different levels could be highlighted. They are detailed in Table 3. These levels range from 1 to 5 where the larger the number, the higher the level of detail. Cost in time: Allows to characterize the time taken by the entire modeling process, including calculations. This cost is evaluated on a scale of 1 to 5 where 1 represents a fast process and 5 a very long process.

Operator cost: Allows to characterize the need for manipulation carried out by the operator. This cost is estimated on a scale of 1 to 5 where 1 represents a semi-automatic process, or even quasi-automatic, and 5 represents a quasimanual process with a lot of manipulation.

Weight: Represents the impact of the model on the computer's memory usage. The weight can range from light to very large. NURBS / SUBD conversion ease: Relative evaluation by experience of the ease of converting a mesh to NURBS or SUBD under Rhino3D software.

Usage: Shows the main uses for each level of modeling.

RENDERING

First, it was important to clarify the difference between a material and a texture, as sometimes there was confusion between the two terms. Texture refers to the envelope of an object, that is, its appearance. Most of the time, images were placed on the object, although there were also more general motifs such as checkerboards, dots, gradients, granite, marble, etc. Material refers to the physical properties of the object that allows it to be seen correctly according to its environment. Among these properties, it is usual to find diffusion, reflection, roughness, transparency, metallic effect, etc. For example, an object without physical properties will not be able to be transparent or reflect light in realistic renderings made using a rendering engine.

In the case of complex surfaces, the problems were not really related to the materials, and therefore to the physical properties, but more on the application of photo-realistic texture. Indeed, thanks to field operations and various studies, it was possible to know or approximate the materials that make up a building. Therefore, it was easy to change the parameters of realistic materials in software to those of the original natural materials. However, the application of texture from images taken in the field, which may even have been used to create the point cloud by photogrammetric calculations, was not possible because of the particular context. This impossibility was because it was difficult to perfectly recreate the object as it existed or as it was represented by the point cloud, because regardless of the mesh operation chosen, it will include a phase of simplification of the model and at least a reduction in the number of triangles. It will then be impossible to perfectly recreate a texture using the images taken in the field, because there will no longer be a correlation between the pixels of the images and the nodes of the resulting mesh [START_REF] Kersten | Automatic texture mapping of architectural and archaeological 3D models[END_REF]. One solution to get around this problem would be to create and apply artificial materials and textures via the tools made available by the software. Indeed, in recent years, we have seen the development and generalization of realistic physical rendering techniques that include all 3D rendering techniques to imitate physical models describing the behavior of light in the real world. These techniques are based on the use of realistic material whose parameters can be modified. In addition, it is also possible to use an image as a reference for each parameter.

In this case, the images are usually shaded in gray, so that the software can vary the intensity of the parameter according to the value of the pixel. Figure 5 shows a realistic rendering of one of the faces of our castle on which we applied a material and an artificial texture of medieval stone type.

Renderings were made in the various useful places for a more detailed analysis of the ruins or for measurement needs before restoration (Figure 6). [START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF], which uses all the hypotheses collected.

CONCLUSION AND PERSPECTIVES

This project made it possible to conduct a study on complex surfaces and 3D modeling approaches. Subsequently, we were able to process the entire cloud, cleaning it, denoising it and then segmenting it, all to allow simplified modelling of the different elements. At present, most of these elements have been segmented either by 3D meshes or by DSM meshes. We were also able to perform an analysis of the influence of point cloud quality on the quality of the resulting mesh. Finally, all the research as well as the practice, allowed resuming the various modeling profiles, as well as to create a level of modeling called LoM. It is important, now, to finish the modelling started of the ruins, which consists in a final improvement of the meshes as well as seams. In addition, the reconstruction of the castle under Rhino3D was carried out in broad outline, so it remains to refine the model and specially to match it with its environment such as the ground.

The previous two approaches can be applied to the entire project. A segmented model of the ruins of the castle has been generated. However, it remains to realize a complete and reworked model, to take over the existing meshes, to manage the topology for a realistic rendering, but also to configure the scenes (environment, light, etc.). Currently, all the meshes have been made with the 3DReshaper software and can be easily exported in OBJ format. It is now necessary to choose a modelling and final rendering software (Rhino3D, Blender, Maya, etc.) to incorporate all the meshes and prepare the rendering. To achieve a realistic rendering, additional research should be carried out on the type of materials present on the site, but also on the possibility of creating custom textures from images taken in the field. Once the final 3D model is completed, it will be possible to easily create a virtual tour of the site, which can be reused for cultural purposes.

Figure 2 :

 2 Figure 2: Overview of the point cloud of the castle and its environment.

  Figure 4a: 2 steps mesh algorithm (3Dreshaper)

Figure 5 :

 5 Figure 5: Rendering with realistic texture

Figure 6 :

 6 Figure 6: Different renderings of characteristic places.

Figure 7

 7 Figure7shows first tries for restitution of the initial castle.

Figure 7a :

 7a Figure 7a: Isometric reconstruction carried out by[START_REF] Biller | Die Burgen des Elsaß. Bd.3 : Der frühe gotische Burgenbau im Elsaß[END_REF], which uses all the hypotheses collected.

Figure 7b :

 7b Figure 7b: Digital reconstruction.

Table 3 :

 3 Level of Modelling (LoM)

Table 3

 3

	shows the different characteristics of the proposed
	modelling levels.
	The modeling levels are characterized by the following
	specifications:
	Level of detail: It represents the level of visual detail that can
	range from low (simplified reality) to very high (very close to
	reality).
	Error rate: The specification trying to characterize the
	percentage of error that can be found if we compare the model
	with reality (point cloud).

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-2/W1-2022 9th Intl. Workshop 3D-ARCH "3D Virtual Reconstruction and Visualization of Complex Architectures", 2-4 March 2022, Mantua, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLVI-2-W1-2022-183-2022 | © Author(s) 2022. CC BY 4.0 License.


Thais Bernardi

Yoann Fleytoux

Jean-Baptiste Mouret

Serena Ivaldi

Learning height for top-down grasps with the DIGIT sensor

We address the problem of grasping unknown objects identified from top-down images with a parallel gripper. When no object 3D model is available, the state-of-the-art grasp generators identify the best candidate locations for planar grasps using the RGBD image. However, while they generate the Cartesian location and orientation of the gripper, the height of the grasp center is often determined by heuristics based on the highest point in the depth map, which leads to unsuccessful grasps when the objects are not thick, or have transparencies or curved shapes. In this paper, we propose to learn a regressor that predicts the best grasp height based from the image. We train this regressor with a dataset that is automatically acquired thanks to the DIGIT optical tactile sensors, which can evaluate grasp success and stability. Using our predictor, the grasping success is improved by 6% for all objects, by 16% on average on difficult objects, and by 40% for objects that are notably very difficult to grasp (e.g., transparent, curved, thin).

I. INTRODUCTION

We consider the case of a robotic manipulator equipped with a parallel gripper and a RGBD camera mounted at the end-effector, which needs to grasp unknown objects (i.e., the object 3D model is unknown) using a 4D grasp, also known as top-grasp. A 4D grasp is generally defined as the (x, y, z, θ), where (x, y, z) are the Cartesian coordinates of the grasp center and θ is the yaw orientation of the gripper on the planar surface. In the last years, several grasp generators based on images have been proposed, and they proved successful in finding 4D grasps. For example, Dex-Net [START_REF] Mahler | Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards[END_REF] generates grasps candidates for an object and ranks them according to a grasp quality functions. GG-CNN [START_REF] Morrison | Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach[END_REF] and GR-ConvNet [START_REF] Kumra | Antipodal robotic grasping using generative residual convolutional neural network[END_REF] are other data-driven methods that generate pixel-wise grasp affordance maps.

However, while the x, y component of the grasp center are set, the grasp height z is often determined by heuristics based on the highest height determined by the point cloud. Widely used grasp representations such as oriented rectangles [START_REF] Jiang | Efficient grasping from RGBD images: Learning using a new rectangle representation[END_REF], [START_REF] Depierre | Jacquard: A large scale dataset for robotic grasp detection[END_REF] or pixel-level grasp maps [START_REF] Asif | Graspnet: An efficient convolutional neural network for real-time grasp detection for low-powered devices[END_REF], [START_REF] Gkanatsios | Orientation attentive robot grasp synthesis[END_REF], [START_REF] Wang | Highperformance pixel-level grasp detection based on adaptive grasping and grasp-aware network[END_REF], [START_REF] Wang | Efficient fully convolution neural network for generating pixel wise robotic grasps with high resolution images[END_REF] take into account the center of the grasp, the distance between two jaws, the size of the gripper and its orientation, but do not encode the grasp height z. Heuristics for the height work most of the time, especially when objects are thick or full, but they are often inadequate when objects have thin parts, transparent parts, holes or curved shapes, because of the intrinsic error in the height estimation from the point cloud Fig. 1. Typical problem of planar grasps: the grasp center height is often set with an heuristic based on the highest surface. If the object's shape is unknown, this can lead to unsuccessful grasps. On the left, a depth grayscale image of an object generated from the point cloud captured by the RGBD camera mounted on the robot's end effector. On the center and on the right, two top-down grasps generated by Dex-Net [START_REF] Mahler | Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards[END_REF] and GPD [START_REF] Pas | Grasp pose detection in point clouds[END_REF], two classical grasp generators. The Dex-Net grasp in the center is too low, the object is not grasped by the gripper jaws, while the GPD grasp on the right is too high to be successful. Without prior knowledge of the object and how thick it is, a wrong height prediction can lead to failure.

or simply the absence of knowledge on what is behind the object's surface, which requires the object 3D model. Figure 1 shows an example of grasp failures in this sense: two stateof-the art grasp generators, Dex-Net [START_REF] Mahler | Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards[END_REF] and GPD [START_REF] Pas | Grasp pose detection in point clouds[END_REF], fail because of the grasp height (in one case too low, in the other too high, in both cases the gripper does not properly touch the object while closing).

In this paper, we address this problem by proposing to predict the best grasp height from the RGBD image, using a regressor that is previously trained with a dataset of the best grasp height from several grasp candidates. We posit that the "best grasp height" can be automatically learned from a sensor-driven data collection, where the robot attempts different heights candidates (z 0 , z 1 , . . . ) on the same 3D candidate grasp (x, y, θ), using the contact information provided by a tactile sensor. In our work, we used DIGIT, an optical tactile sensor that provides a contact ellipsoid measure, which we found experimentally to be a good predictor of grasp stability and grasp success. Our experiments with several objects show that the predicted grasp height slightly improves the grasp success, by 6% overall on all the objects, but most importantly it enables to successfully grasp objects that were otherwise failing, with improvement up to 40%.

The main contributions in this paper are: i) the characterization of the DIGIT sensor, with several experiments to investigate whether its output can be used for stable grasp prediction (note: we found that its output does not relate to the contact force, which is a useful information); ii) the experimental validation of the optical contact ellipsoid as a predictor of grasp stability and success; iii) the method to identify the best candidate grasp using DIGIT's measures and then train a grasp height regressor based on latent grasp patches. All our findings are the result of several realworld experiments with the Franka robot equipped with the standard gripper, mounting two Digit sensors (acquired from GelSight).

II. RELATED WORK

Tactile sensors have become widely used in robotics to extract relevant information about the manipulated objects [START_REF] Dsouza | The art of tactile sensing: A state of art survey[END_REF], such as contact wrench, contact area and location, texture [START_REF] Begalinova | Approaches for intelligent robot grasping and manipulation via human demonstration[END_REF]. Tactile sensors can have a different physical nature [START_REF] Dsouza | The art of tactile sensing: A state of art survey[END_REF]: for example, they can be capacitive, piezoresistive, piezoelectric, inductive, opto-electric. A wide range of materials can be used in their fabrication [START_REF] Wan | Recent progresses on flexible tactile sensors[END_REF], such as substrate materials, active materials, or flexible electrodes, which makes them useful in a range of applications, from robotics to healthcare and even surgery.

In this context, vision-based sensors such as the DIGIT [START_REF] Lambeta | DIGIT: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation[END_REF] have the initial idea of correlating contact force with the changes in the surface of an elastometer. Deformable elastometers have been developed as a medium of contact for this use, notably GelSight [START_REF] Yuan | Gelsight: High-resolution robot tactile sensors for estimating geometry and force[END_REF] (used in the upper part of the DIGIT sensor), which measures high-resolution geometry that can be used to infer local normal and shear force. The DIGIT is designed to be used naturally as a "fingertip" mounted on existing grippers.

In [START_REF] Lambeta | DIGIT: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation[END_REF], the authors describe the design and manufacturing process of DIGIT, the analysis of its properties, and use it in a task that involves learning to manipulate small objects from visual images. With the sensor, the PyTouch library was developed and released as open-source [START_REF] Lambeta | PyTouch: A machine learning library for touch processing[END_REF]: the library can operate on real-world data to provide touch detection, slip and object pose estimations. A simulator for visionbased tactile sensing, supporting DIGIT, is also available [START_REF] Wang | Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors[END_REF]. Its properties make it suitable for precise manipulation and grasping, even of soft objects [START_REF] She | Cable manipulation with a tactile-reactive gripper[END_REF].

An analogous sensor, GelSlim 3.0 [START_REF] Taylor | Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger[END_REF], was used to reconstruct 3D geometry, estimate the spatial distribution of 3D contact forces, and detect slips.

While optical tactile sensors are appealing for data-driven learning methods (especially deep-learning methods that are efficient in processing images), the visual measure is unusual for traditional grasping methods to assess stable grasps, which usually require force information -see [START_REF] Platt | Grasp learning: Models, methods, and performance[END_REF] for a review on grasping methods and performance. To predict grasp success, in [START_REF] Krug | Analytic grasp success prediction with tactile feedback[END_REF] the authors use reasoning in the wrench space of the task; to evaluate the effectiveness of adding tactile feedback to the analytic grasp success prediction, the study relies on the tactile feedback to alleviate contact placement uncertainties. To the best of our knowledge, the grasping performance improvement brought by DIGIT/GelSight is only empirically proved by [START_REF] Calandra | The feeling of success: Does touch sensing help predict grasp outcomes?[END_REF]. The authors use an end-toend learning approach for predicting grasp outcome, training deep neural-networks for vision-based, touch-based and combined vision and touch-based grasp outcome prediction. The subsequent work, [START_REF] Calandra | More than a feeling: Learning to grasp and regrasp using vision and touch[END_REF] uses the GelSight in a grasping robot, employing a deep, multimodal convolutional network that predicts the outcome of a candidate grasp adjustment, and using the raw visuo-tactile data, iteratively selects the most promising actions. The operating performance of the sensors, however, is not fully known. In particular, it is unknown whether the optical measure of the sensor is correlated with the contact wrench or contact normal force, an information that is extremely important to assess the grasp stability and success using traditional measures based on force. For this reason, in this paper, we conducted an experimental characterization of the behavior of DIGIT, before using it for assessing grasping performance.

III. MATERIALS AND METHOD

A. Robotics setup

Our experimental setup consists of a Franka Emika Panda 7 DOF robot arm, equipped with an Intel RealSense D415 Depth Camera mounted on the standard gripper, hosting two DIGIT tactile sensor (Fig. 1).We use the Expanding Space Tree (ESTk) [START_REF] Hsu | Path planning in expansive configuration spaces[END_REF] motion planner from MoveIt to plan the trajectories for the robot, and libfranka 5.0 library to control the arm.

B. DIGIT sensor characterization

DIGIT [START_REF] Lambeta | DIGIT: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation[END_REF] is a vision-based tactile sensor: it consists of an acrylic window that is, in the outermost side, covered by an elastometer, and in the internal side it is lighted by three colored LEDs (red, green and blue). A CMOS camera, inside, captures the acrylic window as an image, which is the sensor's output. When an object pushes on the surface of the sensor, the acrylic window deforms and the effect of the object is noticeable in the output image. All DIGITS used in this work used the reflective elastometer; they were set to the default configurations (image acquisition set at 60 fps, illumination intensity at maximum), except when stated otherwise. Some examples of sensor's outputs for different objects are shown in Fig. 2 and in the Video attachment 1 . Flat surface objects are the least perceived by DIGIT, and not obvious to recognize by the human eye in the output image (Fig. 2(d)), opposing to small objects or objects with edges (a and b). Curved objects, when grabbed anywhere other then the center, also are less perceptible (c).

To quantify the DIGIT response of any contact with the sensor, we consider two metrics. The first is δP , i.e., the multi-channel (since the sensor has RGB channels) sum of the pixel by pixel difference between the current image output P image and a baseline image P no-contact , obtained when there is no contact: δP = 1 n * m n i=0 m j=0 (p i,jimagep i,jno-contact ), where p is the value of each pixel in the [0, 255] range and images are of size n×m. The second is the area of the contact surface of the object with the sensor. The contact area, often difficult to identify with the naked eye, is computed by the ContactArea function from the PyTouch library [START_REF] Lambeta | PyTouch: A machine learning library for touch processing[END_REF], which finds the ellipse enclosing the contact surface; as shown in Fig. 3, ϵ, the contact area approximated by the ellipsoid can be calculated in pixels.

To characterize the behavior of the sensor, we conducted the following tests: Test 1: drift. The goal is to determine if there is a drift on the baseline measure, i.e., the one when the sensor is not in contact with an object, to plan suitable re-calibration or reset procedures if necessary. The test consists of collecting data at 1Hz rate in two separate sessions of 30 and 90 minutes, without any contact. Test 2: sensitivity to environment lightning. To verify if the environment lighting changes the sensor's output, we collect data at 1Hz rate in a 2-minute session, with three different levels of environmental lighting that are artificially generated by a lamp and randomly set. Test 3: relation between the sensor's output and the contact force. The goal is to determine whether the output of DIGIT relates to the contact force. For simplicity, we only conduct the test for the normal force. To carry out the test, we use the Franka gripper, mounting DIGITs on both "fingers", and an Optoforce sensor (model OMD-20-FG-100N) to measure the contact force. The first part of the test consists in closing the gripper with different force and velocity settings, using the libfranka API. Then, the second consists in closing the gripper to grasp a pin (the one

C. Validation procedure of the contact ellipsoid as grasp success predictor

Following the observations of [START_REF] Calandra | The feeling of success: Does touch sensing help predict grasp outcomes?[END_REF], we hypothesize that the contact ellipsoid is a good predictor for grasp success and stability. To test this hypothesis, we design the following procedure. We select N objects, placed one by one on the robot's workspace in the same location, and perform several grasps (e.g., 7-10) with different grasp heights, applying the same force. For each grasp, we evaluate two metrics: the grasp success and grasp stability. The grasp is successful if the object is lifted from the table and brought to a fix location (0.55cm above the table). The grasping is stable if the object does not slip during or after the GRASPA stability procedure, proposed by [START_REF] Bottarel | GRASPA 1.0: GRASPA is a robot arm grasping performance benchmark[END_REF]: the object is held by the gripper (without further squeezing) while the end-effector moves for 40 seconds along an excitation trajectory consisting of fast roto-translations, in particular rotations around the endeffector axis. Fig. 4 shows some postures during this procedure; an example can be viewed in the Video attachment. For each grasp, we log the DIGIT output, the minimal contact area ϵ measured by PyTouch ContactArea, and the binary information of success or fail for both grasp success and stability. Note that the ellipse computation of PyTouch sometimes fails and return nothing, e.g., there is a contact with an object but it is not detected. Then we consider the following cases: True positive: the algorithm finds an ellipse and the grasp is successful; True negative: the algorithm does not find an ellipse and the grasp was unsuccessful; False positive: the algorithm finds an ellipse and and the grasp was unsuccessful; False negative: the algorithm does not find an ellipse and the grasp was successful. Fig. 5. Finding the best grasp height: 5 heights are tried for each 3D grasp, the height associated with the highest ϵ for the two DIGITs is retained. The best height and its grasp, encoded by the VAE of [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF], is saved into a dataset used to train the height regressor.

D. Best grasp height collection procedure

The experiments in the Section IV will show that grasp contact area ϵ is a good predictor of grasp success, which means we can use it to automatize the finding of a good grasping height. To collect a dataset of the best grasp height for given 3D grasp candidates, we design the following procedure. We collect a series of top down grasps, where the gripper's center position (x, y) and orientation θ are manually set, and the initial depth of the grasp (the z-position of the gripper) z i is computed by finding in the height of the point cloud the closest distance to the camera near the grasp position, as done in [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF]. The object is then grasped sequentially at 5 different heights: 2cm above z i , 1cm above z i , at z i , 1cm below, 2cm below. Light and unstable objects are held in place by the experimenter, heights below the workspace plan are ignored. For each grasp, we compute the contact area ϵ of each DIGIT. The height of the grasp with the highest combined contact areas, z c , is considered the best. Fig. 5 illustrates the procedure. The best height associated to each 3D grasp (x, y, θ), encoded by the latent representation (VAE of grasps represented by image patches) of [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF], is added to the dataset of 4D grasp demonstrations, used to train the height regressor described in the next step. We collected 54 demonstrations from 10 objects from the YCB dataset (see Table . III objects 1-10) using this sensor driven collection.

E. Training the grasp height prediction

For a given RGB-D image (W, H, 4), the dataset of the previous section III-D contains grasp demonstrations, represented in the image coordinates by the gripper's center position (x, y), rotated according to the orientation θ and encoded using the same grasp encoding from [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF]. Each grasp is represented as a rotated patch (w, h, 7) centered on the middle of the grasp, the patch is fed to a Variational Auto-Econder (VAE) to get a latent representation m. The patch representation is a practical representation inherited from [START_REF] Mahler | Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics[END_REF], and the latent encoding proved to be data efficient in [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF]. This representation is then used to train a height regressor that learns to predict the correction c in meters between the initial height z i estimation and the height z c found using the two DIGITs (section III-D). In principle, there is not a preferable method for the design of the regressor, so we compared different methods: SVR, Random Forest, AdaBoost, Gaussian Process, Linear Regression and Neural Network. Training can be done offline. Online, the initial height z i is computed using the depth data from the RGB-D camera: we extract an oriented cropped patch of the depth point cloud with the width of the selected grasp and a fixed height (5 pixels), and we use the closest point to the gripper (that is, the highest point of the object). z c is found by adding the output c of the regressor to z i . Fig. 6 illustrates how the height prediction module is incorporated in the grasping pipeline.

IV. EXPERIMENTS & RESULTS

A. Sensor characterization

We report here the results of the 4 tests to characterize the behavior of DIGIT. Fig. 7 shows the results of Test 1 & 2.

Each color channel has values in the range [0,255]. In the first 2 plots (short and long session) the values for each channel are almost constant and close to 0, which is consistent with absence of contact (although there are some visible image stream errors causing spikes2 ). The third plot shows that the lightning does not influence the baseline measure.

Fig. 8 shows the results of Test 3, where 3 set of forces were applied to the two DIGITs. We use the metric δP (section III-B), i.e., the pixel-by-pixel difference averaged on the 3 channels. The pin object was selected such that the contact area ϵ would not change during the experiment. The results for different repetitions is the same: there is no relation between the DIGIT's output in terms of pixel values and the contact force. This means that pixel-bypixel measures with DIGIT cannot be used to distinguish contact forces, but only contact areas.

Finally, Fig. 9 shows the result of Test 4. We executed periodic grasps with the gripper, applying the same force on the pin, 180 times. Two things should be noted. First, the value of δP for the two digits is different: this could be attributed to an asymmetrical distribution of the contact force, even if we used a symmetrical object to avoid this issue. Second, as time progresses we observe strange consecutive "leaps" in the pixel-based measure, appearing at different times for both sensors. We carried out the same experiment many times, and always observed a similar behavior, though the "leaps" seems to happen randomly. This experiment suggests that the pixel response of DIGIT to same contact forces is not consistent over time, but it is subject to additive constant noise. This test further confirms that DIGIT cannot be used in relation to forces across real-world experiments without further investigation.

In addition, we report other issues that we observed during the experiments with DIGIT, that strongly limits its use for repeated grasping in real-world experiments: the elastomer layer is very fragile and easily wear and breaks, which Fig. 6. The grasping pipeline, assuming that the VAE and depth regressor have been trained before and that a suitable top-down grasp candidate generation algorithm is provided (e.g., GR-ConvNet [START_REF] Kumra | Antipodal robotic grasping using generative residual convolutional neural network[END_REF], Dex-Net [START_REF] Mahler | Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards[END_REF], ...). From a RGB-D image, a grasp generator outputs a grasp candidate. The initial grasp height z i is computed using the depth image. The grasp candidate is represented as rotated patches centered on the the gripper's center position (x, y). It is fed to a VAE to get its latent representation m, which is, in turn, the input of the height regressor (Sec. III-E) trained on the dataset collected with the help of DIGIT (Sec. III-D) to obtain the corrected height zc. For all these considerations, DIGIT does not seem to be suitable for repetitive, long-lasting real-world grasping experiments. It seems to be indicated for limited use for contact detection and contact area estimation.

B. Grasp stability and contact ellipsoid

In this experiment we want to evaluate if the contact area ϵ can be used to predict the grasp success and stability. We expect that higher contact areas ϵ lead to more stable grasps. We selected 10 objects with different levels of grasping difficulty and evaluated several grasps (up to 11, for each object) with different height, following the procedure described in section III-C, for a total of 192 images.

Table I shows a bigger value for the false negatives, meaning that there were more experiments were there was an object touching the DIGIT sensor, but it was not perceived by the processing of the sensor image. In particular, as shown in Table II, the plastic bolt, which has a particular shape and ridged texture, was "perceived" by the sensor (visible change in the visual output, at least to the human eye) but PyTouch never detected any ellipse. The accuracy and precision of the perception of an object were 65.81% and 94.21%, respectively. Out of the 114 true positive images, 14 had ellipses that did not correspond to the contour of the object. Fig. 10 shows the relation between the contact area ϵ and grasp success and stability. The results confirm that the higher contact area is associated to higher probability of grasp stability and success.

C. Training the stable grasp height predictor

Using the procedure described in Section III-E, stable grasps are used to train a grasp height predictor. We conducted a 4 fold cross-validation to compare several regressor models (from Scikit-learn) with different parameters found from a grid-search. the same grasp encoding from [START_REF] Fleytoux | Data-efficient learning of object-centric grasp preferences[END_REF] with a VAE trained on a dataset of 2349 RGB-D scenes (339 objects different from the one used in the experiment), with grasps generated using the GR-ConvNet [START_REF] Kumra | Antipodal robotic grasping using generative residual convolutional neural network[END_REF], Dex-Net [START_REF] Mahler | Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards[END_REF] and GPD [START_REF] Pas | Grasp pose detection in point clouds[END_REF] algorithms.The models were trained using a Nvidia GTX 1080, and a Intel(R) Xeon(R) Gold 5118 CPU at 2.30GHz. Fig. 11 compares their performance: the best results were obtained by a simple linear regression algorithm.

D. Testing the stable grasp height predictor

We evaluate the performance of the learned stable grasp height predictor by grasping the 10 YCB objects (objects Fig. 11. Performances of different regressor model using 4 fold cross validation. For each fold the models was trained with the same train and validation data (using 75% of the dataset), a grid-search was used to find suitable parameters, the results above relates the performances of each splits on their remaining test grasp demonstrations. To evaluate the impact of the height regressor model, we compare the grasp and stability success with and without the height correction (i.e., with and without the regressor). All the objects were grasped at a similar (x, y) location in the workspace. Using the regressor led to an improvement of 5-6% over 300 grasps (20 by objects) for both seen and unseen objects, as reported in Table III, showing that the regressor was able to generalise to new scenes of previously seen objects and also novel objects (objects 11-15) . While the improvement is overall very modest, it must be noted that the height correction is crucial to enable grasping of challenging objects (objects 2-6-9-14-15), that otherwise have little to zero chance of being successfully grasped, with an overall improvement of 16% for grasping and 18% in stability.

V. CONCLUSIONS

Despite its limits, DIGIT can be used for automated data collection of stable grasps, as we experimentally found that the higher contact ellipsoid is associated to higher grasp success and stability. We used DIGIT to automatically determine the best grasp height z for top-down grasp candidates (x, y, θ) which otherwise would use heuristics.

The overall improvement in terms of grasp success and stability is relatively modest across all the objects, meaning that the heuristic is often enough for most of everyday objects. However, our height prediction becomes significant for challenging objects (with transparencies, curved shapes, etc.) that would be otherwise very difficult or impossible to grasp with the simple heuristics.

  Fig. 2. DIGIT's output to different types of contact and objects.

Fig. 4 .

 4 Fig.[START_REF] Jiang | Efficient grasping from RGBD images: Learning using a new rectangle representation[END_REF]. Some frames of the GRASPA stability procedure. Once the object is grasped, the robot lifts it and executes an excitation trajectory.

Fig. 7 .

 7 Fig. 7. Test 1 & 2: DIGIT's output as δP to repeatability test over time for different sessions, and for changing environment lighting conditions.

Fig. 8 .

 8 Fig. 8. Test 3: DIGIT's output as δP in relation to different contact forces.

Fig. 9 .

 9 Fig. 9. Test 4: DIGIT's output as δP in relation to repeated contacts of the same force, generated by the gripper.

Fig. 10 .

 10 Fig. 10. Boxplot of the contact ellipse area of the two DIGITs in the fail/success cases, for grasp and stability tests.

TABLE I CONFUSION

 I MATRIX FOR Contact surface METHOD, FILTERING ELLIPSES FOUND FOR THE CONTACT SURFACE AREA BASED IN THE GRASP INFORMATION LOGGED FOR EACH OBJECT AND EXPERIENCE.

			Prediction label (ellipse presence)
			Positive	Negative
	True label	Positive	114	60
	(object grasped)	Negative	7	15

TABLE II RELATION

 II BETWEEN THE SUCCESS RATIO OF GRASP AND STABILITY IN THE PHYSICAL TESTS, AND THE SUCCESS RATIO FOR THE ELLIPSE CALCULATION BY THE ContactArea METHOD, CONSIDERING THE

		UNFILTERED DATA.
	object	Grasp success	Stability success	Ellipse calculated Digit 0 Digit 1
	bottle	9/11	5/11	9/11	9/11
	dust shovel	7/9	6/9	7/9	7/9
	screwdriver	7/8	7/8	7/8	6/8
	white tube	6/8	5/8	7/8	7/8
	green shovel (handle)	10/10	7/10	9/10	4/10
	strawberry	7/7	7/7	3/7	3/7
	green shovel (inside)	10/10	9/10	7/10	9/10
	ear protector	6/9	6/9	5/9	1/9
	golf ball	9/9	7/9	7/9	7/9
	plastic bolt	8/8	8/8	0/8	0/8

TABLE III RESULTS

 III ROBOT GRASPING EXPERIMENT WITH AND WITHOUT USING THE DEPTH REGRESSOR.Table III) seen in training, but presented in new positions, and 5 new and difficult to grasp objects that are not part of the training dataset.

		with regressor	without
	object	Grasp success	Stability success	Grasp success	Stability success
	1 YCB screwdriver	10/10	10/10	10/10	10/10
	2 YCB power drill	7/10	3/10	6/10	2/10
	3 YCB scissors	8/10	7/10	9/10	8/10
	4 YCB orange plastic bolt	10/10	10/10	10/10	10/10
	5 YCB adjustable wrench	6/10	6/10	6/10	5/10
	6 YCB hammer	9/10	3/10	8/10	2/10
	7 YCB glass cleaner	10/10	10/10	10/10	10/10
	8 YCB big spring clamps	10/10	10/10	10/10	10/10
	9 YCB bleach cleanser	10/10	7/10	10/10	3/10
	10 YCB rope	10/10	10/10	10/10	10/10
	11 ear protector	8/10	7/10	8/10	8/10
	12 bottle	10/10	10/10	10/10	10/10
	13 white tube	10/10	10/10	10/10	9/10
	14 green shovel (inside)	4/10	4/10	0/10	0/10
	15 dust shovel	7/10	4/10	5/10	5/10
	1-10 in

Video also available at https://youtu.be/aZ1Hjaziv6Y

During our extensive tests with the DIGIT, we observed frequent image stream errors: the output image is transmitted, but it is shifted and so unusable.

*These authors contributed equally to this work. This work was partially funded by the European projects HEAP and EUROBIN, and the Creativ'Lab facility in Loria.