Youcef Mezouar

Juan Antonio Corrales

I would like to acknowledge my supervisors

Résumé

La manipulation robotique d'objets déformables est difficile mais importante pour étendre les applications robotiques. La préhension et l'asservissement tactile sont deux techniques cruciales pour permettre aux robots d'interagir correctement avec les objets et leur environnement.

Pour la préhension, une méthodologie d'évaluation de la préhension dynamique est proposée pour étendre la théorie et les méthodes de préhension statique utilisées pour les objets rigides vers les objets déformables. Une nouvelle matrice de préhension dynamique est formulée pour décrire l'état de préhension variable sur des objets déformables. Trois métriques de qualité de préhension dynamique sont dérivées de cette matrice de préhension dynamique. En s'appuyant sur ces métriques, une nouvelle méthode de synthèse de préhension pour optimiser la stabilité de la préhension est développée. Des simulations physiques et des expériences robotiques ont vérifié son efficacité. Ce travail devrait fournir une nouvelle perspective dynamique avec des contributions théoriques et pratiques pour la saisie d'objets déformables.

Les empreintes tactiles 3D offrent une résolution spatiale et une sensibilité plus élevées pour l'asservissement tactile. Le système d'éclairage interne du capteur tactile adopté DIGIT offre la possibilité d'une reconstruction 3D basée sur la théorie de la stéréophotométrie. Cependant, son éclairage interne ne peut pas fournir des conditions d'éclairage uniformes sur sa surface de détection, ce qui complique la reconstruction 3D. Comment réaliser la reconstruction 3D optimale sous incertitudes est le problème étudié. La méthode de la table de correspondance mappant les couleurs aux hauteurs s'est avérée instable à différents endroits de la surface de détection. Un réseau de neurones amélioré basé sur la variation de couleur des images tactiles est proposé pour surmonter les incertitudes causées par le défaut du matériel et obtenir de meilleures performances.

L'empreinte tactile 3D peut faciliter les manipulations robotiques. Pour la détection et l'évitement du glissement, le maillage de hauteur dérivé de la forme de contact 3D en temps réel est proposé pour générer un signal de glissement comme indicateur de glissement. Le maillage de hauteur est composé des valeurs de hauteur générées aux pixels sélectionnés dans viii l'image tactile. Un algorithme efficace d'évitement de glissement est développé. Par rapport aux travaux précédents, le maillage de hauteur ne nécessite pas l'intégration des marqueurs physiques à l'intérieur du capteur. Pour le suivi de pose en main de l'objet saisi, un pipeline basé sur la hauteur est proposé pour gérer différents objets dont la forme générale peut être déduite de la forme de contact, comme une clé hexagonale, un câble, etc. L'avantage de ce pipeline est démontré dans des expériences par rapport à la méthode basée sur la couleur.

L'assemblage de la poupée nécessite un haut niveau de coordination sensorielle et motrice. L'étape essentielle est d'ajuster l'interaction physique entre les objets déformables en fonction du retour de toucher. L'homme du métier est capable de sentir la tension extrinsèque entre les deux parties de poupée qui se transmet à sa main. En utilisant des capteurs tactiles, les robots sont également capables d'avoir une sensation de tension sur le contact extrinsèque pendant le fonctionnement. Tirant parti de ce nouveau sens, un pipeline robotique pour effectuer l'assemblage de poupées est proposé. Ses performances robustes sous incertitudes sont validées. L'avantage de l'utilisation de la tension est prouvé dans la comparaison avec le contrôle de position.

En conclusion, cette recherche apporte des contributions substantielles dans le domaine de la préhension, de la détection tactile et de l'asservissement tactile dans le cadre de la manipulation d'objets déformables.

Motivation

The work made in this thesis is driven and inspired by the project SoftManBot. This project aims to develop a complete and advanced robotic system for handling deformable objects in real industrial production. The diagram shown in Figure 1.1 depicts the structure of the developed robotic system. The robot is equipped with the customized industrial gripper. On the gripper, tactile sensors are able to be integrated. The robot communicates with one connected computer on which run the algorithms of perception, grasping and control. The computer also provides the interface for operators to interact with the developed software. The laser scanner and 3D cameras are used to ensure the security of the robotic system. All these hardware should be deployed properly in the workcell to perform tasks cooperating with other machines.

In this project, four specific use-cases drive the development of the robotic system, including the manipulation on toys, tyres, footwear and textiles. Some manual operations are shown in Figure 1.2. The deformation of the manipulated objects brings more uncertainties to the procedure of operation which makes an automation pipeline difficult to be predefined. Instead, skilled workers are able to adjust the operation flexibly by their delicate sensory and motor coordination to overcome the impact of uncertainties. In sole demoulding, workers can adjust their pulling force and direction to find the proper trajectory for demoulding. In tyre assembly, workers can align the two edges of the tyre belt by stretching and observing. In cloth sewing, workers can flatten the wrinkles on the cloth and keep the correct trajectory of sewing. In doll assembly, the connection between the toy parts can be accomplished through intense physical interaction between toys and hands. A stable grasp on toys is fundamental to successfully conduct the insertion. The workers can also flexibly regulate the interaction between toys according to their sensitive tactile feeling. The proper tactile feeling is hard to be described and relies on the experience of workers. As the complexity of these operations, few robotizing attempts are made in the relevant applications.

With an emphasis on the use-case of doll manipulation, the modules "Tactile sensors" and "Grasping" marked in Figure 1.1 constitute the main part of this thesis. Some efforts are also made for the modules "Perception" and "Control".

Grasping is one basic operation in robotic manipulation. For deformable objects, a lack of theory and practice to find the optimal grasping configuration leads to difficulties in manipulation. The work on deformable object grasping in this thesis is committed to extending the grasp theory to explain how one grasp can provide better stability for deformable objects and developing one general grasping synthesis method to find the most stable grasp for robotic manipulation. The requirements on the grasp quality depends on the specific tasks. For a common pick-and-place task, we do not need to find the most stable grasp configuration since many grasps can resist the weight of the gripped object and finish the task. For a task requiring intense physical interaction, the optimized grasp stability can promote the success of manipulation. The weaker grasps are easy to be broken by the large external force. Although there may be some practical limitations that can influence the choice of one grasp configuration, the theoretical analysis of the grasp stability can provide guidance in the grasp synthesis for deformable objects. Tactile sensing is human's fundamental perception to interact with physical world. The same capability is expected for robots to improve their intelligence and dexterity in complicated manipulations, like doll assembly. Rich touching feedback is used to indirectly perceive the contact state between toy parts and make decision for the next step of operation. To achieve this, the image-based tactile sensor is adopted due to its high resolution on perceiving contact. 3D reconstruction methods are developed to recover the richest tactile information in real time from the streamed tactile images. This type of tactile sensors is still in progressing compared to the classical tactile sensors based on capacitive or resistive technologies. It is hard to find a product with the standardized output, which brings difficulty to 3D reconstruction. After several strategies are developed, tested and compared, one improved neural network is determined as the optimal solution to achieve the real-time 3D tactile imprint.

3D tactile imprint enables robots to have a very sensitive touching feedback in manipulations. Leveraging tactile sensing, two general robotic skills are developed. Slip detection and avoidance is one fundamental skill realized by tactile sensing. With the help of 3D touching, the robot is able to detect the subtle slip of the gripped objects. The developed method works well with all toy parts and even textiles. This skill can notify or prevent the slippage or drop of the gripped toy parts in manipulations. The second skill is the in-hand pose tracking of the gripped objects. The method based on the 3D touching provide better tracking stability and accuracy than the color-based method. The acquired relative pose of the gripped object could be used for the control of the orientation of the end-effector. The good tracking performance on toy limbs is validated in experiments.

To address the difficulty of doll assembly, one robotic pipeline is proposed in three steps. In the first step, we use the RGBD camera to detect the relative position between the toy parts and align them accordingly. In the second step, we exploit 3D touching to indirectly perceive the contact state between the toy parts. When a proper contact state is perceived, the third step is launched to insert one toy part to the other by applying a twist. It is hard to define the proper contact state between toy parts. As a novel concept, the sense of tightness is proposed to describe the complicated interaction status between deformable objects. The performance of this pipeline is robust under uncertainties compared to the position control.

Together, the developed techniques related to grasping, tactile sensor and tactile servoing can expand the capability of robots and promote the integration of the developed robotic system in the industrial field. In this chapter, the states of the art about the involved technologies are given in Section 1.2 and Section 1.3. The solved problems and the major contributions are presented in the end.

State of the art of grasping

Robotic grasping synthesis is about determining the optimal grasp configuration for a robotic task. Different tasks have different requirements on the level of the grasp quality. Simple tasks can be finished by a grasp without optimization. As the task complexity increases, one optimized grasp can facilitate the completion of tasks. For instance, the stable grasp can better transfer the twist or wrench applied by the robotic arm to the contact area between toy parts. Then the assembly can be accomplished easier.

How to define the optimal grasp is critical and thus the concept of grasp quality is employed. According to the definition of grasp quality, grasp evaluation process can be performed on different grasp candidates. Then we choose the best one in the ranking list and the robot can apply this grasp with objects. Physical properties of object are crucial for grasping. For rigid objects, the contact status of each fingertip and the entire geometrical configuration between all fingertips are nearly invariable, which eases the difficulty of modelling. For deformable objects, however, the dynamic interaction between fingers and object makes it more challenging. Nevertheless, the fundamental mechanism of grasping is universal for rigid objects and deformable objects, which is depicted in Figure 1.3. The mechanical interface between object and fingertip is the contact. We need to know how to model the effect of this contact. Then we should know how physical variables like force/velocity are transmitted through different frames in the considered grasping system. Guided by this insight, we summarize previous work in the subsection "Contact modeling" and the subsection "Transmission modeling" to clearly position our research in the domain of grasping.

Contact modeling

Contact is the interface between gripper and object, which has significant impact on grasping quality. Different contact models [START_REF] Bicchi | Robotic grasping and contact: A review[END_REF][START_REF] Grady | Contactopt: Optimizing contact to improve grasps[END_REF] are proposed to describe the function of contact in grasping. The simplest contact model is named the frictionless point contact model [START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF], Introduction which is formulated as:

F c i =            0 0 1 0 0 0            f c i f c i ∈ LS 1D c i (1.1)
The exerted force from one fingertip is f c i , which should be limited in the limitation space LS 1D c i . This limitation space contains all positive force values in the normal direction at the contact point. The normal direction points toward the side of the object and f c i only has one element. F c i refers to the corresponding contact force applied on the grasped object. This contact model is used when the friction between the fingertip and the object is low or unknown. It is presumed that even without the help of friction, the object can be manipulated well. This makes grasp synthesis over-conservative. Many feasible grasp candidates could be excluded and only the most secure one is selected.

In most cases, the influence of friction should not be omitted. Thus we have the hard contact model [START_REF] Prattichizzo | Grasping. Springer handbook of robotics[END_REF] as:

F c i =            1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0            f c i f c i ∈ LS 3D c i (1.2)
Compared with Equation (1.1), the exerted force f c i from any fingertip is a 3D force vector.

The limitation space LS 3D c i for this vector is a friction cone FC c i . The friction cone [START_REF] Han | Grasp analysis as linear matrix inequality problems[END_REF][START_REF] Tsuji | Easy and fast evaluation of grasp stability by using ellipsoidal approximation of friction cone[END_REF] can be visualized in Figure 1.4. If the direction of the exerted force from fingertip is out of the range defined by the orange cone, the maximum friction can not compensate the actual tangential force and thus slippery occurs. The shape of friction cone is determined by the friction coefficient between the fingertip and the object. A larger coefficient can result in the larger opening of the friction cone. A larger friction cone implies that a larger set of force could be transmitted toward the object through the interface of contact, which is one advantage for grasping. Another evolution is that the matrix used in Equation (1.2) enables the transmission of variables along all three axes at contact. Soft contact model [START_REF] Prattichizzo | Grasping. Springer handbook of robotics[END_REF] further enables the variable transmission around the normal axis of contact, which is formulated as:

F c i =            1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1            f c i f c i ∈ LS 4D c i (1.3)
Added a torque element, f c i becomes a 4D vector. Hence the corresponding limitation space becomes 4D as well. The coefficient of torsional friction [START_REF] Xydas | Modeling of contact mechanics and friction limit surfaces for soft fingers in robotics, with experimental results[END_REF][START_REF] Li | A review of modeling of soft-contact fingers and stiffness control for dextrous manipulation in robotics[END_REF] is further incorporated for the definition of the limitation space.

If the contact area between the fingertip and the object is small, the soft contact model is enough to describe this mechanical interface. If the contact area is relatively large, the patch contact model should be used in the form as:

F c i =            1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1            f c i f c i ∈ LS 6D c i (1.4)
The exerted force f c i is upgraded to a 6D wrench. All the elements of this wrench can be transmitted. All possible wrenches compose the 6D limitation space LS 6D c i . The previously mentioned contact models are usually used for tackling rigid objects. For deformable objects, their deformation can bring more difficulties in contact modeling. The adopted model is more task-oriented and less general.

When the fingertip is relatively small and it can change the local contact shape of the deformable object, the contact region is variable with the displacement of the fingertip, which is depicted in Figure 1.5. In this case, the contact model maps the displacement of the fingertip and the deformation of the object's surface to the contact force. This contact model should incorporate the specific parameters as the fingertip's dimension, the object's stiffness, etc. When the fingertip moves toward the surface of the object to a certain depth, the contact region can be computed and divided as a set of small facets. After computing the contact force distributed on each facet, the general contact force can be summed up [START_REF] Zaidi | Modelbased strategy for grasping 3d deformable objects using a multi-fingered robotic hand[END_REF]. Fig. 1.5 Variable contact region [START_REF] Zaidi | Modelbased strategy for grasping 3d deformable objects using a multi-fingered robotic hand[END_REF].

Similarly, when the fingertip has a non-planar patch contact with the object as shown in Figure 1.6, the contact patch can be divided into a triangular mesh. The pressure distribution applied on the object can be estimated based on the deformation of the soft pad on the fingertip [START_REF] Danielczuk | Reach: Reducing false negatives in robot grasp planning with a robust efficient area contact hypothesis model[END_REF]. After the initial contact between the fingertip and the object, each time the fingertip moves toward the object one more step, the pressure distribution on the contact region can be updated. Meanwhile, in each triangular facet, the applied normal force and fictional force can be calculated and summed up to compose the general 6D contact wrench. The 6D contact wrench comprises the 3D contact force and the 3D contact torque. Sampling a series of steps, the gathered set of the 6D contact wrenches can be fit into a 6D ellipsoid [START_REF] Xu | 6dls: Modeling nonplanar frictional surface contacts for grasping using 6d limit surfaces[END_REF]. Like the friction cone for the point contact model, this 6D ellipsoid is the limitation space used to define what kind of wrench can be transmitted at the contact. Fig. 1.6 Non-planar patch contact [START_REF] Jingyi | Minimal work: A grasp quality metric for deformable hollow objects[END_REF].

In order to obtain the precise contact model for deformable objects, the computation is heavy. The dimensions and material properties of the fingertip and the object should be known in advance. In many cases, the patch contact model in Equation (1.4) or the soft contact model in Equation (1.3) could approximate the actual contact model. The friction model in the contact area is simplified and some friction features are lost. The adopted limitation space could be a subset of the actual limitation space. In this case, we may neglect some feasible grasp candidates and tend to select the more secure grasps, which is not a big concern and acceptable for many cases.

After a discussion on contact models, we know how variables are converted at the interface between the fingertip and the object. But we still do not know how variables are transmitted from the motors at the fingers' joints to the fingertip and then to the center of the grasped object.

Transmission modeling

Transmission is about how variables like forces or velocities are transmitted to the chosen coordinate frame. Regarding stability, if the grasp wrench defined in the object frame is larger than the disturbing wrench passed to the same object frame, the current grasp is stable. The object frame is usually defined at the center of mass of the object. As depicted in Figure 1.7, the contact forces applied at the three contact locations between the fingertips and the mug can be transferred into the frame of mug to form the resultant grasp wrench. When one grasp aims to stabilize objects, how to find the placements of fingertips on the surface of the object is the main concern [START_REF] Nguyen | Constructing force-closure grasps[END_REF][START_REF] Nguyen | Constructing stable grasps[END_REF]. After fingers are placed at the desired locations, the "Interface" and "Transmission 2" in Figure 1.3 are primarily concerned for grasp stability.

In contrary, regarding dexterity [START_REF] Raymond | On dexterity and dexterous manipulation[END_REF][START_REF] Nikhil Chavan Dafle | Extrinsic dexterity: In-hand manipulation with external forces[END_REF], finger joints are actively controlled to adjust the velocity of the object to follow any desired direction. Involving finger joints, "Transmission 1", "Interface" and "Transmission 2" in Figure 1.3 should be analyzed. The contact placement problem is upgraded to the hand configuration problem. For rigid objects, we use grasp map matrix G [START_REF] Han | Grasp analysis as linear matrix inequality problems[END_REF] to combine the effect of the "Interface" and the "Transmission 2" in Figure 1.3. It can transmit the exerted forces from fingertips into the object frame and form the grasp wrench. Each grasp candidate has its corresponding grasp map G, which embodies the transmission capacity of this grasp. This capacity can be used as quality measures to indicate the grasp quality. The ideal capacity provides the high transmission ratio from the contact forces to the grasp wrench that can resist disturbances from different directions. In order to quantify and rank this transmission capacity, we can extract several indexes from the matrix G. For instance, the minimum singular value [START_REF] Roa | Grasp quality measures: review and performance[END_REF] of the matrix G reflects the minimum transmission capacity of one grasp along one certain direction in the object frame. For a list of grasp candidates, each candidate has its own minimum singular value s min . The candidate with the largest s min should be selected. The volume of grasp wrench space (GWS) [START_REF] Ferrari | Planning optimal grasps[END_REF][START_REF] Andrew | Examples of 3d grasp quality computations[END_REF][START_REF] Qiu | A new approach for grasp quality calculation using continuous boundary formulation of grasp wrench space[END_REF][START_REF] Andrew | Graspit!: A versatile simulator for grasp analysis[END_REF] is one commonly used indicator. When all friction cones related to the contacts of a grasp are transmitted in the object frame, one six-dimensional GWS is formed. If we project this GWS into one three-dimensional space, it can be visualized as shown in Figure 1.7. The gripper has three contacts with the mug. The three friction cones are marked in light red. The projected GWS is depicted in light green in the right. The volume of GWS reflects the general transmission ability of one grasp. One larger volume should be pursued in grasp synthesis.

In some predefined applications, grasp is synthesized to satisfy the requirements of particular tasks. Instead of improving the general transmission capacity of grasp, the specific transmission influencing the execution of tasks is more concerned. The task wrench space (TWS) [START_REF] Ch | Grasp planning: How to choose a suitable task wrench space[END_REF][START_REF] Lin | Task-based grasp quality measures for grasp synthesis[END_REF][START_REF] Lin | Grasp planning to maximize task coverage[END_REF][START_REF] Lin | Task-oriented grasp planning based on disturbance distribution[END_REF] was proposed for this purpose. TWS is defined in the object frame, which is the union of all the disturbing wrenches to be counteracted during the execution of tasks. In the simplest task, only the gravity of the grasped object needs to be overcome. Thus the union only contains one element, the gravitational force, in the object frame. In Figure 1.8, the object frame is marked by the red original spot and two black axes. The red dash arrow refers to the gravitational force, which composes TWS. The dashed blue polygon is the projection of GWS on the plane defined by the two black axes. Comparing the two polygons, the end of the red dash arrow (TWS) is closer to the bottom edge of GWS in the left. It implies that in the direction of gravity, the grasp corresponding to the left polygon can provide less transmission ability than the right one. Thus the grasp generating the right polygon is preferred for the tasks to resist the weight of the gripped object. There are different variants [START_REF] Roa | Grasp quality measures: review and performance[END_REF][START_REF] Sahbani | An overview of 3d object grasp synthesis algorithms[END_REF] of the grasp quality metrics based on the principle of GWS and TWS. They all characterize the transmission ability of a grasp. (a) Tomato picked up [START_REF] Lin | Picking up a soft 3d object by "feeling" the grip[END_REF] (b) Grasped rubber [START_REF] Jia | Grasping deformable planar objects:squeeze, stick/slip analysis, and energy-based optimalities[END_REF] Fig. 1.9 Objects with small local deformation.

For deformable objects, the transmission ability of a grasp is not static anymore and presents a dynamic behavior due to deformation. However, this fundamental problem is always ignored in the existing literature. For instance, the static grasp map G is adopted to compute the required minimal contact forces for manipulating deformable hollow objects [START_REF] Jingyi | Minimal work: A grasp quality metric for deformable hollow objects[END_REF]. When small deformation is introduced during the execution of tasks, the impact of the dynamic transmission is not significant. There are two situations in which deformation can be maintained in small scale. Firstly, the stiffness of the grasped object is large enough and then the transmission is treated as quasi-static. Secondly, the disturbing force is relatively small. To balance small disturbances, the magnitude of contact forces could be kept to a lower level and thus no large deformation is introduced. Like the pick-and-place of one light object, this kind of tasks could be finished without considering the dynamic transmission. In these cases, the local contact status has a larger impact on grasp quality [START_REF] Lin | Picking up a soft 3d object by "feeling" the grip[END_REF][START_REF] Jia | Grasping deformable planar objects:squeeze, stick/slip analysis, and energy-based optimalities[END_REF][START_REF] Zaidi | Modelbased strategy for grasping 3d deformable objects using a multi-fingered robotic hand[END_REF][START_REF] Zaidi | Grasp planning pipeline for robust manipulation of 3d deformable objects with industrial robotic hand + arm systems[END_REF]. If the goal is to find one grasp able to finish a simple task, the requirement on grasp quality is relatively low. However, if the synthesized grasp needs to withstand heavy loads or large disturbing forces, or the manipulated object has low stiffness that tends to deform a lot, the quality of the dynamic transmission is critical for the optimization of the grasp. To our best knowledge, the relevant work is absent to consider the impact of the dynamic transmission on grasp quality with these assumptions.

After a survey and summary of the existing work on grasp modelling and synthesis, we found that the property of dynamic transmission is fundamental to constitute the theory of deformable object grasping. Since no previous work is found in this field, the effort is dedicated to establish the theoretical basis of dynamic transmission and provide guidance to apply dynamic transmission in grasp synthesis.

State of the art of tactile sensing

Besides vision, tactile sensing is another important perception for robots, especially in manipulation. The feedback of touching can give robots the information about the physical interaction between gripper, object and environment. The sensed contact area can determine the type of the contact model to be employed. The sensed compression can help to estimate the contact force/torque or the properties, geometry and in-hand pose of the manipulated object. Then robots can react accordingly. Various tactile sensors [START_REF] Silva Girão | Tactile sensors for robotic applications[END_REF][START_REF] Chi | Recent progress in technologies for tactile sensors[END_REF] are invented for tactile sensing. The evolution of tactile sensors' design manifests the pursuit toward higher resolution of touching and more compact mechanical structures. Compared to vision, the hardware of tactile sensors is still being developed and improved constantly. At the same time, new usages and applications of tactile sensing are being expanded.

Pursuit of resolution

The most basic tactile sensor degrades to a force sensor. The sensor made by SingleTact is one example as shown in Figure 1.10. It can measure the precise normal force applied on its round and flat unit. Besides this normal force, we can not know much detail about the contact. This sensor can measure very large normal force and is small enough to be integrated on the gripper. Its structure is simple and robust. This measurement unit can be duplicated and arranged as one array. Figure 1.11 demonstrates one two-finger gripper equipped with two tactile arrays. Their measurements are visualized at the bottom of the figure, respectively. The small end of screw can activate four tactile units in the center and the large end of screw activates eight tactile units in contact with the circular edge of the end. Although the 2D distribution of normal force is available, the measurement is still a rough approximation of the real contact due to its discreteness. Although it is difficult to know the exact contact shape from this kind of measurement, this resolution could be sufficient for some simple tasks. The planar units based on the resistive change can only measure normal force but the 3D unit is able to measure 3D contact force. One series of tactile sensors composed by 3D measurement units is shown in Figure 1.12. Each hemispherical unit can provide 3D contact force applied on its surface by inferring from the optical variation inside the cavity of the unit. The unit is made by silicone and deformable by external contacts. Its deformation causes the light change inside it. By processing the measurements from all soft pillars on board, a general 6D wrench comprising contact force and torque on the entire sensor can be estimated and provided by the official APIs. However, the spaces between units are out of measurement. The complete tactile information is still missing.

Contactile is a representative of the optics-based tactile sensor but not an image-based sensor. On a small scale, it makes measurement on the change of the light intensity. Some photodiodes are used to process the light rather than a camera. It is not able to have a complete measurement on the entire sensing surface. To measure on a larger scale, we can exploit the image from a camera.

So far, the image-based tactile sensors can provide the highest resolution of touching. One classical instance is GelSight [START_REF] Yuan | Gelsight: High-resolution robot tactile sensors for estimating geometry and force[END_REF]. The appearance of GelSight is shown in Figure 1.13a. Its internal design is depicted in Figure 1.13b. When the sensor is working, the LEDs inside light the transparent elastomer gel. As the gel is soft, it can conform with the shape of the object in contact and change the distribution of internal lights. The camera can capture and stream this varying lighting at a certain frame rate. The output of this kind of sensor is image frames. As the minimum measurement unit for images is the pixel, the resolution of touching encoded by these contact images is dramatically increased. This high resolution can improve the sensitivity of perception and promote the usage of tactile sensors in dexterous manipulations. Following a similar principle, various image-based tactile sensors [START_REF] Donlon | Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger[END_REF][START_REF] Alexander | Visuotactile sensors with emphasis on gelsight sensor: A review[END_REF][START_REF] Hameed Shah | On the design and development of vision-based tactile sensors[END_REF][START_REF] Zhang | Hardware technology of vision-based tactile sensor: A review[END_REF] have been developed with improvements or adaptations on the mechanical design. GelSight Wedge [4] is designed for reaching narrow spaces, which is demonstrated in Figure 1.14. Another compact version is GelSlim [START_REF] Taylor | Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger[END_REF] as shown in the left of Figure 1.15. The sensor is shrunk by optimizing the optical path from illumination sources to the camera. (a) OmniTact [START_REF] Padmanabha | Omnitact: A multi-directional high-resolution touch sensor[END_REF] (b) Round GelSight [START_REF] Romero | Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation[END_REF] Fig. 1.16 Sensors with a round tactile surface.

Another research direction is to customize the sensor's effective area into non-planar shapes [START_REF] Padmanabha | Omnitact: A multi-directional high-resolution touch sensor[END_REF][START_REF] Romero | Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation[END_REF][START_REF] Won | Densetact: Optical tactile sensor for dense shape reconstruction[END_REF][START_REF] Sandra | Gelsight fin ray: Incorporating tactile sensing into a soft compliant robotic gripper[END_REF][START_REF] Liu | Gelsight baby fin ray: A compact, compliant, flexible finger with high-resolution tactile sensing[END_REF], which can enlarge the sensible area of sensors and promote the dexterity of fingertips. The representative sensors shown in Figure 1.16 are OmniTact [START_REF] Padmanabha | Omnitact: A multi-directional high-resolution touch sensor[END_REF] and Round GelSight [START_REF] Romero | Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation[END_REF]. As the field of view of sensors is largely increased, how to design the internal optical path and the position of camera becomes the main challenge.

Besides the advancement of hardware, algorithms can further enrich tactile signals. Without processing, the tactile image from sensors is two-dimensional. The information in the normal direction of the contact surface is missing. To recover the complete tactile status, the 3D tactile imprint is reconstructed [3-5, 51, 48]. Two reconstruction methods are reported with the series of GelSight sensors. The first one is to calibrate sensors with an object like a ball bearing. During the calibration, the ball is pressed on the surface of sensor. The depth of imprint is calculated by the spherical geometry model. The color difference between the non-contact tactile image and the contact tactile image can be obtained. Then the mapping between the color difference and the depth is saved in a lookup table. This process is repeated with a bunch of contact images. When one calibrated sensor touches a new object, the depth is generated by searching the table. This method requires many tactile images to guarantee the quality of calibration. This kind of calibration can be automatized by a CNC machine [START_REF] Romero | Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation[END_REF].

Training a neural network can replace the calibration table. It requires less tactile images to prepare a labelled dataset. As the dataset is pixel-wise, one tactile image can provide abundant data. Some 3D tactile imprints reconstructed by this method are shown in Figure 1.17. So far, the 3D tactile imprint can provide us the richest touching information. How to implement the 3D reconstruction on other image-based sensors and How to use this 3D effect properly in different applications are still two scarcely explored questions. Fig. 1.17 From top row to bottom: objects, tactile images, and inferred 3D shapes [4].

Pursuit of dexterity

With the advancement of tactile sensors, their application expands rapidly. One basic usage of image-based sensors is to identify the properties of objects in touch. For instance, the texture and thickness of fabrics can be discriminated [START_REF] Li | Sensing and recognizing surface textures using a gelsight sensor[END_REF][START_REF] Yuan | Active clothing material perception using tactile sensing and deep learning[END_REF][START_REF] Luo | Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition[END_REF]. The hardness of objects can be inferred from a sequence of frames [START_REF] Yuan | Estimating object hardness with a gelsight touch sensor[END_REF][START_REF] Yuan | Shape-independent hardness estimation using deep learning and a gelsight tactile sensor[END_REF]. When the contact between the object and the surface of the sensor proceeds, the object with larger hardness shows less deformation. This pattern can be captured by images.

For grasping, if the incipient slip can be detected by the tactile sensor, the gripper can react to avoid slippage and the failure of grasp [START_REF] Li | Slip detection with combined tactile and visual information[END_REF]. When slippage is happening, the small motion of the object with reference to the surface of sensor can be captured in a few consecutive frames. When this relative motion is larger than a threshold, the gripper imposes more contact force to stabilize the object [START_REF] Dong | Improved gelsight tactile sensor for measuring geometry and slip[END_REF].

As a 3D tactile imprint can be converted into a point cloud, the registration of the point cloud [START_REF] Huang | A comprehensive survey on point cloud registration[END_REF] can be used to estimate and track the pose of the object in touch. In Figure 1.18, the pose of a transparent triangular prism on the tactile sensor is tracked by the Iterative Closest Point (ICP) algorithm [START_REF] Zhang | Fast and robust iterative closest point[END_REF]. Because this object is transparent and vision could be partially blocked, pose tracking based on tactile sensing is more effective than using a camera in this case.

The potential of tactile sensors in dexterous operations is demonstrated. SwingBot [START_REF] Wang | Swingbot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation[END_REF] shown in Figure 1.19 is able to swing one object to the desired angle using two tactile fingertips. Before swing, SwingBot needs to estimate the manipulated object's mass, center of mass and moment of inertia by tilting it with the help of a trained neural network. By shaking the object, the surface friction can be estimated. Then the acquired knowledge about the object is fed to one learned forward dynamics model to produce the parameters of motion control. The training dataset is composed by objects' physical parameters and tactile patterns. In more practical contexts, tactile sensing is useful for manipulating clothes [START_REF] Sunil | Visuotactile affordances for cloth manipulation with local control[END_REF] and cables of electrical devices [START_REF] Li | Localization and manipulation of small parts using gelsight tactile sensing[END_REF][START_REF] She | Cable manipulation with a tactile-reactive gripper[END_REF]. Figure 1.20 depicts how a tactile image is used to orient the cable in hand. When one end of the cable is fixed, the gripper is able to follow the cable toward the other end while maintaining the cable at the desired position in hand. Two controllers are used in this manipulation. One grip controller is used to adjust the griping force between two fingers in order to allow the sliding of the cable and avoid dropping down. The mean displacement of the black dot markers on the tactile image as shown in Figure 1.20 is the value to be controlled. According to this value, one PD controller modulates the opening of the gripper. Another controller is used to orient the cable in hand. The contact with the cable is always in the form of an ellipse. As shown in Figure 1.20, the contact shape can be estimated from the reconstructed depth image since the depth in the contact region is larger than that in the non-contact region. Processing the estimated contact shape, the axes of the contact ellipse can be tracked as shown in the "Pose" image in Figure 1.20. The ideal pose of the cable is in the center of contact and parallel to the edge of the sensor. Using one linear dynamic model to approximate the cable-gripper dynamics, one Linear Quadratic Regulator (LQR) controller is implemented to adjust the pose of the cable in hand. Intrinsic contact sensing [START_REF] Bicchi | Contact sensing from force measurements[END_REF] addresses the problem of localizing contacts on the surface of a robot with known geometry. Usually, the contact location between the fingers and the object is estimated using the Force/Torque sensor on robot. Recently, by contrast, the concept of extrinsic contact sensing is proposed to estimate the contact location between the grasped object and the environment using tactile sensors. One example is shown in Figure 1.21. The screwdriver keeps a contact with the base. When the fingertips move the screwdriver in a way, the motion of the object frame attached to the screwdriver can be inferred by observing a sequence of tactile images. Solving some transformation equations, the position of this contact point can be estimated in the world frame. This method works under some assumptions:

• Point contact or line contact between tool and environment.

• The grasped object is rigid.

• The environment is rigid.

• The object remains in contact with environment.

• The grasp of the object is stable. Extrinsic contact sensing is useful for insertion tasks. As shown in Figure 1.22, the tactile images sampled during contact phase are fed to a Reinforcement Learning (RL) model to output an action for the next attempt. The training is performed in real experiments because the simulated tactile images during the contact phase are not precise enough. This approach demonstrated some progress in the insertion of different objects in regular shapes. A similar work [START_REF] Kim | Active extrinsic contact sensing: Application to general peg-in-hole insertion[END_REF] is made to improve the training phase. Instead of directly using the high-dimensional tactile images as the input of the RL model, the contact line between the object and the hole, as a low-dimensional representation of the contact, is adopted as the intermediate variable for the policy training. The contact line is depicted in Figure 1.23.

Assumed that the bottom surface of the object and the top surface of the hole are flat, the contact line is formed by connecting the intersected points between the polygon of the bottom surface of the object and the polygon of the top surface of the hole. Thanks to this contact line, the policy training can be performed in a simple 2D geometric simulation without the need to collect training data in a real experiment. The trained RL insertion policy can take the contact line as input and produce a robotic action as output. Some restrictions are declared for this approach:

• Object bottom surfaces and hole top surfaces are flat.

• The misalignment between the object and the hole is an SE(2) displacement in the plane of the contact.

• Objects and holes are un-chamfered. We think the concept of the extrinsic contact sensing can be generalized as the indirect tactile sensing. Extrinsic contact sensing aims to solve the localization problem of the contact between the object and its environment. Localization retrieves one type of the contact information useful for manipulation. The contact force, slippage or other contact state between the object and its environment can also provide guidance for manipulation. Inferring these external information from the sensed contact between the sensor and the object embodies the indirect tactile sensing. Compared with the commonly used direct tactile Fig. 1.23 Red contact line between the object and the hole [START_REF] Kim | Active extrinsic contact sensing: Application to general peg-in-hole insertion[END_REF].

sensing committed to perceive the contact state between the sensor and the object, the indirect tactile sensing expands the scope of the capability of the tactile sensing and enables more contact-rich applications.

In summary, image-based sensors are widely adopted because their high resolution can provide more informative perception, which is important for some complicated robotic manipulations. Also, tactile images bridge the existing vision techniques with the domain of tactile sensing and servoing, which can inspire new research. The currently reported results are quite diverse but still far from the real applications in production. As a new field, the way to properly utilize tactile images is still in the stage of exploration. Different proposals continue coming up. All attempts to solve a specific problem could lead to some advancement.

Contribution

After the presentation of the motivation of our project and the current progress of the relevant research, we can better position our work and clarify our contributions.

Contact and transmission, as two theoretical bases for grasping, are well researched for rigid objects. For deformable objects, existing research works focus on the modelling of contact but ignore the modeling of the dynamic transmission property. With this insight, we chose our research line and made the following contributions:

• The dynamic transmission ability of a grasp is modeled to fill in a gap in the grasping theory for deformable objects.

• The dynamic evaluation methodology is proposed according to the nature of deformation of objects during grasping.

• Based on the extended theory, a novel grasping synthesis method to optimize grasp stability is developed and verified in both of simulations and experiments.

The whole work has contributions in grasping modeling and grasping synthesis for deformable objects, which is presented in Chapter 2. We adopted the image-based tactile sensor DIGIT [START_REF] Lambeta | Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation[END_REF] as hardware and implemented the real-time 3D tactile imprint. Several 3D reconstruction methods are developed and compared to achieve the quality 3D tactile imprint in order to enable the high tactile resolution and sensitivity. The internal illumination of DIGIT has high uncertainties which can impair its 3D performance. One improved neural network based on the color variation of tactile images is implemented and validated to provide the optimal 3D reconstruction effect for DIGIT among all the developed methods. The work related to the 3D tactile imprint is presented in Chapter 3.

In Chapter 4, empowered by 3D tactile imprints, some contributions are made in the specific robotic applications:

• One novel slip detection method is proposed based on the tactile mesh derived from the reconstructed height map on the tactile sensing surface. The effectiveness of the proposed method is validated by various objects with different physical properties. The slippage occurring in different directions can be detected. As the basic tactile unit in the tactile mesh is a pixel, we can change the density of the mesh by selecting and using a certain part of all pixels in the tactile image. The impact of the density of the tactile mesh is studied on the performance of detection. One slip avoidance strategy is implemented to flexibly react to the varying slip state in manipulation.

• One height-based pipeline to track the in-hand pose of the gripped object is developed.

Compared to the open-source color-based method in Pytouch [START_REF] Lambeta | Pytouch: A machine learning library for touch processing[END_REF] using the color change between the current frame and the base frame without contact to estimate the pose, the proposed pipeline can provide more reliable results. In experiments, the height-based pipeline is proved to be able to track different objects having different contact areas with the sensor.

• In order to facilitate doll assembly through indirect tactile sensing, the concept of the sense of tightness is proposed. The tactile sensor can perceive the contact tension between the gripped toy part and the other toy part to be connected with. The pattern of the tightness value evolving in the operation is demonstrated and how to select a desired tightness value is explained. Using tightness to regulate the interaction between the object and its environment can provide high robustness to the uncertainties in the operation. Based on the sense of tightness, a robotic pipeline with high sensory and motor coordination is proposed to conduct the doll assembly, which outperforms the strategy using position control. The demonstrated novel sense has the potential to be applied in more scenarios requiring the awareness of the contact tension between the gripped object/tool and its environment.

The final chapter gives out the conclusion of this research and points out the future directions.

Chapter 2

Dynamic evaluation of deformable object grasping

Introduction

Robotic grasping synthesis consists on determining the optimal grasp configuration for robotic tasks. The foundation of grasping synthesis is the definition of effective quality measures to evaluate grasp candidates. For rigid objects, a bunch of quality measures has been proposed and studied in detail [START_REF] Roa | Grasp quality measures: review and performance[END_REF]. These methods attempt to solve two problems: contact location [START_REF] Bicchi | On the closure properties of robotic grasping[END_REF][START_REF] Rimon | On force and form closure for multiple finger grasps[END_REF] and hand configuration [START_REF] Shimoga | Robot grasp synthesis algorithms: A survey[END_REF]. This article is devoted to solve the problem about contact location. The common measures to determine the contact placement include the algebraic properties of the grasp map matrix [START_REF] Li | Task-oriented optimal grasping by multifingered robot hands[END_REF][START_REF] Kim | Optimal grasping based on non-dimensionalized performance indices[END_REF], the grasp wrench space [START_REF] Borst | A fast and robust grasp planner for arbitrary 3d objects[END_REF], the task wrench space [START_REF] Ch | Grasp planning: How to choose a suitable task wrench space[END_REF] and other measures based on geometric relations [START_REF] Rubert | On the relevance of grasp metrics for predicting grasp success[END_REF]. Two important characteristics of the measures for rigid objects are:

• These measures reflect the capacity of one grasp to transfer contact forces as the resultant grasp wrench through contact interfaces in a direct or indirect form.

• These measures are static because the corresponding transfer capacity of a grasp candidate is static since the stiffness of rigid bodies is large enough to resist deformation that can change this capacity.

However, deformable objects have smaller stiffness and their deformation during grasping can bring instability to the transfer capacity. How to evaluate this dynamic transfer capacity is one problem to be solved in this work. First of all, we need to clarify the type of the studied deformable objects since deformable objects can be of different natures with totally different physical properties, such as ropes, paper, clothes, 3D toy pieces, plastic bottles, etc [START_REF] Sanchez | Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey[END_REF]. In this study, volumetric deformable objects with certain stiffness are considered. For common volumetric deformable objects, grasp stability and deformation are two main concerns in grasp evaluation. The grasp stability represents the resistance of a grasp to external disturbances. For fragile objects or deformable liquid containers [START_REF] Jingyi | Minimal work: A grasp quality metric for deformable hollow objects[END_REF], preventing large deformation should be the priority. For non-fragile deformable objects, grasp stability is the main problem, which is the focus of our study. After a survey on the existing literature about deformable object manipulation, several features can be summarized:

• 2D deformable object handling [START_REF] Gopalakrishnan | D-space and deform closure grasps of deformable parts[END_REF][START_REF] Jia | Grasping deformable planar objects:squeeze, stick/slip analysis, and energy-based optimalities[END_REF][START_REF] Ramirez-Alpizar | Dynamic nonprehensile manipulation for rotating a thin deformable object: An analogy to bipedal gaits[END_REF] is more studied than 3D objects.

• More efforts are dedicated to analyzing the influence of local changing contact states [START_REF] Lin | Picking up a soft 3d object by "feeling" the grip[END_REF][START_REF] Zaidi | Modelbased strategy for grasping 3d deformable objects using a multi-fingered robotic hand[END_REF] than to analyzing the varying global wrench transfer capacity.

• The applications are usually limited to some simple and light-loaded situations [START_REF] Lin | Picking up a soft 3d object by "feeling" the grip[END_REF][START_REF] Zaidi | Grasp planning pipeline for robust manipulation of 3d deformable objects with industrial robotic hand + arm systems[END_REF].

To pick up an eggplant or transport a foam does not challenge much on the grasp quality.

In applications, the grasped object could bear different loads to finish tasks. For example, the grasp only needs to overcome the self-weight of the object in transportation but has to resist large wrenches in assembly. In this case, the interaction between the fingertips and the object should be adapted accordingly. Certainly one static score cannot carry all the information we need to evaluate one grasp during interaction. How to represent and assess this dynamic grasp stability is still an open question.

In this work, a dynamic evaluation method based on the dynamic grasp map matrix is proposed to convey and analyze the dynamic performance of grasp candidates for 3D deformable objects. This method provides three main contributions:

• The transmission ability of a grasp is dynamically modelled to complement the general grasping theory for deformable objects.

• A new methodology to evaluate grasps on deformable objects in a dynamic manner, which is helpful to inspire new research lines.

• A new guidance for grasp synthesis. The specific steps to find the optimal grasp are given based on the analysis of the proposed dynamic metrics.

In order to present the methodology and techniques of dynamic evaluation on deformable objects grasping, the remainder of this work is organized as follows: Section 2.2 introduces the mathematical formulations of the grasp map matrix and discusses the selection of the contact model. Then the dynamic grasp map matrix, three derived dynamic grasp metrics and the specific usage of the metrics in grasp synthesis are presented in Section 2.3. In Section 2.4, physical simulations demonstrate the procedure of the dynamic evaluation method and verified its effect. Section 2.5 presents the experimental results achieved by one dual arm robotic system to further validate the effectiveness of our dynamic evaluation method. Discussions and concluding remarks are presented in the last section. To describe a grasp, the object frame and the contact frames are set up as shown in Fig. 2.1. The origin of contact frame is attached on the contact point and the Z-axis of contact frame normally points inward the tangential surface of the contact. Each contact point has its own contact frame. The origin of object frame is set on the center of mass of the grasped object. The geometric relationship between the contact frame C i and the object frame O is encapsulated on g oc i :

Mathematical formulation of grasping 2.2.1 Grasp map matrix

g oc i = (p oc i , R oc i ) ∈ SE(3), (2.1)
where p oc i represents the relative translation between the origins C i and O. R oc i refers to the rotation matrix between the contact frame and the object frame. In the contact frame C i , the contact model usually has the form as:

F c i = B c i f c i f c i ∈ FC c i , (2.2)
where F c i represents the contact wrench applied by the fingertip. f c i is a vector representing the contact force magnitude in each dimension. For one certain contact type, the wrench basis B c i is used to define in which dimensions the contact force or torque could be applied. The friction cone FC c i is determined by the friction coefficients between the fingertip and the grasped object [START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF]. Through the adjoint transformation matrix, the contact wrench F c i can be transferred into the object frame O as F o :

F o = Ad T g -1 oc i F c i = R oc i 0 poc i R oc i R oc i B c i f c i f c i ∈ FC c i . (2.3)
One contact map can be defined as:

G i = Ad T g -1 oc i B c i . (2.4)
In order to derive the general form of the grasp map with m contact points, the total wrench in the object frame can be written as the following sum:

F o = G 1 f c 1 + ... + G m f c m = G 1 ... G m    f c 1 ... f c m    = G f c . (2.5)
Combining (2.4) and (2.5), the grasp map G can be obtained as:

G = Ad T g -1 oc 1 B c 1 ... Ad T g -1 oc m B c m (2.6)
Observing Eq. (2.6), the grasp map is related to the geometric transformation between the contact frames and the object frame, and the wrench basis B c i defined by contact model. The geometric transformation is consistent for rigid bodies but dynamic for deformable objects. If the dynamic geometric transformation can be tracked, we can calculate the grasp map in a dynamic manner and use this dynamic grasp map to analyze the dynamic performance of a grasp, which is one of the main novelties in this study.

Contact model

The grasp quality is usually defined by two general factors. One factor is the wrench transfer capacity formulated by Eq. (2.6). Another factor is the limitation space as explained in Subsection 1.2.1. The limitation space defines the range of the transferable contact wrenches in the contact frames. As an instance of the limitation space, the friction cone FC c i of the hard contact model is formulated by:

F = ( f 1 , f 2 , f 3 ) FC c i = {F ∈ R 3 : f 2 1 + f 2 2 ) ≤ µ f 3 , f 3 > 0} (2.7)
where µ is the coefficient of friction between the fingertip and the object and f 3 is the normal component of the contact force vector F. Only the contact force F limited in the friction cone is transferable. This limitation space is visualized in Figure 1.4.

One contact model usually consists of two components. One component defines the wrench basis B c i which impacts the wrench transfer capacity in Eq. (2.6). Another component exactly defines the limitation space. Since the emphasis of this work is to study the dynamic wrench transfer capacity of a grasp, we should decouple the influences of these two components. The wrench basis B c i should be adjusted according to the actual situation. If a gripper has small contacts with an object, the point contact models can be used. If the contact area is large, we adopt the wrench basis B c i of the patch contact model. It has the form of 6D unit matrix, which means 3D force and 3D torque can be transferred through the patch contact interface.

In order to equalize the impact of the limitation space, for one object, only the grasp candidates with similar limitation spaces are considered. For example, the grasp candidates with the similar contact pattern and friction coefficient are selected. In this way, we can better reveal the influence of the dynamic wrench transfer capacity on the actual grasp quality instead of confusing too many factors at the same time. Therefore, in the following contents, the particular limitation space is excluded from discussion because it is the same or assumed similar for all grasp candidates of one object. This condition holds when the same contact model is adopted and the friction coefficient between the fingertip and the object is consistent.

The different contact models have their particular wrench basis B c . In practice, if 6D wrenches are generated at the contact, the wrench basis of the patch contact model should be used in the form of 6D unit matrix. If certain contact force or torque along or around one axis is zero or relatively small, the corresponding row of the wrench basis can be set as zero. The degraded wrench basis is corresponding to the model of line contact or point contact. For example, the common soft finger contact can transfer contact forces along three axes and one contact torque around the inward normal of the tangential contact surface. In total 4D wrench is transferable in this model. More explanations about wrench basis B c can be found in [START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF]. For complicated non-planar contacts, a model was proposed to consider the patch contact between the compliant fingertip and rigid objects [START_REF] Danielczuk | Reach: Reducing false negatives in robot grasp planning with a robust efficient area contact hypothesis model[END_REF]. This method divides the patch contact into different triangles. The total wrench is the sum of the wrench applied on each triangle. To describe the friction of the non-planar patch contact, the 6D limit surface models were defined in [START_REF] Xu | 6dls: Modeling nonplanar frictional surface contacts for grasping using 6d limit surfaces[END_REF]. For the planar patch contact between fingertips and soft objects, the 3D limit surface models [START_REF] Kao | Contact modeling and manipulation[END_REF] are used to define the set of the possible friction. The limit surface is the boundary of the set of friction whose determination requires a lot of computation. The friction and the normal contact force together compose the contact wrench. The set of the possible contact wrenches is the limitation space.

Dynamic grasp metrics and dynamic evaluation 2.3.1 Dynamic grasp metrics

The dynamic grasp map G dy (t) can be expressed in time domain as:

G dy (t) = Ad T g -1 oc 1 (t)B c 1 ... Ad T g -1 oc m (t)B c m Ad T g -1 oc i (t) = R oc i (t) 0 poc i (t)R oc i (t) R oc i (t) (2.8)
As B c i is fixed with the adopted contact model, the dynamic property of G dy (t) is determined by the relative position p oc i (t) and the relative rotation R oc i (t) between the object frame and each contact frame during grasping. This dynamic grasp map G dy (t) represents the dynamic grasping status on the deformable object.

This map is a matrix whose size depends on the number of fingers and the wrench basis B c i . Each element of G dy (t) is defined by Eq. (2.4). If we use soft contact model, as Ad T g -1 oc i is a 6 × 6 matrix and B c i is a 6 × 4 matrix, the size of each element of G dy (t) is 6 × 4. The element number is equal to the number of fingers. The dynamic grasp map G dy (t) could be a 6 × 12 matrix for a 3-finger gripper.

As a matrix, the dynamic grasp map itself cannot be directly used as the grasp quality metric. However, each instance of the dynamic grasp map at a moment can be used to calculate grasp metrics or generate grasp wrench space. Concatenating all these values or states together, dynamic grasp metrics can be achieved to assess grasp quality for the entire grasping process. In order to present the idea and method without much complexity, we propose the dynamic grasp metrics based on the algebraic properties of the dynamic grasp map G dy (t). Other quality metrics can also be integrated in this dynamic evaluation methodology as future research work.

A full-rank grasp map G has 6 singular values which are the positive square roots of the eigenvalues of GG T . Each singular value quantifies the wrench transfer ability of one grasp in one direction. A higher singular value can transfer the contact wrenches applied in the contact frames as a larger grasp wrench defined in the object frame to stabilize the grasped object. Based on this understanding, the minimum singular value of the dynamic grasp map matrix can be used as the first dynamic grasp quality metric, which is expressed as:

Q 1 (t) = σ min (G dy (t)) (2.9)
When the smallest singular value is approaching to zero, the grasp is close to a singular configuration where the grasp is losing resistance to the external wrench in the corresponding direction [START_REF] Li | Task-oriented optimal grasping by multifingered robot hands[END_REF]. Q 1 (t) reflects the dynamic minimum transmission gain from contact forces to the grasp wrench in the object frame [START_REF] Kim | Optimal grasping based on non-dimensionalized performance indices[END_REF]. Thus an effective grasp should keep a large Q 1 (t).

The dynamic volume of the ellipsoid in the wrench space is defined as the second dynamic grasp metric. If we set the constraint || f c || = 1 (a unitary sphere), (2.5) can transfer this sphere into an ellipsoid in the wrench space. The volume of this ellipsoid reflects one grasp's global transfer ability from contact forces to the grasp wrench [START_REF] Li | Task-oriented optimal grasping by multifingered robot hands[END_REF], which has the form:

Q 2 (t) = det(G(t)G T (t)) = σ 1 (t)σ 2 (t)...σ 6 (t) (2.10)
where σ i (t) denotes each singular value of the dynamic grasp map. This quality metric considers all the singular values with the same weight and should be maximized to achieve the maximum general wrench transfer ability. Since it cannot reflect the wrench transfer ability of one grasp in a specific direction, we need to use it with other complementary metrics.

After knowing the minimum wrench transfer ability and the global wrench transfer ability, another metric is expected to reflect the equilibrium of transfer ability in all directions. The dynamic grasp isotropy index [START_REF] Kim | Optimal grasping based on non-dimensionalized performance indices[END_REF] is defined for this purpose:

Q 3 (t) = σ min (G(t)) σ max (G(t)) (2.11)
where σ min (G(t)) and σ max (G(t)) represent the minimum and maximum singular values of the dynamic grasp map. The value of this metric is between 0 (worst case, singular grasp) and 1 (optimal case, isotropic grasp). Using these three dynamic metrics together, we can have the thorough knowledge on the dynamic wrench transfer ability of one grasp during grasping. The analysis on these dynamic values can provide the insight to choose the optimal grasp.

Based on our need, these proposed metrics can also relate to other dynamic variables instead of time. When common grippers with two parallel jaws are considered, the metrics can be formulated as

Q 1 ( f ), Q 2 ( f ), Q 3 ( f ).
f refers to the value of contact force.

Stability of wrench transfer ability

Among the proposed three dynamic grasp metrics, we use Q 2 to represent the global wrench transfer ability (GWTA) of a grasp. During grasping, Q 2 may have three directions to go: increase, stay constant and decline. We hope it can increase, but in most cases it declines. It could be constant if the gripped object is rigid. If it has to decline, we hope it could decline as slowly as possible when contact forces are increasing. We use the term "Stability of Wrench Transfer Ability"(SWTA) to describe the ability of one grasp to keep its GWTA. SWTA can be quantified by the gradient of Q 2 . We guess that the current local stiffness of the object around the fingertips has great influence on SWTA. In order to reveal its linkage to the physical world, we derived the analytical solution of Q 2 using Eq. (2.10) in the simplified case. Then its gradient with reference to contact force is achieved.

To reduce the complexity of computation, 2-finger grips are studied. We set the contact frames of the fingertips as the black frames marked in Fig. 2.3b. The right frame is termed C 1 and the left frame is termed C 2 . The red object frame is located in the center of mass of the object. The Y axis of each frame points outward page. Using Eq. (2.4) and Eq. (2.6), the grasp map matrix of one typical 2-finger grip can be obtained as:

G 2 f =            -1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 -1 0 0 0 1 0 0 -z 1 -y 1 0 0 -z 2 y 2 0 -z 1 0 x 1 0 z 2 0 -x 2 0 y 1 x 1 0 -1 -y 2 x 2 0 1            (2.12)
where (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) refer to the coordinates of C 1 and C 2 in the object frame, respectively. Then we can compute Q 2 as:

Q 2 = det(G 2 f G T 2 f ) = 2(z 1 -z 2 ) (x 1 -x 2 ) 2 + (y 1 -y 2 ) 2 + (z 1 -z 2 ) 2 (2.13)
Two fingers are opposite and in this specific case (x 1 , y 1 ) is equal to (x 2 , y 2 ). Thus Q 2 can be simplified as:

Q 2 = 2(z 1 -z 2 ) 2 (2.14)
If Q 2 is changing with the contact force f , its derivation is:

d(Q 2 ( f ))/d f = 4(z 1 -z 2 )(d(z1 -z2)/d f ) = 4D/K (2.15)
where z1 -z2 is termed as D that means the distance between two contacts and K is the current stiffness of object between two fingers. Therefore, for 2-finger grips, we know that a larger stiffness can improve SWTA.

Dynamic evaluation

To select the grasp with the optimal dynamic wrench transfer ability, three factors need to be considered, including SWTA (defined by the gradient of

Q 2 such as d(Q 2 ( f ))/d f ), GWTA (defined by Q 2 )
and the specific transfer ability (defined by Q 1 and Q 3 ). At first, we should pay attention to SWTA, which is the prerequisite for discussing other metrics. If SWTA of two grasps is different, the capacity of these two grasps to keep the current wrench transfer ability is different. Even though their absolute value of the related metric is the same at the moment, we should choose the grasp that can better keep its wrench transfer ability. SWTA demonstrates how solid the computed wrench transfer ability is for a grasp. Also, SWTA is related to the object's stiffness. The fundamental impact of stiffness on grasp quality is also discussed in [START_REF] Jingyi | Minimal work: A grasp quality metric for deformable hollow objects[END_REF]. For deformable hollow objects, the contact location with higher stiffness is preferred.

If SWTA of grasp candidates is similar, we need to further compare other metrics. Thinking from whole to part, we should compare the absolute value of GWTA, which reflects the general wrench transfer ability of a grasp. The larger GWTA is preferred. If GWTA of grasp candidates is similar too, we further consider the distribution problem of their wrench transfer ability (Q 1 and Q 3 ).

The specific steps of dynamic evaluation can be described as following:

• Compute the dynamic metrics for all grasp candidates.

• Compare SWTA. SWTA is computed by d(Q 2 ( f ))/d f . Since the value of Q 2 declines, the value of SWTA is negative and a smaller value implies a worse SWTA. If we draw the Q 2 curve for the grasp with worse SWTA, it possesses a steeper downward slope.

If SWTA of a grasp is obviously worse than others, it should be excluded.

• Compare GWTA. Among the grasps with similar SWTA, we choose the one with the largest GWTA.

• Compare other factors. If two grasps with similar SWTA and GWTA, we can further compare Q 1 and Q 3 . A larger Q 1 or Q 3 indicates a better grasp.

For instance, some possible dynamic Q 2 curves are visualized in Fig. 2.2. Firstly, Grasp2 is excluded due to its weak SWTA. Then we do not choose Grasp4 because its GWTA is too weak. Finally, Grasp3 is the desired one among these candidates as its good performance in SWTA and GWTA. In the next two sections, this method is applied to analyzing the dynamic metrics in physical simulations and robotic experiments.

Simulated dynamic evaluation

The simulator DefGraspSim [START_REF] Huang | Defgraspsim: Simulation-based grasping of 3d deformable objects[END_REF] is adopted as it is empowered by Isaac Gym [START_REF] Makoviychuk | Isaac gym: High performance gpu-based physics simulation for robot learning[END_REF]. The complex object geometry, object deformation, gripper-object dynamics, and large perturbations are explicitly modeled in Isaac Gym. The deformable objects are represented by tetrahedral meshes and simulated based on 3D co-rotational FEM. Its accuracy in simulating deformable objects is validated in [START_REF] Narang | Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections[END_REF].

Grasp quality prediction

Using DefGraspSim, the contact forces between the gripper finger and deformable objects can be fetched to compose the "metrics-force" curves. The grasped object is a deformable brick with the size 6cm × 3cm × 2cm. We set its Young's Modulus as 2e 5 Pa, Poisson's Ratio as 0.4, density as 3000kg/m 3 . The Young's Modulus and Poisson's Ratio correspond to the material of foam, which determine the deformation feature of the object. In order to produce a larger inertial force on the object in the acceleration tests as described in Subsection 2.4.2, the density of object is increased. The default density 1000kg/m 3 in the simulator is hard to cause an adequate inertial force to break the proposed grasp before the end of test. The Franka 2-finger gripper is adopted whose fingers' width is 2cm. Four grasp candidates are proposed as shown in Fig. 2.3b. The red object frame is defined at the center of the brick. The black contact frames are defined at the center of the contact patch. The Y axis points outward page. We denote the configurations indicated by the green, red, yellow and black arrows as Grasp1, Grasp2, Grasp3 and Grasp4, respectively. Grasp1 and Grasp2 are chosen to investigate the impact of stiffness. The comparison among Grasp1, Grasp3 and Grasp4 is helpful to reveal the effect of the relative position. The influence of the contact area's size is eliminated because the four grasps have the same contact size as 2cm × 2cm during grasping. This size is determined by the thickness of the grasped object and the width of the finger of the gripper. The gripper can downward penetrate the table a bit to grasp the brick since this conflict is set as ignored in the simulation. The steps to achieve the dynamic "metrics-force" curves are described as following:

• For one candidate, initialize the gripper's pose as the predefined configuration and maximize its opening.

• Slowly close the fingers. After the contact between the finger and the brick is detected, we start to record the contact forces and the corresponding positions of fingertips.

• Using the recorded series of the fingertip position, we can follow Eq.(2.1)-(2.6) to obtain the series of the grasp map G. Based on the results of the singular value decomposition on each G, we can easily compute the series of the metrics by Eq.(2.9)-(2.11).

• Visualize these metrics changing with contact force.

For these four grasps, their dynamic metrics are illustrated in Fig. 2.4. Based on the method in Subsection 2.3.3, we excluded Grasp2 at first because its SWTA is much weaker than other grasps. Since Q2 of Grasp1, Grasp3 and Grasp4 are similar, we further compare other factors. In both of Q1 and Q3, Grasp1 is the best and then Grasp3 is better than Grasp4. Therefore, we found "Grasp1 > Grasp3 > Grasp4 > Grasp2" according to the analysis of these dynamic curves. This prediction is validated by the stability test in the next subsection.

Grasp quality validation

Using DefGraspSim, we can implement the acceleration tests to verify grasp stability. The steps of test can be described as:

• The gripper is initialized to the desired grasp configuration and the desired contact force.

• Select the directions to accelerate. The direction and the jerk of the motion of the gripper are predefined in the simulator [START_REF] Huang | Defgraspsim: Simulation-based grasping of 3d deformable objects[END_REF].

• The gripper is accelerated slowly along the chosen directions and stops until it loses the grasped brick. The maximum accelerations the gripper can reach before the grasp fails are recorded. The grasp with larger maximum accelerations should be the most stable grasp because it can bear larger inertial forces. The four grasps are tested along the directions "0, 1, 2, 3" defined in Fig. 2.5 [START_REF] Huang | Defgraspsim: Simulation-based grasping of 3d deformable objects[END_REF]. The desired contact force of each finger is set as 5N. The test results are listed in Tab. 2.1. The average value of the maximum accelerations for each grasp verified the prediction from the dynamic evaluation in the last subsection.

Experimental dynamic evaluation

This section presents the experimental validation of our method. The "metrics-force" curves are achieved with the help of tactile sensors. By analyzing these curves, the better grasp candidate can be predicted. Then the grasp stability experiments are designed and implemented on a dual arm robotic manipulation platform. We chose a deformable hollow doll head made from rubber as the grasped object, because in the next phase of our project we need to assemble the doll head with doll's other components. In robotic assembly, the doll head needs to bear a large external wrench thus determining a stable grasp configuration for doll heads is crucial. The experimental results proved the effectiveness of the dynamic evaluation to find the more stable grasp disturbed by large external wrenches.

Grasp quality prediction

In experiments, the tactile sensors are used to measure contact forces. A pair of tactile sensors (Contactile Sensing Array) is integrated on the 2-finger Robotiq gripper as shown in Fig. 2.6. Considering the specific design and shape of the doll head, two grasp candidates are proposed to assure that the fingers can achieve almost full patch contacts with the object. The relative poses of the two fingers are depicted by the arrows in Fig. 2.7. It is difficult to find more reasonable candidates who can avoid the area of nose, eyes and ears of the toy head. We followed the same procedure to generate the dynamic curves as the steps in simulation. To compute G, the wrench basis B c i of the patch contact model is adopted because we observed that the fingertips fast achieve the patch contact with the toy head and then keep a contact on almost its full rectangle surface. The achieved dynamic curves for two grasps are illustrated in Fig. 2.8. It can be observed that Grasp2 has similar gradient and higher values in Q2 compared to Grasp1. It means Grasp2 has similar SWTA and better GWTA. The overlapped curves in Q1 reflect that two grasps have the same minimum transfer ability in one certain direction. Based on Eq. (2.11), Grasp1's higher value in Q3 implies that the maximum transfer ability of Grasp1 in one direction is weaker than Grasp2. In conclusion we can judge that Grasp2 is more stable than Grasp1. In order to validate the judgement made in the last subsection, the following "graspingdisturbing-measuring" experiments are designed and implemented. The experimental setup is shown in Fig. 2.9. The Robotiq 2-finger gripper on the end of the left UR10 robotic arm is used to grasp the toy head in the configurations defined by Grasp1 and Grasp2. The Robotiq 3-finger gripper on the end of the right UR10 arm is holding the disturbing bar stably. After the 2-finger gripper achieves the desired grasp, the disturbing bar is driven to break the grasp from different directions. During "disturbing", the caused forces and torques are measured by the force/torque sensor on the wrist joint of the right arm. The frame of the force/torque sensor is marked on the right of Fig. 2.9. The angle between the vertical direction and the Y axis has 45 degrees.

Grasp quality validation

Using the tactile sensors, the map relationship between the gripper separation (the deformation of toy head) and the contact force has been established. For Grasp1, the gripper is controlled to achieve the separation at 48mm that corresponds to the contact force around 16N. For Grasp2, the gripper stops at the opening of 57mm which also maps to the contact force around 16N. Before the start of "disturbing", the grasp is initialized to these states.

The right arm is controlled to adjust the posture of the disturbing bar. Disturbing from different directions, three groups of experiments are conducted. The initial posture of the bar for each group is shown in Fig. 2.11. In each group, the initial posture, the moving direction, speed and displacement of the disturbing bar are predefined as the same with reference to the 2-finger gripper. We rotate the toy head to switch between the configurations of Grasp1 and Grasp2. The disturbing bar is driven to move toward the toy head until the grasp ends in failure. The force/torque sensor on the right arm recorded the interaction during the disturbing process. The measured actual wrenches in these three groups of experiments are visualized in Fig. 2.12. In Group1 and Group2, the applied external forces/torques on Grasp2 are obviously larger than Grasp1. Grasp2 can resist a larger external wrench until getting invalid, which means Grasp2 can better stabilize the toy head. For Group3, the measured forces on Grasp2 are much larger than Grasp1. However, the measured torques for Grasp1 have a different pattern compared to Grasp2. In fact, this phenomenon is caused by the final status of the toy head stuck between the disturbing bar and the palm of gripper, which is depicted in Fig. 2.10. In general, the results of these three groups of experiments validated that Grasp2 is more stable than Grasp1, which agrees with the prediction from the dynamic grasp quality curves. The effectiveness of the dynamic synthesis method is thus verified.

Conclusion

A dynamic evaluation methodology on deformable object grasping has been proposed, demonstrated and validated to boost the grasping synthesis from static level toward dynamic level. How to exploit the interaction data between a gripper and a deformable object to guide grasping synthesis is presented in details.

The dynamic grasp map is proposed to describe the varying grasp configuration during the deformation of object. Three dynamic grasp metrics are formulated to quantify the changing wrench transfer ability of a grasp. The linkage between dynamic metrics and physical properties is explained. The analysis method on the dynamic metric curves is provided. These dynamic metrics and the evaluation method are applied in both physical simulations and robotic experiments. The achieved results proved the effectiveness of the dynamic evaluation in finding the grasp with optimal stability. This dynamic method brings the possibility to analyse SWTA of grasp candidates, which is the crucial criterion to find the optimal grasp in consideration of the deformation property of objects. The static methods for rigid objects cannot reflect this factor and cannot be directly applied for deformable objects. For example, when one candidate has high GWTA but low SWTA, it can be excluded by the dynamic method but selected by the static methods. In the static methods, the absolute value of GWTA is cared but SWTA is ignored. In practice, the low SWTA leads to the poorer grasp stability as explained in Section 2.3.

The methodology of dynamic evaluation inspires a new way to analyze, compare and synthesize the grasps for deformable objects. Other grasp quality metrics, algorithms and applications could be filled in this framework to adapt to the requirements defined by the context of handling deformable objects.

Chapter 3 3D reconstruction of tactile imprints 3.1 Introduction

When humans touch objects, they have a general feeling on the shape of contact. This perception can help us to know the property of the touched object, like texture and softness, identify the type of objects, estimate the pose of the object in hand, and feel grasp stability, etc. In robotics, the sense of touch can be digitalized by tactile images with high resolution. Recovering the 3D information of contact from these 2D raw tactile images, even a more precise 3D perception than humans can realize.

Summarizing the methods for tactile 3D reconstruction, four main technical routes are reported. Firstly, the photometric stereo algorithm [START_REF] Robert | Photometric method for determining surface orientation from multiple images[END_REF][START_REF] Micah | Retrographic sensing for the measurement of surface texture and shape[END_REF] is applied in the series of GelSight sensor [START_REF] Yuan | Shape-independent hardness estimation using deep learning and a gelsight tactile sensor[END_REF]. Using the technique of photometric stereo, the surface normals of the contact can be estimated by observing that contact region under different lighting conditions. A lookup table is usually established to map the observed color intensities to the surface normals. Secondly, learning-based pipelines are proposed to realize 3D reconstruction for the sensors with 3D curved sensing surfaces [START_REF] Won | Densetact: Optical tactile sensor for dense shape reconstruction[END_REF][START_REF] Won | Densetact 2.0: Optical tactile sensor for shape and force reconstruction[END_REF]. A fisheye camera is installed inside their sensor which can provide a larger view on the curved sensing surface. Thirdly, the tailored depth sensors [START_REF] Alspach | Softbubble: A highly compliant dense geometry tactile sensor for robot manipulation[END_REF][START_REF] Kuppuswamy | Soft-bubble grippers for robust and perceptive manipulation[END_REF] are employed to directly measure the 3D contact shape. The dimension of this kind of sensors is too large to be installed as a fingertip. Lastly, the binocular stereo vision systems [START_REF] Zhang | Learning-based six-axis force/torque estimation using gelstereo fingertip visuotactile sensing[END_REF][START_REF] Zhang | High-precision 3d reconstruction study with emphasis on refractive calibration of gelstereo-type sensors[END_REF][START_REF] Cui | In-hand object localization using a novel high-resolution visuotactile sensor[END_REF][START_REF] Cui | Self-supervised contact geometry learning by gelstereo visuotactile sensing[END_REF] are developed to measure the 3D contact shape based on the ray tracing method [START_REF] Hu | Gelstereo palm: A novel curved visuotactile sensor for 3d geometry sensing[END_REF]. Two cameras are integrated inside the sensor. Most of them require special hardware configurations to achieve a good 3D reconstruction and it is hard to reproduce their sensors.

The application of 3D tactile imprints is less widespread than the direct use of raw 2D tactile images. 2D tactile images are usually used to train neural networks for different robotic tasks [START_REF] Li | Slip detection with combined tactile and visual information[END_REF][START_REF] Nathan F Lepora | Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch[END_REF][START_REF] Kim | Active extrinsic contact sensing: Application to general peg-in-hole insertion[END_REF]. 3D tactile imprints can be used for in-hand object tracking [START_REF] Sodhi | Patchgraph: In-hand tactile tracking with learned surface normals[END_REF][START_REF] Bauza | Tactile mapping and localization from high-resolution tactile imprints[END_REF] and shape mapping [START_REF] Wang | 3d shape perception from monocular vision, touch, and shape priors[END_REF][START_REF] Smith | 3d shape reconstruction from vision and touch[END_REF][START_REF] Smith | Active 3d shape reconstruction from vision and touch[END_REF][START_REF] Suresh | Shapemap 3-d: Efficient shape mapping through dense touch and vision[END_REF]. Shape mapping aims to reconstruct the whole shape of the object by combining the local 3D contact shapes measured by the tactile sensor. Their studied objects are rigid. The performance of their methods relies on the quality of tactile 3D reconstruction. The selected image-based tactile sensor in this research is DIGIT [START_REF] Lambeta | Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation[END_REF] because its availability, robust mechanical design, and compatibility with the robotic gripper used in our lab. The structure of DIGIT tactile sensor is shown in Figure 3.1. The key components include elastomer, lighting PCB and camera PCB. Elastomer is the interface for external contacts. When an object has a contact with this layer, it conforms with the shape of the object. The physical properties of the elastomer can be customized by using different mixture of materials to achieve the desired stiffness. When the sensor is used in high-loaded situations, stiffness should be properly increased. The thickness of this layer also affects the sensible range of depth. The surface of the elastomer is usually coated by a thin membrane of reflective material, which can reflect the internal light for the internal camera and block the external light from the environment. The lighting PCB comprises three LED lights at three different locations. They provide red, green and blue light sources inside the sensor and these lights are reflected by the coat of elastomer. When the elastomer is deformed by contact, the reflected lighting condition changes accordingly. This change is captured by the camera beneath. The photos shown in Figure 3.2 provide more realistic demonstration about the main components of DIGIT. Its technical specifications are listed in Table 3.1.

Nevertheless, DIGIT is not specially designed for 3D reconstruction due to the uncertainties of its internal illumination. On the sensing surface, the distribution of light and how the light changes under contacts are not well controlled. There are other commercial image-based tactile sensors, such as GelSight, with a similar internal lighting principle but with a more uniform light distribution on its surface that improves their use for 3D reconstruction. This difference comes from two factors. Firstly, the internal illumination of DIGIT is not designed to guarantee the uniform distribution of light on its sensing surface. Secondly, the fabrication of DIGIT is not strictly controlled since it is based on manual operations. Thereby, its light distribution is not uniform and quite uncertain, creating the aforementioned uncertainties. Therefore, we propose to develop a more robust 3D reconstruction procedure in order to use the DIGIT sensor, that is freely available to the scientific community with open-source components, in tactile-based robotic applications requiring 3D contact estimation where it could not be applied before.

In this chapter, the principle of 3D reconstruction is formulated and explained at first. Two different technical routes to implement 3D tactile imprint are presented in Section 3.3 and Section 3.4. The implemented 3D reconstruction methods are evaluated and compared in Section 3.5. The last section concludes this chapter.

Principle of 3D reconstruction

According to the theory of photometric stereo [START_REF] Robert | Photometric method for determining surface orientation from multiple images[END_REF][START_REF] Micah | Retrographic sensing for the measurement of surface texture and shape[END_REF][START_REF] Anish R Khadka | Object 3d reconstruction based on photometric stereo and inverted rendering[END_REF][START_REF] Durou | A comprehensive introduction to photometric 3d-reconstruction[END_REF], the normal of objects' surfaces can be estimated by observing the object under different illumination. The deformation of the elastomer in a vision-based tactile sensor changes the reflectance of the membrane layer coating on the top of the elastomer. The RGB 3-channel light's intensity captured by the camera is influenced by this reflectance. There is a nonlinear relationship between the light intensity and the surface normal of deformation. This relationship is used to implement the 3D reconstruction.

The height map of a surface could be described as:

z = H(x, y) (3.1)
where (x, y, z) refers to the 3D coordinates of one pixel in the tactile image. The surface normal at the location (x, y) can be expressed as [START_REF] Weisstein | Normal vector[END_REF]:

N(x, y) = ( ∂ H ∂ x , ∂ H ∂ y , -1) (3.2)
where ∂ H ∂ x (x, y) and ∂ H ∂ y (x, y) represent the height gradients along x and y direction, respectively. The relationship between the light intensity I(x, y) received by the camera and the shape of the reflective surface can be described by:

I(x, y) = R( ∂ H ∂ x (x, y), ∂ H ∂ y (x, y)). (3.3)
For each light source, we can have a function R to map the surface deformation to the observed light intensity. Because the light sources are mounted at the side of the sensing surface, one light source is not enough to cover all the deformed space. For DIGIT, we have three formulas as Equation 3.3 corresponding to each LED light. The inverse function of R can take the observed light intensity I(x, y) as input and output the height gradients. However, this reverse function R -1 is nonlinear and hard to be explicitly established. Height gradients can be integrated by solving Poisson Equation to get the height map H. Poisson Equation has the form as:

∇ 2 H = ∂ ( ∂ H ∂ x ) ∂ x + ∂ ( ∂ H ∂ y ) ∂ y (3.4)
where ∇ 2 refers to the Laplace operator.

In conclusion, the difficulty is to establish the relationship R -1 . Two different approaches are developed in the next two sections.

Lookup table for 3D

We can build a lookup table to map color intensity variation to height gradients. Color intensity refers to the three RGB values contained in the tactile image from the camera. Each set of color intensity has three values from 0 to 255 corresponding to the red, green and blue channels. The color intensity variation refers to the regulated color intensity difference between any tactile image frame and the base tactile image. The base tactile image is the one without any contact. There are two questions to answer. Firstly, why we do not use color intensity directly?

• For one tactile image from the camera, it is formed by the light from two sources. The first one is the light reflected by the reflective membrane on the surface of the sensor. This part is variable according to the deformation of the elastomer. The other one is the light statically existing inside the sensor. Using the light intensity variation can reduce the influence of the non-reflected light and create the conditions to apply Equation 3.3.

• The color intensity could contain three values up to 255, which means that if we want to cover all possible values, we should prepare an array table in the size of (255, 255, 255, 2). The "2" here is reserved for the two height gradients. For instance, the color intensity of a pixel is denoted as (r, g, b) that is the index for search. The corresponding height gradients stored at the location (r, g, b, 2) can be retrieved. This is not a small table to search values, which can lower the efficiency of the whole program.

Secondly, why we cannot use color intensity difference directly? The color intensity difference could be negative or positive, but the index of an array cannot be negative. To store all possible negative values in the lookup table, we need to extend the dimension of array, which leads to a lower search efficiency.

To avoid these drawbacks, we regulate the color intensity difference to compose a lookup table in the size of [START_REF] Wang | 3d shape perception from monocular vision, touch, and shape priors[END_REF][START_REF] Wang | 3d shape perception from monocular vision, touch, and shape priors[END_REF][START_REF] Wang | 3d shape perception from monocular vision, touch, and shape priors[END_REF][START_REF] Jingyi | Minimal work: A grasp quality metric for deformable hollow objects[END_REF]. This table can balance efficiency and accuracy. First of all, we use a ball bearing to deeply press on different locations of the sensing surface. By counting the maximal and minimal value in the array containing the color variation, we found that the color intensity difference is in the range of (-90, 90) as: ∆I raw ∈ (-90, 90).

(3.5)

By adding 90 to ∆I raw , we have:

∆I add ∈ (0, 180). ( 3.6)
Then we divide ∆I add by 180 and keep the float value as:

∆I f loat ∈ (0, 0.999). (3.7)
Finally we scale ∆I f loat by 90 and convert them as integers as:

∆I f inal ∈ (0, 90). (3.8)
∆I f loat can be scaled by other values, which is a trade-off between table efficiency and accuracy. A larger scale can increase the resolution of identification but also enlarge the size of the table.

After knowing how to dispose the three color intensity variations, we explain the method to handle height gradients. The methodology is to use one object whose dimension is known in advance to press on the sensor. We can calculate the height map of the contact using its geometrical model. Then the height map can be converted as height gradients by performing a convolution operation on each element of the height map. The adopted convolution operators are defined in Equation 3.12. Finally, these height gradients are mapped to the corresponding color variations in the lookup table.

The chosen object for pressing is a ball bearing. To start this calibration routine, two parameters should be determined. The first one is the diameter of the ball bearing. We used two ball bearings in different sizes. Measuring them by a caliper, the diameter of the larger ball is 11.10mm and the smaller one has 6.07mm. The second important parameter is the dimension of one pixel. The resolution of the tactile image is set to 320 × 240. To calibrate this parameter, we press a caliper on the sensor, as shown in Figure 3.3. The opening of caliper is 5.71mm. When it is pressed on the sensor, we can see its two top ends from tactile images. We manually select these two top ends and then the pixel distance between them can be computed. Given these data, we can know the dimension of one pixel. For each tactile image, we measure the pixel distance between two top ends of the caliper four times. The results of measurement are listed in Table 3.2. The average measurement is 18.6pixel/mm. Hence the dimension of one pixel is the reverse as 0.0538mm.

When one ball bearing is pressed on the sensor, its circular edge of contact on the tactile image can be detected. By image processing, we can know the center position and the radius of this circle in pixel. We can convert these values in millimeter by multiplying 0.0538mm. The method to calculate the depth at each pixel is illustrated in Figure 3.4. The point A is located on the circular edge. The distance between the ball center O and the point A is the radius of the ball R. The distance D AA ′ between the point A and the point A ′ is the diameter of the circular edge. Given the half of this distance and the ball radius, the vertical distance d1 can be computed as:

d1 = R 2 -(D AA ′ /2) 2 (3.9)
The point B represents any point on the contact surface. Its corresponding pixel in the tactile image is inside the circular edge of the contact. The distance between this pixel and the center of the contact circle can be calculated given the coordinates of this pixel. Converting the unit of this distance in millimeter, we can get the half value of the distance D BB ′ between the point B and the point B ′ . Then the vertical distance d2 is calculated as:

d2 = R 2 -(D BB ′ /2) 2 (3.10)
Therefore, the depth d at the point B is the difference between d2 and d1 as:

d = d2 -d1 (3.11)
For all the pixels inside the circular edge, the same method is applied to compute their corresponding depth. Therefore, for one tactile image, we can establish its height map by element-wise computation. d1 is constant for all pixels. d2 is updated for each pixel according to its distance to the center of the contact circle. The height of the element outside the circular edge is set as zero. In order to calculate gradients, we use the following two arrays to perform convolution with the height map, respectively:

A x =    0 0 0 0.5 0 -0.5 0 0 0    , A y =    0 0.5 0 0 0 0 0 -0.5 0    (3.12)
A x is used to compute Gradx and A y is for Grady. The pair of color variations and gradients is stored in the lookup table. Different pixels can have the same color variation. If the same set of color variations appears more than once, we compute and keep the average values of gradients. To be noted, only elements inside the circular edge are valid and used to set up the lookup table.

The determination of the circular edge comprises two stages, which is depicted in Figure 3.5. In the first stage, an image processing routine based on OpenCV [START_REF] Bradski | The OpenCV Library[END_REF] is set up to handle the difference array between the base frame and the frame with one circular imprint. The difference array has the form (320, 240, 3). The third dimension contains the three color variations. The maximum absolute value of these three color variations is kept and the other two are discarded to produce a grayscale image in the form (320, 240, 1). By setting a threshold, this difference image can be processed as a binary image. Then OpenCV's "findContours" function is used to calculate the contours in the binary image. The contour with the largest area is selected to be processed by OpenCV's "minEnclosingCircle" function To fill in the lookup table, we prepared one hundred tactile images pressed by ball bearing at different locations and depth. The location of press distributes on the entire sensing surface. At one location, we usually apply a hard press and a light press manually. The working flow to process each sample and build up the final table is illustrated in Figure 3.6. Most steps are explained in the above content. After handling all one hundred of the tactile images, some color intensity entries may not find their corresponding gradients. Therefore, the step "smoothing table" is used to fill in these potential vacancies and smooth the table. In the program, the occurrence of each color entry is counted. In the end, if the occurrence of one color entry is zero, its closest color entry paired with gradients can be found by computing their distance based on the coordinates (r, g, b). The gradients of its closest color entry are adopted. The smoothed table is exported as a file for later usage. Loading this table file, we can perform real-time 3D reconstruction. We use various objects to interact with the sensor. Some qualitative results of 3D reconstruction are shown in Figure 3.8. In each sub-figure, the top picture is the object touching the sensor. In the middle, the raw tactile image is on the left and the corresponding depth map is on the right. At the bottom is the rendered point cloud of the 3D tactile imprint. To prepare the point cloud, we get the coordinate (x, y) of each pixel in millimeter by using the calibrated dimension of pixel. The coordinate z is set as the reconstructed height. The point cloud is rendered with the help of Open3D [START_REF] Zhou | Open3D: A modern library for 3D data processing[END_REF]. Therefore, the sensor is able to capture the details of contact. In Figure 3.8d, the signature on the toy head is reconstructed with fidelity. The quantitative evaluation and comparison are presented in Section 3.5.

The working flow of the real-time 3D reconstruction is depicted in Figure 3.7. To generate a height map from a tactile image, it takes around 0.00725s. The lookup table file and the base tactile image are preloaded in the program. The tactile sensor produces raw tactile images in the resolution of 320 × 240 at 30 FPS. Processing each raw tactile frame with the base frame, the color intensity variation in the form of (320, 240, 3) can be generated, which is the input for the lookup table. The output of the table is in the form of (320, 240, 2). The third dimension of the array is for storing the two gradients. Using the Fast Poisson algorithm [START_REF]Fast poisson reconstruction in python[END_REF] to integrate the gradients, the height map of the whole tactile image can be obtained. The adopted algorithm is able to solve the Poisson Equation defined in Equation 3.4. Given the dimension of each pixel, the position in pixel can be converted to the actual position in millimeter. Given the height map, we can know the 3D real position of each pixel in the tactile image. Then this 3D tactile imprint can be rendered for visualization or used as the input of some tactile servoing pipelines.

Observing the rendered 3D effect in Figure 3.8, the color distribution is not continuous which implies some noise. There are two main reasons. Firstly, the lookup table only contains the information of color variation, which means that the same rule is applied on all pixels in the entire tactile image. In the central area of sensor, the lighting condition is relatively uniform and static. In the corners, however, the lighting condition is more complicated and dynamic. Secondly, the lookup table is discrete and further scaled down to the size of (90, 90, 90, 2). The height resolution is reduced. Even a small color variation could introduce some perceptible height noise. To improve this problem, we may have two possibilities:

• Can we build up a table with the information of pixel position? The current table has 4 dimensions. If 2D pixel position is further incorporated in the table, we have to prepare a 6D table which is too large to be feasible.

• We can use a neural network to replace the lookup table. To train this network, we can prepare the data comprising the coordinates of the pixel. In addition, neural networks can produce continuous height values. Thus it is a reasonable choice for the DIGIT sensor.

Neural network for 3D

As previously discussed, a neural network is expected to replace the previous lookup table.

A multi-layer perceptron (MLP) is adopted in our use-case since its performance is validated on GelSight [4]. The network has five input variables, including two positional variables and three color-related variables. Compared with the lookup table, the addition of two positional variables is expected to balance the performance of 3D reconstruction under different lighting conditions on the surface of sensor. The network has three output variables as the three coordinates of the normal direction of one location on the sensing surface which corresponds to one pixel in the tactile image. The vector of the normal direction is normalized. Instead of using the two gradients, the three elements of the vector of the normal direction are easier to be normalized in the range [0, 1], which can facilitate the training process. Since normal and gradients can be converted mutually, it is not a problem. Between the input layer and the output layer, three hidden layers are added to compose a neural network in the form of (5 -32 -32 -32 -3). The adopted activation function between layers are ReLU and Dropout function. The linear transformation is used to connect layers. The first step before training is to collect some tactile images pressed by a ball bearing. Since a small ball bearing can produce a sharper deformation thus a larger range of gradients, the 6.07mm ball bearing is used. In total, 30 tactile images pressed by ball bearing are collected. We manually choose the spots of pressing and try to distribute them evenly on the rectangle area of the tactile image. In fact, the goal is to use less tactile images to finish the calibration. In one tactile image, the press spot takes up a certain area of the sensing surface. In 30 tactile images, the press spots can almost cover the whole sensing surface which can result in a more complete dataset containing the data generated from the whole sensing surface. A smaller amount of tactile images cannot cover the whole sensing surface well. We also tried to use more tactile images to prepare the dataset but the performance of the trained network is not clearly improved. In contrast, more tactile images are used to set up the lookup table because we found that increasing the number of the used tactile images is able to reduce the vacancy of the table and achieve the averaged gradients that can better represent the whole sensing surface. Then we label these tactile images by following the same routine as shown in Figure 3.5. For each tactile image, we save its file name, the pixel position of the contact center and the contact radius in a csv file for the next step.

Using these 30 tactile images and the labelled csv file, we can prepare our dataset for training. For each tactile image, we generate a corresponding "normal image". Based on the annotation in the csv file, we can create a mask for the contact circle. The normal of Fig. 3.9 Computation of the normal of one contact point. pixels outside the circle is set as (0, 0, 1), which means that it is flat. Inside the circle, we can calculate the normal direction for each pixel based on the theory of the spherical coordinate system [START_REF] Weisstein | Spherical coordinates[END_REF]. We set up one frame at the center O of the ball bearing as shown in Figure 3.9. The goal is to calculate the normal vector at an arbitrary contact point P that corresponds to a pixel in the tactile image. The surface XOY defined by the x-axis and y-axis is parallel to the sensing surface. In the section MNPV of the ball bearing, the line MN is parallel to the y-axis and the line PV is parallel to the line HO in the surface XOY . In the base frame, if the coordinate of the point P is (x, y, z) and the coordinate of the center O is (x c , y c , z c ), then the value a of the angle PV M can be computed as:

a = arctan x -x c y -y c (3.13)
The distance between P and V is calculated as:

d PV = (x -x c ) 2 + (y -y c ) 2 (3.14)
The distance between P and O is the radius r of the ball bearing, then we can know the value b of the angle V PO as:

b = arccos d PV r (3.15)
Given a and b, the normal (n x , n y , n z ) of the point P can be determined by:

n x = cos a cos b n y = sin a cos b n z = sin b (3.16)
Then each component of the normal vector can be normalized by: After this processing, each tactile image can generate a paired "normal image" in the form of (320, 240, 3). The three channels contain the three normal components. Each "normal image" is saved as a csv file for later usage. In the csv file, the first row contains five items (x, y, n xnr , n ynr , n znr ).

n xn = n x /(n 2 x + n 2 y + n 2 z ) n yn = n y /(n 2 x + n 2 y + n 2 z ) n zn = n z /(n 2 x + n 2 y + n 2 z ) (3.17
To prepare the color data, two strategies are proposed. Firstly, the absolute RGB values of each pixel are directly used because this strategy is validated on GelSight [4]. For one tactile image, a csv file is created to store (x, y, r, g, b) for its pixels. Since each color absolute value is in the range [0, 255], we can normalize it in the range [0, 1] by dividing 255. Similarly, x is divided by 240 and y is divided by 320. As a result, all data are regulated in the range [0, 1]. For 30 tactile images, 30 csv files are created for (x, y, n xnr , n ynr , n znr ) and 30 csv files are created for (x, y, r, g, b). Then we combine and merge all these data together to create a larger csv file to store (x, y, r, g, b, n xnr , n ynr , n znr ). According to the value of the normal, data are classified into the zero group and the non-zero group. The zero group stores the data of pixels outside the labelled contact circle. The non-zero group stores the data of pixels inside the circle. We can sample a certain percentage of zero data to be merged with all non-zero data to compose the clean dataset. The suitable percentage of the zero data can be determined by trial and error. We evaluate the trained network in the manner as described in Section 3.5.

In our case, twenty percent is applied. Finally the clean dataset is split as the training set and the test set by ratio 4 : 1. However, as the comparison detailed in the next section, we found that in the non-contact region, the performance of the first strategy to prepare color data is not very good. Hence, the second strategy is proposed to improve it. We use the color variation between the contact tactile image and the non-contact tactile image to replace the absolute color value. The color variation can be normalized in the range [0, 1] by following the process defined by Equation 3.5, 3.6 and 3.7.

The proposed neural network is trained in the pixel-wise way. Each image can provide 320 × 240 pixels. Hence 30 tactile images are enough to provide a large dataset. The labelling and training is fast.

In real-time 3D reconstruction, we need to preprocess the streamed tactile frame before feeding it to the trained model. We transfer the image array as a list containing the coordinates and color information of each pixel. All these data should be normalized in the range [0, 1]. The color information could be the absolute color value or the color variation between the current tactile frame and the non-contact tactile image. Then the trained model can output a list of normal information for each pixel. We reshape this list in an array (320, 240, 3). Then we restore the normal values in the range [-1, 1] by the reverse operation defined in Equation 3.18.

Comparison and discussion

In this section, we present a thorough comparison and evaluation on the performance of the 3D reconstruction implemented by different methods. To study the influence of ball bearing in the calibration, two lookup tables are calibrated by two different ball bearings. The configurations of these two tables are listed as Table-11 and Table-6 in Table 3.3. To study the impact of color information, NN-abs and NN-var are trained based on the color absolute value and the color variation, respectively. The detailed training procedure is described in the previous section. We compare the results of the 3D reconstruction from four candidates with the "ground truth". The ground truth is generated by the ball bearing using the same principle as depicted in Figure 3.4. At first, we determine the contact circle in the tactile image by the routine defined in Figure 3.5. Then inside the circle, the geometrical model of the ball bearing is used to compute the 3D contact shape. Outside the contact circle, the height is set as zero.

The six pressing spots on tactile images are marked in Figure 3.11a. Three ball bearings are used to press around the spots on the sensor, whose diameter is 6.07mm, 7.14mm and 11.10mm, respectively. Each ball bearing is pressed around each spot twice. One press is hard and another is light. In total, 36 pressing images are generated for comparison.

At one spot, we use three different ball bearings to press six times. Thus we have six tactile images for testing. For one tactile image, besides the ground truth, four methods listed in Table 3.3 are employed to generate the corresponding height map. To better visualize and compare the results, we choose two profiles of each height map and plot them in the same figures. The profile is the contour of a section of the 3D contact shape on the sensing surface. Instead of using the entire 3D contact shapes, profiles are easier to be compared and the details of the difference can be visualized more clearly.

First of all, we find the maximum value of the ground truth and its row index and column index. The row that contains this maximum value is used as the first profile. The column with this maximum value is used as the second profile. These two profiles reflect the general We apply the same operation on all the height maps generated by four different candidates. The used indexes are same as these for the ground truth. These row profiles and column profiles are able to reveal the shape features of the reconstruction. We can know whether the reconstructed shape is smooth and natural, and if it is close to the shape of the ground truth, etc. We also compute the root mean square error (RMSE) between the entire height map built by each candidate and the entire ground truth. This indicator can help to compare the general error of the reconstruction. The profiles are demonstrated in Figure 3.12-3.17. Each figure corresponds to the situation at one spot. In each sub-figure, the upper plot is about row profiles and the lower plot reflects column profiles. Based on these measurements, we can have several observations:

• In general, the tracking performance of the two candidates using neural networks is more stable than the two candidates using lookup tables. For instance, the reaction of Table-6 and Table-11 at spot 1 is quite strong compared to the ground truth. However, at spot 5, their reaction could become quite small as shown in Figure 3.16a and Figure 3.16d. In most cases, the profiles generated by NN-var and NN-abs stay closer to the profiles of the ground truth. Lookup table applies the same color-deformation relationship on the whole sensing surface whose lighting condition is varied. In contrast, neural network is trained by the data incorporating positional information, which can better balance the performance of reconstruction at different spots.

• The curves of the profiles of NN-var and NN-abs are smoother than lookup table is discrete and its resolution is further reduced to scale down the table size.

In the smoothing step of the table as introduced in Section 3.3, some table vacancies could be filled by their neighbour's values. All these factors add some uncertainties to its prediction.

• In non-contact regions, the neural networks still produce some non-zero values. The output of the neural network is continuous. When we train NN-var and NN-abs, a certain number of zero data is added to the training set. We found that when the percentage of zero data increases, the prediction of the network at non-contact areas is closer to zero. But at the same time, its tracking performance at contact areas is weakened. Thus there is a trade-off. Usually, we determine an appropriate percentage of the zero data by trial and error to prioritize the tracking performance at contact areas.

• Comparing NN-var and NN-abs, we can find that NN-var is better at controlling the noise at non-contact areas. In the method of NN-var, the color variation at non-contact areas is maintained in a certain small range around zero. In other words, the zero data after processing present a certain pattern for the neural network. It is easier to learn a good reaction to the color variation in a fixed small range. In contrast, the absolute color value corresponding to the noise at non-contact areas is varied. It is hard to learn an uniform reaction to reduce noise in a large range. To reduce the noise from NN-abs, we have to add more zero data in the training set, which can result in the worse tracking performance at contact areas. It is difficult for NN-abs to achieve the balanced performance. NN-var can improve this point.

• For the lookup table, in most figures, its profiles at non-contact areas are quite flat and near zero. In fact, the integrated height at non-contact areas is quite unstable and usually has a negative value. In program, all negative values are truncated to zero. Therefore, its performance at non-contact areas is not natural and could become apparent like the profiles shown in Figure 3.15a and Figure 3.17a.

• The reconstructed profiles around the contact area are usually wider than the ground truth. When we use the geometry model of the ball bearing to generate the ground truth, the height map outside the perceptible contact circular edge is set as zero. However, in reality, around the contact circle some deformation is introduced as shown in Figure 3.18. The gel of the sensor cannot completely conform with the shape of the ball due to its softness. If a softer material is used to fabricate the gel layer, the conformation status between the object and the gel layer can be improved. The small color variation around the contact circle can be detected by the lookup table or the neural network to produce estimation. The RMSE between each candidate and the ground truth is listed in Table 3.4. The average RMSE of the lookup table is smaller because their output is usually zero at the non-contact area. The output of the neural network at the non-contact area is hard to be totally eliminated. But at the contact area, their profiles are more stable and smooth. Their RMSE values for different presses are closer. Conversely, the maximum RMSE of the lookup table could reach 0.0583 and the minimum value could be 0.0065. In general, the performance of the network is more stable at different locations of the sensor. Comparing NN-var and NN-abs, the average error of NN-var is much smaller than NN-abs. The advantage of NN-var on noise control and profile tracking is validated.

Conclusion

Using the DIGIT image-based tactile sensor, several 3D reconstruction methods of tactile imprints are developed to achieve high tactile resolution and sensitivity. The first method is based on a lookup table which can map the color variation of a pixel to its gradients. Integrating the gradients of all pixels in the tactile image, we can get the corresponding 3D height map. The dimension of the lookup table is designed to achieve a balance between efficiency and accuracy. The procedure to calculate data using the ball bearing geometry model and fill them in the table is demonstrated. As the internal illumination of DIGIT is not uniformly distributed on its surface, it brings difficulty to apply one fixed table to the whole surface of the sensor. In addition, the discrete output of the lookup table could aggravate noise.

To improve these shortcomings, a neural network is employed. Instead of merely using color information, the positional information of each pixel in the tactile image is incorporated in the dataset. For the convenience of normalizing data, the normal vector corresponding to each pixel is chosen as the output of the network. The algorithm to convert normal vectors into gradients is explained. Once gradients are known, the Fast Poisson algorithm is used to integrate them as the height map. Initially, the absolute color value is adopted as the input for the neural network. However, at non-contact areas, the output of the network is hard to converge to zero because in the training set, zero data's corresponding absolute color value is largely varied and hard to be distinguished from non-zero data's color value. To improve this problem, the color variation is adopted to replace the absolute color value. The color variation at non-contact areas is limited in a small range and can be better distinguished from the color variation pattern in contact areas. This improvement is validated in contrast experiments.

To evaluate all developed methods, a series of contrast experiments are designed and performed. The results validated that the performance of the neural network is more stable than the lookup table across the surface of a sensor with varying illumination. The network based on color variation has better performance than the network based on absolute color value in both of noise control and profile tracking. For image-based tactile sensors without uniform illumination design like DIGIT, the neural network based on color variation is one method to overcome the design defect of hardware and achieve better 3D reconstruction performance. 3D tactile imprints are able to facilitate various robotic manipulations as demonstrated in the next chapter.

Chapter 4

Tactile sensing for deformable object manipulation

Introduction

In this chapter, we demonstrate how to exploit the 3D reconstruction ability of vision-based tactile sensors in the context of object manipulation, especially for deformable objects.

The first researched scenario is about slippage detection which is a popular topic for the application of tactile sensors. Previous works explored different technical principles [START_REF] Romeo | Methods and sensors for slip detection in robotics: A survey[END_REF] to realize a sense of slip for robotic hands. One main methodology is to monitor the ratio between the tangential contact force and the normal contact force [START_REF] Song | A novel dynamic slip prediction and compensation approach based on haptic surface exploration[END_REF][START_REF] Song | Efficient break-away friction ratio and slip prediction based on haptic surface exploration[END_REF][START_REF] Kanno | Slip detection using robot fingertip with 6-axis force/torque sensor[END_REF][START_REF] Zhang | Multifingered robot hand dynamic grasping control based on fingertip three-axis tactile sensor feedback[END_REF][START_REF] Okatani | A mems slip sensor: Estimations of triaxial force and coefficient of static friction for prediction of a slip[END_REF]. If this ratio increases beyond the static friction coefficient between the robotic hand and the object, slip occurs. To measure multi-axial forces, a variety of sensors is designed based on sensing technologies like piezoresistive [START_REF] Arpa Mingrino | Slippage control in hand prostheses by sensing grasping forces and sliding motion[END_REF] and capacitive [START_REF] Novak | Initial design and analysis of a capacitive sensor for shear and normal force measurement[END_REF]. As multi-axial forces are needed, the structure of the sensor is relatively complicated. In addition, the adopted simplified friction model is not valid for soft materials. Vibration generated by the sliding between two contacted surfaces can be utilized as an indicator of slip. A piezoelectric sensor [START_REF] Jae S Son | A tactile sensor for localizing transient events in manipulation[END_REF][START_REF] Xin | Pvdf tactile sensors for detecting contact force and slip: A review[END_REF][START_REF] Yamada | Tactile sensor with 3-axis force and vibration sensing functions and its application to detect rotational slip[END_REF][START_REF] Shirafuji | Detection and prevention of slip using sensors with different properties embedded in elastic artificial skin on the basis of previous experience[END_REF][START_REF] Fujimoto | Development of artificial finger skin to detect incipient slip for realization of static friction sensation[END_REF] can produce a signal of dense fluctuation when slip occurs which is distinguishable from the normal signal. Temperature has high influence on its performance. Signal processing technologies like transform operations [START_REF] Cheng | Data correlation approach for slippage detection in robotic manipulations using tactile sensor array[END_REF][START_REF] Fernandez | Microvibration-based slip detection in tactile force sensors[END_REF][START_REF] Agriomallos | Slippage detection generalizing to grasping of unknown objects using machine learning with novel features[END_REF][START_REF] Antonio | Identification of slippage on naturalistic surfaces via wavelet transform of tactile signals[END_REF] and filters [START_REF] Vatani | Force and slip detection with direct-write compliant tactile sensors using multi-walled carbon nanotube/polymer composites[END_REF][START_REF] Romeo | Slippage detection with piezoresistive tactile sensors[END_REF][START_REF] Zollo | Restoring tactile sensations via neural interfaces for real-time force-and-slippage closed-loop control of bionic hands[END_REF] can help to isolate and reveal the slip signal. There are also some efforts to detect slippage using image-based tactile sensors. One method is to integrate some markers inside the deformable layer on the top of the sensor. Tracking the movement of these markers can retrieve the information of slip [START_REF] Yuan | Measurement of shear and slip with a gelsight tactile sensor[END_REF][START_REF] Ito | Robust slippage degree estimation based on reference update of vision-based tactile sensor[END_REF][START_REF] Dong | Improved gelsight tactile sensor for measuring geometry and slip[END_REF][START_REF] Taylor | Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger[END_REF]. The integration of markers increases the difficulty of the fabrication of sensors. The distribution of markers on the sensing surface is relatively sparse, which lowers the resolution of its sensing ability. One small object in contact may fail to cause an identifiable movement of markers. Some learning-based methods can process a series of consecutive image frames to identify whether slip occurs [START_REF] Li | Slip detection with combined tactile and visual information[END_REF][START_REF] Lambeta | Pytouch: A machine learning library for touch processing[END_REF]. But these methods require a time-consuming training procedure and more computation resources. Besides, waiting for multiple image frames to output a result can lead to a delay on the slip detection and prevention. The height-based slip detection approach proposed in this research can provide customizable high sensing resolution without the integration of the physical markers. Compared with learning-based methods, its prediction is more physically intuitive and faster by using just two consecutive image frames. Its effectiveness and generality are validated by different kinds of unknown objects. One slip avoidance scheme is implemented to regulate the action of the gripper accordingly.

The second scenario is to estimate and track the relative pose between the gripped object and the finger with the tactile sensor. PyTouch [START_REF] Lambeta | Pytouch: A machine learning library for touch processing[END_REF] is an open-source library for touch processing that enables the robotics community to process raw touch data from tactile sensors. This software library modularizes a set of commonly used tactile-processing functions valuable for various robotic tasks. In PyTouch, the color variation between the base frame without any contact and the current frame is exploited to estimate the contact area. This contact area can help to predict the contact pose between the object and the sensor. However, for DIGIT tactile sensors, the pattern of color variation under contact is not consistent on the whole sensing surface, which can lead to errors between the colorbased estimation and the real contact area. Since we can reconstruct the real-time contact shape as detailed in Chapter 3, one height-based method is proposed to estimate the contact area and then predict the contact pose. This contact pose can provide online guidance for manipulation. A similar method is reported in cable manipulation [START_REF] She | Cable manipulation with a tactile-reactive gripper[END_REF] using GelSight sensors. They demonstrated how to track the in-hand pose of USB cables. In our research using a DIGIT sensor, we further validated the performance of the height-based method with different objects in different dimensions. This method can estimate the orientation of the contact area. Sometimes the orientation of the contact area is the only concern of the problem like in the cable manipulation [START_REF] She | Cable manipulation with a tactile-reactive gripper[END_REF]. Sometimes we need to infer the pose of another part of the gripped object indirectly by estimating the contact area at first. For instance, by grabbing a pen, the pose of the pen tip can be obtained since we know the orientation of the contact area corresponds to the length direction of the pen. Given the relationship between the contact area and the concerned part of the gripped object, the inference can be achieved. This relationship is a geometrical transformation. The quality of the estimation on the contact area is the basis.

The third scenario is about the indirect contact-based tactile servoing. Compared to the common direct tactile servoing, its key feature is that the perceived contact between the object and the sensor is used to infer the contact state between the object and its environment. This inferred contact state provides guidance for the manipulation. As one typical application in the project SoftManBot, toy assembly can be facilitated by this technique. In factory, toy assembly requires diverse skills of proficient workers. A typical process can be separated into three steps:

1. Localize the two parts to be connected and approach one part against the other part.

2. Adjust the interaction between two parts until the worker feels it right. In fact, the contact tension between two parts can be perceived indirectly by workers' tactile feedback. Then workers can react to it. The felt contact state between the hand and the gripped object is used to perceive the contact state between two toy parts to be assembled, which is a scenario where we can apply the indirect contact-based tactile servoing.

3. Apply a proper twist to finish the insertion.

These steps are illustrated in Figure 4.1. In step one, workers use vision to localize two parts and get them closer. In step two, workers use feeling to adjust the contact status between the male part and the female part. When worker feels it right, one twist is applied and the connection is made as shown in Figure 4.1c. The involved feeling in this operation mainly relates to the tactile sensing. Inspired by this manual routine, cobots are expected to conduct the same operation to liberate workers from this repetitive and physical job. Therefore, an automation pipeline is proposed to implement this robotic manipulation task. In the first stage, a depth camera is used to perceive the 3D location of the hole of the toy head fixed by a robotic gripper. Then the toy body gripped by a 2-finger gripper is moved toward the detected hole and stopped at a certain distance. In the second stage, the toy neck is pushed toward the hole until a proper contact tension between the two toy parts is perceived by the tactile sensor. In the final stage, a predefined twist movement is applied by the 2-finger gripper to finish the insertion. Then the gripper can release the toy component.

The usage of tactile sensing is crucial for the second stage to achieve a proper interaction between the two parts to be assembled. If the interaction is too intensive, the protective stop due to high payload is triggered on the UR10 robotic arm used to drive the toy body. The manipulation is interrupted. If the compression between two toy parts is not tight enough, the correct connection cannot be accomplished in the third stage. The tactile sensor can perceive the extent of the interaction and regulate the relative motion between the two parts at the right moment in the second stage. The uncertainties existing in the end of the first stage make the usage of tactile sensing more indispensable. For instance, the distance between the toy neck and the hole is uncertain in the end of the first stage, which can easily cause a failure if only position is considered in the control of the robotic motion.

As this scenario of manipulation involves deformable objects and intensive physical interaction, it is difficult to find similar references in literature. Some existing works focus on the peg-in-hole task of rigid objects using tactile sensing [START_REF] Dong | Tactile-rl for insertion: Generalization to objects of unknown geometry[END_REF][START_REF] Kim | Active extrinsic contact sensing: Application to general peg-in-hole insertion[END_REF]. The dimension of the peg is smaller than the hole, the insertion task is converted as a geometrical alignment problem with ideal assumptions. In our case, the dimension of the peg is larger than the hole, a correct alignment is not enough to guarantee the success of the insertion. The deformation and compression between the toy parts should be considered and regulated under uncertainties. Finding a proper way to apply tactile sensing is challenging in this context. The concept of the sense of tightness is proposed for robots to perceive the interaction between gripper, object and environment. In light of this new sensory ability, robots are enabled to have an awareness of tightness between hand, tool and environment and react accordingly. This capability is useful when robots need to perform one delicate operation under uncertainties or explore unknown environments.

Slip detection and avoidance

To enable slip detection and avoidance, we exploit the real-time 3D reconstruction of the DIGIT tactile sensor. The NN-var method is employed because of its better overall performance which is explained and validated in Chapter 3.

The resolution of a raw tactile image is 320 × 240. In theory, we can have 76800 pixels to be regarded as the basic tactile units because each pixel is associated with one reconstructed height value. In practice, we can activate one pixel every ten pixels in a row and one pixel every ten pixels in a column. Thus a evenly distributed height mesh 32 × 24 is formed on the sensing surface with 768 tactile units in total that is still more numerous than most traditional tactile sensors like piezoresistive or capacitive sensors.

To judge the occurrence of slip, we compare two consecutive frames. The absolute variation of the height value on each tactile unit is summed up as a slip signal S , which can be formulated as:

S = n ∑ i=1 |h i t -h i t-1 | (4.1)
where n denotes the number of the activated tactile units, h refers to the height value at one pixel and t means the current tactile image frame. If this signal is larger than a threshold, slip is found. If the signal is always lower than the threshold, grasp is deemed to be stable. The threshold is usually set to twice the average value of the static slip signal when no slip occurs, which is proved effective in experiments. Using the overall variation of the contact shape to monitor slippage is physically intuitive and robust. In the following contents of this section, the advantages of this method are demonstrated with concrete examples.

Slip detection with unknown objects

We prepared some common objects in daily life to test our method, like tissue, socks, paperboard, plastic package, USB cable, etc. Some of the tested objects are listed in Figure 4.2.

For the objects less volumetric, like the towel, the tissue, the cable, etc, the 2-finger gripper executes a normal command to grab them first. Then we pull the grabbed object randomly to cause a slippage. If the signal S presents a distinguishable change at the moment when the slip occurs, we can associate this kind of signal change to slip. We plot the recorded signal S in Figure 4.3 to demonstrate its status before slip, in slip and after slip. For all these tested objects, slip can cause a dramatic signal change which can be clearly distinguished from the normal status and be regarded as the indicator of slippage. Even for fabrics like Fig. 4.2 Some objects used in the slip test. First row: towel, tissue, sock, paperboard, plastic package. Second row: USB cable, carton box, toy leg, toy body, toy head. socks, the signal change caused by slip is large enough to be a valid indicator. The tested objects in Figure 4.3 are deformable and their physical properties, like texture, hardness, geometry, etc, are quite diverse and representative. In the tests, the sock is grabbed from its opening and bottom, respectively, because the fibers used at these two locations are different.

When one object is gripped, the reconstructed contact shape on the tactile sensor is unique. When slip is triggered, this contact shape varies. The distributed tactile units capture this variation and the calculated slip signal can reflect it quantitatively. For USB cables, its slippage is an evident relative motion between the fingertip and itself. Thus the corresponding variation of the contact shape is also apparent. However, for the layer-like objects like fabrics, how to capture the subtle variation of the contact shape and generate an adequate slip signal when slip occurs? Thanks to the high resolution of 3D reconstruction, this is achieved. We can measure the contact details even the texture of fabrics. Although the general contact shape is less variable, the sliding of texture can stimulate the distributed tactile units and produce a slip signal large enough. The 3D reconstruction results of some tested objects are visualized in Figure 4.4. In the left of each sub-figure, it is the raw tactile image from DIGIT sensor which records the local contact information. In the right, it is the corresponding result of 3D reconstruction. For the USB cable, the general contact shape is clear. For other layer-like deformable objects without a solid general shape, we can still get their 3D textures. When they are sliding in grip, the distribution of texture changes, which leads to a detectable change on the slip signal. Identifying the change of the physical contact shape as the indicator of slip provides high intuitivity and universality. Also, the height map overcomes the problem of the uneven distribution of lights on the sensing surface. Thus the sensitivity of slip detection is more uniform on the whole sensing surface.

Slip detection in different directions

For the objects less volumetric, like tissue, cable, etc, it is more practical to pull it from grip to introduce a slippage. However, for volumetric objects, contact conflicts with the environment could introduce a slip in all directions with reference to the coordinate frame of the gripper. How the height-based method performs when slip happens in different directions is the focus of this subsection. Four different objects are studied, including the carton box, toy leg, toy body and toy head as shown in Figure 4.2.

The carton box is a hexahedron and each of its faces is quite flat. In the tests, the grip is initiated with the two largest surfaces to avoid the influence of edges. We tried to control the relative motion of the carton box in the horizontal plane which is parallel with the surface of fingertips. In the first trial, the relative motion of the box is faster and we moved the box more slowly in the second trial. In the direction vertical to the surface of fingertips, we applied a relative motion like pressing the box toward the sensor. This motion is also regarded as a kind of slip since the contact between the object and the fingertip without the sensor becomes less tight. The corresponding slip signals are depicted in Figure 4.5. In all these situations, the slip signal corresponding to the status of slip is easy to be identified from the normal status. For the slip caused by a sudden interference in the tests, a signal impulse can be observed. For a slower relative motion, a continuous signal variation presents in a longer period.

Besides the carton box in a regular shape, we also tested the toy components in complicated shapes. In order to define the direction of slip, one coordinate frame is adopted as illustrated in Figure 4.6. Z-axis is parallel to the axis of gripper. X-axis is perpendicular to the top side of the tactile sensor and Y-axis is perpendicular to other two axes. The dashed A-axis denotes the geometrical axis of the toy component itself. After each toy component is gripped, a small translation is applied on the toy component along each axis in the coordinate frame, respectively. A small translation along X-axis is possible since the surface of the sensor is deformable. Besides, a small rotation is applied around its own axis. We did not test the small rotation around each axis in the coordinate frame because this kind of motion is similar to the previous tested motion on a small scale. We performed the same tests on toy leg, toy body and toy head. The recorded slip signals are plotted in Figure 4.7. For the tests along X-axis, the signal peak of the toy leg is much higher than the other toy components. The hardness of the toy leg is higher than the others since the hollow structure inside the toy head and the toy body is larger which can lower the resistance of the surface to external contacts. A harder object can give stronger stimulation to the sensing area of the tactile sensor. Along Y-axis, the signal peak of the toy leg is lower than other tests on it and is close to the signal peaks of the toy body and the toy head. The reason is related to its initial pose in grip as shown in Figure 4.2. When a small translation is introduced, the sensed contact area changes less than the tests on Z-axis and A-axis. The sensed contact depth changes less than the test on X-axis. Along Z-axis, the signal peaks of the toy body and the toy head are close to their peaks measured along Y-axis and are lower than the signal peak of the toy leg. Because the dimensions of the toy body and the toy head are larger than the sensing surface of the tactile sensor, a larger contact area is already achieved in the initial grip. A small translation in the plane defined by Y-axis and Z-axis could not cause a larger variation on the achieved contact area. In addition, the higher softness of the toy body and the toy head makes the contact region flatter, which is not in favour of the generation of higher signal peaks. In contrast, the initial contact between the toy leg and the tactile sensor occupies a smaller percentage of the whole sensing surface. A small translation can lead to a displacement of the whole contact area on the sensing surface, which can cause an obvious signal change. The rotation can result in a large signal peak for all toy parts. We can conclude that the height-based method is able to detect the slip that occurs in different directions.

Density of the tactile mesh

The density of the used tactile units can be easily adjusted by choosing certain pixels in the tactile image. The selected tactile pixels compose a sensing array, which is termed as a tactile mesh. In the densest case, all the pixels are used to form a 320 × 240 tactile mesh. Picking up some pixels in a certain pattern can create a tactile mesh in the form 160 × 120, 32 × 24 or 4 × 3, etc. In the sparser tactile meshes, the height values on the pixels that are not selected are discarded. Then the question would be what impact the density of the tactile mesh has on the performance of slip detection. To reveal this concern, two groups of experiments are conducted.

Firstly, the towel shown in Figure 4.2 is tested. By setting the adjacent spatial interval between each activated pixel, tactile meshes are prepared in the form of 320 × 240, 64 × 48, 32 × 24, 16 × 12 and 4 × 3. Each tactile mesh is evenly distributed on the entire sensing surface of the tactile sensor. Without changing hardware, we switch these tactile meshes to measure the slip signals when one slip is introduced by pulling the gripped towel. The parameters to control the initial grip are set as the same for all tests. The recorded slip signals are plotted in Figure 4.8.

For these five tactile meshes, a slip is able to provoke a similar reaction. The difference is reflected on the absolute value of their slip signals. For a denser tactile mesh, the value of its slip signal is higher, including the signal before slip, during slip and after slip. Before slip and after slip, the accumulated noise from more used tactile units can reach a higher value. During slip, more tactile units are impacted and counted to produce a higher signal peak. However, the scale between the signal peak and the signal before or after slip is quite similar for all these tactile meshes. It means that using any tactile mesh from them can make slip distinguishable easily. The influence of the tactile density on the capability of slip detection cannot be verified in this situation. The possible reason is that the contact between the towel and the tactile sensor always takes up the whole sensing surface. When the towel is pulled externally, this action is able to stimulate the entire sensing surface. When the density of the tactile mesh is changed, the value of slip signal is scaled but the pattern of signal keeps.

A higher resolution should present its advantage when a subtle slip needs to be detected in a small contact region. To verify this supposition, the following experiments are performed. Three hex keys in different dimensions are tested as shown in Figure 4.9a. In the first step, a grip is initiated with the small hex key, the medium hex key and the large hex key, respectively. Their initial tactile images after grip are shown in Figure 4.10. Then we slightly push one end of the gripped hex key to introduce a slight slippage. Each hex key is tested twice with a 32 × 24 tactile mesh and a 8 × 6 tactile mesh. In the tactile mesh, the tactile units are evenly distributed on the sensing surface.

The results of tests are illustrated in Figure 4.11. For the large hex key, both of the tactile meshes are able to generate an identifiable signal peak to mark the occurrence of slip. For the small hex key and the medium hex key, the denser tactile mesh can function as usual.

Nevertheless, the sparse tactile mesh cannot produce a distinguishable signal at the moment of slip. Thus the density of the tactile mesh influences its capacity to detect the slight slip of objects having a small contact region with the sensing surface. When a sparse tactile mesh is used and the contact region is small, a slight slip can merely cause a small change on this contact region that cannot stimulate sufficient tactile units to produce a large signal peak.

In contrast, a dense tactile mesh is more sensitive because a small change of the contact region can swipe on a lot of tactile units. If the density is not high enough, we have to expect that the small contact region changes dramatically or the contact region is large enough in order to swipe on more tactile units. The guidance is that if the manipulated objects having a large contact region with tactile sensors, the low density can be used, but if the contact region is relatively small, the high density is necessary to guarantee the high sensitivity of slip detection.

Slip avoidance

After the detection of slip, robot should react accordingly to prevent the further slippage and even the drop of the gripped object. A slip avoidance pipeline is developed as depicted in Figure 4.12. The left column describes the workflow to achieve the slip signal from the tactile image. The real-time slip signal is fed to the state monitor which decides whether the gripped object is in slip. After the initial grasp on the object is completed, the state monitor is launched. Before the state monitor is able to work, it sends a request to the module named "Adaptive threshold" to get the proper threshold value to judge if the received slip signal indicates a slip. Usually the "Adaptive threshold" module calculates the average value of the 30 consecutive slip signals in the normal state without slip and returns twice this average Fig. 4.13 Grip in two modes. In the left, the larger grip force makes the contact tighter and thus there is much less space between the sensor and the object.

value as the response to the state monitor. Twice the average value of the no-slip signal is proved as an effective threshold in experiments. Then the state monitor uses this returned value as threshold to be compared with the real-time slip signal. If the slip signal is always lower than the threshold, the program stays in the "No slip" mode. The normal grip force is applied as the initial grip. In this case, no action is taken by the gripper. The robot can execute its operation as normal. When a slip signal larger than the threshold is sent to the state monitor, it turns on the slip mode and a larger grip force is applied by the gripper. Then the timer starts to count if the state monitor does not output "slip" state in continuous three seconds. If the "slip" is counted, the program is still running in the slip mode, the large grip force is not released and the timer restarts the counting in another three seconds. If the timer does not receive any slip output from the state monitor in three seconds, the program judges that the slippage is stopped or passed and thus switches back to the "No slip" mode. The normal grip force is reapplied to avoid the possible damage or any negative effect on the gripped object. However, changing grip force can cause the variation of the contact shape and result in a large slip signal that could be mistakenly regarded as "slip" by the state monitor. To avoid this kind of misjudgement, the "Monitor manager" module is added to the program. When the gripper is in motion, the state monitor is disabled temporally by the monitor manager. After the motion is finished and the gripper becomes static again, the monitor manager can relaunch the state monitor. Each time the state monitor is relaunched, it requests an updated threshold from the module "Adaptive threshold" since the average value of slip signals could vary a bit after the change of the contact shape between the object and the tactile sensor.

This pipeline is tested with different objects. We manually interfere the gripped object to create some slips and then observe the reaction of the gripper. When the slip mode is activated, the object is gripped by the larger force. If the interference persists, the slip mode keeps. When the timer and the state monitor decide to turn on the no-slip mode, the grip force is reduced. When new interference is applied, the slip mode is switched on again. This process runs smoothly and the different modes can be told by observing the action of the gripper and the resulted contact status between the object and the gripper. For instance, the toy head gripped in the slip mode and the no-slip mode is shown in Figure 4.13a and Figure 4.13b, respectively. The grip in slip mode is stronger. From the perspective of the slip signal, we can also validate the effectiveness of the proposed pipeline. The toy arm is initially gripped as shown in Figure 4.14. Then we interfere the grip five times from different directions. Each interference caused the slip mode and the no-slip mode is recovered after a period without interference. The recorded slip signal can reflect this mechanism, as depicted in Figure 4.15. From left to right, every two signal peaks are in a pair as highlighted by the red rectangle. We have five pairs in total corresponding to five interferences. In each pair, the first peak represents the signal variation caused by the slip and the stronger grip, and the following peak represents the signal variation caused by the reduction of grip force when there is no more slip. With the help of the module "Monitor manager", the second signal peak in each pair is not treated as slip. The proposed pipeline of slip avoidance works efficiently when slip occurs and flexibly when slip is gone.

In-hand pose estimation 4.3.1 Height-based pipeline

When the vision system is absent or blocked, tactile sensors can still function. In some operations, the relative pose between the gripped object and the finger of the gripper should be sensed to guide the movement of robots. For example, a pen is grasped by a 2-finger gripper. Before it can be put back in a pen container, the orientation of this pen should be determined at first. Then the robotic arm can adjust its pose accordingly to make sure that the direction of the pen is proper to be fit in the container. The tactile sensor can measure this orientation by making use of the reconstructed 3D contact shape. The proposed pipeline is described as below:

1. Get the height map from the tactile image by the 3D reconstruction method NN-var that is demonstrated in Chapter 3.

2. Filter the noises in the height map. In the height map, all height values less than 0.06mm are set as zero. 0.06mm is determined because the measured maximum noise is usually around 0.05mm when the tactile sensor has no contact with anything.

3. Get the maximum value of the current height map. The half of the maximum value is set as the current threshold to transform the current height map as a binary image. Each height map generated from a tactile frame produces its own threshold. In the height map, the height values lower than the threshold are set as zero and the values larger than the threshold are set as 255. Choosing the half of the maximum height as the threshold is for segmenting the most essential part of the contact shape. As the orientation of this segmented part is used to infer the orientation of the gripped object in the end, a reasonable segmentation is important. The full contact shape or the smaller contact region near the height peak risks losing the correspondence between the segmented contact shape and the general shape of the gripped object. The full contact shape may contain some uncertainties in the region of light contact. The smaller contact region near the height peak could be highly impacted by the local design of the object. In trials, setting the half of the maximum height value as the threshold can provide more stable performance. To be noted, as the maximum height value of each tactile frame is varying, the threshold is updating dynamically.

The threshold should be larger than 0.06mm. If not, the continuous execution of the program is suspended until this dynamic threshold is greater than 0.06mm again. The reason is that 0.06mm is also the value used to filter noise. There could be a conflict between the noise filtering and the contact shape segmentation. Fortunately, in practice, a normal grip on a volumetric object can easily cause a height value much higher than 0.12mm, which means the threshold is larger than 0.06mm.

4. With the help of OpenCV [START_REF] Bradski | The OpenCV Library[END_REF], find contours in the binary image.

5. Fit an ellipse for the largest contour. The orientation of this ellipse represents the orientation of the contact shape and thus the part of the object in contact with the sensor. The assumption is that the contact shape in the sensor can be mapped to the general shape of the gripped object around the contact area. For instance, the contact shape of pens or cables in the sensor can be mapped to the surface of the object. The axis of the contact shape corresponds to the axis of the object.

Another method to measure the orientation of the gripped object is implemented by tracking the color variation between the current tactile frame and the reference frame in the state of no contact. This method is open-sourced with PyTouch library [START_REF] Lambeta | Pytouch: A machine learning library for touch processing[END_REF]. The pipeline of this method is described as below:

1. Get the color variation between the current frame and the reference frame. The color variation contains three channels. For each pixel, the channel with the largest absolute value is kept and the other two channels are discarded. In the end, the tactile image in the form (320, 240, 3) is transformed as a grayscale image in the form (320, 240, 1).

2. Blur the grayscale image using a 2D filter.

3. Convert the blurred grayscale image as a binary image by threshold. Then erode it.

There is no basis for the choice of threshold. It cannot be generalized.

4. With the help of OpenCV [START_REF] Bradski | The OpenCV Library[END_REF], find contours in the binary.

5. Fit an ellipse for the largest contour. The orientation of this ellipse represents the orientation of the contact shape and thus the part of the object in contact with the sensor.

Compared with the height-based pipeline, the color-based method has serious defects because the color variation of the DIGIT sensor under contact possesses high uncertainties. We conducted several experiments to demonstrate the advantage of the proposed height-based method and the drawback of the color-based method.

Static experiments

In static experiments, we press the hex key and the toy leg on the DIGIT sensor, respectively. Then two pipelines are launched to process the tactile image and infer the in-hand pose. The results are depicted in Figure 4.16 and Figure 4.17.

In Figure 4.16b, the contact area with the height larger than the half of the maximum height is segmented in the binary image. Fitting the white region in an ellipse, the long axis of the ellipse can represent the orientation of the contact. The final result shown in the second window is quite precise since the estimated red axis is closely aligned with the ridge line of the hex key. The color-based method needs more steps of image processing to get results. In Figure 4.16c, the first window records the raw color difference between the tactile frame in contact and the reference frame without contact. The second window records the grayscale image after processing. Two color channels are trimmed off. In the kept channel, the color values larger than a threshold is reset as the same to be distinguished from the other area with only a small value of color variation. This window reveals that the color variation around the contact area is not uniform. We can see a dark line in the middle of the gray area caused by the ridge of the hex key and the scattered gray dots at the bottom of the contact area. Thanks to the blurring, we can get a more complete contour in the third window. The fourth window records the binary image after threshold and the final contour after eroding is shown in the fifth window, which is fitted by an ellipse. Although the final matching effect in the last window is not bad, the used contour cannot reflect the real contact shape.

For the objects with a simple exterior contour like hex keys, the problem of the colorbased method is not very obvious. But for objects with more natural exterior shapes, like the toy leg, the height-based method clearly outperforms the color-based method. In Figure 4.17c, the difficulty to handle the complicated color variation of the DIGIT sensor is exposed. Due to the uneven color change, the grayscale image in the second window fails to constitute a complete contact shape and leads to the malfunction in the following steps. In the end, the prediction marked by the ellipse is wrong in both of orientation and location. In contrast, the prediction depicted in Figure 4.17b is much more reasonable. For each object, we rotate it twice in the direction marked by the green arrow in the figure.

The height-based method and the color-based method are adopted, respectively. The initial position of the object in the grip is kept as similar as possible. During rotation, the inferred orientation is recorded as the angle value between the gripped object and the tactile sensor.

For the hex key, a sequence of the tactile images produced by the height-based method is demonstrated in Figure 4. [START_REF] Xydas | Modeling of contact mechanics and friction limit surfaces for soft fingers in robotics, with experimental results[END_REF]. The images in one column are in a pair generated at the same moment. The binary image is segmented by height and the estimated ellipse is drawn in the raw tactile image. In the row, the images record the varying in-hand pose of the hex key in the chronological order. The binary images reveal that the height-based segmentation is quite stable during tracking. The second row presents the good tracking precision. We can compare the direction of the red long axis with the contact edge of the hex key. As a dynamic test, we plot the output angle of the height-based method during rotation in Figure 4.22a. The output angle is defined as the angle between the red major axis of the ellipse and the long side of tactile images as depicted in Figure 4.20. One arrow represents the direction of the red major axis and another represents the direction of the image edge. The angle formed by these two arrows is marked by the green arc. Observing the angle curve, its overall variation is smooth and its noise in the local scale is inconspicuous. For comparison, the color-based method is applied to estimate the angle during the rotation of the hex key. This process is recorded in Figure 4.21. Each row contains the four key steps to get the estimation for one tactile image. From left to right, there are the grayscale image of the color difference, the blurred grayscale image, the binary image and the estimated ellipse drawn in the raw tactile image. From top to bottom, each row corresponds to one moment in the rotation. Except the first row and the last row, the inference in other rows has obvious error. The quality of the binary images are not stable during rotation because the pattern of the color difference is not stable. The output angle of the color-based method is depicted in Figure 4.22b. After the hex key reaches 100 degrees, the output angle becomes unstable and the local noise is amplified. The color distribution of the DIGIT sensor is uneven and unstable on the sensing surface. Relying on the color difference is not wise. In contrast, the developed 3D reconstruction technique can compensate the hardware defect as detailed in Chapter 3 and lead to a more stable tracking performance. The pen is rotated in the reverse direction of the hex key. The progress of the height-based method is shown in Figure 4.23 and the progress of the color-based method is shown in Figure 4.24. The height-based method is able to provide more reliable estimation on the in-hand poses. The color-based method has obvious error in the third row seeing that it is hard to get the correct segmentation from the fragmented grayscale image in the first column. We further compare their angle curves in side of the tactile image) and then the angle is immediately reset as zero. The maximum angle value is 180 degrees in the program. When the red major axis is perpendicular to the short side of the tactile image, 0 degree and 180 degrees are equivalent. After this switch moment, the performance of the color-based method becomes quite unstable. In the first column of Figure 4.24, we can find that the contents in the image become more fragmented after the switch moment. When a human is operating tools, the tightness between the fingers, the tools and the environment can be felt and adjusted to complete a dexterous task, like driving a screw in a hole. In operation, when the tissue around the contact location is fully stretched or pressed, we have a tight feeling. If human tissue is replaced by the sensing surface of the tactile sensor, the tactile image should be quite static without much variation. In contrast, if the tool is not well held in hand, the state of tissue in contact is unstable since the tool can move around in the grip. Similarly, the tactile image should vary unstably. Before the start of screwing, a tight contact status between the hand, the tool and the screw is expected. Humans use this awareness of tightness to facilitate the manipulation on all kinds of objects. By making use of tactile sensors, robots are also able to feel this kind of tightness in order to perform difficult tasks, like the assembly of deformable objects. The signal S defined by Equation 4.1 can be used to measure the stability of the contact shape. A stable signal S suggests a stable contact between the sensor and the gripped object, and hence a grip tight enough for the current robotic operation. When this signal changes drastically, the grip is not tight enough anymore in the current operation. When this signal gradually converges toward a static value, the grip is being tightened again. By feeling the tightness, we can regulate the action of robots according to the requirements of operations.

When the approximately uniform motion is applied on the end-effector, the value of the signal S in a time interval can reflect the evolution of the physical tightness between the sensor and the gripped object. The motion of the end-effector in a time interval can be regarded as a stimulation on the signal S. The approximately uniform motion can provide the approximately equivalent stimulation in a time interval. Under this stimulation, a tighter grasp state can lead to a smaller signal S. Reversely, if the measured signal S decreases during operations executed at roughly constant speed, the tightness of the contact state between the gripper, the gripped object and the environment can be inferred. During the approximately uniform motion of the end-effector in a robotic operation, the relative tightness T is represented by the value of the signal S at the moment t n , denoted as:

T = S t n (4.2)
The sampling frequency F on the signal S has significant impact on the value of the tightness, which is equal to:

F = 1 t n -t n-1 (4.3)
where t n denotes the current sampling time and t n-1 refers to the last sampling time. If the frequency F is too high, the value of the signal S could be too variable to reflect the general trend of the felt tightness. If the frequency F is too low, the measured tightness loses the real-time feature. The adequate frequency also depends on the motion speed of the end-effector. When the end-effector moves at a higher speed, the real-time feedback is important to regulate the action of the end-effector in time to avoid the damage on the operated objects or tools. Hence a higher frequency F is needed. If we slow down the speed of the motion of the end-effector, a relatively low frequency F like 10 Hz can be used. This frequency can better capture the tightness change in a longer time interval to fully perceive the evolution of the tightness of the contact between the gripper, the object and the environment in the operation. Usually, a smaller value of T suggests a tighter status. The approximately uniform motion can comprise the translation and the rotation. The value of the signal S characterizes the degree of the height signal change on the sensing surface of the sensor. The degree of the height signal change reflects the degree of the stability of the contact. The speed of the motion of the end-effector characterizes the degree of the stimulation to the height signal change. Without this stimulation, the height signal stays static. The tightness implies how the height signal changes under the similar stimulation. A tight enough contact can reduce the impact of the stimulation, resulting in a small height signal change.

Choice of tightness

As explained in Section 4.1, the assembly is performed in three steps:

1. Visual alignment.

2. Interaction based on tightness.

Insertion by a twist.

The most crucial step is to regulate the interaction between the two toy parts according to the felt tension, i.e., the second step. Before we can answer what kind of tightness is desired between the two toy parts, we should have a look at the profile of the felt tightness during the operation. The hardware setup is shown in Figure 4.32. In this subsection, we focus on the step related to the tactile servoing. The neck of the toy body is already aligned with the hole on the toy head. Then the toy body is driven toward the hole at the speed around 25mm/s. The 3-finger gripper is used to fix the toy head and does not move. At the beginning of the motion, there is no contact between the two toy parts. Then the toy neck has the initial contact with the toy head. The intensity of the physical interaction between two parts is increased while the motion of the robotic arm continues. In the end, we can plot the signal S generated in this process in Figure 4.33. The motion of the end-effector starts from around 1.5s. The contact between toy parts occurs at around 3s since at this moment the signal S has a drastic change. Then the signal reaches its peak fast. As the two toy parts are gradually compressed, the signal goes down until the motion of the end-effector is stopped forcibly by the robotic arm just before 5s. This triggered protective stop due to high payload causes a drastic signal change that forms the second signal peak in the plot. Between the two signal peaks, we can have a sense of tightness by monitoring the decline of the signal. Before the physical contact of the toy parts is too tight to lead to the protective stop of the robotic arm, we can regulate the motion of the end-effector actively.

To understand the feature of this curve, we should observe the process of interaction. From the moment of the initial contact to the occurrence of the protective stop, the interaction between two toy parts can be divided into two stages. In the first stage, the slip of the toy body causes the drastic change of the signal. The slip occurs because the bottom of the toy body near the palm of the gripper is not supported at the beginning, as shown in Figure 4.34a. This problem is noted as "Uncertainty 2". The compression between the toy neck and the toy head can generate a force to push the toy body toward the palm of the gripper. As shown in Figure 4.34b, after the toy body is stabilized by the support of the palm of gripper, the interaction enters the second stage. The compression between the toy neck and the toy head can be transmitted to the contact area between the tactile sensor and the side of the toy body. With the intensification of the compression between the toy neck and the toy head, the compression between the tactile sensor and the toy body is also intensified. This phase leads to the convergence of the signal. The whole interaction must experience the second stage about compression. If initially the toy body is closely put next to the palm of the gripper, the first stage about slip may be not obvious. The transition between the two stages is continuous without a clear demarcation. Declining from the peak, different signal values reflect different tightness. We should choose a certain signal value in order to mark our desired tightness. If a very tight contact status is pursued, we should choose a desired value slightly larger than the value of the static signal before the contact between two objects or the final convergent value. As the demand on the tightness of the contact decreases, we can increase the desired tightness value. For example, the signal value before contact in Figure 4.33 is around 0.9. A signal value 1.2 refers to a very tight contact status in operation. A signal value 2.0 refers to a medium tightness. The proper way to determine an ideal tightness is to know the signal value before contact at first. Based on this value, we can increase it to lower the tightness. Once one tightness is chosen, we can validate it in operation. If the achieved contact status satisfies the requirements of operation, we keep this tightness for usage. If not, we can further increase or reduce it until we find the proper tightness value for operation.

Specifically, we want to find the contact status with a medium tightness for the two toy parts. If the contact is too tight, it runs the risk of triggering the protective stop of the robotic arm due to high load or breaking the grip on the toy head. These two situations lead to the failure of operation. If the contact is too light, the final insertion step cannot make the two toy parts connected correctly. Therefore, we need to regulate the tightness in an adequate range to guarantee the success of operation. One strategy is to monitor the evolution of the signal value in operation. When the signal value reaches the desired tightness, the robotic arm is halted and the next operation is ready to be executed.

Why a relatively tight contact can make the toy assembly easier? There are two reasons. In tests, we found that when the contact between two toy parts are tightened, the deformation of the toy parts is well released and they become stiffer. Wrenches are easier to be transmitted between two stiffer objects to overcome the resistance against the connection. The theory about wrench transmission is explained in Chapter 2 about grasping. Another reason is related to the shape of the region around the hole on the toy head. The deformed region can reduce the resistance for connection.

In some experiments, the Moveit library [START_REF] Görner | Moveit! task constructor for task-level motion planning[END_REF] is used to control the motion of the endeffector. After the toy neck and the hole of the toy head is aligned, we set a desired Cartesian position goal for the end-effector. Before the end-effector can reach that position, the motion of the end-effector is stopped in advance according to the measured tightness. From the initial contact between the toy parts to the moment that the robot is commanded to stop, the motion of the end-effector can be assumed as the approximately uniform motion by setting the speed limit of the robot. The 2-finger gripper moves along the Z-axis marked in Figure 4.32 toward the hole of the toy head. In the step of insertion, a twist comprising a rotation of 120 degrees around the Z-axis and a translation of 15mm along the Z-axis is applied to accomplish the assembly.

When the desired tightness is set as different values, the motion of the end-effector is stopped at different contact states. We keep the motion parameters as the same but change the value of the desired tightness that can determine when to stop the motion of robot. The recorded tightness in the tests is shown in Figure 4.35. In the first test, the desired tightness value is set as 4. In Figure 4.35a, when the tightness value declines to 4, the robot is commanded to stop and then the tightness signal fast drops down to the static values. The resultant contact state between the toy parts is not tight enough and the following insertion did not succeed. In the second test, the desired tightness is set as 2.5. In Figure 4.35b, the motion of the end-effector lasts longer and stops after the sensed tightness reaches 2.5. When the signal value fast drops to the level of the static signal value, it means that the motion of the end-effector is terminated. This tightness makes the following insertion succeed. In the third test, we set the desired tightness value as 1.2. Before this value can be achieved, the protective stop of the robot is triggered first which leads to the failure of the operation. Therefore, choosing a proper signal value as the desired tightness is important to achieve the ideal interaction between objects, which can decide the failure or success of tasks. As "loose" and "tight" is a relative concept, setting the desired value in a reasonable range is usually enough for the success of operation. Humans only have a rough feeling on tightness but that is enough to enable their manipulations. How to find this reasonable range depends on the initial signal value before contact and some trial and error. At first we can plot a signal curve in operation as a reference. Once a proper tightness value is determined, we can apply it to the same task under uncertainties. The robustness of the tightness to uncertainties is explained in the next subsection.

Robustness of tightness

As indicated in Figure 4.34a, some uncertainties exist in the experimental configuration. The "Uncertainty 1" refers to the distance between the toy neck and the hole, as it is not precisely controlled. After the step of visual alignment, this distance may have error with reference to the target distance. How the toy head is placed can also influence this distance. Hence it is hard to set a universal position goal for position control. The "Uncertainty 2" refers to the unknown distance between the toy body and the palm of the gripper. Just controlling the position of the toy neck cannot ensure a stable contact between the bottom of the toy body and the palm of the gripper. This contact is important to keep the stability of the grasp on the toy body in the final step of the assembly. Thirdly, the pose, deformation and physical properties of the gripped toy head could be uncertain. For instance, we need to deform the region around the hole to make it stiffer to facilitate the final insertion step. But the stiffness of each toy head could vary and it is hard to set a predefined movement to ensure the ideal state. In addition, even for a specific toy part, the ambient temperature can affect its physical properties, especially its stiffness.

One unified solution to mitigate the effects of all these uncertainties is to compress them together properly. We use the value 2.5 as the tightness threshold. Before the start of the second step of assembly, the toy neck and the hole of the toy head is aligned and their distance is initialized to around 15mm, 30mm and 45mm, respectively. This distance is measured when the toy body has contact with the palm of the gripper. The corresponding position of the gripper is recorded as the starting location of the motion of the gripper. In the tests, the distance between the toy body and the palm of the gripper is uncertain. The configuration of the motion control based on Moveit is kept as the same in each test. In the phase of the physical compression between the toy parts, the speed of the end-effector is around 33mm/s. From each starting location, three attempts of assembly are made. In total, all nine attempts succeeded. The measured tightness data are plotted in Figure 4.36. In fact, if the velocity of the gripper is regulated rather than its position, the impact of the "uncertainty 1" is not significant. The "uncertainty 2" and "uncertainty 3" always exist in each experiment since they are hard to be controlled as the same. For "uncertainty 2", we intentionally enlarge the initial distance between the toy body and the palm of the gripper than usual. The compression and the insertion between two toy parts can also be completed properly. The specific impact of this distance on the tightness signal is demonstrated in Figure 4.37. To get these data, the toy neck is driven toward the hole of the toy head until the protective stop of the robot is triggered. The speed of the motion is around 15.4mm/s. After the initial contact between the two toy parts is achieved, the tightness signal takes longer to reach its peak and then declines as usual. As this initial distance is longer, the toy body is able to slide across the sensing surface of the tactile sensor for longer time, which can give the sensor more stimulation. When the signal starts to decrease from its peak, the tightness threshold to halt the motion is still effective. Fig. 4.37 Tightness signal when the initial distance between the toy body and the palm of the gripper is longer.

The robustness of the tightness to the "uncertainty 3" is proved in the experiments on the different sets of toy pieces. Three toy heads and three toy bodies are prepared as shown in Figure 4.38a. We initialize the 2-finger gripper to a certain location and then use the pipeline demonstrated in the next subsection to conduct assembly on these toy pieces. The alignment between the toy neck and the hole of the toy head is achieved with the help of a RGBD camera. The compression between two toy parts is implemented by the same motion control configuration based on Moveit. During the compression, the tightness signal is monitored to determine the proper contact state. When the desired tightness value is achieved, the motion of the 2-finger gripper is halted. Lastly, the insertion step is performed by a twist applied by the 2-finger gripper. These toy pieces have the same design but their physical properties have uncertainties. They are put in three pairs. In the first round, all three pairs are assembled successfully as shown in Figure 4.38b. In the second round, the heads and the bodies are re-paired and then the assembly procedure repeats. All of them can achieve the desired tightness and complete the assembly. These experiments validated the robustness of the tightness-based method to regulate the interaction between deformable objects. Although three kinds of uncertainties exist in these tests, the assembly can still be accomplished.

Pipeline of assembly

Leveraging the concept of tightness, a robotic pipeline to conduct the assembly of doll components is proposed in three main steps: alignment, compression and insertion.

We assume that the toy head is already gripped by the 3-finger gripper and the gripper is moved to a certain location. The 2-finger gripper is able to grasp the toy body and keep the axis of the toy body roughly in the direction of the Z-axis as indicated in Figure 4.32. The Z-axis attached to the 2-finger gripper is nearly perpendicular to the palm of the 3-finger gripper. In this way, the circle of the hole is easier to be detected by the camera.

In alignment, we use the RGBD camera on the wrist of the robotic arm to detect the 3D position of the center of the hole. The calculated 3D position of the center of the hole is defined in the camera frame. As the transformation between the camera frame and the gripper frame is known, we can get the 3D position of the center of the hole in the gripper frame. Then the gripper is moved to the position facing the hole. After the alignment, the distance between the toy neck and the hole is around 25mm. To get the 3D position of the hole, we need to know its pixel position in the image. For a RGB image, we use the OpenCV [START_REF] Bradski | The OpenCV Library[END_REF] pipeline of Hough transform to detect the circle. We can get the pixel position of the circle center and its radius in pixel. Then we search the corresponding depth of the circle center in the aligned depth image. However, due to the hollow design of the hole, the corresponding depth is usually unavailable. Thus, we have to pick up four pixels around the circle. The average depth of these four pixels is used to represent the depth of the circle center. Apparently, this approximation has some uncertainties. Given the pixel position of the hole center, the depth of the hole center and the intrinsic parameters of the camera, the 3D position of this hole center in the camera frame can be calculated based on the pinhole camera model. In compression, we set the tightness threshold as 2.5. In the framework of Moveit, the toy neck is driven toward the hole at a speed around 20mm/s after the initial contact between the toy parts. Once the signal value declines to 2.5 or below, the robotic arm is commanded to stop. In insertion, the gripper twists the toy body toward the toy head in a predefined mode. The twist consists of a rotation of 120 degrees and a translation of 15mm. After the insertion, the two grippers can release the assembled toy.

To verify this pipeline, we randomly selected three different starting locations for the 2-finger gripper. The relative positions between the toy head and the camera on the wrist of the robotic arm are reflected in the camera view as shown in From each starting location, the pipeline is executed for five times. In total, the pipeline is tested for 15 times. All tests were successful to finish the assembly. One typical assembly procedure started from the location 3 is demonstrated in Figure 4.40. In the first figure, the RGBD camera detects the hole as marked in pink and the 3D position of the hole center is calculated. In the second figure, the gripper approaches the hole according to the calculated relative position between the hole center and the gripper frame. The step of alignment is finished. In the step of compression, the tightness signal is updating based on the reconstructed 3D contact shape from the tactile sensor. The third figure corresponds to a light contact state between the toy head and the toy body. When the tightness signal declines below the threshold value, the motion of the robotic arm is terminated. The fourth figure records the state when the desired tightness is achieved. Observing the raw tactile images in the small windows, we can see the evolution of the contact state. After the step of compression, a twist is applied to the toy body as shown in the fifth, sixth, seventh figures. Finally, the toy body is released by the gripper and the two toy parts are well assembled. When the motion of the robotic arm is asked to be terminated, the gripper may shake slightly before stopping. This momentary unstable interaction could be reflected by the second signal peak on the curves. After this, the contact is stable again and the curves are calmed. This phenomenon does not impact the assembly. The shaking phenomenon may be very subtle or absent. In addition, when the lower ambient temperature makes the material of the toys stiffer, the second signal peak is more likely to take place.

Comparison

In order to demonstrate the advantage of the proposed assembly pipeline, we compare it with the strategy of position control. The alignment step and the insertion step are kept as the same. The compression step based on tightness is replaced by the step of position control. In this step, the tightness value is not monitored anymore. Instead, the robotic arm is commanded to arrive at a certain position. But how should we set the target position for it? We counted the movement distance of the robotic arm in the compression step for each assembly test using tightness. The results are listed in Table 4.1. The average distance 53.55mm is set as the target position for the position control.

The assembly test based on position control is repeated five times from each starting location. All the tests launched from the location 1 fail because of the protective stop of the robotic arm. The protective stop is triggered when the contact between two toy parts is too strong. The tests launched from the location 2 fail four times. The failures are also caused by the protective stop of the robotic arm. The tests launched from the location 3 all succeed. The main reason is related to the detection of the hole. In the cases of failure, maybe the pose of the toy head is different, the calculated 3D position of the hole center is located at a further point than usual. Thus the movement in the alignment step is adjusted toward a further point. This detection error leads to a shorter distance between the toy neck and the hole after the alignment step. In position control, the robotic arm executes the predefined command without an awareness of tightness on the real contact status. This leads to a large number of failures. The results of assembly tests are summarized in Table 4.2. Using tightness can provide good robustness and adaptability to uncertainties.

In the case of failure due to the protective stop of the robotic arm, the typical tightness signal is depicted in Figure 4.44. The second signal peak is caused by the protective stop. Before its occurrence, the tightness signal is indicating the tighter and tighter state. If we intervene in time, the protective stop can be avoided.

In a review on the code of the assembly pipeline, we found the received tactile images have a lag around 0.3s in the experiments of the pipeline starting from three locations. The effect of this lag is to make the compression step last around 0.3s longer than it should be. Since the speed of the gripper in the phase of the physical compression is around 20mm/s, the 2-finger gripper settles at a location around 6mm further. We can consider that when the desired tightness is achieved, the gripper is commanded to move further around 6mm. As this lag problem has the same influence on all tests on the pipeline, the comparison between the pipeline and the position control is still valid.

After this lag problem is fixed, we further performed 15 assembly tests starting from three random locations. 14 of 15 are completed. Then the averaged movement of the gripper in the 14 successful attempts is used as the target position of the position control for 15 assembly tests starting from the same three locations. 10 of 15 are completed. In some failed cases, the distance between the toy neck and the hole of the toy head is a little longer than others after the alignment step using camera. A fixed movement cannot achieve the ideal tightness for the final insertion step.

Another concern is about whether the force/torque sensor on the wrist of the robotic arm can also perceive the tension between the two toy parts. In the compression step, the robotic arm moves until its protective stop. This process is repeated twice. The measured force along the Z-axis of the force/torque sensor and the tightness signal are plotted in Figure 4.45. The Z-axis points to the hole. To better visualize and compare the signals in the same figure, the quarter of the measured force value is plotted since the original force value is much larger than the tightness signal. In contrast with the tightness signal, the measured force does not present a clear pattern to be matched to the contact state between the two toy parts. After the force increases than a certain value, its variation becomes irregular. But at the same time, the downward trend of the tightness signal is clear. The felt tightness on the fingertip is not equivalent to the measured force on the wrist. The tactile sensor is closer to the source of tension and thus the transmission of the contact state is more direct. For the force/torque sensor, the unpredictable deformation of the toy parts can easily affect its measurement. Using the measured force to infer the contact state between two toy parts has many uncertainties.

To set a threshold for the tightness signal, we can start from the signal value in the static state since the signal eventually goes down. However, the force signal goes up. Touching different objects with different stiffness can lead to different responses of the force/torque sensor. It is hard to find a proper threshold which works in more cases. In addition, the size, weight and price of a tactile sensor are also favorable.

Conclusion

This chapter demonstrated how to unleash the power of the 3D tactile sensing to facilitate real applications. The first studied application is the slip detection and avoidance. The proposed tactile mesh can handle various deformable or rigid objects, even textiles. The developed method can detect the slip occurring in different directions. The impact of the density of the tactile mesh is studied and verified in experiments. One effective slip avoidance algorithm is developed and validated with different objects.

The second studied application is the estimation of the in-hand pose of objects. Based on the reconstructed 3D tactile shape, one height-based pipeline is developed to estimate the orientation of the gripped object in real time. To reveal its advantage, we designed the static experiments and the dynamic experiments to compare it with the color-based method. The height-based method performs much better in stability and accuracy.

The last application is the assembly of deformable objects. The concept of the sense of tightness is proposed to describe the complicated contact status between hand, tool and environment. Robots are enabled to have a feeling of tightness in manipulation or interaction. The guidance to determine a proper tightness value is provided and demonstrated. The robustness of the tightness-based method under uncertainties is validated in experiments. Leveraging the sense of tightness, a pipeline to assemble toy parts is proposed and evaluated in experiments. The results of evaluation are encouraging. The advantage of the usage of tightness is proved in comparison with the position control and the usage of a force/torque sensor.

In one word, exploiting the 3D tactile shape makes complicated tasks easier. Due to the complexity of the assembly of deformable objects, this type of tasks is still performed by skilled workers. The proposed indirect contact control strategy using a sense of tightness provides a direction to robotize this kind of tasks.

Chapter 5 Conclusion

Conclusion

In the first stage of this research, efforts are committed to solve the difficulty of deformable object grasping. The dynamic evaluation methodology is proposed according to the nature of deformation of objects during grasping. The transmission ability of a grasp is dynamically modelled to complement the general grasping theory for deformable objects. Based on this extended theory, a novel grasping synthesis method to optimize grasp stability is developed and verified in both of simulations and experiments. The whole work has contributions in the grasping theory and the specific techniques of the grasp synthesis.

In the second stage of this research, we put efforts in the 3D reconstruction of the tactile image to achieve high tactile resolution and sensitivity. A neural network based on the color variation is developed to optimize the 3D reconstruction performance for the image-based tactile sensors without uniform illumination like DIGIT. The extensive experiments validated that in both of noise control and profile tracking, the proposed network outperforms the lookup table and the network based on absolute color values.

Leveraging the power of the real-time 3D tactile shape, three kinds of applications based on tactile servoing are studied in the third stage of this research. In the first application, one effective slip detection and avoidance method is developed based on the reconstructed height mesh. It works effectively with different deformable objects. In the second application, a height-based pipeline is developed to estimate and track the in-hand pose of the gripped object. Compared with the color-based method, its performance is much better in stability and accuracy. In the last application, the sense of tightness is proposed to give robots a new sense of perception. How to exploit the sense of tightness to overcome the different uncertainties in deformable manipulation is demonstrated in experiments. An effective pipeline to assemble deformable objects is proposed and evaluated in experiments. The results of evaluation are encouraging. The advantage of the usage of tightness is proved in the comparison with the position control.

In general, this research provides substantial contributions in the field of grasping and tactile servoing using vision-based tactile sensors in the context of the deformable object manipulation.

Outlook

The grasp synthesis method for deformable objects can be further promoted by incorporating the impact of the varying contact status between the fingertips and the gripped object. The varying contact status can be modelled by a dynamic friction space. We can use the dynamic grasp map to transfer the dynamic friction space as the dynamic grasp wrench space.

Analyzing the dynamic property of the dynamic grasp wrench space, like the volume of the space or the minimal distance from the center of the space to its limitation surface, can provide guidance for the grasp synthesis on deformable objects. In this manner, the influence of the contact model and the transmission model on grasp quality is unified in a dynamic geometrical space. From the static grasp wrench space to the dynamic grasp wrench space, one more systematic theory on grasping can be constituted. The 3D tactile imprint is worth pursuing. However, the non-uniform illumination on the sensing surface of the tactile sensor is the major obstacle to further improving the precision of 3D reconstruction. Some modern deep networks should be tested and compared in order to find the optimal solution to tackle with the uncertainties associated with the hardware and achieve the uniform performance on the non-uniform sensing surface.

In manipulation, the 3D tactile imprint can facilitate different tasks. In the tackled applications, we did not fully exploit the high resolution of the 3D tactile shape. The 3D shape registration of the deformable object could be a suitable topic to employ the high resolution. 3D shape registration refers to the procedure to match two point clouds by finding their common shape feature. A transformation between these two point clouds is returned in the end of the procedure. We can already achieve the point cloud of the tactile shapes as shown in Figure 5.1a. The small point cloud is extracted from the reconstructed 3D tactile shape. The large point cloud is divided from the toy body model. The goal is to match the small point cloud to the corresponding location on the large point cloud. In this way, we can estimate the precise in-hand pose of the gripped object by touching. In Figure 5.1b, the two point clouds are downsampled. In Figure 5.1c, the global registration is performed to map the contact shape to the toy body. The registration result has some errors to the real contact location. The reason is that the loaded point cloud of the toy body is the model without external contacts. Thus it is not deformed. However, the extracted small point cloud corresponds to the real contact shape deformed by contact forces. How to update the point cloud of the gripped object according to the sensed contact status is the difficulty to perform the 3D registration on deformable objects since their shapes change dynamically under external forces.

3. 1

 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Principle of 3D reconstruction . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Lookup table for 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Neural network for 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Comparison and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Tactile sensing for deformable object manipulation 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Slip detection and avoidance . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Slip detection with unknown objects . . . . . . . . . . . . . . . . . 4.2.2 Slip detection in different directions . . . . . . . . . . . . . . . . . 4.2.3 Density of the tactile mesh . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Slip avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 In-hand pose estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Height-based pipeline . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Static experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Dynamic experiments . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Sense of tightness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Definition of tightness . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Choice of tightness . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Robustness of tightness . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Pipeline of assembly . . . . . . . . . . . . . . . . . . . . . . . . .4.4.5 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusion 5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References List of figures 1.1 Overview of the robotic system in the project SoftManBot. . . . . . . . . . 1.2 Manual operations in industrial sectors. . . . . . . . . . . . . . . . . . . . 1.3 Mechanism of grasping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Friction cone. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Variable contact region [1]. . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Non-planar patch contact [2]. . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Grasp wrench space projected in the three-dimensional space. . . . . . . . 1.8 Two grasp candidates' GWS and TWS projected in a two-dimensional space. GWS is depicted by the blue polygon and TWS is depicted by the red arrow. 1.9 Objects with small local deformation. . . . . . . . . . . . . . . . . . . . . 1.10 Sensors made by SingleTact (https://www.singletact.com/). . . . . . . . . . 1.11 Sensor made by PowerOn (https://www.poweron.one/). . . . . . . . . . . . 1.12 Sensors made by Contactile (https://contactile.com/). . . . . . . . . . . . . 1.13 Design of GelSight [3]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.14 GelSight Wedge for reaching narrow spaces [4]. . . . . . . . . . . . . . . . 1.15 GelSlim vs GelSight [5]. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.16 Sensors with a round tactile surface. . . . . . . . . . . . . . . . . . . . . . 1.17 From top row to bottom: objects, tactile images, and inferred 3D shapes [4]. 1.18 Pose estimation by the registration of point clouds [6]. . . . . . . . . . . . 1.19 Principle of SwingBot [7]. . . . . . . . . . . . . . . . . . . . . . . . . . . 1.20 Cable manipulation system based on tactile sensing [8]. . . . . . . . . . . . 1.21 One example of extrinsic tactile sensing [9]. . . . . . . . . . . . . . . . . . 1.22 Insertion task regulated by a reinforcement learning policy [10]. . . . . . . 1.23 Red contact line between the object and the hole [11]. . . . . . . . . . . . . 2.1 Contact frames and object frame. . . . . . . . . . . . . . . . . . . . . . . . 2.2 A set of Q 2 curves. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii List of figures 2.3 Grasp candidates in simulation. . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Dynamic metrics in simulation. . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Directions of acceleration test. . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Integrated tactile sensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Two grasp candidates for toy head. The left grasp candidate is named Grasp1 and the right one is named Grasp2. . . . . . . . . . . . . . . . . . . . . . . 2.8 Dynamic metrics in experiments. . . . . . . . . . . . . . . . . . . . . . . . 2.9 Experimental setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 Stuck toy head. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Initialization of experiments. . . . . . . . . . . . . . . . . . . . . . . . . . 2.12 Measured wrenches in experiments. . . . . . . . . . . . . . . . . . . . . . 3.1 (a) DIGIT sensor. (b) Exploded view of DIGIT sensor. A) elastomer, B) acrylic window, C) snap-fit holder, D) lighting PCB, E) plastic housing, F) camera PCB, G) back housing. [12] . . . . . . . . . . . . . . . . . . . . . 3.2 Main components of DIGIT. (a) LEDs and camera. (b) Lighted LEDs. (c) Reflective membrane. (d) Transparent window and elastomer. . . . . . . . . 3.3 Calibration of the dimension of pixel. . . . . . . . . . . . . . . . . . . . . 3.4 Depth calculation by pressing a ball. . . . . . . . . . . . . . . . . . . . . . 3.5 Location of circular edge. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Diagram to build up the lookup table. . . . . . . . . . . . . . . . . . . . . 3.7 Diagram of real-time 3D reconstruction. . . . . . . . . . . . . . . . . . . . 3.8 3D reconstruction based on lookup table. . . . . . . . . . . . . . . . . . . . 3.9 Computation of the normal of one contact point. . . . . . . . . . . . . . . . 3.10 3D reconstruction based on neural network. . . . . . . . . . . . . . . . . . 3.11 Six pressing spots for comparison. (a) Marked pressing spots. (b) Some pressing instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Profile comparison at spot 1. . . . . . . . . . . . . . . . . . . . . . . . . . 3.13 Profile comparison at spot 2. . . . . . . . . . . . . . . . . . . . . . . . . . 3.14 Profile comparison at spot 3. . . . . . . . . . . . . . . . . . . . . . . . . . 3.15 Profile comparison at spot 4. . . . . . . . . . . . . . . . . . . . . . . . . . 3.16 Profile comparison at spot 5. . . . . . . . . . . . . . . . . . . . . . . . . . 3.17 Profile comparison at spot 6. . . . . . . . . . . . . . . . . . . . . . . . . . 3.18 Edge of contact. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Manual assembly of doll head and doll body. . . . . . . . . . . . . . . . .

Fig. 1 . 1

 11 Fig. 1.1 Overview of the robotic system in the project SoftManBot.

Fig. 1 . 2

 12 Fig. 1.2 Manual operations in industrial sectors.

Fig. 1 . 3

 13 Fig. 1.3 Mechanism of grasping.

Fig. 1 . 4

 14 Fig. 1.4 Friction cone.

Fig. 1 . 7

 17 Fig. 1.7 Grasp wrench space projected in the three-dimensional space.

Fig. 1 . 8

 18 Fig. 1.8 Two grasp candidates' GWS and TWS projected in a two-dimensional space. GWS is depicted by the blue polygon and TWS is depicted by the red arrow.

Fig. 1 .

 1 Fig. 1.10 Sensors made by SingleTact (https://www.singletact.com/).

Fig. 1 .

 1 Fig. 1.11 Sensor made by PowerOn (https://www.poweron.one/).

Fig. 1 .

 1 Fig. 1.12 Sensors made by Contactile (https://contactile.com/).

Fig. 1 .

 1 Fig. 1.13 Design of GelSight [3].

Fig. 1 .

 1 Fig. 1.14 GelSight Wedge for reaching narrow spaces [4].

Fig. 1 .

 1 Fig. 1.15 GelSlim vs GelSight [5].

Fig. 1 .

 1 Fig. 1.18 Pose estimation by the registration of point clouds [6].

Fig. 1 .

 1 Fig. 1.19 Principle of SwingBot [7].

Fig. 1 .

 1 Fig. 1.20 Cable manipulation system based on tactile sensing [8].

Fig. 1 .

 1 Fig. 1.21 One example of extrinsic tactile sensing [9].

Fig. 1 .

 1 Fig. 1.22 Insertion task regulated by a reinforcement learning policy [10].

Fig. 2 . 1

 21 Fig. 2.1 Contact frames and object frame.

Fig. 2 .

 2 Fig. 2.2 A set of Q 2 curves.

Fig. 2 . 3

 23 Fig. 2.3 Grasp candidates in simulation.

Fig. 2 . 4

 24 Fig. 2.4 Dynamic metrics in simulation.

Fig. 2 . 5

 25 Fig. 2.5 Directions of acceleration test.

Fig. 2 . 6

 26 Fig. 2.6 Integrated tactile sensors.

Fig. 2 . 7

 27 Fig. 2.7 Two grasp candidates for toy head. The left grasp candidate is named Grasp1 and the right one is named Grasp2.

Fig. 2 . 8

 28 Fig. 2.8 Dynamic metrics in experiments.

Fig. 2 . 9

 29 Fig. 2.9 Experimental setup.

Fig. 2 .

 2 Fig. 2.10 Stuck toy head.

Fig. 3 .

 3 Fig. 3.1 (a) DIGIT sensor. (b) Exploded view of DIGIT sensor. A) elastomer, B) acrylic window, C) snap-fit holder, D) lighting PCB, E) plastic housing, F) camera PCB, G) back housing.[START_REF] Lambeta | Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation[END_REF]

Fig. 3 . 2

 32 Fig. 3.2 Main components of DIGIT. (a) LEDs and camera. (b) Lighted LEDs. (c) Reflective membrane. (d) Transparent window and elastomer.

Fig. 3 . 3

 33 Fig. 3.3 Calibration of the dimension of pixel.

Fig. 3 . 4

 34 Fig. 3.4 Depth calculation by pressing a ball.

  Fig. 3.5 Location of circular edge.

Fig. 3 . 6

 36 Fig. 3.6 Diagram to build up the lookup table.

Fig. 3 . 7

 37 Fig. 3.7 Diagram of real-time 3D reconstruction.

Fig. 3 .

 3 Fig. 3.8 3D reconstruction based on lookup table.

) 2 ( 3

 23 In this way each component of the normal vector is in the range [-1, 1]. We regulate them in the range [0, 1] by: n xnr = (n xn + 1)/2 n ynr = (n yn + 1)/2 n znr = (n zn + 1)/

Fig. 3 .

 3 Fig. 3.10 3D reconstruction based on neural network.

Fig. 3 .

 3 Fig. 3.11 Six pressing spots for comparison. (a) Marked pressing spots. (b) Some pressing instances

  (a) Hard press by 6.07mm ball bearing. (b) Light press by 6.07mm ball bearing. (c) Hard press by 7.14mm ball bearing. (d) Light press by 7.14mm ball bearing. (e) Hard press by 11.10mm ball bearing. (f) Light press by 11.10mm ball bearing.

Fig. 3 .

 3 Fig. 3.12 Profile comparison at spot 1.

Fig. 3 .

 3 Fig. 3.13 Profile comparison at spot 2.

Fig. 3 .

 3 Fig. 3.14 Profile comparison at spot 3.

Fig. 3 .

 3 Fig. 3.15 Profile comparison at spot 4.

Fig. 3 .

 3 Fig. 3.16 Profile comparison at spot 5.

Fig. 3 .

 3 Fig. 3.17 Profile comparison at spot 6.

Fig. 3 .

 3 Fig. 3.18 Edge of contact.

Fig. 4 . 1

 41 Fig. 4.1 Manual assembly of doll head and doll body.

  Sock grabbed from opening. (e) Sock grabbed from bottom. (f) USB cable.

Fig. 4 . 3

 43 Fig. 4.3 Slip signal in tests with some common objects.

Fig. 4 .

 4 Fig. 4.4 3D reconstruction of some tested objects.

Fig. 4 . 5

 45 Fig. 4.5 Slip signal in tests on carton box.

Fig. 4 . 6

 46 Fig. 4.6 Coordinate frame defined for slip tests on toy components.

  (a) Leg translation X. (b) Body translation X. (c) Head translation X. (d) Leg translation Y. (e) Body translation Y. (f) Head translation Y. (g) Leg translation Z. (h) Body translation Z. (i) Head translation Z. (j) Leg rotation. (k) Body rotation. (l) Head rotation.

Fig. 4 . 7

 47 Fig. 4.7 Slip tests on toy components.

  (a) 320 × 240. (b) 64 × 48. (c) 32 × 24. (d) 16 × 12. (e) 4 × 3.

Fig. 4 . 8

 48 Fig. 4.8 Slip signals of towel.

  (a) Hex keys in three dimensions. (b) Gripped small hex key.

Fig. 4 . 9

 49 Fig. 4.9 Tested hex keys.

  (a) Small hex key. (b) Medium hex key. (c) Large hex key.

Fig. 4 .

 4 Fig. 4.10 Tactile images of hex keys.

  (a) Small hex key, 32 × 24. (b) Medium hex key, 32 × 24. (c) Large hex key, 32 × 24.(d) Small hex key, 8 × 6.(e) Medium hex key, 8 × 6.(f) Large hex key, 8 × 6.

Fig. 4 .

 4 Fig. 4.11 Slip signals of hex keys.

Fig. 4 .

 4 Fig. 4.12 Pipeline of slip avoidance.

  (a) Grip in slip mode. (b) Grip in no-slip mode.

Fig. 4 .

 4 Fig. 4.14 Initial grip on toy arm.

Fig. 4 .

 4 Fig. 4.15 Slip signal in slip avoidance.

  (a) Contact. (b) Height-based method.(c) Color-based method.

Fig. 4 .

 4 Fig. 4.16 Static experiments on hex key.

  (a) Contact. (b) Height-based method.(c) Color-based method.

Fig. 4 .

 4 Fig. 4.17 Static experiments on toy leg.

Fig. 4 .

 4 Fig. 4.18 Four objects used in dynamic experiments.

4. 3 . 3

 33 Dynamic experiments Besides the static experiments, more dynamic experiments are performed to test the performance of the height-based method in tracking. Four objects are used as shown in Figure 4.18.

Fig. 4 .

 4 Fig. 4.19 Dynamic test of the height-based method on hex key.

Fig. 4 .

 4 Fig. 4.20 Output angle.

Fig. 4 .

 4 Fig. 4.21 Dynamic test of the color-based method on hex key.

Figure 4 . 25 .

 425 The curve of the height-based method is smooth with much less local noise. The sudden change of angle in the middle corresponds to the moment when the red long axis just surpasses 180 degrees (perpendicular to the short (a) Height-based method. (b) Color-based method.

Fig. 4 .

 4 Fig. 4.22 Angle plots of hex key in dynamic experiments.

Fig. 4 .

 4 Fig. 4.23 Dynamic test of the height-based method on pen.

  (a) Height-based method. (b) Color-based method.

Fig. 4 .

 4 Fig. 4.25 Angle plots of pen in dynamic experiments.

Fig. 4 .

 4 Fig. 4.27 Dynamic test of the color-based method on toy arm.

  (a) Height-based method. (b) Color-based method.

Fig. 4 .

 4 Fig. 4.28 Angle plots of toy arm in dynamic experiments.

Fig. 4 .

 4 Fig. 4.29 Dynamic test of the height-based method on toy leg.

  (a) Height-based method. (b) Color-based method.

Fig. 4 .

 4 Fig. 4.31 Angle plots of toy leg in dynamic experiments.

Fig. 4 .

 4 Fig. 4.32 Setup for doll assembly.

Fig. 4 .

 4 Fig. 4.33 Tightness signal in interaction.

  Fig. 4.34 Stabilized toy body.

  (a) Desired tightness value is 4. (b) Desired tightness value is 2.5. (c) Desired tightness value is 1.2.

Fig. 4 .

 4 Fig. 4.35 Robot is stopped according to the sensed tightness.

Fig. 4 .

 4 Fig. 4.36 Tightness signal in operation. The initial distance between the toy neck and the hole is around 15mm, 30mm and 45mm. All nine attempts are successful and their signals are plotted in the sub-figures, respectively.

  (a) Toy pieces before assembly.(b) Toys after assembly.

Fig. 4 .

 4 Fig. 4.38 Different toy pieces used in experiments.

Fig. 4 .

 4 Fig. 4.39 Three starting locations in the camera view.

Figure 4 .

 4 [START_REF] Zaidi | Grasp planning pipeline for robust manipulation of 3d deformable objects with industrial robotic hand + arm systems[END_REF] Figure 4.39a, the coordinates of the hole center in the camera frame is around (49.83, -9.89, 229.00). The unit is millimeter. The coordinates of the hole center is around (57.76, 14.61, 240.33) in Figure 4.39b. The coordinates of the hole center is around (77.38, -28.95, 267.67) in Figure 4.39c.

Fig. 4 .

 4 Fig. 4.40 Demonstration of the assembly pipeline.

123

 123

Fig. 4 .

 4 Fig. 4.41 Tightness signals in the assembly tests starting from the location 1.

Fig. 4 .

 4 Fig. 4.43 Tightness signals in the assembly tests starting from the location 3.

Fig. 4 .

 4 Fig. 4.44 Tightness signal in the case of the protective stop.

Fig. 4 .

 4 Fig. 4.45 Tightness vs force.

Fig. 5 .

 5 Fig. 5.1 3D registration. (a) Point clouds of the contact and the divided toy body. (b) Downsampled point clouds. (c) Registration.

Table 2 .

 2

		1 Results of acceleration tests
	Direction Grasp1 Grasp2 Grasp3 Grasp4
	0	83.73	52.27	71.60	56.80
	1	62.53	61.67	62.53	69.27
	2	87.67	53.73	68.57	57.73
	3	64.07	62.07	90.00	62.53
	Average	74.50	57.44	73.18	61.58

Table 3 .

 3

	1 Parameters of DIGIT sensor
	Parameter
	size: l × w × h (mm)	36 × 26 × 33
	weight (g)	20
	Maximal sensing field (mm)	25 × 19
	resolution (pixel)	320 × 240
	Frame rate per second	30

Table 3 .

 3 2 Results of pixel calibration

	Measurement number	Pose 1	Pose 2
		Pixels per millimeter Pixels per millimeter
	1	18.915	18.430
	2	18.565	18.768
	3	18.565	18.472
	4	18.215	18.873
	average	18.6

Table 3 .

 3 3 Four candidates for comparison

		Table-11	Table-6	NN-abs	NN-var
	Method	lookup table lookup table neural network neural network
	Ball bearing	11.10mm	6.07mm	6.07mm	6.07mm
	Sample number	100	100	30	30
	Color mode	variation	variation	absolute value	variation

Table 3 .

 3

			4 RMSE of the candidates
	Spot	Press	RMSE
			NN-var NN-abs Table-6 Table-11
		6.07mm-hard 0.0339 0.0319 0.0335	0.0446
		6.07mm-light 0.0244 0.0258 0.0220	0.0253
	1	7.14mm-hard 0.0321 0.0428 0.0453 7.14mm-light 0.0259 0.0379 0.0212	0.0534 0.0279
		11.10mm-hard 0.0286 0.0334 0.0366	0.0412
		11.10mm-light 0.0225 0.0315 0.0244	0.0268
		6.07mm-hard 0.0281 0.0228 0.0477	0.0437
		6.07mm-light 0.0161 0.0243 0.0118	0.0106
	2	7.14mm-hard 0.0281 0.0386 0.0218 7.14mm-light 0.0236 0.0403 0.0071	0.0148 0.0094
		11.10mm-hard 0.0249 0.0305 0.0304	0.0311
		11.10mm-light 0.0224 0.0327 0.0194	0.0178
		6.07mm-hard 0.0398 0.0410 0.0217	0.0409
		6.07mm-light 0.0180 0.0287 0.0065	0.0088
	3	7.14mm-hard 0.0335 0.0495 0.0179 7.14mm-light 0.0255 0.0430 0.0096	0.0188 0.0154
		11.10mm-hard 0.0305 0.0407 0.0227	0.0277
		11.10mm-light 0.0253 0.0386 0.0126	0.0114
		6.07mm-hard 0.0315 0.0252 0.0583	0.0440
		6.07mm-light 0.0187 0.0235 0.0190	0.0135
	4	7.14mm-hard 0.0334 0.0413 0.0254 7.14mm-light 0.0251 0.0390 0.0116	0.0175 0.0205
		11.10mm-hard 0.0249 0.0274 0.0290	0.0251
		11.10mm-light 0.0213 0.0298 0.0199	0.0182
		6.07mm-hard 0.0328 0.0369 0.0267	0.0284
		6.07mm-light 0.0206 0.0307 0.0110	0.0147
	5	7.14mm-hard 0.0298 0.0442 0.0160 7.14mm-light 0.0242 0.0468 0.0131	0.0152 0.0169
		11.10mm-hard 0.0281 0.0402 0.0189	0.0188
		11.10mm-light 0.0214 0.0345 0.0101	0.0113
		6.07mm-hard 0.0272 0.0292 0.0436	0.0315
		6.07mm-light 0.0178 0.0246 0.0242	0.0123
	6	7.14mm-hard 0.0293 0.0414 0.0240 7.14mm-light 0.0289 0.0438 0.0082	0.0173 0.0183
		11.10mm-hard 0.0225 0.0310 0.0315	0.0260
		11.10mm-light 0.0215 0.0313 0.0282	0.0243
		Average	0.0262 0.0349 0.0231	0.0234

Table 4 .

 4 1 Movement distance in the compression step

	Test number		Start location
		Location 1 Location 2 Location 3
	1	52.14	53.97	51.28
	2	52.39	48.59	55.69
	3	53.20	52.39	54.15
	4	55.25	55.49	53.14
	5	54.83	54.40	56.29
	Average		53.55

(a) Group1-Grasp1. (b) Group1-Grasp2. (c) Group2-Grasp1. (d) Group2-Grasp2. (e) Group3-Grasp1. (f) Group3-Grasp2.Fig. 2.11 Initialization of experiments.

(a) Group1-Grasp1. (b) Group1-Grasp2. (c) Group2-Grasp1. (d) Group2-Grasp2. (e) Group3-Grasp1. (f) Group3-Grasp2.Fig. 2.12 Measured wrenches in experiments.

Acknowledgements

3D reconstruction of tactile imprints

The conversion between the normal vector and the gradients is implied in Equation 3.2. Using the following equations, we can get the value of the gradients from the components of the normal vector:

Integrating the gradients by Fast Poisson algorithm [START_REF]Fast poisson reconstruction in python[END_REF], we can get the height map. Some results of 3D reconstruction are shown in Figure 3.10. To generate a height map from a tactile image, the method using the absolute color takes around 0.0147s and the method using the color variation takes around 0.0156s.

height-based method outperforms the color-based method in both of stability and accuracy. In Figure 4.28, the instant when the angle jumps to zero implies that the contact between the toy arm and the sensor is lost. Unfortunately, the results of the color-based method are extremely unstable and thus unusable. The third column in Figure 4.30 looks like we are operating different objects. The estimations in the first two rows are totally wrong. Its angle plot in Figure 4.31b proved how bad its performance is. The starting angle has an error around 90 degrees, which should be like the starting angle as shown in Figure 4.31a. This error is so big because the estimated contact shape in the binary image is totally wrong. During rotation, the curve of the color-based method oscillates abnormally. Hence, when the contact area increases, the height-based method is able to keep the quality of its output, but the output from the color-based method is no longer usable.

Through dynamic experiments, the advantage of the height-based method in stability, accuracy and robustness is validated over the color-based method.