Zineb Abderrahmane

Gowrishankar Ganesh

André Crosnier

Andrea Cherubini

Haptic Zero-Shot Learning: Recognition of Objects Never Touched Before

Keywords: Haptic recognition, Zero-shot learning, Attribute-based description, Robotic hand

Object recognition is essential to enable robots to interact with their environment. Robots should be capable, on one hand of recognizing previously experienced objects, and on the other, of using the experienced objects for learning novel objects, i.e. objects for which training data are not available. Recognition of such novel objects can be achieved with Zero-Shot Learning (ZSL). In this work, we show the potential of ZSL for haptic recognition. First, we design a zero-shot haptic recognition algorithm and, using the extensive PHAC-2 database [1] as well as our own, we adapt, analyze and optimize the ZSL for the challenges and constraints characteristic of haptic recognition. Finally, we apply the optimized algorithm for haptic recognition of daily-life objects using an anthropomorphic robot hand. Our algorithm enables the robot to recognize eight of the ten novel objects handed to it.

Introduction

Transferring human perceptual capabilities to robots can enhance their autonomy, and hence facilitate their integration in human environment. Object recognition is a powerful human capability that, when transferred to a robot, can optimize grasping and manipulation. For instance, recognizing a mug and its functionalities can guide a robot to hold it vertically to avoid spilling the liquid.

Object recognition has been extensively studied in computer vision and tremendous advances have been made [START_REF] Grauman | Visual object recognition[END_REF]. Visual recognition of an object however, requires it to be in the camera field of view, and is limited by lighting constraints and visual occlusions. One way to overcome these constraints is by the integration of visual recognition with additional sources of information, such as haptics; i.e., touch and proprioception. Haptics can not only help overcome the constraints associated with visual recognition, but also provide additional information regarding the shape, size, texture and compliance of an object, that can arguably improve recognition.

An object encountered by a robot can be one that it has enough experience with (that it has enough training data on). In this case, the object can be recognized using the available haptic recognition algorithms that use multiclass classifiers to efficiently deal with challenging robotic setups [START_REF] Johnsson | Haptic perception with self-organizing anns and an anthropomorphic robot hand[END_REF][START_REF] Madry | St-hmp: Unsupervised spatio-temporal feature learning for tactile data[END_REF] and object sets [START_REF] Corradi | Bayesian tactile object recognition: learning and recognising objects using a new inexpensive tactile sensor[END_REF][START_REF] Schmitz | Tactile object recognition using deep learning and dropout[END_REF], using various machine learning and optimization techniques [START_REF] Schneider | Object identification with tactile sensors using bag-of-features[END_REF][START_REF] Martinez-Hernandez | Active haptic shape recognition by intrinsic motivation with a robot hand[END_REF][START_REF] Liu | Object recognition using tactile measurements: Kernel sparse coding methods[END_REF]. On the other hand, an encountered object may be one with sparse, or in the worst case, no training data [START_REF] Yu | Attribute-based transfer learning for object categorization with zero/one training example[END_REF][START_REF] Elhoseiny | Write a classifier: Zero-shot learning using purely textual descriptions[END_REF][START_REF] Lampert | Attribute-based classification for zero-shot visual object categorization[END_REF][START_REF] Cheng | Random forest classifier for zero-shot learning based on relative attribute[END_REF]. In our study, we focus on the problem of recognizing new objects that have never been experienced before. To the best of our knowledge, this problem has never been addressed in regard to haptic recognition.

Recognition of novel objects that have not been trained on, is referred to as Zero-shot learning or ZSL. The key idea underlying ZSL is the exploitation of object relationships in order to transfer the knowledge learned from known objects, to recognize novel ones. In this work, we adopt an attributes-based approach. An attribute is a high-level object property that may represent, for example object size (e.g. small), or object texture (eg. rough). By describing all objects using attributes, the system can be trained to learn how to map haptic data to attributes. This mapping is authors of [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] considered attributes unreliability and proposed a statistic solution to leverage errors in attributes. In [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF], the two stages of learning attributes and inference were replaced by a simplistic implementation.

In this work, we choose to use semantic binary attributes as they are intuitive and yet shown to perform well on visual data [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF]. To define a set of attributes describing the haptic sensation of objects, we refer to an important work for describing haptic sensation using words [START_REF] Chu | Robotic learning of haptic adjectives through physical interaction[END_REF]. This work proposed a framework to describe 60 different objects using a list of 24 binary haptic adjectives based on human blind haptic exploration of objects. We believe this is an important step toward improving the answer of a robot while touching objects: instead of returning the name or the identity of the object, the authors propose to return a vector of binary values indicating the presence of each haptic adjective in the object. In addition to the multiple applications of this adjective-based description presented in [START_REF] Chu | Robotic learning of haptic adjectives through physical interaction[END_REF], we investigate its application to zero-shot learning.

Zero-shot learning

Problem formulation

ZSL consists of training and testing a recognition system by using disjoint training and test class sets. Specifically, the system is trained on D train ⊂ X × Y and tested on D test ⊂ X × Z, where X is the feature space in which data is represented and Y = {y 1 , . . . , y N } and Z = {z 1 , . . . , z L } two disjoint sets of classes 4 . A multi-class classifier will fail to learn a mapping function f : X -→ Z from D train in such a case since Y ∩ Z = ∅.

Solution overview

The key idea behind ZSL is to exploit relationships between Y and Z to transfer knowledge learned from Y to recognize objects in Z through a layer of attributes, as illustrated in Fig. 1. All objects from O = Y ∪ Z are described using a set of attributes A = {a 1 , . . . , a M } (e.g. A = {thin, so f t, rigid}). This results in the object-attribute matrix K (see Fig. 1a), in which the row i is the attributes-based description a o i = a o i 1 , . . . , a o i M of an object o i ∈ O, such that a o i m = 1 if attribute a m is present in object o i , and a o i m = 0 otherwise (e.g. vector a = [1, 0, 1] for object class pencil). Then, D train and attribute-based descriptions of training objects {a y n , y n ∈ Y} are used to learn a mapping from X to A: for any x ∈ X, this mapping predicts a probabilistic attribute-based description p(a 1 |x), . . . , p(a M |x), where p(a m |x) is the probability of the presence of a m in x (e.g. p(thin|x) = 0.8, p(soft|x) = 0.2, p(rigid|x) = 0.5 for x collected from a pencil)(see Fig. 1b). Finally, for a test sample x ∈ X, the returned posterior p(a m |x) of each attribute a m is used by a DAP model [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] along with the attribute-based descriptions of test objects {a z l , z l ∈ Z} to infer the final classification z x = f (x) (see Fig. 1c).

In this scenario the cost of adding a new object to the test set is dramatically lower than with traditional learning. In fact, it breaks down to adding a row (the attribute-based object description) to matrix K. This is a very low cost as compared to collecting haptic data for the new object to retrain the multi-class classifier.

Direct Attributes Prediction (DAP)

The attribute-based zero-shot recognition system of [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] proposed two models: Direct Attributes Prediction (DAP) and Indirect Attributes Prediction (IAP). When comparing models recognition accuracies [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF][START_REF] Lampert | Attribute-based classification for zero-shot visual object categorization[END_REF], the difference is relatively small, with a slightly better accuracy given by DAP. Thus, in this work, we focus only on DAP.

DAP is a probabilistic model that makes use of attributes learned from trained objects Y, to classify a test sample x ∈ X as one of a set of novel objects Z, using object-attribute matrix K. During the training phase, DAP learns a mapping from X to A. By assuming attribute independence, for each attribute a m , an independent probabilistic binary classifier g m :

X -→ [0, 1] is trained with D m train = {(x, a y m ) s.t. (x, y) ∈ D train }.
Each g m returns the attribute posterior p(a m |x) for a given x ∈ X.

During the test phase, to recognize a sample x as one of Z objects, DAP computes a posterior probability p (z l | x) for each object z l ∈ Z. For this, both the object's attribute-based description a z l (provided by matrix K) and its posterior probability p(a z l |x) are needed. To compute the z l object posterior, the attributes layer is introduced and Bayes' rule is applied to obtain:

a M a 1 A y 1 y N 0 z 1 z L Y Z 1 0 0 1 1 1 0 a 2 1 1 1 0 K (a) a M a 1 A y 1 y N D train a 2 x N,1 ,...,x N,n N x 1, 1 ,...,x 1,n 1 train a 2 train a 1 train a M a 1 a 2 a M x 2, 1 ,...,x 2,n 2 y 2 0 1 0 1 1 0 0 0 1 (b) z 1 z L Z 1 0 1 0 1 1 K a 1 a 2 a M x p(a 1 |x) p(a 2 |x) p(a M |x) DAP z x 1 1 0 z 2 (c)
p(z l | x) = a∈{0,1} M p(z l | a) p(a | x) = a∈{0,1} M p(a | z l ) p(z l ) p(a) p(a | x), (1)
Since each object z l is associated with a discriminating attributes vector a z l , p(a | z l ) = 1 if a = a z l and 0 otherwise. Replacing in (1) yields:

p(z l | x) = p(z l ) p(a z l ) p(a z l | x), (2)
where test class priors are considered uniform. DAP assumes attributes independence so p(a z l |x) = M m=1 p(a z l m | x) and (2) becomes:

p(z l | x) = p(z l ) p(a z l ) M m=1 p(a z l m | x), (3)
Finally, mapping function f : X -→ Z recognizes object z x , as the one having the highest posterior:

z x = f (x) = argmax z l ∈Z p(z l | x). (4)

Evaluation of zero-shot learning for haptic recognition

So far, we presented the theoretical background of attribute-based ZSL. However, its use with haptic data collected by a robot is not trivial. The specific nature of haptic data and robotic limitations add challenges that require various adaptations of the aforementioned theoretical framework. First, haptic data collection is costly, requiring the optimization of training data. Second, both the choice of attributes and the nature of haptic data depend heavily on a robot's sensing capabilities, hindering comparisons and generic solutions. Third, data gathered from the robot can be multimodal (e.g. joints, temperature. . . ) requiring processing and fusion techniques. Fourth, the spatial limitation of robot exploration leads to sparse and noisy, or missing data, which is challenging to the recognition algorithm. To assess the use of ZSL on haptics in the presence of all these difficulties, we started by applying it to the state of art PHAC-2 dataset [START_REF] Chu | Robotic learning of haptic adjectives through physical interaction[END_REF].

The performance of a recognition algorithm may be evaluated by its classification accuracy, computation speed and memory requirement. Here we concentrate on classification accuracy as we are using an offline recognition method and are not constrained by the training response time. Furthermore, we suppose that enough resources are available to run our algorithms. This was the case throughout our experiments which were carried out successfully on a PC with an Intel(R) Core(TM) i7-3840QM processor having a speed of 2.8 GHz and a RAM of 8 GB.

PHAC-2 database description

The PHAC-2 dataset5 was designed to study how a robot can learn to describe its haptic perception using words. The developers of PHAC-2 use haptic adjectives which are binary high-level physical properties of objects such as 'hard', 'absorbent' and 'bumpy'. By considering the PHAC-2 adjectives as attributes, we can apply an attribute-based approach to perform ZSL.

Several characteristics motivated us to use PHAC-2 to evaluate our ZSL framework. First, it is probably the single largest object database with objects labeled by their haptic characteristics [START_REF] Corradi | Bayesian tactile object recognition: learning and recognising objects using a new inexpensive tactile sensor[END_REF][START_REF] Gu | Haptic perception of unknown object by robot hand: Exploration strategy and recognition approach[END_REF][START_REF] Regoli | Controlled tactile exploration and haptic object recognition[END_REF]. Second, the objects in the database encompass a variety of physical properties in terms of texture, material and stiffness. Third, the adjective definition and the binary associations between objects and adjectives have been developed using an arguably unbiased procedure. And finally, PHAC-2 provides data from multiple explorations of the same object, allowing for the testing of algorithms for robustness to inter-trial variabilities.

The PHAC-2 database contains 60 objects, labeled by a broad range of material, texture and stiffness related adjectives. Twenty four haptic adjectives were defined by human volunteers for the set of objects. For our analysis we utilized 19 adjectives (after removing adjectives present in less than 3 objects). The adjectives used by us were A = {absorbent, bumpy, compressible, cool, f uzzy, hard, hairy, metallic, porous, rough, scratchy, slippery, smooth, so f t, solid, springy, squishy, textured, thick}.

In addition to the adjectives, the PHAC-2 database provides 48-dimensional vectors of raw haptic signals. These have been obtained using the Willow Garage PR2 robot 6 gripper, equipped with two BioTacs sensors 7 . The gripper provides the following kinesthetic information: aperture X g and height Z g in the robots torso coordinate frame. Additionally, each BioTac provides cutaneous information: core temperature T DC , heat flow T AC , static pressure P DC , dynamic pressure P AC and the voltage change for each of the 19 impedance sensing electrodes [E 1 . . . E 19 ]. These data items (X g , Z g and 23 signals from each BioTac) sum up to define the 48-dimensional vector. Ten vectors are provided for each object, one corresponding to each trial that had four exploration procedure, or EP (including a squeeze, hold, slow slide and fast slide on the object), making a total of 600 samples (60 objects × 10 trials).

Splitting the object set O

To perform ZSL using the PHAC-2 dataset, we split the 60 objects into two disjoint sets. To ensure that the results are independent from the chosen splits, we generated 5 splits {(Y s , Z s ), s = 1, . . . , 5} where each pair respects two constraints: Y s ∪ Z s = O and Y s ∩ Z s = ∅. To generate each pair (Y s , Z s ), we randomly chose for the test set 10 objects out of the 60: Z s = {o k ∈ O , k ∈ rand(10, 60)} 8 . The remaining 50 objects were used for training: Y s = O -Z s . However, some objects have the same attributes vector and since ZSL identifies each test object by its attributes vector, we verified that the attributes vectors of all objects in Z s were mutually different. If not, the random selection (of the 10 objects) was repeated until the condition was satisfied.

Feature Extraction

As in [START_REF] Chu | Robotic learning of haptic adjectives through physical interaction[END_REF], we considered the minimum gripper aperture, mean aperture and travelled distance for each exploration as features. This results in a vector of 12 kinesthetic features (3 features × 4 EP). For the BioTac readings, we exploit the results from [START_REF] Hoelscher | Evaluation of tactile feature extraction for interactive object recognition[END_REF][START_REF] Xu | Tactile identification of objects using bayesian exploration[END_REF][START_REF] Chu | Using robotic exploratory procedures to learn the meaning of haptic adjectives[END_REF]. First, we removed the baseline activity from the BioTac readings by subtracting the mean of the first 100 readings. Next, the average of each signal was computed over the exploration time. The computed features were then normalized to have zero mean and standard deviation of 1. Since the BioTacs are not calibrated in the dataset, we considered each BioTac separately. Thus, by concatenating features of the 4 EP, we obtained a vector of 92 features for each BioTac (23 BioTac features × 4 EP). This vector was concatenated with the gripper's 12 features, which resulted in a vector of 104 features for each BioTac. This high dimension was reduced using Principal Component Analysis (PCA) to a 25-dimensional vector that justifies 95% of the variance. To sum up, an exploration trial results in 2 vectors of haptic features x b1 , x b2 ∈ R 25 , one for each BioTac.

Attributes classification

Attributes classification aims at estimating the presence of attribute a m for a given feature vector x, i.e., to derive p (a m |x). By assuming attributes independence, we trained a Support Vector Machine (SVM) classifier for each attribute and each BioTac. When a test sample x is given to a SVM classifier, it returns a score s (x) corresponding to the distance from x to the decision boundary. The attribute posterior p (a m |x) is estimated by transforming s(x) into a probability using a sigmoid function.

We Fig. 2a shows the average classification accuracy for each attribute. We averaged the accuracies obtained from the two BioTacs classifiers that were tested on the 5 different test object sets. The results show that attributes classifiers were able to learn from training data whether an attribute is present or absent in a novel object. However, performance varied from one attribute to the other: we obtained an accuracy of 92% for attribute solid whereas we obtained modest, though still better than random, performance for attributes smooth (64%) and textured (64%). To evaluate the influence of the object splits, we plot in Fig. 2b the attributes classification accuracies for each of the five splits (Sect. 4.2), averaged on the 2 BioTac and on all attributes. The figure shows that classification accuracies are not heavily influenced by the choice of object splits: the average accuracy is 78%, with a maximum difference of only 3% between splits 2 and 5.

While above we report the choice of attribute classification assuming attribute independence, we did also consider attribute classification without the assumption of independence. For this, we trained a multi-label SVM classifier for each BioTac b i using D train,b i = {(x b i , a y ) s.t. (x, y) ∈ D train }, with the Python scikit-multilearn10 library implementing the classifier proposed in [START_REF] Read | Classifier chains for multi-label classification[END_REF], and tuned the SVM parameters C and γ using a leave-one-out cross validation. This multi-label classifier predicts, given a test sample x, a vector of posteriors, one for each attribute. However, with this multi-label classifier, we obtained an average recognition accuracy of 0.31, which is lower than that 0.39 obtained assuming independent attributes (see table 1).

Recognition of novel objects

The final step consists of combining the attribute posteriors to infer the identity of the distinct test sample x (see Sect. 3.3). Since we distinguish features from each BioTac: x = (x b1 , x b2 ), attributes posteriors p(a z l m |x b1 ) and p(a z l m |x b2 ) must be combined to infer p(a z l m |x). For this, we propose two methods. The first (that we name MAXDAP) considers only the highest posterior, by assuming that it is the most confident value between the two:

p(a z l m |x) = max(p(a z l m |x b1 ), p(a z l m |x b2 )). ( 5
)
The second method (AVGDAP) considers the average of both posteriors to eliminate the influence of misclassification:

p(a z l m |x) = p(a z l m |x b1 ) + p(a z l m |x b2 ) 2 . ( 6
)
Attributes classification posteriors are used by DAP to perform zero-shot recognition according to equation ( 3). First, we compare in table 1 the use of MAXDAP and AVGDAP to combine BioTac posteriors by assuming uniform attributes posteriors. Since we use the MAP estimator in equation ( 4) to distinguish the ten test objects, the random classification accuracy for the ZSL algorithm is 0.1. The reported accuracies were therefore significantly higher than random chance. Performances of the two methods are similar, since the objects have homogeneous properties on their surfaces, giving close attributes posteriors for the two BioTacs. We compared three methods for measuring the attribute prior p(a z l m ): 1. based on its presence in training objects: p(a m = 1) = 1 N N n=1 a y n m , 2. based on its presence in test objects: p(a m = 1) = 1 L L l=1 a z l m , 3. based on a uniform distribution: p(a m = 1) = 0.5.

Split

We refer to these methods as train-prior, test-prior and uni-prior respectively. Note that test-prior is feasible because the attribute vectors of unknown objects are available to the robot in ZSL. We report in table 2, results obtained using AVGDAP method. Results show that the three methods give similar accuracies with a slight improvement with uni-prior. This is coherent with results reported in [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], which show that attribute prior estimation is not crucial.

Robustness to the choice of objects

Considering that recognized objects are experienced for the first time, the results listed in tables 1 and 2 show that our ZSL can be used for haptic recognition. However, as one would expect, we found that the recognition results vary depending on the training and test set (see the difference in performance across the 5 splits). This is an interesting finding because Fig. 2 shows that attributes classification accuracies themselves do not differ between the splits. This discrepancy highlights the influence of attribute space, that is that object closer in attributes space are more difficult to distinguish from one another, and motivated us to test the robustness of our method with more challenging data sets, where the similarities between the training and test sets are minimal. Previously, we reported results for five random training/test splits. To minimize the similarities between the training and test data, and make the recognition more challenging for our ZSL algorithm, we considered our objects in a 2-dimensional representation of haptic similarities (Fig. 3). To obtain this representation, we used PCA to reduce the dimension of each trial feature vector from R 25 to R 2 (by considering the first two principal components to represent the objects). Following this, we averaged the 10 vectors corresponding to the 10 trials of each object so as to obtain one vector that represents the object in the 2-dimensional space. In this representation space, note that near objects feel similar when touched, and are harder to distinguish by a haptic recognition system. Therefore, to create challenging splits, we first used lines y = x and y = -x (red) and divided the object space into 4 partitions. Since the partition on the left of the graph is significantly denser than the others, we divided it again in two with line y = 0 (blue), to yield 5 partitions in total. We then considered 5 splits, by using each time one partition as test set and the 4 others for training. Since attribute-based ZSL cannot distinguish between objects having identical attribute vectors, in each test set we kept only objects having different attributes vectors. Next, to increase the attributes classifier's ability to generalize to new objects, we ensured that each attribute classifier is trained using at least 3 different objects. Overall, the above procedure helped us create challenging object splits that ensure that, a) test objects are different from training objects, and b), test objects are similar, making them harder to distinguish.

1 st principal component 2 nd principal component
In table 3, we report the classification accuracies with the challenging splits. Overall, we note a decrease in accuracies, as compared to the results obtained using random splits. This was expected since we intentionally made the ZSL task more difficult. Despite the decrease in accuracy, the recognition accuracy was above chance, highlighting the ability of the algorithm to recognize object very different from those it has trained on.

Influence of attribute numbers

Finally, we investigated the influence of the number of attributes on the ZSL accuracy. As mentioned before, we used 19 attributes to perform ZSL. To estimate the accuracy when using j attributes, we randomly picked up to 1000 combinations of j attributes out of the 19, and then averaged the accuracies of ZSL obtained with each of the combinations. We do so with j varying from 7 to 19. We started from 7 since it is the size of the minimum subset of attributes that allows to distinguish between objects. Results, reported in Fig. 4, show that, as expected, increasing the number of attributes generally improves the performance. Increasing the number of attributes improves the separation between objects, which are now represented in a higher dimensional space, and having more distinct attribute vectors between objects alleviates the influence of misclassifying them.

Summary

To summarize, we developed a variant of ZSL for haptic recognition of novel objects. Using the PHAC-2 dataset, we analyzed the influence of several factors on the performance of the haptic ZSL, including the choice of attribute prior, the way of combining the attribute posterior, the number of attributes, and the object set split. Furthermore, we showed the robustness of our algorithm by minimizing the similarity between the training and test set. We show that even in the worst case, when the training and test objects are very different, the algorithm can still give above chance recognition accuracies.

Next, we applied the developed haptic ZSL on an anthropomorphic hand system, to test its performance in a real experimental setting.

Recognition of hand held daily life objects by a robotic hand

Motivation: Beyond PHAC 2

PHAC-2 offers an important amount of objects, attributes and haptic data allowing us to test the application of ZSL on haptics. However, the database was built in a controlled setting and with objects of regular shape, that may not always be the case in a real robotics application. Therefore, in our experimental setup we used a less controlled exploration, more realistic objects with heterogeneous surface properties, and different modalities of haptic data. The goal was to show how ZSL can be applied to real life robotic applications, and to recognize any new constraints the recognition system could face in such applications. Specifically: 1. We did not have a planned object exploration strategy (like in PHAC 2), and our object exploration is achieved during open-loop random grasps. Our robot grasped the object in an unknown position and orientation, and used whatever it senses as exploration data. This made the exploration short and coarse, but crucially more realistic. 2. We used daily-life objects with semantic meanings (e.g. bottle, mug and box). The objects were not of any particular shape. In addition, we allowed objects to have heterogeneous physical properties on their surfaces, meaning that the object can feel differently depending on the touched part. This requisites the exploration to be incremental and to include different sources of information. 3. We used a dexterous anthropomorphic robotic hand that offered not only cutaneous information, but also rich kinesthetic data, typical of whole hand object grasps. 4. We made use of available (online dictionary based) textual descriptions of the object in order to avoid the timeconsuming human exploration process and minimize the human effort needed in this procedure. This is more suitable for real scenarios, since we aim at minimizing the cost of adding a new object to set O. In Fig. 5, we summarize the different steps of our robotic experimentation. First, we collected two disjoint sets of daily life objects, one for training and the other for testing. Then, we defined a list of attributes allowing us to describe both sets of objects, resulting in the object-attribute matrix K. Next, we trained binary attribute classifiers: using BioTac readings for material attributes and robot joint readings for shape attributes. Finally, haptic readings collected by exploring a new test object are introduced to the attributes classifiers and the resulting posteriors are used by the DAP model to infer the test object's identity.

Robot hand setup

Our robot setup consists of a cable-driven Shadow Dexterous Hand 11 with a BioTac sensor mounted on each fingertip. The encoders on the hand's 19 joints provide kinesthetic information x shadow = {q 1 , . . . , q 19 }, where q i is the angular position of joint i, and each BioTac provides cutaneous information x biotac = {T AC , T DC , P DC , P AC , E 1 , . . . , E 19 }.

Daily life object set

We collected a set of different daily life objects to form our object set O. Our choice was based on multiple state of art studies that established lists of real objects that are interesting for robotic manipulation. The authors of [START_REF] Choi | A list of household objects for robotic retrieval prioritized by people with als[END_REF] propose a list of objects ordered according to their relevance for automatic retrieval after surveying people with amyotrophic lateral sclerosis (ALS). A larger list of daily life objects categorized according to their use is given in [START_REF] Matheus | Benchmarking grasping and manipulation: properties of the objects of daily living[END_REF]. The YCB database [START_REF] Calli | Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols[END_REF] also regroups a set of physical objects to be used for object manipulation benchmarking. We chose our objects from these previous works that were big enough to be grasped by the Shadow Hand, and which were not hot, or have sharp edges that could damage the BioTacs. Finally, we selected the 20 objects illustrated in Fig. 6

Attribute-based description

For attribute-based ZSL, the design of the dataset requires defining: (1) training objects Y, (2) test objects Z, (3) the set A of attributes that can be derived from the data collected with our setup and (4) matrix K. First, we randomly split the object set O into two equal disjoint sets: Y = {cardboard box, glass bottle, plastic bottle, round container, mug, thermal mug} and Z = {ball, rectangular container, tube, blender leg, bowl, glass, plastic cup, measuring cup, jar, salter} as illustrated in Fig. 6. Next, we defined a set of attributes A appropriate for describing the haptic sensation of Y and Z. Here, we deal with a specific type of attribute which is semantic and binary-valued. Considering nonsemantic [START_REF] Farhadi | Describing objects by their attributes[END_REF] and real-valued attributes [START_REF] Parikh | Relative attributes[END_REF] is beyond the scope of this paper. Multiple works have studied the definition of attributes by relying on human expressing capabilities [START_REF] Chu | Robotic learning of haptic adjectives through physical interaction[END_REF][START_REF] Osherson | Default probability[END_REF] or linguistic knowledge databases [START_REF] Russakovsky | Attribute learning in large-scale datasets[END_REF][START_REF] Rohrbach | What helps where-and why? semantic relatedness for knowledge transfer[END_REF]. In PHAC-2, authors relied on human experiments to define the adjectives list, which can depend on the participants' choice of words. Here, we chose a more objective approach by making use of objects names to extract their textual descriptions from online dictionaries 12 . From these descriptions, we chose attributes that could be sensed by our robot. For instance, from the definition of a bottle: a glass or plastic container that has a narrow neck and usually has no handle, we extracted the underlined statements as attributes. Overall, by analyzing all objects descriptions, we extracted a list of 11 shared attributes : A = {porcelain, plastic, glass, cardboard, steel, cylindrical, round, rectangular, concave, has a handle, has a narrow part}.

Next, we set a o m = 1 if the attribute a m is used to describe the object o as a required property (e.g. has a narrow part for a bottle), and a o m = 0 if the attribute is an undesired or unnecessary property (e.g. has no handle for a bottle). Using this procedure we obtained the object-attribute matrix K illustrated in table 4. Training objects y n having identical a y n are indicated only once in the table, this is why we have 6 instead of 10 training objects. Note that each test object z l has a specific a z l that will be used to distinguish it from other objects during recognition. porcelain plastic cardboard glass steel cylindrical round rectangular concave has handle has narrow part cardboard box 0 0 1 0 0 0 0 1 0 0 0 glass bottle 0 0 0 1 0 1 0 0 0 0 1 plastic bottle 0 1 0 0 0 1 0 1 0 0 1 round container 0 1 0 0 0 0 1 0 0 0 0 mug 1 0 0 0 0 1 0 0 1 1 0 thermal mug 0 1 0 0 1 1 0 0 1 1 0 ball 0 1 0 0 0 0 1 0 0 0 0 rectangular container 0 1 0 0 0 0 0 1 0 0 0 tube 0 1 0 0 0 1 0 0 0 0 0 blender 0 1 0 0 0 1 0 0 0 0 1 bowl 1 0 0 0 0 0 1 0 1 0 0 glass 0 0 0 1 0 1 0 0 1 0 0 plastic cup 0 1 0 0 0 1 0 0 1 0 0 measuring cup 0 1 0 0 0 1 0 0 1 1 0 jar 0 1 0 1 0 0 0 1 0 0 0 salter 0 0 0 1 1 1 0 0 0 0 1

Data collection

The next step was data collection from the training objects, to learn attributes. We note that our attributes can be decomposed into 2 categories: material and shape attributes. We referred to the state of art on haptic exploration to choose the best procedure a hand must perform in order to recognize shapes and materials. According to [START_REF] Lederman | Hand movements: A window into haptic object recognition[END_REF], materials can be measured by performing a static contact between sensors and object surface. Shape can be inferred from the hand grasping/enclosing the object. Thus, we chose to explore each object by grasping/enclosing, which combines the exploration procedures required to perceive both, material and shape.

During the training phase, each training object was handed to the Shadow hand by an experimenter 10 times in random positions and orientations. To grasp the object, the robot fingers were spread out to their joint limits. Then, they were closed by setting a desired constant current to each joint actuator. The currents (and corresponding cable tensions) were kept low enough to avoid damage during contact with object, or in the case of self collisions, while ensuring a "good" contact between the BioTac and object. Once all joints stopped (either because the finger contacted the object, or because it reached its joint limit), the contact was maintained for 20 seconds to obtain the thermal equilibrium between sensors and object. Different examples are illustrated in Fig. 7.

Since our attributes have different nature, we assume attribute independence and learn each one separately. From each grasp, we gathered BioTac readings x biotac serving as inputs for material attributes classifiers, and encoders readings x shadow serving as inputs for shape attributes classifiers. After exploring all objects from Y, the collected data were used to build a training set D a m train for each shape attribute and D a m train,b for each material attribute-BioTac pair. Each test object was grasped up to 15 times, again using the same grasping procedure as during training, to build the test set D test . Grasps in which none of the fingers touched the object were dropped as they do not include material information. A video of the experiments is provided with this paper 13 .

Attributes classification

For material attributes, we chose the same feature extraction technique as for the previous experimental setup (see Sect. 4.3). We used a time average of the features and linear SVM as suggested in [START_REF] Hoelscher | Evaluation of tactile feature extraction for interactive object recognition[END_REF]. But in contrast to that work, since we performed a static contact, we did not consider the vibrations signal P AC from the BioTac. We obtained a feature vector of 22 normalized means which was reduced using PCA to a 8-dimensional vector x mat that explained more than 98% of the variance. The resulting x mat was used for the classification of the material attributes.

For each shape attribute, a binary classifier was trained using joints position measurements from the Shadow hand x shadow . Joints that did not contribute to the closing procedure (e.g. the wrist) were excluded, resulting in a feature vector x sh of 10 angular positions input to the shape attributes classifiers. The classification was performed using nonlinear SVM with a Gaussian kernel. We used the same sigmoid function as in Sect. 4.4 to convert the SVM classification score to an attribute posterior.

DAP -handling robotic constraints

To classify a test sample x, we introduced it in each attributes binary classifier, to obtain a set of posteriors {p(a m | x), m = 1, . . . , 19}. The attributes posteriors and object-attribute matrix are used by DAP (see Sect. 3.3) to return the object class.

When the hand grasps an object, it provides a data sample x = (x sh , x mat ) where x mat = (x 1,mat , . . . , x B,mat ) with 1 ≤ B ≤ 5 depending on the number of BioTacs in contact. Thus, the attributes classifiers return p(a m | x sh ) for shape attributes and {p(a m | x b,mat ), b = 1, . . . , B} for material attributes. These posteriors must be combined to infer the final attribute posterior required in equation [START_REF] Johnsson | Haptic perception with self-organizing anns and an anthropomorphic robot hand[END_REF].

Furthermore, the attribute posteriors have to take into account several constraints posed by our realistic experimental setup, which were absent when we worked with the PHAC-2 database:

1. Test objects can be made of multiple materials. Thus, BioTacs on different fingers can be in touch with different materials (e.g. the salter grasp in Fig. 7), requiring the integration of information from different fingers to estimate the material attributes. 2. Objects can be heterogeneous and a grasp may provide only a local view. For example, in Fig. 7 (right) the hand grasps the lower part of the bottle and misses the presence of the narrow neck. Therefore, we need to deal with information missing from touched parts and to combine grasps to obtain a global view. 3. The number of touching fingers B can vary from one grasp to another, giving a different size of the test sample each time, which must be taken into account in multi-grasp classification.

In the next sections, we propose and test different solutions proposed to take into account these constraints. 13 The video can also be found on the IDH YouTube channel: https://youtu.be/Ekd28b0BiQs

Single grasp classification -Combining information from various BioTacs

Since the test object is grasped in an unknown pose, the number of BioTacs B making contact may vary for each grasp. The contact of a BioTac is detected when the difference in static pressure exceeds a given threshold. For each material attribute a m , we obtain a set of posteriors from the contacting BioTac classifiers {p(a m | x 1,mat ), . . . , p(a m | x B,mat )}. To obtain the final attribute posterior for the material attributes, we tested both MAXDAP and AVGDAP (see Sect. 4.5) to combine classifications from the contacting BioTacs. MAXDAP considers only the BioTac that is most confident about the presence of the a m , and was implemented as:

p(a m = 1 | x) = max b=1,...,B p(a m = 1 | x b,mat ). (7)
AVGDAP on the other hand, combines all contacting BioTacs by averaging their posteriors:

p(a m = 1 | x) = 1 B b=1 p(a m = 1 | x b,mat ). (8)
Finally, while the material attribute posteriors were obtained using either MAXDAP or AVGDAP, the shape attributes were assessed as:

p(a m = 1 | x) = p(a m = 1 | x sh ).

Single grasp classification -Handling local view

In regard to both material and shape attributes, a grasp may miss some attributes that are not present in the touched part of the object. This implies that if p (a m = 1 | x) < 0.5, the attribute could be absent from the whole object or only from that particular grasp. To alleviate the effect of a possible misclassification, we replace the attribute posterior with a uniform distribution if the attribute is absent from x. For shape attributes:

p(a m = 1 | x) = max(0.5, p(a m = 1 | x sh )), (9)
and for material attributes:

p(a m = 1 | x) = 1 B B b=1 max(0.5, p(a m = 1 | x b,mat )). (10)
We refer to this method as 'Local DAP' or LDAP.

Multi-grasp classification -Developing a global view

While single grasp classification can recognize objects by making some assumptions about absent attributes, combining several grasps is obviously advantageous as it gives a wider view of the object, and thus is expected to improve recognition performance [START_REF] Schmitz | Tactile object recognition using deep learning and dropout[END_REF]. Grasping an object T times in different positions results in a set x = {x (1) , . . . , x (T ) } of test samples. We compare two approaches to exploit information from multiple grasps: data fusion and decision fusion.

Data fusion merges data from the T grasps to form one "super grasp", that can be used to classify an object just like a single grasp. Formally, for shape attributes, we have:

p(a m = 1 | x) = 1 T T t=1 max(0.5, p(a m = 1 | x (t) sh )), (11)
and for material attributes:

p(a m = 1 | x) = 1 T T t=1 1 B t B t b=1 max(0.5, p(a m = 1 | x (t) b,mat )). (12)
We refer to this method as 'Data Fusion for Multi-grasp DAP ' or DF-MDAP. Decision fusion separately classifies each grasp in the sequence starting from x (1) until k similar classifications are made or until x (T ) is reached. The classification that reaches k is taken as final decision on the object. We refer to this method as 'Similar classifications for Multi-grasp DAP' or SC-MDAP.

Experimental evaluation and results

We made a series of experiments to evaluate the multiple adaptations of DAP to our experimental setup. For better analysis, we decomposed the test set Z into 3 subsets: objects having homogeneous material and shape properties Z hom ={ball, rectangular container, tube}, objects whose shape properties can differ according to the touched part Z het,sh ={blender leg, bowl, glass cup, plastic cup, measuring cup} and objects made of multiple materials: Z het,mat ={jar, salter}.

Attributes classification

First, we evaluate the performance of the binary classification of attributes. In Fig. 8, we present the accuracies of attributes classifiers on the test set. All accuracies are averaged across the test trials. Material attributes are additionally averaged over the five BioTacs. Overall, attribute classification achieved a satisfying average accuracy of 89%. However, attribute plastic had an accuracy of 45% which is considerably lower than the rest. This was probably due to the variety of plastic types used in the training and test sets. For example, the round container is made of a softer plastic than the blender leg.

S l,m =|| a z l m - 1 | D z l test | x∈D z l test p(a m = 1 | x) || 1 , (13)
D z l test represents the set of test samples gathered from object z l . We note that the majority of distances are lower than 0.5, indicating that objects are well classified by the corresponding attribute classifier. However, some classifiers perform badly on some objects, probably because these are too different from the ones they have trained on. For instance, the plastic attribute column presents high distances for the ball, rectangular container and jar because of the difference between the plastic material constituting these objects, and that constituting the training objects.

Single grasp DAP

Recognition accuracies of test objects using a single grasp are presented in table 5. First, it is clear that the accuracy obtained with homogeneous objects is better than that obtained with heterogeneous ones. This is because all object properties can be felt from a single grasp. Furthermore, all BioTacs touch the same material over a homogeneous object, "collaborating" to give a more confident classification. However, this collaboration becomes more delicate for objects belonging to Z het,mat , which explains the accuracy deterioration. Moreover, we note that AVGDAP performs better than MAXDAP for heterogeneous objects. This is probably because AVGDAP averages the decision from the BioTacs, and is hence less sensitive to errors from individual sensors.

However, LDAP outperforms all methods because we found that more often than not, random grasps on daily life objects lead missing some parts of the object, and the LDAP can deal efficiently with absent attributes. Note that table 5 should however be viewed considering the fact that single grasps, by nature, are constrained by local object properties. Consider the three example grasps on the measuring cup during the single grasp classification (Fig. 10). We found that for 83% of grasps having the fingers touching the cylindrical part, the measuring cup is classified as a tube. All the grasps touching the upper side lead to the measuring cup being classified as the pink cup, and 66% of grasps touching the handle are classified as a measuring cup. These cases should not all be considered as misclassifications because this is the best decision that can be made from the given local grasp. For instance, a human touching the cylindrical part of the measuring cup can also not be sure if this is the tube, the pink cup or the measuring cup, since the three objects share the same local shape. Hence, table 5 shows the limitation of single grasps for object recognition.

Multi-grasp DAP

Using a single grasp, LDAP gave the best recognition accuracy of 59.63% over all objects, which is modest, hence encourages multi-grasp recognition. Next, we checked the recognition accuracies during multiple grasps, using both DF-MDAP and SC-MDAP. In Fig. 11, we analyzed the DF-MDAP and SC-MDAP performances by changing their parameters T and k, respectively. We note that DF-MDAP accuracy increases constantly with the number of merged grasps T . This is understandable as an increase of T improves the information contained in the "super grasp". Since LDAP gave the best accuracy, we used it to perform each single grasp classification for SC-MDAP. For SC-MDAP, the best accuracies were obtained between k = 2 and k = 4. The presence of an optimal k can be explained by the fact that, while increasing the number of required similar classifications helps removing noisy classifications, it becomes harder to find k similar classifications when k becomes large. Overall, both methods improve the performance compared to single grasp recognition (k = 1, T = 1). Figure 12 shows the confusion matrices with different DAP choices. The best classifications were achieved with SC-MDAP. We note that multi-grasp recognition was very beneficial to correct Z hom misclassifications, to the point that SC-MDAP reached an accuracy of 100% for all objects. For Z het,sh , we note that the blender, bowl and glass are correctly classified by both multi-grasp methods. The plastic cup is still frequently confused with a tube by DF-MDAP but these misclassifications were overcome with SC-MDAP. However, all methods performed weakly on the measuring cup and on the jar, which were often confused with the tube. This was because the attributes classifications were poor on these objects (see distance matrix in Fig. 9), subsequently influencing their classification.

Overall we could analyze and develop a haptic ZSL algorithm for an anthropomorphic hand that enables very good recognition of daily life objects that the robot encounters for the first time (see Fig. 12c).

Discussion and Conclusion

In this work, we proposed a haptic zero shot learning algorithm that enables a robot to recognize novel objects, which it has never been trained on before. First, we applied ZSL to probably the best haptic object database for ZSL (the PHAC-2), which includes 60 objects with a wide variety of texture, material and stiffness properties. This allowed us to analyze the possibilities and constraints associated with the application of ZSL to haptic recognition. We used this analysis to understand if and how the choice of attribute classifier (Sect. 4.4), the way of combining attribute posteriors (Sect. 4.5), the choice of training set (Sect. 4.6 ) and the number of attributes (Sect. 4.7) affect recognition performance. The algorithm developed from this analysis was then applied on an anthropomorphic robot hand, to make it recognize unexplored objects (Sect. 5). In this realistic scenario, we further optimized our algorithm and enabled it to account for heterogeneous objects (Sect. 5.7) and to integrate information from multiple grasps (Sect. 5.7.3). Our final algorithm enabled the robot to correctly recognize eight out of ten objects, that it grasps for the first time (third panel of Fig. 12).

Note that the recognition rate in our real robot experiment (Fig. 12) was much better than in the PHAC-2 database recognition (Tables 1 and2). Comparison is difficult, as different objects were used in the two cases. However, it is still interesting to note that the real robot implementation worked better, considering that it used coarse grasps to explore heterogeneous objects of different shapes, compared to recognition with the PHAC-2 database in which objects were homogeneous, regularly shaped, and explored using a regular and well controlled procedure. However, we believe that the results of our robot experiment are in fact a better indicator of the capabilities of our algorithm for haptic object recognition. This is because of several reasons. Primarily, the PHAC-2 database focuses on the tactile properties of objects, and not on shape (all objects were in fact of the same shape in the dataset). On the other hand, shape is definitely a fundamental feature for haptic recognition during grasps, and our robot is able to integrate both tactile and shape information (through its joint angles) efficiently to improve object recognition. Moreover, while homogeneous objects (as in PHAC-2) may intuitively feel to be easier to recognize, this may not always be true. Most real life objects are heterogeneous and in fact the heterogeneous nature of an object, if explored well (we propose one way in Sect. 5.5), can act as a signature for the object, making it easier to recognize. This was probably the case in our robot experiment as well. Finally, the recognition of the two discussed features, i.e material heterogeneity and shape, are further improved by our algorithm by using multiple grasps in our robot experiment. In PHAC-2 on the other hand, the homogeneity of objects and regularity of explorations make the data similar across exploration trials, making simulated multiple grasps irrelevant.

The results obtained with our robotic setup show the promising capabilities of haptic ZSL for object recognition. This is very encouraging, given that this is still a prototype system that can be improved in several aspects. First, in this work we focused on the ZSL problem by assuming that all evaluated objects have never been experienced before. However, robots encounter previously explored as well as unexplored objects, and the integration of ZSL with other suggested multi-class object recognition systems can enable them to recognize known objects and progressively integrate novel ones. Second, since the goal of this study was to introduce and adapt ZSL to haptics, we adopted a simplistic approach using semantic binary attributes and the DAP model. Semantic representation and learning can be improved by investigating new techniques as those presented in [START_REF] Xian | Zero-shot learning -the good, the bad and the ugly[END_REF]. Finally, in this work we study haptic object recognition by assuming that vision is unavailable, with the prospect in the future of combining the results with visual object recognition to improve general object recognition by machines [START_REF] Gao | Deep learning for tactile understanding from visual and haptic data[END_REF]. In summary, our work highlights the efficiency of ZSL for haptic recognition, and we hope it will provide a start to works for optimizing haptic object recognition by autonomous robots in uncertain environments.

Figure 1 :

 1 Figure 1: Attribute-based zero-shot learning: system overview.

Figure 2 :

 2 Figure 2: Recognition accuracy of attributes binary classification.

Figure 3 :

 3 Figure 3: Splitting the PHAC-2 objects set into five partitions.

Figure 4 :

 4 Figure 4: Recognition accuracy vs. the number of attributes utilized for ZSL.

Figure 5 :

 5 Figure 5: Our zero-shot recognition system overview.

  to form our object set.

Figure 6 :

 6 Figure 6: Disjoint training (left) and test (right) object sets.

Figure 7 :

 7 Figure 7: Examples of object grasps by the Shadow Hand.

Figure 8 :

 8 Figure 8: Recognition accuracy of attributes binary classification. To analyze the performance of attributes classifiers on each object, we averaged p(a m = 1 | x) for each objectattribute pair across the test trials of each object. Ideally, if x is collected from object z l then p(a m = 1 | x) should be close to a z l m since p(a m = 1 | x) ∼ 1 if a z l m = 1 and p(a m = 1 | x) ∼ 0 otherwise. Matrix S presented in Fig. 9 measures L 1 distance between attributes binary labels and their posteriors:

Figure 9 :

 9 Figure 9: Distance between attributes binary labels and their posteriors for test objects.

Figure 10 :

 10 Figure 10: Representative grasps on the "measuring cup" show how different grasps may indicate distinct kinematic features.

Figure 11 :

 11 Figure 11: Recognition accuracy vs. number of combined grasps T for DF-MDAP (a) and vs. number of similar classifications k for SC-MDAP (b).

Figure 12 :

 12 Figure 12: Confusion matrices of LDAP , DF-MDAP (T = 5) and SC-MDAP (k = 4) respectively. SC-MDAP achieved perfect recognition of 8 out of 10 new objects.

Table 1 :

 1 ZSL accuracy for independent attributes classifiers.

		MAXDAP AVGDAP
	1	0.48	0.48
	2	0.38	0.36
	3	0.38	0.38
	4	0.35	0.35
	5	0.35	0.36
	Average	0.39	0.39

Table 2 :

 2 DAP accuracy for independent attributes classifiers.

	Split	uni-prior test-prior train-prior
	1	0.48	0.24	0.36
	2	0.38	0.33	0.28
	3	0.38	0.39	0.32
	4	0.35	0.39	0.38
	5	0.35	0.33	0.29
	average	0.39	0.34	0.33

Table 3 :

 3 DAP accuracy for challenging splits.

Table 4 :

 4

Class-attribute matrix K for training (upper) and test (lower) objects.

Table 5 :

 5 Recognition accuracy of DAP on test objects from a single grasp.

Email adresses: firstname.lastname@lirmm.fr

Email adress:g.ganesh@aist.go.jp

Merriam-Webster dictionary

Since we study object recognition, a class here represents an object category.

https://github.com/IanTheEngineer/Penn-haptics-bolt

http://www.willowgarage.com/pages/pr2/overview

https://www.syntouchinc.com/sensor-technology/

rand(k, n) returns k random numbers in range 1, . . . , n.

http://scikit-learn.org/stable/

http://scikit.ml/

shadowrobot.com

We used merriam-webster.com and en.oxforddictionaries.com.

Acknowledgement

We would like to thank the authors of [1] for providing us with the data and the support for using the PHAC-2 database. Zineb Abderrahmane is supported by the Ministry of Higher Education and Scientific Research of Algeria through the Excellence Fellowship.


Olivier Flasseur

Loïc Denis

Corinne Fournier

Éric Thiébaut

Robust object characterization from lensless microscopy videos

Lensless microscopy, also known as in-line digital holography, is a 3D quantitative imaging method used in various fields including microfluidics and biomedical imaging. To estimate the size and 3D location of microscopic objects in holograms, maximum likelihood methods have been shown to outperform traditional approaches based on 3D image reconstruction followed by 3D image analysis. However, the presence of objects other than the object of interest may bias maximum likelihood estimates. Using experimental videos of holograms, we show that replacing the maximum likelihood with a robust estimation procedure reduces this bias. We propose a criterion based on the intersection of confidence intervals in order to automatically set the level that distinguishes between inliers and outliers. We show that this criterion achieves a bias / variance trade-off. We also show that joint analysis of a sequence of holograms using the robust procedure is shown to further improve estimation accuracy.

I. INTRODUCTION

Thanks to the massive development of low-cost imaging sensors and the simultaneous increase in computational capabilities, lensless microscopy is emerging as a method of choice for time-resolved analysis of microscopic objects. It is already used in several fields where the accurate estimation of 3D location and size over time is crucial, such as in the study of fluid flows or biomedical imaging [START_REF] Greenbaum | Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy[END_REF].

Analysis of digital holograms is traditionally performed by first reconstructing the 3D volume by light back-propagation, then analyzing this 3D volume to segment, locate and size objects [START_REF] Kreis | Handbook of holographic interferometry: optical and digital methods[END_REF]. However, this approach suffers from artifacts due to the imperfect reconstruction step, in particular to spatial distortions close to the image borders and spurious diffraction patterns known as twin images in holography [START_REF] Kreis | Handbook of holographic interferometry: optical and digital methods[END_REF]. Location and size estimates can be significantly improved by using a maximum likelihood estimator, i.e., by fitting a model of the diffraction pattern generated by the object of interest directly on the hologram rather than by reconstructing the image volume. This method is very successful for the study of isolated objects [START_REF] Soulez | Inverse-problem approach for particle digital holography: accurate location based on local optimization[END_REF], [START_REF] Wang | Holographic characterization of protein aggregates[END_REF], but some bias appears when unwanted objects in the field of view alter the hologram. These objects may have complex or unknown shapes and their influence on the hologram may be difficult to explicitly model and account for.

We suggest replacing the maximum likelihood estimator with a robust estimator so that any notable discrepancies between the model of the pattern of interest and the data due to unwanted objects have less influence on the final estimate. Section II provides a brief presentation of the lensless microscopy principle and the model of the diffraction pattern generated by spherical objects behaving as opaque objects such as droplets, cells or cocci bacteria. Section III then describes the proposed robust estimation procedure to characterize microscopic objects. Special attention is paid on the setting of the level to distinguish between inliers and outliers. An automatic criterion based on the intersection of confidence intervals is proposed to achieve a bias / variance tradeoff. Section IV illustrates the performance of the proposed method on holographic videos.

II. CHARACTERIZATION OF MICROSCOPIC OBJECTS WITH

LENSLESS IMAGING

Lensless microscopes are based on the in-line holographic setup originally proposed by Dennis Gabor [START_REF] Gabor | A new microscope principle[END_REF]. The objects are illuminated by an incident monochromatic plane wave of wavelength λ. After free-space propagation over distance z p , a hologram d is recorded by a monochromatic image sensor, see Fig. 1.

Based on diffraction theory, it is possible to model the diffraction pattern created on the hologram plane by an object of interest of given 3D shape, absorption and optical index. The case of opaque spherical objects is of particular practical interest since it covers opaque particles introduced within the flow in fluid mechanics, and colored spherical objects such as cocci bacteria or cells in flow cytometry in biology. When the distance z p is such that z p 4π.r 2 p λ , the diffraction pattern of an opaque sphere located at (x p , y p , z p ), with radius r p , is [START_REF] Tyler | Fraunhofer holography applied to particle size analysis a reassessment[END_REF]:

m p (x, y) = π r 2 p λ z p J 1c 2π r p ρ p λ z p sin π ρ p λ z p , (1)
where ρ p = (x -x p ) 2 + (y -y p ) 2 is the transversal distance to the center of the sphere and J 1c (u) = J 1 (u)/u is the cardinal Bessel function of first order. More generally, in the following m θ (a, b) denotes the intensity of the diffraction pattern generated at the pixel location (a, b) on the sensor by an object of interest. This object of interest is fully characterized by a vector of parameters θ (e.g., for an opaque sphere θ = {x p , y p , z p , r p , }). Maximum likelihood estimation of the parameters θ of the object of interest, under the hypothesis of additive white Gaussian noise, amounts to solving a nonlinear least-squares problem [START_REF] Soulez | Inverse-problem approach for particle digital holography: accurate location based on local optimization[END_REF]:

θ(ML) = arg min θ A a=1 B b=1 w(a, b) • [m θ (a, b) -d(a, b)] 2 , (2)
where d(a, b) represents the recorded data, w(a, b) ∝ 1/ (a, b) 2 is a weight that is set to zero for defective pixels and that can account for a non-stationary noise variance (a, b) 2 , and A and B are the height and width of the sensor, in pixels.

III. PROPOSED ROBUST ESTIMATION SCHEME

A. Robust estimation by reweighted least squares

To reduce the impact of outliers on the estimation, Huber introduced the so-called M-estimators [START_REF] Huber | Robust statistics[END_REF] that replace the least squares by another objective function ρ in order to reduce the penalization of the largest deviations between the model and the data. Minimization problem (2) is then replaced by:

θ(M) = arg min θ A a=1 B b=1 w(a, b)•ρ m θ (a, b) -d(a, b) s , (3)
where ρ is a non-negative, continuous and symmetric function with a minimum equal to 0 at 0, and s is a parameter that scales the residuals by setting the level that distinguishes between inliers and outliers. Depending on the choice of the function ρ, the minimization problem (3) can be difficult to solve. A simple algorithmic strategy leading to a local minimizer consists in solving a sequence of least-squares problems of the form (2): the Iteratively Reweighted Least Squares (IRLS) [START_REF] Holland | Robust regression using iteratively reweighted least-squares[END_REF]. In this scheme, the estimate θk+1 at iteration k + 1 is obtained by reweighting the squares with weights w k (a, b) derived from the residuals of iteration k:

θk+1 := arg min θ A a=1 B b=1 w k (a, b) • [m θ (a, b) -d(a, b)] 2 , (4)
where the weights w k (a, b) are computed from the residuals

r k (a, b) = m θ k (a, b) -d(a, b
) at iteration k according to an update rule that depends on the objective function ρ:

w k (a, b) = w(a, b) • s r k (a, b) • ∂ρ(u) ∂u u=r k (a,b)/s . (5)
Many different objective functions ρ and their associated weight update rules have been studied and classified in the literature [START_REF] Dennis | Techniques for nonlinear least squares and robust regression[END_REF]. So-called hard-redescenders are functions ρ that are constant above a given threshold, thereby leading to assign zero weights to data points that display a misfit larger than the chosen threshold. So-called soft-redescenders are functions ρ whose limit at infinity is finite, thus asymptotically producing zero weights for large residual values. For lensless applications, we selected a soft-redescender to achieve a good compromise between bias reduction and loss of efficiency of the estimator. In the following, we use the Cauchy objective function:

ρ(u) = 1 2 log(1 + u 2 ) , (6)
which gives the following weight update:

w k (a, b) = w(a, b) • 1 + (r k (a, b)/s) 2 -1 . (7)

B. Automatic tuning of parameter s: distinguishing inliers and outliers

Adequate tuning of the parameter s is essential in robust estimation schemes [START_REF] Rousseeuw | Robust regression and outlier detection[END_REF]. On the one hand, if s is too low, most residuals r k (a, b) will be (much) larger than s and the weights of the corresponding data points will decrease in the following iteration, thereby discarding most of the signal of interest. In this case, the result will be an increase in the variance of the estimation. On the other hand, if s is too large (much larger than the typical discrepancy of outliers), the M-estimator tends to the least squares estimator and loses its robustness properties. In this case, reduction in the bias caused by outliers is no longer obtained.

A common method to estimate the scale parameter s in the case of i.i.d. noise is based on the median absolute deviation (MAD) estimator, which provides a robust estimate of the residuals standard-deviation [START_REF] Huber | Robust statistics[END_REF]:

ŝ(MAD) = 1.48 • median [ |r -median(r)| ] , (8)
where r denotes the collection of all residuals values r(a, b)

for a ∈ [1, A] and b ∈ [1, B].
MM-estimators have been proposed to achieve high robustness to the presence of outliers while conserving high efficiency in the absence of outliers [START_REF] Yohai | High breakdown-point and high efficiency robust estimates for regression[END_REF]. These estimators consist in two steps: (i) estimation of the scale of the residuals s using an M-estimator defined by a hard-redescender ρ; and (ii) actual estimation of the parameters of interest θ with a more efficient penalty function ρ (such as a soft-redescender).

One drawback of these methods of setting parameter s is that they do not account for the actual impact of the outliers on the estimation. When considering unwanted objects within the field of view, the location of the objects with respect to the object of interest may or may not lead to biases. In other words, while ŝ(MAD) is suitable in cases where the outliers are uniformly distributed in the measured signal, a more appropriate choice of s could be made in cases of nonuniform distribution of outliers.

We propose to estimate parameter s using a method that accounts for the actual impact of the outliers on the estimation. Starting from the value ŝ(MAD) , which may be overconservative, we apply the intersection of confidence intervals (ICI) rule to decide up to what point parameter s can be increased to reduce estimation variance before being subject to bias due to outliers. The ICI rule has been successfully applied in image denoising to locally select the largest neighborhoods on which to perform denoising by local polynomial approximation [START_REF] Katkovnik | Adaptive window size image de-noising based on intersection of confidence intervals (ici) rule[END_REF]. Its application only requires knowledge of the estimate along with its standard deviation.

The vector of parameters θ is estimated using the robust IRLS estimation scheme (4) by progressively increasing parameter s until the new estimate θ(M) s obtained with parameter s leads to a confidence interval [ θ(M) s -δ s , θ(M) s + δ s ] with an empty intersection with the preceding confidence intervals:

ŝ(ICI) = max s ∈ ŝ(MAD) , ∞ ∃θ, ∀s0 ∈ ŝ(MAD) , s , θ ∈ θ(M) s 0 -δs 0 , θ(M) s 0 + δs 0 . ( 9
)
This principle is illustrated in Fig. 2 considering a single parameter θ.

To build the confidence intervals, we used the Cramér-Rao Lower Bounds (CRLBs) [START_REF] Stuart | Kendall's Advanced Theory of Statistics : Distribution Theory[END_REF]. The CRLB δ(i) represents the minimum standard-deviation for any unbiased estimator of the parameter θ(i). Under additive white Gaussian noise, the least squares estimator (2) asymptotically reaches the CRLB given by:

δ 2 (i) = I(θ) -1 i,i , (10)
where Fisher information matrix I is defined by:

[I(θ)] i,j = A a=1 B b=1 w(a, b) ∂m θ (a, b) ∂θ(i) ∂m θ (a, b) ∂θ(j) , (11)
In the presence of outliers, we approximate the confidence intervals by computing the CRLB with the weights obtained by the IRLS procedure.

To reduce the computational cost of estimating parameter s according to the ICI rule (9), we consider a non-uniform discretization of the set ŝ(MAD) , ∞ such that two successive s values lead to a given decrease in the CRLBs. In practice, each s value is obtained by bisection to achieve the prescribed CRLBs reduction.

We now illustrate the tuning of parameter s based on the ICI rule (9) on a simple 1D problem: estimating the x p Fig. 3. Estimation of the position of a pattern by a robust approach using the MAD and the proposed ICI rule. Evolution of ŝ and MSE are respectively in dash and solid lines. location of a Gaussian-shaped pattern m. We simulate 100 noisy realizations d by adding to the model a Gaussian white noise (with a peak signal-to-noise ratio of 10) and a constant outlier formed by a rectangular function with 3 samples in width (with a peak signal-to-noise ratio of 1), see Fig. 3(a).

The ICI rule leads to values of s that are about 5 times larger than ŝ(MAD) when the outlier is too far to introduce a significant bias, and to values of s close to that of ŝ(MAD) when the outlier affects the estimation, see Fig. 3(b). When ŝ(ICI) is higher than ŝ(MAD) (resulting in a weaker penalization of the residuals), the mean square error (MSE) on the location of the pattern is reduced by a factor between 2 and 4. In practice, the empirical standard-deviation reaches the CLRB when the outlier has little impact on the estimate. A difference of a few tens of percent is observed when the constant outlier has an impact on the estimate since the estimator is no longer asymptotically unbiased. Determination of s with an ICI rule is still possible. In conclusion, choosing the parameter s with the ICI rule improves the mean square error compared to a MAD estimator.

IV. APPLICATION TO LENSLESS MICROSCOPY VIDEOS

In this section, the performance of the robust estimate scheme applied in lensless microscopy is evaluated on two lensless imaging applications. The goal is to accurately estimate the radius r p of spherical objects. We also evaluate the performance of a robust estimation performed jointly over several consecutive frames (super-resolution). In these examples, the image formation model m θ is non-linear with respect to the parameters θ, as given by equation [START_REF] Greenbaum | Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy[END_REF]. As a result, the minimization problem (2) and each IRLS iteration in (4) is solved using a few iterations of Levenberg-Marquardt algorithm. The original videos and the results of the robust estimation are available online at http://perso.univ-st-etienne. fr/deniloic/robust lensless video.html.

A. Holographic video of an object of constant size

We consider a 200 frames video of a slightly moving particle located near an unwanted object. The object of interest is a circular chromium deposit on a glass plate, with a calibrated radius of 50 µm (Optimask TM , diameter ± 1 µm, roundness error ± 0.25 µm). The unwanted object is a fixed human hair [START_REF] Verrier | Digital holography super-resolution for accurate three-dimensional reconstruction of particle holograms[END_REF].

Since the outlier is fixed throughout the video, it is possible to roughly remove outlier pixels by applying a static binary weighting mask w(a, b) = 0 if pixel (a, b) is in the masked region, w(a, b) = 1 otherwise. Figure 4 shows two frames from the video and the manual removal of the outlier by masking. Due to the mask w, part of the signal due to the particle of interest (diffraction rings) is not accounted for.

Since the radius r p of the particle remains constant during acquisition, it is possible to jointly estimate this parameter in several consecutive frames. This super-resolution approach has already been successfully applied in lensless microscopy considering the non-robust minimization problem (2) [START_REF] Verrier | Digital holography super-resolution for accurate three-dimensional reconstruction of particle holograms[END_REF]. We propose to combine this super-resolution approach with a robust estimation scheme (minimization problem (4)) considering 40 stacks of 5 successive frames.

In each the vector of parameters θ is estimated by least squares fitting, Eq. ( 2), with constant weights (hereafter denoted single -without mask) or with the binary mask (denoted single -with mask), and using the robust Mestimator, Eq. ( 4) and ŝ(MAD) (denoted single -robust). In the case of a joint estimation, the term single is replaced by the term joint in the denomination of the methods tested. Figure 4 is an example of the weights used for the last IRLS iteration. The weights are close to a segmentation of the pixels impacted by the unwanted object. The diffraction rings characterizing the particle of interest are well preserved (weights close to 1). Figure 5 shows the evolution of the estimated radius during the IRLS iterations for the first 5 frames of the video. This shows that the algorithm converges in a few iterations and that the joint estimate of the radius carried out by super-resolution is very close to the ground truth equal to 50 µm.

Table I lists the mean estimated radius r(M) p , the corresponding standard-deviation σ and the CRLB δ obtained with ŝ(MAD) for each method tested. The manual removal of the outliers appears to reduce the estimation bias. The bias is further reduced when a robust weighting of residuals by IRLS is applied since it preferentially eliminates the outliers while retaining the signal of interest. As a result, the estimation using a robust approach leads to a very small increase in CLRB which is negligible compared with the reduction of the bias. The combination of a robust estimation method with a super-resolution scheme (joint -robust) exploiting the information redundancy between different frames significantly reduce the estimation standard-deviation. In agreement with statistical results [START_REF] Robinson | Statistical performance analysis of superresolution[END_REF], we observed a decrease in the standarddeviation by a factor at least equal to √ 5. This joint robust estimation approach enables accurate (i.e. low bias and low standard-deviation) estimates of the radius of the holographic objects and thus improves the overall performance of a conventional least squares estimation process.

B. Holographic video of evaporating droplets

We used a 160 frames video of an evaporating diethyl ether droplet dispersed in a homogeneous and nearly isotropic turbulence that is tracked over time. As the particle evaporates, it produces a plume of vapor that diffuses around it. The vapor plume creates outliers since its signal mixes with the signal produced by the particle of interest. Other evaporating particles passing close to the particle of interest also act as outliers. The evaporation rate of these particles was recently studied using the least squares approach (2) [START_REF] Marié | Digital holographic measurement of the lagrangian evaporation rate of droplets dispersing in a homogeneous isotropic turbulence[END_REF].

Figure 6(b) shows two frames from the video and the 2D trajectory of the particle over time (superimposed color line).

Figure 6(a) shows the evolution of the rescale factor of the residuals obtained with a MAD estimator and with the proposed ICI rule as well as the IRLS weights for two different frames. The ICI rule leads to a larger value of s than MAD but without notably modifying the estimation of the radius r p . This is the case when outliers do not disturb the estimate like in frame 60 (left column) since the particle of interest is isolated and the plume of vapor is rather small. In contrast, ŝ(MAD) and ŝ(ICI) are very close when the particle of interest is strongly disturbed by outliers. This is illustrated by frame 97 (right column) where another evaporating particle is close to the particle of interest and disturbs its signal. These observations underline the fact that the MAD estimator only accounts for the presence of outliers strongly deviating from the model but does not account for their spatial localization and hence, their actual impact on the estimate. Figure 6(c) illustrates the evolution of confidence intervals for increasing values of s. Figure 6(d) shows the weights obtained at the end of the IRLS procedure for ŝ(MAD) (first row) and ŝ(ICI) (second row). On frame 60, ŝ(ICI) is significantly larger than ŝ(MAD) , leading to larger weights, while on frame 97 ŝ(MAD) = ŝ(ICI) and the weights are identical.

Figure 7 shows the evolution of the estimated radius over time with the three methods of estimation tested. The robust estimation scheme leads to an average increase of 0.4 µm in the estimated radius. We also observed that it reduces the estimation standard-deviation. These experimental results are explained by the fact that the moving outliers (vapor plume plus other particle) cause a non-stationary disturbance of the signal of interest over time. The robust approach considering a MAD or ICI rule to set the parameter s lead to close results in this application. This is due to the fact that the weighting matrix obtained with ŝ(MAD) already guarantees a high level of efficiency (around 90% of the maximum efficiency obtained without reweighting). V. CONCLUSION

We have shown that the use of a robust approach to estimate the 3D position and size of objects in lensless microscopy videos reduces the estimation bias caused by unwanted objects. We proposed a simple method to estimate the scale parameter that distinguishes between inliers and outliers in the robust estimation scheme. This method is based on an intervals confidence intersection rule and achieves a better bias / variance trade-off than the usual MAD rule. Finally, we have shown that the combination of the proposed robust approach with a joint estimation scheme also reduces the standard-deviation of the estimation and consequently improves the overall performance of the method.

Fig. 1 .

 1 Fig. 1. Lensless microscopy setup and object fitting.

Fig. 2 .

 2 Fig. 2. Estimation of s with the ICI rule.

Fig. 4 .

 4 Fig. 4. Calibrated particle near an outlier -example of a frame.

Fig. 5 .

 5 Fig. 5. Estimated radius as a function of the number of IRLS iterations.

Fig. 6 .

 6 Fig. 6. Robust detection and tracking of an evaporating droplet. (a) evolution of ŝ(MAD) and ŝ(ICI) . (b), (c) and (d) respectively give the holograms, confidence interval diagrams, and robust weighting maps for two video frames. (holographic video credit: Fluid Mechanics and Acoustics Laboratory LMFA, CNRS-Université de Lyon, France)

Fig. 7 .

 7 Fig. 7. Estimated radius over time with the different methods tested.

  Work supported by CNRS grant DETECTION (DEFI IMAG'In 2015)

TABLE I MEAN

 I ESTIMATED RADIUS, STANDARD-DEVIATION σ AND CRLB δ

	FOR THE DIFFERENT METHODS TESTED
	Method:	r(M) p	(µm) σ (µm)	δ (µm)
	single -without mask	49.71	0.31	0.014
	single -with mask	49.78	0.25	0.016
	single -robust	49.91	0.26	0.015
	joint -without mask joint -with mask joint -robust	49.71 49.78 49.93	0.13 0.08 0.09	0.014 / 0.016 / 0.015 /	√ √ √	5 5 5