Franck Gonzalez
email: franck.gonzalez@cea.fr

Florian Gosselin
email: florian.gosselin@cea.fr

Wael Bachta
email: bachta@isir.upmc.fr

A Framework for the Classification of Dexterous Haptic Interfaces Based on the Identification of the Most Frequently Used Hand Contact Areas

Keywords: Dexterous haptic interfaces, Interaction taxonomies, Hand contact surfaces, Quantified design trade-off

Haptic interfaces are purposed for the simulation of physical interactions with a virtual environment in a realistic way. Their design is often based on a compromise between the interaction capabilities and mechanical design complexity. This paper is aimed at providing a methodology that helps tuning this trade-off. A survey of both manipulation and exploration taxonomies allows the identification of elementary hand contact areas with their associated frequency of use. The relative importance of these areas is illustrated by the building of an interaction map for the hand. Some combinations of these elementary areas, with their frequency of use, are then organised into a graphical tree as a function of their complexity. This tree allows to review some existing devices and to propose some guidelines for improved designs.

INTRODUCTION

Haptic interfaces are purposed for the simulation of physical interactions with a virtual environment in a way as realistic as possible. This asks for the highest level of transparency, i.e. the user should be able to grasp and move the interface in the same way as real objects and get the same feedback as for real interactions. Therefore, such systems should be designed to perfectly track the hand motions and to apply forces on any area of its surface. This challenges the current available technological solutions. Interaction capabilities of haptic interfaces are then often reduced to keep an acceptable level of technological complexity: a trade-off is made between interaction and design efficiencies.

Throughout the literature, three categories of haptic interfaces can be considered. The first one, including most of the commercially available interfaces, is based on universal 3 or 6 degrees of freedom (DOF) mechanisms, equipped with handles or tool-shaped props. Although allowing a realistic tool manipulation simulation, these devices offer a limited subset of tasks and reduce the users dexterity. The second category is made up of interfaces that focus on finger interactions. Because of the difficulty to address all degrees of freedom of the hand, developers of such systems often focus on specific activities. These tasks are analyzed to narrow the number of DOF to the required minimum. Two examples can be found in [START_REF] Gosselin | UE MUVII, spécifications préliminaires de l'interface haptique IKD[END_REF] and [START_REF] Trannoy | IHS10, Etude des interactions conducteur commandes véhicule d'une voiture Peugeot 308[END_REF]. In the former, several educational applications involving fine manipulation of small objects are reviewed: it led to the conclusion that a two-finger device allowing for interactions with the index and thumb fingertips is sufficient for such tasks. An interface based upon these specifications was developed with only 12 passive and 6 active DOF (respectively 6 passive and 3 active DOF for each finger) [START_REF] Gosselin | Design of a wearable haptic interface for precise finger interactions in large virtual environments[END_REF] and proved useful for practical teaching applications [START_REF] Christodoulou | Haptic devices in virtual reality used for education: designing and educational testing of an innovative system[END_REF]. In the latter, a study of hand interactions with a car cockpit led to the development of a three-finger interface [START_REF] Gosselin | Guidelines for the design of multi-finger haptic interfaces for the hand[END_REF] addressing the thumb, index and middle fingertips. Both interfaces allow efficient interactions but are limited to the considered applications. The third category is composed of the encounter-type interfaces, i.e. reconfigurable devices that can physically replicate various object shapes in the real world. The hand is theoretically thought to interact with these objects in a realistic way. As an example, the authors in [START_REF] Yokokohji | Designing an encountered-type haptic display for multiple fingertip contacts based on the observation of human grasping behavior[END_REF] roughly extracted contact profiles from grasps of various daily life empirical objects. They concluded that the contact points are distributed over a sphere which centre is located in the middle of the back of the hand. Their device was designed to conform to this shape as much as possible. It is however limited to such geometries.

This short survey reveals that haptic interfaces design is usually application-dependent, and points out the lack of a general methodology allowing to easily tune the trade-offs between the interaction capabilities and mechanical complexity. This paper is aimed to provide a tool that could help in choosing an appropriate design. It consists in a graphical tree organising different combinations of elementary contact areas of the hand, associated with their frequency of use during technical manipulations, as a function of their complexity. The tree is constructed thanks to a survey and a review of the existing hand manipulation and exploration taxonomies. The use of the proposed tree is illustrated by a ranking of some available haptic interfaces.

The remainder of the paper is organised as following: the review of the existing taxonomies is achieved in section 2. This allows extracting elementary contact areas, which are first organized into a hand interaction map to highlight their relative importance. The construction of the graphical tree is then presented. In section 3 the ranking of some available haptic interfaces is developed, and some guidelines for improved designs are discussed.

CLASSIFICATION OF THE HAND INTERACTIONS

Analysis of Hand Interaction Patterns

Hand interactions can be divided into manipulation, in which the hand is mostly used to act on the environment, e.g. moving an object, and exploration, characterizing movements performed in order to acquire further information about the surrounding objects, e.g. their temperature. Both aspects are to be investigated hereinafter.

Manipulation has been widely studied in the field of ergonomics. Much of grasping research refers to the work of Schlesinger, summarized in [START_REF] Taylor | The anatomy and mechanics of the human hand[END_REF]. Six different grasps are depicted: cylindrical, fingertip, hook, palmar, spherical and lateral, driven by the object shape. Napier introduced a distinction between power grasps, utilized in case of tasks involving strength, and precision grasps for fine control [START_REF] Napier | The prehensile movements of the human hand[END_REF]. The posture of the hand is said to be mostly influenced by the object being grasped, and by the intended activity. Kamakura et al. proposed another classification of static prehension patterns with novel intermediate grip and adduction grip categories along with the power and precision ones [START_REF] Kamakura | Patterns of static prehension in normal hands[END_REF]. The most known taxonomy of human grasps was proposed by Cutkosky [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF]. It is based on previous works, esp. [START_REF] Taylor | The anatomy and mechanics of the human hand[END_REF] and [START_REF] Napier | The prehensile movements of the human hand[END_REF], and observations in workshops, using task dexterity and precision as discriminants. As a whole, 16 different patterns are depicted. Contrary to [START_REF] Kamakura | Patterns of static prehension in normal hands[END_REF], only two categories are considered, i.e. power and precision. Intermediate graps are missing, e.g. tripod grip and its variations used for example for holding a pen. Besides, numerous variations on the grasps are noticed to depend on the context (sizes of the hand and object, personal pref-erences). These works were extended by Feix et al. [START_REF] Feix | A comprehensive grasp taxonomy[END_REF]: based on a review of 17 taxonomies, mostly variations on [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF], 33 characteristic patterns are proposed and sorted in an array. The classification is based on the power/precision requirements and considers power, intermediate and precision grasps. The role of the thumb is also emphasized, being in adducted or abducted position. One can note that other approaches exist for the classification of human manipulation behaviours. As an example, [START_REF] Bullock | Classifying human manipulation behavior[END_REF] proposes to also take the hand's or objects' motions into account, esp. within hand. However, the categorization is less detailed for prehensile patterns. Consequently, we will refer to [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF] and [START_REF] Feix | A comprehensive grasp taxonomy[END_REF] for our work.

As regards exploration, researchers seem to have come to the agreement that some invariant properties during movements are maintained in order to evaluate the weight, shape, temperature, size and stiffness of a material [START_REF] Jones | Human Hand Function[END_REF]. Six different patterns are highlighted, each one associated to a specific property [START_REF] Klatzky | Object recognition by touch. In Blindness, brain plasticity and spatial function[END_REF]: Lateral motion (texture): fingers rub across the surface; Unsupported holding (weight): the object is held in a hand and there is hefting of the arm and wrist; Pressure (hardness): normal forces are applied; Enclosure (global shape and volume): the hand is in contact with as much of the object's envelope as possible; Static contact (temperature): one hand passively rests on the object; and Contour following (global and exact shape): the hand moves dynamically while in contact with the object.

Each pattern can be further analyzed in order to get the associated hand-environment interaction area. From the observation of the 34 manipulation patterns (the 33 patterns from Feix et al. [START_REF] Feix | A comprehensive grasp taxonomy[END_REF] plus the non-prehensile power grasp defined by Cutkosky which is not included in Feix et al.'s array), in the framework of [START_REF] Kamakura | Patterns of static prehension in normal hands[END_REF], we identified 21 characteristic interaction areas as a whole, from the inner side of the index and middle fingers (M 1 ) to the whole inner hand surface (M 21 ). Similarly, from the 6 stereotypical exploration patterns, we extracted 4 characteristic contact areas, from the index fingertip (E 1 ) to once again the whole inner hand surface (E 4 ).

Figure 1 maps these interaction patterns with the corresponding areas. A dark grey box means that a given pattern (in the box' column) can be performed if a given interaction area (in the box' row) is involved, and conversely that this interaction area allows performing this manipulation or exploration pattern (in order to limit the number of columns, some of them gather several patterns which involve the same contact areas). Of course these areas are inclusive, as shown with light grey boxes (if a contact area encompasses another one, it allows performing the included one's patterns). White boxes mean incompatible patterns and contact areas.

As a whole, a given interaction area allows performing all interaction patterns marked with a light or dark grey box in the corresponding line. The whole hand allows all kinds of interactions while the index fingertip allows only few exploration patterns. Similarly, a given pattern can be performed provided that any contact area marked with a light or dark grey box in the corresponding column be involved.

In each column, the dark grey box is the lowest grey box. It refers to the minimal contact area strictly necessary to perform the pattern(s) in this column, which we will call in the sequel reference interaction area for this (these) pattern(s). Conversely, in each row, it is the left-most (resp. right-most) grey box for manipulation (resp. exploration) patterns. It refers to the pattern(s) that cannot be performed if other interaction areas which do not encompass it are involved. Such pattern(s) will be called below reference interaction pattern(s) for this area.

Adding Interaction Patterns Frequencies

Figure 1 shows how many types of interaction patterns can be performed for a given interaction area. Intuitively, the number of available interactions increases in accordance to the size of the interaction area. This information alone is however not sufficient to guide the design of a generic haptic interface. The most important goal is not to be able to perform the highest number of patterns but those which are the most frequently used. This requires the knowledge of the relative frequency as to which each pattern is used, i.e. how long each interaction pattern occurs over a given amount of time.

As regards manipulation, this knowledge can be retrieved from [START_REF] Zheng | An investigation of grasp type and frequency in daily household and machine shop tasks[END_REF]. The authors give the frequency of use of each grasping pattern for machinist activities. This data corresponds to the associated reference interaction area defined in section 2.1 and can be associated with the dark grey box in the referring pattern's column and area's row (no percentage is associated with the light grey boxes). As for exploration, [START_REF] Klatzky | Object recognition by touch. In Blindness, brain plasticity and spatial function[END_REF] indicates the typical duration of each exploration pattern. Given an a priori equal importance to all environment properties, a pattern frequency (and thus the frequency of the corresponding reference interaction area) can be obtained by dividing its duration by the sum of the durations of all patterns. In a similar way to manipulation, frequencies are connected to the related dark grey boxes. It is worth noting that a reference interaction area may correspond to several manipulation or exploration patterns (e.g. M 6 ). In this case the frequency given in the area's dark grey box is the sum of the frequencies associated with each pattern.

Considering typical applications of haptic interfaces (e.g. digital design, fitting, ergonomic studies, exploration of virtual worlds and their physical properties -esp. inaccessible ones like nanoscale behaviour), we hypothesized that most applications require exploration (as defined in [START_REF] Klatzky | Object recognition by touch. In Blindness, brain plasticity and spatial function[END_REF]) and manipulation in workshop-like environments (as studied in [START_REF] Zheng | An investigation of grasp type and frequency in daily household and machine shop tasks[END_REF] for a machinist), with a ratio of 3/4 for manipulation and 1/4 for exploration (for specific activities, their relative importance may be different). The aforementioned frequencies of use are combined through these ratios to obtain a weighted average for each reference interaction area. It is worth noticing that a null exploration (resp. manipulation) contribution is assigned for the exclusive manipulation (resp. exclusive exploration) reference interaction areas. This way, the relative frequency of use of each of the 24 interaction areas (21 for manipulation, 4 for exploration, 1 being common to both) can be obtained and reported in the lines of figure 1 (one can notice that the machinist does not use 3 of the 34 manipulation patterns. They are not reported in figure 1, so as for the associated interaction areas).

Building a Hand Interaction Map

By overlapping the 24 interaction areas and combining their associated frequencies, a hand interaction map can be obtained (see figure 2). As a whole, we can distinguish more than 20 different elementary interaction areas, i.e. skin surfaces that can be combined to produce any of the interaction areas. As an example, the thumb fingertip, involved in the interaction areas M 2 to M 13 and M 15 to M 21 , is used 77 % of the time as a whole.

At first glance for such technical activity, the map fits the intuition: the fingertips are the most utilized parts of the hand, then the rest of the fingers, and the palm comes in the end. A closer look reveals that the thumb and index fingertips are the most used, then comes the middle, then equally the lateral surface of the index and the ring pad.

It is worth noting that the palm involves 9 elementary interaction areas between which the frequency of use does not vary from more than 2 percent. Similarly, the metacarpus involves 6 areas among which 5 have no more than 2 percent difference. We chose to gather these zones. This way a simplified hand interaction map is obtained, based on only 12 simplified interaction areas (SIAs). All interaction areas can still be obtained from these SIAs, however some of them make use of a limited part of the palm and metacarpus areas. Figure 3: A partial tree of the interaction areas and their combinations as a function of the number of their constitutive simplified interaction areas (SIAs) and the percentage of time they allow to interact naturally with the environment.

Building a Hand Interaction Tree

The interaction map gives an insight on how long and which areas are utilized by a particular worker, over his actual activities.

Although it gives some cues on which elementary areas are to be taken into account for the building of a generic haptic interface, this information is not sufficient since it does not allow sorting the capabilities of a haptic interface as a function of its complexity.

To make this classification possible, a tree is built (figure 3) using the interaction areas and some of their combinations. They are sorted as a function of a complexity index, defined as the number of SIAs that are part of each area. If the palm or metacarpus is partially involved, it is accounted for 1/2 SIA (e.g. M 11 ). For example, the interaction area M 5 is composed of 6 SIAs (thumb, left and right side of index, left and right side of middle, and ring fingertips, see figure 2). It is therefore located above the areas composed of 5 or less SIAs and below those composed of 7 or more SIAs.

This tree is composed of the 21 interaction areas from figure 1 and some of their combinations. Among the numerous possible combinations, only those which significantly increase their potential efficiency index are retained. The efficiency index of a given area is defined as the sum of the frequencies of use of the patterns it grants access to, since this value gives an estimation of the time during which this area will allow a natural interaction with the environment. This index is computed as the sum of the frequencies of use of the reference interaction areas encompassed in the considered area. It can also be obtained as the sum of the indexes of the lower areas it is connected to, minus the indexes that are accounted for several times due to multiple links, plus the percentage of this area. As an example, the index of the area M 5 is the sum of the indexes of the areas E 2 and M 4 , minus the index of the area E 1 which is included in both E 2 and M 4 , plus the specific frequency of M 5 (as given in figure 1), i.e. 19.1 +30.2 -15.5 +5.7 =39.5 %.

Two areas are linked if one of them encompasses the other. Multiple links are possible if a given area is included in several others. However, redundant links are to be avoided. As an example, area M 17 includes area E 1 but no link is drawn between them as E 1 is already taken into account through M 3 , M 4 , M 5 and M 6 , M 6 being linked to M 17 .

One can note from figure 1 that some reference interactions areas are rarely used (e.g. M 8 is used only 1.3% of the time). As a consequence, they are considered as being less interesting as they allow only few more natural interactions while being more complex (e.g. M 8 has a complexity index of 5 and allows interacting naturally with the environment 20.4% of the time, i.e. only 1.3% more than M 3 for an added complexity index of 2). Those areas are displayed in grey boxes in figure 3.

The tree in figure 3 illustrates the complexity and efficiency of the various interaction areas in a synthetic way. A critical path appears in thick line that distinguishes, in function of the number of SIAs, the areas to be taken into account to grant access to the most utilized patterns over time.

It is worth noting that the efficiency index of a given area represents the maximum amount of time it allows to interact naturally with the environment. Building a generic haptic interface allowing to track and apply forces on this area (i.e. on each of its constitutive Table 1: A summary of human abilities [START_REF] Jones | Human Hand Function[END_REF][START_REF] Brooks | Telerobotic response requirements[END_REF][START_REF] Kunesch | Invariant temporal characteristics of manipulative hand movements[END_REF][START_REF] Jones | Dextrous hands: Human, prosthetic, and robotic[END_REF]. elementary areas) is not sufficient to reach this index. Therefore tracking and feedback must be perfect (i.e. along all directions, over full position and force range and bandwidth, with a stiffness from 0 to the minimum perceived as infinitely stiff, etc.). In practice, no interface allows such perfect local interaction and the theoretical maximum efficiency index is hardly reached. The tree of interaction remains however a useful tool to theoretically sort the devices as a function of their potential (i.e. at best) interaction efficiency, along with other criteria like for instance the number of DOF with force feedback, the maximum amount of force or the feedback bandwidth. As an example, table 1 gives an insight on the required performances for a perfect interaction at the fingertips. Similar data could be obtained for the phalanxes and palm.

3C LASSIFICATION OF DEXTEROUS HAPTIC INTERFACES

Use of the Interaction Tree to Sort Haptic Interfaces

Dexterous haptic interfaces can be associated with the areas they allow interacting with (i.e. with the leaves of the interaction tree). This way, they can be sorted as a function of their complexity and their efficiency. The higher the complexity index (i.e. the more SIAs considered), the more independent effectors required to track the hand surface and apply forces on it. The higher the efficiency index (i.e. the higher associated percentage), the larger amount of time the patterns they grant access to are used. A perfect dexterous haptic interface is due to simulate every interaction. This would require the use of the entire hand glabrous skin, i.e. the whole hand interaction area on top of the interaction tree. To our knowledge, no existing interface allows such whole Figure 4: Some existing interfaces sorted as a function of their complexity and efficiency.

hand interaction with haptic feedback. In practice, dexterous haptic interfaces have limits and can only render at best the patterns allowed with the interaction area they grant access to. Some examples will be given below.

In order to simplify the discussion, we will focus on force feedback and hypothesize that rendering spatial and temporal high frequency information can be obtained with miniature tactile feedback devices mounted on the end effectors of the considered haptic interfaces (see e.g. [START_REF] Wang | Compact, portable, modular, high performance, distributed tactile transducer device based on lateral skin deformation[END_REF], [START_REF] Benali Khoudja | Vital: an electromagnetic integrated tactile display[END_REF], [START_REF] Sarakoglou | A high performance tactile feedback display and its integration in teleoperation[END_REF] for a few examples of such devices).

Review of Existing Dexterous Haptic Interfaces

Without being exhaustive, a few examples are given in figure 4: • Two thumb and index fingertip devices are first considered, the WHIPFI [START_REF] Gosselin | Design of a wearable haptic interface for precise finger interactions in large virtual environments[END_REF] and the PERCRO PFHE [START_REF] Frisoli | Kinematic design of a two contact points haptic interface for the thumb and index fingers of the hand[END_REF]. Both devices have a complexity index of 3. Both are composed of two 6-DOF robots with 3 motors each. They have a sufficient workspace and allow force feedback in all directions. They theoretically allow simulating patterns used 19.1 % of the time. The WHIPFI however has a limited force capability (0.8 N continuous and 4.7 N peak) and its maximum perceived stiffness is only 900 N m -1 . It is not sufficient to simulate stiff surfaces and its practical efficiency index is below the theoretical value of 19.1 %. The PFHE has higher force capability (more than 4 N continuous and 25 N peak) and stiffness (minimum mechanical stiffness in the worst case: 5900 N m -1 ).

Is allows a better fingertip interaction and an efficiency index close to 19.1 %, however at the expense of a more cumbersome and heavier design (it must be fixed on the forearm);

• A three fingertips device is also considered, the IHS10 Glove [START_REF] Gosselin | Guidelines for the design of multi-finger haptic interfaces for the hand[END_REF]. Its complexity is ranked as 5. It is composed of three 5-DOF robots with 3 motors each. Its workspace is sufficient for transparent use and it allows a force feedback in all directions at the fingertips. This device theoretically allows the simulation of the patterns used 30.2 % of the time. Its force capability (4 N continuous and 10 N peak) and stiffness (up to 5000 N m -1 control stiffness in reference configuration) allow approaching this theoretical value;

• The Rutgers Master II [START_REF] Bouzit | The rutgers master 2 -new design force-feedback glove[END_REF] is given as an example of a four fingertip haptic interface. The theoretical complexity of the interaction area it is associated with equals 6 and it theoretically allows simulating the interaction patterns used 39.5% of the time. It is composed of 4 small air cylinders located inside the palm and attached to the fingertips. It is very light and compact. However, this technological solution limits the fingers movements and provides force feedback (16 N max continuous force) only in hand closure direction. In practice, its complexity is below 6 (it makes use of only 4 motors) but its efficiency index is well below 39.5 % (although a precise value is difficult to provide, as a first approximation, an estimate between 10 and 15 % seems reasonable, i.e. about 1/3 of the maximum theoretical value as it allows force feedback in only 1 out of 3 directions at fingertips);

• Finally, the Cybergrasp from CyberGlove Systems company (www.cyberglovesystems.com) is given as an example of a five fingers dexterous haptic device. Its complexity index equals 7 and it theoretically allows simulating the interaction patterns used 45.7 % of the time. In practice it makes use of a semi-exoskeleton architecture, and allows applying forces on the fingertips and middle phalanxes. These movements are however coupled and the corresponding areas cannot be addressed separately. It is therefore considered as a fingertip device. Its workspace is sufficient to cover the whole fingers movements. However, in order to limit its weight, a remote actuation is implemented. Transmission cables introduce friction and flexibility and the force feedback is of medium quality. Moreover, the force feedback is only provided in hand opening -closure movement. Consequently, the actual efficiency index is well below the theoretical value of 45.7 % (here again, it can be estimated between 10 and 15 % as force feedback is provided in only 1 direction -12 N max continuous -while 3 would be necessary to simulate any interaction at fingertip).

It is worth noting that the tree is only valid for dexterous haptic interfaces allowing to track and apply forces on different skin areas independently. One could argue that a universal device like the Virtuose 6D 35-45 from Haption company (www.haption.com) allows whole hand interactions. It is however untrue. It allows only simulating small diameter, medium wrap, adducted thumb and index finger extension in a realistic manner, plus ring and inferior pincer. Those grasps are used only 14.3 % of the time. In this case the whole hand surface is involved but with a fixed shape. Consequently, the percentage of time the associated patterns are used drops from 100 % to 14.3 %.

Guidelines for the Design of a Novel Interface

Though not exhaustive, figure 4 shows a representative panorama of existing dexterous haptic interfaces. Let alone some complex laboratory prototypes, no four-or five-finger device exists that allows force feedback in 3 independent directions on fingertips. As a result, existing interfaces have a limited real efficiency. It is suggested to do research on simple alternative technologies allowing the development of such a device without relying on a too complex architecture. A particular attention should be paid to kinematics, design of the joints and actuation in order to allow for a sufficient workspace and force amplitude yet with a light structure. This way an efficiency up to 39.5 (for a four fingers interface associated with M 5 ) or 45.7 % (for a five fingers device associated with M 6 ) could be reached.

Considering figure 3 and its critical path, an interesting alternative could be the interaction area obtained as a combination of M 5 and M 2 . It appears as a good compromise between complexity and efficiency. At the expense of a limited added complexity with respect to M 5 , which itself seems attainable with the same technologies as used for example on the IHS10 Glove, it allows a higher theoretical efficiency index, up to 53.6 % (even more than M 6 ).

Finally, to approach the maximum above-mentioned theoretical efficiencies, it is also necessary to do further research on the association of such force feedback devices and miniature tactile interfaces allowing to render complementary touch information.

CONCLUSION AND DISCUSSION

In this paper, we presented how hand interactions patterns can be analysed to extract associated interaction areas which can be used to set up both a hand interaction map and a hand interaction tree. The interaction tree gives a synthetic overview of the complexity and efficiency of the various interaction areas. It can be used to sort dexterous haptic interfaces as a function of the areas they grant access to. This way, theoretical complexity and efficiency indices can be associated with various interface designs. It was used to critically review some existing devices and to propose guidelines for the design of novel interfaces. Other aspects of the haptic rendering (e.g. temperature, texture, etc ...) should be taken into account to effectively reach the maximum efficiency index.

This analysis is based on the hypothesis that the interactions of a machinist are representative of generic VR applications. It could be enriched with data covering a larger set of applications. The authors encourage people involved in the study of humans at work to enrich Feix et al.'s database (i.e. frequency of use of the interaction areas for various activities).

Figure 1 :

 1 Figure 1: The interaction areas and their associated frequency of use as a function of manipulation and exploration patterns.

Figure 2 :

 2 Figure 2: Hand interaction map for a machinist (for 75 % manipulation, 25 % exploration), (a) elementary interaction areas, (b) simplified interaction areas.

ACKNOWLEDGEMENTS

The authors wish to thank Prof. Vincent Hayward (ISIR) for his valuable comments and advice in writing this paper.


Emir Vela
email: evela@utec.edu.pe

Moustapha Hafez
email: moustapha.hafez@cea.fr

Stéphane Régnier
email: stephane.regnier@upmc.fr

Automated Vision-Based System for Parallel Contactless Micromanipulation

This paper presents the automated parallel noncontact manipulation of glass beads ranging from 30 up to 300 µm in size under water. Non-contact micromanipulation is performed by generating controllable laser-induced thermocapillary flows that are capable of dragging the beads. Automated manipulation process is achieved with visual servoing in order to accurately manipulate the beads in a parallel and high-speed manner. Image correlation allowed to detect the bead positions and to provide these positions to a mirror scanner that addressed the IR laser beam at a certain position from the bead. By scanning the laser beam from one bead to another, automated parallel manipulation of beads at speeds in the range of mm•s -1 was demonstrated.

I. INTRODUCTION

The growing miniaturization, development and production of microsystems in both the industrial and research field require versatile and autonomous micromanipulation systems of microcomponents. The diversification, in consequence complexity, of these systems such as MEMS, MOEMS, bioMEMS, hybrid electronic devices, needs the use of several microfabrication processes in order to engineer microcomponents made of different geometries, materials and physical properties. These components have to be assemble into microsystems for broad applications. Due to the potential market of micro and nano devices, industries need to improve their throughput. Indeed, the micromanipulation and assembly of microcomponents rely on operators thus limiting large production. However, autonomous micromanipulation and assembly systems could achieve cost-effective devices with high throughput. This field is rapidly progressing as autonomous systems are of great interest to massively parallel manipulate and assemble microscale parts. Several research groups are working on micromanipulation methods. Most works are based on mechanical or contact micromanipulation modes [START_REF] Sitti | Two-Dimensional Fine Particle Positioning Under Optical Microscope Using a Piezoresistive Cantilever as a Manipulator[END_REF], [START_REF] Dionnet | Vision and Force Based Autonomous Micromanipulation[END_REF], [START_REF] Probst | A Microassembly System for the Flexible Assembly of Hybrid Robotic MEMS Devices[END_REF]. However, contact micromanipulation does not allow parallel manipulation capabilities at high-speed due to the limited displacement speed of microtools (tens of microns per second). Non-contact micromanipulation are more suitable for parallel manipulation. Among non-contact methods such as magnetic fields [START_REF] Lee | Manipulation of biological cells using a microelectromagnet matrix[END_REF], [START_REF] Furlani | Magnetophoretic separation of blood cells at the microscale[END_REF], dielectrophoresis [START_REF] Abidin | High-gradient electric field system for the dielectrophoretic separation of cells[END_REF], [START_REF] Rosenthal | Dielectrophoresis Traps for Single-Particles Patterning[END_REF], electroosmosis [START_REF] Hwang | Electro-osmotic Propulsion of Helical Nanobelt Swimmers[END_REF], electrowetting [START_REF] Ren | Dynamics of electro-wetting droplet transport[END_REF], acoustic waves [START_REF] Shi | Acoustic tweezers: patterning cells and microparticles using standing surface acoustic waves (SSAW)[END_REF], [START_REF] Manneberg | Flowfree transport of cells in microchannels by frequency-modulated ultrasound[END_REF], thermocapillary flows [START_REF] Basu | Virtual microfluidic traps, filters, channels and pumps using Marangoni flows[END_REF], [START_REF] De Saint | Laser switching and sorting for high speed digital microfluidics[END_REF], [START_REF] Hu | Micro-Assembly Using Optically Controlled Bubble Microrobots In Saline Solution[END_REF], optical tweezers have achieved a more advance level of automation [START_REF] Arai | Synchronized laser micromanipulation of multiple targets along each trajectory by single laser[END_REF], [START_REF] Tanaka | Automated manipulation of non-spherical microobjects using optical tweezers combined with image processing techniques[END_REF]. Nevertheless, it is limited to objects up to tens of microns in size, transparent spherical objects and it reaches about tens of picoNewtons in forces. In this work a fully automated non-contact manipulation system capable of dealing with the challenges mentioned above is presented. The non-contact method used was the generation of highly localized microflows in a thin water layer using an infrared (IR) laser beam. The focused IR laser irradiated from below the liquid water layer thus producing a localized heat source by laser absorption. As a consequence, the established heat gradient at the liquid free surface generated unbalanced surface stresses. These stresses caused convection flows in order to reach an energetic equilibrium in the liquid medium, the Marangoni effect. The flows are toroidal shaped around the heat source. A significant advantage of this method is that microobjects can be dragged by the flows regardless of their shape, material NI Card scanner electronic board Fig. 2. Experimental setup is composed by a commercial inverted microscope, a high-speed monochrome camera, a laser shot system, a mirror scanner, a NI card, electronic boards and a computer. The IR laser beam is directed by the mirror scanner through the microscope objective towards the sample, where microobjects are to be manipulated. The inset shows the mirror scanner integrated under the nosepiece and the high-speed camera located at the left port of the microscope.

composition and range size. While the IR laser beam is irradiating the liquid, the objects, inside the convection flows, move towards the laser beam. When the IR laser beam is switched off, the heat gradient disappears and therefore the convection flows become null (Fig. 1). As a result microobjets stop moving [START_REF] Vela | Non-contact mesoscale manipulation using laser induced convection flows[END_REF].

To accurately manipulate the microobjects using this method, it was necessary to automate the manipulation process. A visual servoing system was implemented with a high-speed camera that provided the images containing the objects while moving. Then using image processing methods the objects positions were obtained in order to be used as positions feedback in the implemented controller algorithm. Thus the IR laser shot was controlled regarding the positions of the microobjects. Furthermore, a strategy to accurately manipulate two glass beads in a parallel manner and at speeds in the range of mm•s -1 was demonstrated.

The scenarios of interests are the manipulations of hybrid microscale parts for self-assembly for high-throughput microsystems production, or even the contactless manipulations of biological cells by only using liquid flows that could help to investigate in cells behaviors.

II. EXPERIMENTAL SETUP

The experimental setup consists of an inverted microscope (Olympus IX71) with a 4× objective (numerical aperture NA= 0.1; working distance = 18.5 mm) and an 1.6 magnification lens giving a total magnification of 6.4, it allowed to have a visible workspace of about 5×5 mm 2 . The manipulated objects were as large as 0.3 mm in size. An IR laser beam of 1480 nm wavelength, 120 mW of maximum power and a collimated radius of 3 mm was used, it was provided by OCTAX Microscience GmbH. The laser shot system was mounted at the microscope rear port. The IR laser has an absorption coefficient in liquid water of 23.45 cm -1 . Distilled water was used as the liquid medium. The filling depth was up to 0.6 mm for most experiments while maintaining a free surface, water-air interface. As a container, a petri dish made of soda-lime-glass with 60 mm in diameter and 12 mm in height was used. Considering all the optical components that the laser beam past through, including the petri dish, a 80 mW of power was measured at the output (120 mW of input power). This output power is used to irradiate the liquid layer, according to the Beer-Lambert law and taking 0.6 mm of water depth, a power of 3.13 mW is obtained at the liquid-air interface. As a result, about 96% of the laser power is absorbed by the water layer. In order to monitor the sample and movement of the microobjects, a high-speed DALSA Genie Monochrome CMOS camera with 1 inch CMOS sensor and 1400×1024 pixels at a frequency acquisition rate of 60 fps was installed in the left port of the microscope. This camera allowed to take pictures by varying the image pixels and frame rate. Additionally, a C-mount Olympus lens adapter of 0.33× magnification was coupled to the camera in order to enlarge the field of view displayed on the computer screen. Fig. 2 illustrates the experimental setup, where a dichroic mirror (from Octax GmbH) was installed under the microscope nosepiece. This reflected the IR laser beam into the entrance pupil of the microscope objective in order to be focused into the sample. Furthermore, the mirror was mounted on a 2 degree of freedom electromagnetic actuator to move the mirror and in consequence to direct the laser beam at any x-y position of the sample workspace. A NI USB-6210 DAQ card was used to drive the mirror scanner and the laser beam shot.

A. Two DOF Tip-Tilt Mirror Scanner

As the setup is based on a commercial microscope, all components of the system were designed or chosen in order to be mounted on it. A 2-DOF mirror scanner was needed in order to direct the laser beam at the sample workspace and control it. Due to compactness, a custom-made TIP/TILT mirror scanner (Fig. 3(a)) was preferred to galvanometric mirrors [START_REF] Guillemot | High-throughput laser printing of cells and biomaterials for tissue engineering[END_REF], [START_REF] Bennet | Optically trapped microsensors for microfluidic temperature measurement by fluorescence lifetime imaging microscopy[END_REF]. This device was developed by the CEA-LIST. IIt is a compact mechatronic device actuated by four electromagnets in a pushpull antagonistic configuration. The forces exerted by the actuators are illustrated in Fig. 3(a. The magnitudes of these forces are identical for each magnet. They are placed on four cardinal positions around the pivot of the mirror holder. Hence a couple of electromagnets gives 1 rotational degree of freedom. Due to its dimensions (50×50×15 mm 3 ) this device can be easily mounted on an inverted microscope, under the nosepiece as shown in Fig. 3(c). A dichroic mirror is mounted on this device, at a 45 • position, in order to reflect the IR laser beam into the objective. In this configuration, the dichroic mirror makes a 45 • angle with the incident laser beam and the microscope optical axis. It allowed to direct the laser beam at the sample without interfering with the imaging light directed towards the high-speed camera, thus reducing considerably the laser beam optical path and distortions.

III. AUTOMATION

A. System Calibration

The micromanipulation system had to be calibrated, due to geometrical distortions introduced by the mirror scanner and optical components, in order to have an accurate and precise control of the microobjects positions and manipulation. The IR laser beam, directed by the TIP/TILT mirror scanner into the sample, was not projected exactly on the desired position in the sample. For instance, if a square position reference was given to the scanner controller, in the sample the expected projected square trajectory of the laser beam was a curved trajectory. These distortions are explained in [START_REF] Hafez | Compact fast-steering tip/tilt laser scanner for high power material processing applications[END_REF]. To correct the distortions, a third degree polynomial approximation was used as follows:

x c = a x x+b x y +c x x 2 +d x xy +e x y 2 +f x x 2 y +g x x 3 +h x y 2 x, (1) y c = a y x+b y y +c y x 2 +d y xy +e y y 2 +f y x 2 y +g y x 3 +h y y 2 x,

(2) where x c and y c are the corrected positions to be sent to the scanner controller after calculation of eq. ( 1) and ( 2), x and y are the needed positions of the laser beam in the sample. To calculate the coefficients of this equation system, at least eight positions values of x and y are needed to obtain an equation system of eight equations and eight unknowns respectively . The indexes 1 up to 8 represent the eight measured laser positions in the sample as follows:

   x 1c . . . x 8c    X =    x 1 y 1 x 2 1 • • • x 3 1 y 2 1 x 1 . . . . . . x 8 y 8 x 2 8 • • • y 3 1 y 2 8 x 8    P ×    a x . . . h x    C , (3)
where X is the corrected position vector, P is the matrix containing the terms of the polynomial and C is the coefficient vector of the polynomial, this gives X = P C. In order to solve this system, that means to find the coefficient vector, a reference position set of eight values was used for X. The values of P were calculated with eight measured laser positions (Fig. 4(a)). The eight laser positions (x, y) in the sample were captured by a camera and calculated by image segmentation. In other words, the mirror scanner was commanded by eight X values that directed the laser beam at eight positions in the sample, then the eight laser positions (x, y) were measured in order to compute the matrix P , finally the coefficients C were calculated according to C = P -1 X. The same process was performed for Y vector. Once C obtained, the polynomial equations system (Eq. ( 1) and ( 2)) for X and Y was introduced in the scanner controller. For instance, Fig. 4(b) shows the desired linear trajectory positions of the laser beam in the sample (square marks), these are introduced to Eq. ( 1) and ( 2) in order to compute the command positions to be given to the mirror scanner (circle marks). As the scanner was commanded with voltage and the measured positions in the sample were in pixels, a linear correspondence was calculated in order to convert pixels to volts.

B. Image Processing

In the calibration process image segmentation was performed to obtain the laser positions from the sample. For the visual servoing of the automated micromanipulation system, an image correlation method was preferred because it is robust, fast and accurate for small regions of interests (ROIs). Fig. 5 shows the image correlation used to automate the manipulation process with position feedback. In order to track an object, a template of this (inset: top right of the figure) is first captured. The template is then correlated with the image ROI to calculate the correlation coefficients (Fig. 5(b)). The position of the maximum coefficient gives the position of the object, the brightest dot in the image. The image processing methods were performed using the OpenCV library.

C. Control Algorithm

The automation of the micromanipulation system was possible due to the predictable behavior of the convection flows and negligible inertial effects at this scale length. The control algorithm was developed in order to automatically manipulate microbeads in a parallel manner and at high speed. For instance, the handling of two glass spherical beads ranging from 30 to 90 µm in diameter are reported. In addition, with the algorithm presented below, spherical beads up to 300 µm can be automatically manipulated. This algorithm was developed based on a visual servoing in which the beads are tracked while moving until they reaches their target position. The control algorithm was as follows (Fig. 6):

1) Initialization: Detection of the glass beads using image segmentation within an image frame of 500×500 pixels in size. 2) Image templates of the beads are captured and regions of interest (ROI) are located at the beads centre of mass. 3) Reference target positions are introduced by an user with a computer mouse or written in the program graphical unit interface (GUI). 4) Closed-loop starts: Image correlation starts to track the beads and give the calculated positions to the mirror scanner controller. 5) A vector path is computed between the bead and target positions.

6) The laser beam is directed by the scanner at a distance d from the bead. The laser is shot during an exposure time t e The laser position shot is located on the computed vector path. 7) The ROI position is updated to the current bead position. 8) The bead is positioned at the target position, the mirror scanner is turned off and the manipulation process ends.

The beads were tracked in order to compute their position during the whole process. Their positions provided to the system the feedback position required to compute the location where the laser had to be shot until the beads reached their target position. The image correlation time depended on the size of the ROI. A size of 100×100 pixels was used and the time to find a bead position (size of a bead was about 10 up to 20 pixels in diameter) within the ROI was about 2 ms. After detection of the beads by segmentation, the templates were captured and the ROIs were positioned at the center of mass of the beads. Once the correlation was performed in the whole ROIs, the bead position were obtained. A vector path was then calculated between the bead and target positions. The laser is shot a distance d from the bead. The bead was dragged by the laser shot. The image correlation was performed in order to compute the new bead position. Once the bead position computed, the ROI updated its location in order to have the new bead position at the centre of the ROI. For a new bead position, vector path was recalculated. The calculation of the bead position is permanently tested, if the bead moves too fast and goes out of the image ROI, the process restarts in order to detect the current bead location. For each iteration, the distance between the bead and the target positions was compared to a tolerance distance introduced in the program. When the bead-target distance was smaller than a tolerance distance, the process ended. The tolerance gave the accuracy of manipulation. The highest accuracy that the system can achieve was one pixel, which is the position error given by the image correlation process. In the micromanipulation system, one pixel corresponded to 5 µm. The time to achieve the displacement of a bead in a closed loop was given by the addition of the laser exposure time (50-90 ms) and the scanner response time (125 ms) that gave 215 ms. The correlation time (2 ms) was not taken into account because the image correlation was performed in a parallel thread.

calculation of the bead and laser position calculation of the ROI position calculation of the bead positioncorrelation

bead laser target Fig. 6. Schematic of the tracking method using image correlation. The laser is shot in order to the bead be dragged, the bead moves and the ROI updates its location at the new bead position. Once the bead position is computed, the laser is shot again.

The automated manipulation process was implemented using the C++ language in a windows operating system. A multithreaded program was necessary to perform the tracking process, the IR laser shot with a determined exposure time and the GUI display in three threads respectively. A serial programming manner blocked these processes while one of them is being performed. For this purpose, the OpenMP application programming interface (API) was used. Fig. 7 shows the threads used in the program. Thread 1: Image acquisition and GUI display; thread 2: image processing; and thread3: DAQ NICard and control of the laser scanner and IR laser shot. These threads communicate each other through shared variables, two for the image acquisition and processing and two for the positions calculations.

IV. EXPERIMENTAL RESULTS

The automated non-contact manipulation of two beads in parallel manner is depicted in Fig. 8. The glass beads were 50 and 85 µm in diameter respectively. The water depth was 450 µm. Firstly, the beads were located by the user in an image frame of 500 × 500 pixels (Fig. 8(a)). One pixel corresponded to 5 µm in the sample workspace. Then the detection process was activated on the GUI by the user. A detection filter was implemented to avoid the detection of particles or beads smaller than 30 µm and larger than 150 µm in diameter. These parameters can be changed in the program. Once the beads were detected, the user introduced the target positions or destinations by clicking with a computer mouse on the displayed video (Fig. 8(b)). The target positions were shown by a black circle of 50 µm in radius. This dimension also represented the positioning tolerance. That means when the beads centre of mass was located at a distance smaller the 50 µm from the target position, the automated manipulation process ended. This tolerance could be changed in the program. The highest accuracy measured was of 5 pixels (25 microns) with the implemented algorithm controller. However an accuracy of 1 pixel could be achieved with a more suitable controller. After the introduction of the target positions, the laser was shot at a distance of 450 µm from each bead and during 90 ms of exposure time (Fig. 8(c)). These parameters are according to [START_REF] Vela | Laser-Induced Thermocapillary Convection for Mesoscale Manipulation[END_REF] in order to drag the beads at a speed in the range of mm•s -1 . The laser position was switched from one bead to the other thus each bead was dragged one after the other (Fig. 8(c-h)). This strategy was implemented because of the radial shape of flows. Due to the different distance destinations, one of the two beads arrived at its target position faster (Fig. 8(g)). As a result, the laser shot focused on the remaining bead until it arrived at its destination. When the two beads were positioned in their respective target positions, the automated process stopped. Pictures in Fig. 8(b-h) are shown at an interval of 1 s, taking a time of 6 s since the user introduced the target positions until the beads arrived at their respective final positions. The trajectory of the beads were not straight, this was due to the radial shape of flows, however the algorithm controller achieved to position the beads at their destinations because the linear trajectories were computed at every position of the beads.

V. CONCLUSION AND PERSPECTIVE

A. Conclusion

In order to contacless manipulate microobjects in a parallel and at high speed manner, an automated non-contact micromanipulation system is presented. This system is capable of manipulating objects regardless of the material composition and in addition within a large range size from 30 up to 300 µm. The automated manipulation of two glass beads was achieved and reported. A single 2-DOF mirror scanner allowed to adressed an IR laser at a specific position in the sample. A visual servoing was implemented to compute the bead position while moving and send the position to the controller, thus the mirror scanner and laser shot were controlled. This was based on an image correlation method. It was demonstrated that convection microflows can be used to automatically manipulate objects under water without the need of mechanical grippers or tweezers. This is of great interest for the fabrication of hybrid microdevices, self-assembly or production with batch processes at the microscale for high throughput. With a faster mirror scanner and a more suitable algorithm controller the manipulation time could be improved. Moreover the induced microflows could be reconfigurable according to the required manipulation, for example flows can be engineered by changing the pattern of the laser beam irradiation such as circles, lines, multiple dots or other specific geometry, and in an automated manner.

B. Perspective

The automation of this method was implemented by a simple position feedback, in which the distance between the bead and goal position was computed in order to be compared to the tolerance position. A more suitable regulator could be implemented to improve the performances of the system, it could increase the speed and accuracy of the automated process. For instance, in the program the distance at which the laser was shot was not variable as well as the exposure time. If this parameters could be changed with respect to the computed distance between a bead and its destination, the bead could be moved faster when it is far from the goal position and slower when it is close to target position. Hence, a position accuracy of 1 pixel can be reached in a shorter time as 1 pixel is the error due to the image correlation. The main drawback of this method for a multiple bead manipulation is the toroidal shape of the engendered microflows, this is due to the radial symmetry of convection flows with respect to the heat source. The implemented control algorithm did not take into account this effect. A more suitable manipulation strategy could be implemented with a sophisticated regulator. Apart from the control algorithm, a faster mirror scanner could be used so that the laser could switch from one bead to another much faster or to generate dedicated patterns faster in order to manipulate the objects at higher speeds according to [START_REF] Vela | Laser-Induced Thermocapillary Convection for Mesoscale Manipulation[END_REF]. Future work will deal with the mentioned problems mentioned above, study of a better control and image processing algorithms for multiple beads and non-spherical micro-objects manipulation in an autonomous manner. Furthermore, it would be very interesting in using this method to manipulate biological cells such as ovocytes [START_REF] Steager | Automated biomanipulation of single cells using magnetic microrobots[END_REF]. This could be possible because cells are spherical and flows are capable to drag them.

Fig. 1 .

 1 Fig. 1. Manipulation method using the Marangoni effect. (a) static microobject within a thin water layer, (b) IR laser beam is shot thus generating convection flows and microobject is dragged, (c) laser beam is shot into another position so that microobject follows it (inset shows a picture of the convection flows around the laser beam), (d) laser beam is turned off in order to place the microobject in a final position.

Fig. 3 .

 3 Fig. 3. Two DOF mirror scanner. (a) 2-DOF TIP/TILT electromagnetic device, (b) Forces exerted by a couple of electromagnets, (c) Mirror scanner mounted on a commercial inverted microscope, the IR laser beam is directed by the controllable dichroic mirror at any planar position in the sample.

Fig. 4 .

 4 Fig. 4. Calibration of the mirror scanner. (a) Laser positions are obtained from the workspace by image segmentation of the captured image, (b) Laser positions in a linear trajectory obtained with a curve command positions given to the mirror scanner. Image frame of 500 × 500 pixels.

Fig. 5 .

 5 Fig. 5. Image correlation method used to track microobjects. (a) A template of a microbead is correlated to an image ROI of the workspace containing the object, inset: template of the microobject, (b) A map of the position of the correlation coefficients, the brightest dot corresponds to the microbead position.

Fig. 7 .

 7 Fig. 7. Block diagram of the implemented multithreaded program. The main thread is divided into three independent threads which communicates between each other through shared and protected variables. I 0,1 and P 0,1 are the image and position shared variables.

Fig. 8 .

 8 Fig. 8. Image sequence depicting the automated manipulation of two glass beads. (a) The bead positions are detected by the program using image segmentation. (b) The user introduces the respective target positions with a computer mouse by clicking on the image frame. (c-h) The IR laser is shot by switching its position from one bead to the other until the beads arrive at destination. The pictures are shown at intervals of 1 s. Scale bar of 200 µm.

ACKNOWLEDGMENT

This work was supported by the European GOLEM project, Bio-inspired Assembly Process for Mesoscale Products and Systems, FP 6, NMP, contract No. STRP 033211. The authors acknowledge OCTAX Microscience GmbH for providing the laser shot system.


L Delezoide

P Lucat

F Sayd

Prêteux

Adrien Chan-Hon-Tong

Nicolas Ballas

Catherine Achard

Bertrand Delezoide

Laurent Lucat

Patrick Sayd

Franc ¸oise Prêteux

Skeleton Point Trajectories for Human Daily Activity Recognition

Keywords:

Skeleton point trajectories for human daily activity recognition

INTRODUCTION

Human activity recognition is becoming a major research topic (see [START_REF] Aggarwal | Human activity analysis: A review[END_REF] for a review). The ability to recognize human activities would enable the development of several applications. One is intelligent video surveillance in a medical context to monitor, at home, people with a limited autonomy (elderly or disabled person). Such systems could detect, in a non-invasive way, events affecting people safety such as falls or fainting and warn automatically the medical assistance. Human activity recognition could also lead to the construction of gesturebased human computer interface and vision-based intelligent environment.

During the last decade, the analyse of natural and unconstrained videos has known many improvements, such as [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Liu | Recognizing realistic actions from videos 'in the wild[END_REF][START_REF] Rodriguez | Action mach: a spatio-temporal maximum average correlation height filter for action recognition[END_REF]. These improvements were driven by recent progresses in object recognition in static image. In this field, most of the state-of-the-art approaches are based on the standard bag-of-words pipeline [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] that couples low-level features like [START_REF] Heikkilä | Description of interest regions with local binary patterns[END_REF]Lazeb-nik et al., 2005;[START_REF] Lowe | Object recognition from local scaleinvariant features[END_REF] with semantic understanding.

Besides, some works taking place in a videosurveillance setting, with a constrained environment, explore the characterization of human activities using 3D features [START_REF] Li | Action recognition based on a bag of 3d points[END_REF][START_REF] Ni | Rgbd-hudaact: A color-depth video database for human daily activity recognition[END_REF], or, middle-level information related to person poses. 3D features based methods mainly involve bag-of-words pipeline and extends low-level features by adding 3D information provided by depth-map.

Skeleton based methods rely on the extraction of the body-part positions in each frame of a video or motion capture sequence. Despite having a suitable accuracy when based on markers like in pioneer works [START_REF] Campbell | Recognition of human body motion using phase space constraints[END_REF], focused on specific body parts in [START_REF] Just | Hmm and iohmm for the recognition of mono-and bi-manual 3d hand gestures[END_REF] or being adapted to dedicated applications such as human sign language recognition [START_REF] Bashir | Object trajectory-based activity classification and recognition using hidden markov models. Image Processing[END_REF], skeleton based methods were first not suited to the recognition of various natural actions. But, recent devices like low-cost accelerometers [START_REF] Parsa | Rigid-body pose and twist estimation using an accelerometer array[END_REF] or KINECT allow the use of skeleton information to characterize general activities.

In this work, we take advantage of these recent Figure 1: RGBD-HuDaAct dataset [START_REF] Ni | Rgbd-hudaact: A color-depth video database for human daily activity recognition[END_REF].

progresses and present an algorithm for human-action sequence recognition. This algorithm is based on skeleton data, and on trajectory descriptors. Trajectory of each skeleton articulation is associated with a descriptor. This descriptor is invariant to rotations (toward the vertical axis) and translations of the body, and, captures both trajectory short-term information (throw an extension to 3D of [START_REF] Ballas | Trajectories based descriptor for dynamic events annotation[END_REF] state-of-the-art descriptors) and trajectory middle-term information (throw elementary gesture recognition). These descriptors (one for each skeleton articulation) form the sequence signatures which are analysed by a Multiple-Kernel Learning (MKL).

In addition, we study the effect of missing skeleton on our pipeline: despite large academic efforts to deal with general pose estimation [START_REF] Baak | A data-driven approach for real-time full body pose reconstruction from a depth camera[END_REF][START_REF] Girshick | Efficient regression of generalactivity human poses from depth images[END_REF][START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF], local failures in skeleton extraction are common in real-life color-depth-videos (RGBD-video). Hence, we extend works of [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF] where effects of noise on joint positions on detection results is studied, by focusing on our classification results according to the degree of failures in skeleton extraction on the daily-life sequences provided in the RGBD-HuDaAct dataset (figure 1).

In this paper, related works are first reviewed in section 2. Studied trajectory descriptors are presented in section 3 and tested in section 4 on public datasets (figure 2) where complete skeleton data are provided (as part of the dataset). Then, impact of non-artificial failures in skeleton extraction on the classification performance of our algorithm is presented in section 5.

cooking lowering an object drinking opening a drawer using a computer closing a drawer

RELATED WORKS

Trajectory features have been introduced in the context of human action recognition to capture video motion patterns through long-term analysis. In [START_REF] Matikainen | Trajectons: Action recognition through the motion analysis of tracked features[END_REF], trajectory motion vectors form directly non-fixed-length trajectory descriptors.

In [START_REF] Raptis | Tracklet Descriptors for Action Modeling and Video Analysis[END_REF], the average of descriptors of points composing the track over time is taken as the trajectory descriptor, resulting in a fixed-size descriptor. However, this approach discards the temporal information of the trajectory. To tackle this issue, a Markov process is used in [START_REF] Messing | Activity recognition using the velocity histories of tracked keypoints[END_REF][START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF]: elementary motions are quantized and transitions between motions words are modelled through a Markov model to represent trajectory. In [START_REF] Mezaris | Local invariant feature tracks for high-level video feature extraction[END_REF], Multiple Haar filters extract motion information at different time-scales, and, in [START_REF] Ballas | Trajectories based descriptor for dynamic events annotation[END_REF], motion and velocity information are combined to form trajectory descriptor.

After the descriptors extraction, an aggregation scheme transforms the local descriptors into a global video signature. In [START_REF] Ballas | Trajectories based descriptor for dynamic events annotation[END_REF][START_REF] Raptis | Tracklet Descriptors for Action Modeling and Video Analysis[END_REF][START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF], this aggregation relies on the bag-of-words model, while in [START_REF] Messing | Activity recognition using the velocity histories of tracked keypoints[END_REF], the aggregation is obtained through a Gaussian Mixture Model.

All these approaches rely on numerous salient patches with low semantic meaning to construct the video signature. However, it has been noticed by the well-known moving light experiment [START_REF] Johansson | Visual perception of biological motion and a model for its analysis[END_REF]) that a human is able to recognize a human action only from the set of skeleton articulations. This experiment, coupled with the recent success in skeleton extraction based on depth map provided by active captor, invites to explore the idea of human action analysis based on skeleton data.

Previous works have explored human skeletonbased features for human action recognition. In [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF], distances between skeleton articulation are used as weak features and Hough-forests framework as detector. They prove that Hough-forests using both skeleton and image features achieve better performance than forests using only image features. In [START_REF] Raptis | Real-time classification of dance gestures from skeleton animation[END_REF], a system based on skeleton pose estimation is presented. The system considers 120 frames-long skeleton trajectories and applies the maximum normalized cross correlation framework to recognize dance gestures. In [START_REF] Sung | Human activity detection from rgbd images[END_REF] a twolayered maximum entropy Markov model (MEMM) is built on pose based features, while in [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF], pose based features are considered in a conditional random field (CFR) context. Finally in [START_REF] Barnachon | Interprétation de mouvements temps réel[END_REF], activities are represented as words on a gesture alphabet, this allows to use an automaton to recognize activity languages. Those previous works have proven the relevancy of the skeletonbased features for human activity recognition.

In this work, we apply trajectory descriptors to skeleton data to form sequence signatures, invariant both in rotation and translation. These signatures are then used to classify sequences throw a machine learning algorithm. The offered algorithm is tested on both CUHA dataset [START_REF] Sung | Human activity detection from rgbd images[END_REF] and TUM dataset [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF].

In addition, we study the impact of skeleton extraction failure on our classification results. In every previous works on human action recognition based on skeleton data, skeleton extraction failure is rarely considered. In [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF], the correct-detection rate is evaluated when adding noise on joint positions. The offered algorithm provides a Gaussian noise robustness on detection performance up to a standard deviation of 0.27m. Following this work, we present the performances of our system when skeleton extraction is only intermittent as frequently observed in natural sequences.

SKELETON DATA BASED SEQUENCE SIGNATURE

The computation of our sequence-signature based on skeleton data is divided into several steps. First, trajectories are normalized in order to be invariant according to rotation around the vertical axis and translations. Then, one descriptor capturing the trajectory information is computed for each articulation. These descriptors handle intermittent data to manage cases where skeleton extraction failed. Finally, this set of trajectory descriptors forms the final sequence signature.

Pose Extraction and Normalization

In order to be independent of the skeleton data across datasets and existing systems (such as NITE), we choose to use only trajectories of feet, knees, hips, shoulders, head, elbows and hands articulations as input trajectories.

In these trajectories, the 3D coordinates of the skeleton articulations are expressed in a system of coordinates which is linked to the camera position. To recognize activity under various viewpoints, a normalization scheme has to be applied resulting in articulation coordinates invariant to geometric transformations. For instance, translation, and rotation invariance is achieved in [START_REF] Raptis | Real-time classification of dance gestures from skeleton animation[END_REF][START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF]. In [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF], skeleton information is reduced to a set of features based on distances between articulations. In [START_REF] Raptis | Real-time classification of dance gestures from skeleton animation[END_REF], the skeleton articulation coordinates are expressed in a new system of coordinates defined by the principal axis of torso points. However, with complete rotation invariance, some activities such as lie-down and standing become indistinguishable. To ensure the discriminative power of our final descriptor, we choose to be invariant only toward skeleton rotations around the vertical-axis and to skeleton global translations. To reach this invariance, articulation coordinates are expressed in a new system of coordinates (O,ux,uy,uz) where the origin is the center of the shoulders, u z is an estimation of the vertical vector (built using the video vertical vector), u y is the orthogonal projection of the vector connecting the left shoulder to the right shoulder on the horizontal plane determined by u z , and u x is set so that (u x , u y , u z ) defines an orthonormal system (figure 3).

Using these new coordinates, we define 6 shortterm descriptors which model one frame information and a middle-term descriptor which models elementary gestures.

Short-term Descriptors

Short-term modelling focuses on capturing the distribution of instantaneous information in the sequence represented as (v 1 , ..., v I ): v i (i ∈ {1, ..., I} where I is the length of the sequence) is a vector of position, velocity (difference of position between two consecutive frames) or acceleration (difference of velocity between two consecutive frames). To achieve scale invariance, vectors are normalized according to the trajectory maximum vector magnitude max i=1..I ||v i || 2 . Then, the vectors v i are quantified to estimate their distributions. We propose a 3D extension of the quantification from [START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF] where polar grid is used to quantize both vector direction and vector magnitude. As final classification performances are empirically stable relatively to this quantification, we introduce the least change from [START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF] quantification. Hence, we use spherical coordinates with 3 equal bins for vector magnitude and respectively 2 and 4 uniform bins for inclination and polar angles. An additional bin is added to represent the null vector resulting in a 25 bins quantification as in [START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF].

Two kinds of models are considered for the trajectories. First, a simple histogram is estimated resulting in a 25 dimensional vector.

As an histogram is an orderless representation, it does not take into account the temporal relation between the elementary vectors. To complete the previous representation, we also consider the Markov Stationary Features [START_REF] Ni | Contextualizing histogram[END_REF][START_REF] Sun | Hierarchical spatio-temporal context modeling for action recognition[END_REF] that enforces the temporal consistency by considering the temporal co-occurrence statistics. The stochastic matrix counting the co-occurrences of successive bins is computed. As this matrix belongs to a high dimensional space (25 × 25), it can not be directly used as descriptor and the stationary distribution associated to the Markov process [START_REF] Breiman | Probability. Society for Industrial and Applied Mathematics[END_REF]) is used instead.

Both representations are L 1 normalized in order to be invariant to the action duration.

It results in 6 different descriptors for each articulation: Short Term Position, Short Term Motion, Short Term Acceleration, Short Term Markov Position, Short Term Markov Motion, Short Term Markov Acceleration. In addition to these descriptors, a descriptor based on middle-term modelling is also introduced.

Gesture based Descriptor

An activity can be described as a succession of elementary gestures, defined by several frames of the sequence. Contrary to [START_REF] Barnachon | Interprétation de mouvements temps réel[END_REF] where a rough segmentation is performed based on trajectory discontinuities to extract non-overlapping gestures, we choose to adopt a dense and overlapping approach to be independent from ad-hoc gesture-segmentation algorithm, and, to ensure that all discriminative gestures are extracted (figure 4.a). We consider each window (with different allowed sizes) of each trajectory as a gesture. In this work, the set of allowed sizes is {0.2, 0.4, 0.6, 0.8} second. Hence, let (v 1 , ..., v I ) be the sequence of positions measured at frequency f , then for all size s ∈ {0.2, 0.4, 0.6, 0.8} and for all offset i ∈ {1, ..., Is f } the sub-trajectory v i , v i+1 , ..., v i+s f is a gesture (when skeleton is available in all frames of the window).

A bag-of-words model [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] is then used to capture the gesture distribution contained in a trajectory. A vocabulary of gestures is learned through an unsupervised clustering performed by the K-means algorithm (figure 4.b). K-means is done for each gesture size and articulation separately. Then, each articulation trajectory is described with the corresponding histogram (figure 4.c) which forms our bag-of-gestures descriptor (BOG).

Classification based Multiple Kernel Learning

Our sequence signatures are composed by 13 trajectory descriptors -one for each skeleton articulation.

To exploit this multiple channels representation, a Multiple-Kernels support machine is used along with a χ 2 kernel. The χ 2 distance between two vectors u, v of size N is given by

D N (u, v) = 1 2 N ∑ n=1 (u n -v n ) 2 u n + v n
Let A and B be two sequence signatures and let H be the size of the corresponding trajectory descriptor, then the kernel value for A, B is

K(A, B) = exp(- 13 ∑ s=1 β s D H (A s , B s ))
where β s is the weight associated to the channel s (a specific articulation here). Optimal β s are automatically determined using training samples through MKL implemented by SHOGUN library [START_REF] Sonnenburg | The shogun machine learning toolbox[END_REF].

EVALUATION ON TUM AND CUHA DATASETS

We evaluate all introduced trajectory-based sequences signatures on two public datasets using the same MKL algorithm for classification. In addition, we also evaluate combination of trajectory descriptors.

Datasets

The CUHA dataset (for Cornell University Human Activity) is presented in [START_REF] Sung | Human activity detection from rgbd images[END_REF]. It deals with 14 classes of daily-life activities performed by four people (figure 2). It contains 68 sequences around 30 seconds each. The evaluation process consists in a leave-one-subject-out cross validation in precision-recall terms.

The TUM dataset is presented in [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF]. It deals with 10 action classes occurring when setting a table (figure 2). It contains 19 realistic sequences around 2 minutes each, performed by 5 people. Evaluation from [START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF] consists to split data between training and testing, and to output result in correct-classification rate term. In this dataset, each frame is associated to one action label.

As our system expects segmented data (contrary to those from [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF][START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF])), we split each sequence each time the action label changes. this gives around 1000 sub-sequences with homogeneous label which are used as input.

Results

The evaluation of our trajectory signatures are presented in table 1 (in percentage). These results are completed by the results of 3 well-known methods of the literature:

• The first one (MEMM) proposed by [START_REF] Sung | Human activity detection from rgbd images[END_REF] uses a two-layered maximum entropy Markov Model on pose-based features.

• The second one [START_REF] Yao | A hough transform-based voting framework for action recognition[END_REF][START_REF] Yao | Does human action recognition benefit from pose estimation?[END_REF] uses the Hough forests to classify actions associated to weak classifiers based on distances between skeleton or visual feature.

• Finally, a Dynamic Time Wrapping (DTW) algorithm combined with the nearest neighbour classification approach [START_REF] Fengjun | 3d human action recognition using spatio-temporal motion templates[END_REF][START_REF] Müller | Motion templates for automatic classification and retrieval of motion capture data[END_REF] has also been evaluated on these datasets.

Only the four best short-term descriptors are presented (Short Term Position, Short Term Motion, Short Term Acceleration, Short Term Markov Motion). The number of centroids for K-means algorithm which provides the best trade-off on all experiments is 60 for BOG. As we do not see any improvement by using multiple windows in the BOG descriptor, we use only 0.4 second windows for gesture extraction.

One result of this experiment is that the introduced descriptors provide complementary results depending on the databases. Short Term Position has the best performances among other Short Term descriptors on the CUHA dataset which deals with several static actions (standing still, relaxing on a couch) whereas Short Term Motion performs better on the TUM dataset, which contains dynamic actions (opening a door, closing a door). Bag-of-gestures (BOG), which combines pose and movement on a middleterm duration, has better results than any individual short term descriptor on CUHA and TUM datasets. It can also be noted that both short-term descriptors and bag-of-gestures lead to results similar or better than those of the literature on the two databases.

The size of short-term descriptor and bag-ofgesture descriptor are really different since the first vector size is 25 × 13 = 325 while the second one is 60 × 13 = 780. Moreover, as the best short-term descriptor depends on the database, we propose to combine these short-term descriptors into a single one with size 325 × 4 = 1300. As shown in Figure 1, this new descriptor called short-term combination leads to better results on the TUM database and is less effective on the CUHA database. As its size is much larger, a good compromise is to use the bag-of-gesture descriptor which leads to better results than the state-ofthe-art methods, regardless of the test database.

As it has been noticed by previous work, skeleton based approaches provided high performances. In this work, we show that on datasets where skeleton is provided (CUHA, TUM), our approach leads to high results and outperforms the state-of-the-art. However if skeleton based approaches seem to be robust, one may wonder what happens on real sequences where the skeleton is obtained only intermittently.

Indeed, some algorithms such as DTW combined with the nearest neighbour classification framework which relies directly on comparison between skeleton articulations positions can not deal with missing observation. In the last section, we describe the behaviour of our algorithm on intermittent data. More precisely, we show that the BOG descriptor is flexible enough to adapt to missing data and provides results comparable to state-of-the-art ones.

BEHAVIOUR OF THE METHOD ON INTERMITTENT DATA

In order to evaluate the performance of our system in presence of missing skeleton data, incomplete sequences could be generated from complete ones by removing data. However real-life failure can be correlated with specific part of the action and in the worst cases, with the most discriminative parts of the action. Hence, we decide to study the performance of our system on real intermittent data instead of artificial ones.

In that purpose, we use the NITE software suite to extract skeleton from RGBD-HuDaAct dataset. As this database is not designed for skeleton extraction, some problems occur with side or back views of people, or when the subject has very specific postures that does not allow skeleton extraction. This provides incomplete sequences, and, we evaluate our algorithm according to various percentages of missing data.

Dataset

The RGBD-HuDaAct dataset is presented in [START_REF] Ni | Rgbd-hudaact: A color-depth video database for human daily activity recognition[END_REF]. It deals with 12 classes (+ one random class) of daily-life activities performed by 30 people. It is composed by 1189 RGBD-video sequences, around 1 minute each. Contrary to other datasets like [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF][START_REF] Sung | Human activity detection from rgbd images[END_REF], RGBD-HuDaAct is not designed for skeleton extraction. In TUM dataset [START_REF] Tenorth | The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition[END_REF], skeleton stream is provided by a body-part tracker which is manually helped when tracking failures occur. In CUHA dataset [START_REF] Sung | Human activity detection from rgbd images[END_REF], skeleton stream is provided by the general public NITE software suite, but there is no skeleton extraction failures, maybe be- cause actions are fronto-parallel to the camera and relatively constrained. In RGBD-HuDaAct, sequences seem more natural (actions are not fronto-parallel to the camera...) and skeleton extraction based on NITE software suite suffers from many failures. NITE software provides for each skeleton articulation, a boolean describing if the position is considered as reliable by the system. Hence, we consider that the skeleton is correctly estimated in a frame, if and only if, one skeleton only is extracted from the frame and for all body-parts of interest (feet, knees, hips, shoulders, head, elbows and hands) the corresponding booleans are true.

In order to measure the intensity of failures in skeleton extraction, the ratio between the number of frames where skeleton is correctly estimated and the total number of frames is computed for each sequence of the dataset. This ratio expressed in percentage is called, in this work, SER for Skeleton Extraction Ratio, and is used as a measure of the sequence validity.

The histogram corresponding to the number of sequences versus SER is presented in figure 5.

We can already notice that for more than ten percents of the sequences, the skeleton is never extracted during the entire sequence.

Figure 6 shows the average SER for each action of the database. It varies between 51.3% for stand up (T) and 82.3% for put on jacket (D). It can be noticed that the sequences where the skeleton is not well extracted do not rely on the performed actions.

The large range of SER presented in figure 5 allows us to evaluate our action classification system with different levels of skeleton extraction failures. For this purpose, we extract all sequences with a SER greater than λ% (where λ is an integer varying from 0 to 100) and evaluate our algorithm on this subset using the leave-one-subject-out scheme as in [START_REF] Ni | Rgbd-hudaact: A color-depth video database for human daily activity recognition[END_REF].

Classification

As the number of sequences vary with the level of SER, we have to deal sometimes with classes having a small number of sequences. Hence, we use a multiple-C-SVM framework (as suggested in [START_REF] He | Rare class classification by support vector machine[END_REF]) instead of MKL to perform classification.

Basically, in binary C-SVM, data are mapped to point in some vector space and a plane is designed to minimize the number of misclassification between positive and negative points. But, it has been emphasized in [START_REF] He | Rare class classification by support vector machine[END_REF] that SVM classifiers efficiency decreases when positive and negative misclassification rates (number of misclassifications over number of points) are too different, which is common when data are not balanced. Hence, in binary 2C-SVM, a plane is designed to minimize both the number of misclassifications and the difference between positive and negative misclassification rates.

In practice, this is performed by dividing the misclassification cost of a point by the number of points of the corresponding class. The standard multipleclasses classifier built on the poll of one-vs-one binary 2C-SVM is then used to perform classification [START_REF] Hsu | A comparison of methods for multiclass support vector machines[END_REF]. Linear-SVM implementation is provided by LIBSVM [START_REF] Chang | Libsvm: a library for support vector machines[END_REF].

Results on Intermittent Data

The evolution of the correct-classification rate versus λ (minimal SER allowed) is presented in figure 7. We do not evaluate the system for λ greater than 88 as only 10% of sequences from RGBD-HuDaAct have a SER greater than 88%. The correct-classification rate is low both when there are too few data (λ close to 88) or when data contains some heavily corrupted samples (λ close to 0). However, for λ = 69 (corresponding to 442 sequences) a good compromise is found between the number of sequences and their quality: the algorithm has 82.41% of correct-classification rate.

In order to link these results to the state-of-the-art, let us remind that the best known results [START_REF] Ni | Rgbd-hudaact: A color-depth video database for human daily activity recognition[END_REF] on this dataset is a correct-classification rate of 81.5% on 59% of all sequences (655 sequences ran-domly sampled). Two multi-modal strategies, combining color and depth information, have been developed: spatio-temporal interest points (STIPs) and motion history images (MHIs). These two methods do not use skeleton and thus are not sensitive to the failure problems during skeleton extraction. It is not easy to predict their results on the complete dataset, or, on the 442 sequences leading to our 82.41% of correct-classification rate. Moreover, as the 655 used sequences were randomly selected, it is not possible to build their testing database in order to compare both results. However, our algorithm is competitive with their method as the recognition rates are similar. Hence, we can conclude that skeleton based approaches like our algorithm provide state-of-the-art results even on intermittent data and thus can be used for action recognition in real-life.

CONCLUSIONS

In this paper, we evaluate 7 trajectory descriptors in context of human action recognition based on skeleton trajectory. We first presented 6 short term trajectories descriptors based on position, speed or acceleration. The last descriptor is more original since it extends the well-known bag-of-words approach to the bag-of-gestures ones, defined only on 3D position of articulations. To our knowledge, this is the first time that bag-of-gestures are defined using 3D points.

Performances of each descriptor and combination of them associated with a same MKL classifier have been evaluated on the public CUHA and TUM datasets, for which skeleton stream is provided. The main result is that the descriptor based on bag-ofgestures outperforms three very recent methods of the state-of-the-art on the two databases : 88.9% of F 0.5 measure on CUHA database and 84.5% of correct classification on TUM dataset.

We also study the proposed algorithm on a more difficult database not designed to extract skeleton: the public RGBD-HuDaAct database. During the skeleton estimation performed with NITE software, some problems occurred with side or back views of people, or when the person has very specific postures that do not allow skeleton extraction. Even if the RGBD-HuDaAct database is really challenging to perform action recognition based on skeletons, it has been considered here to represent some conditions of video surveillance system at home. The main result of these tests if that even in these conditions, and with a significant amount of missing data, our descriptor achieves the state-of-the-art performance of 82% of recognition rate. To our knowledge, it is the first time that failures in skeleton extraction are considered during action recognition.

In future works, we will continue to explore the links between high-level human actions and elementary gestures and design a framework to learn middlesemantic gestures which are the most relevant for human action recognition. Such framework will allow us to recognize an action from a small number of middle-semantic gestures whereas the algorithm presented in this work recognizes actions from the set of all gestures. Hence, such framework is likely to both speed up and increase accuracy of the system. In addition, we will extend current bag-of-gestures descriptor by taking into account the co-occurrence relation of different articulations and co-occurrence relation of pairs of successive gestures of a given articulation.

Figure 2 :

 2 Figure 2: CUHA dataset (first column-(Sung et al., 2011)) and TUM dataset (second column-(Tenorth et al., 2009)).

Figure 3 :

 3 Figure 3: Schema of skeleton normalization: u z = e z , u y is the vector between the two shoulders projected on the plane z = 0 and the 0 is the center of the two shoulders.

Figure 4 :

 4 Figure 4: Bag-of-gesture descriptor for a trajectory.

Figure 5 :

 5 Figure 5: Number of sequences with a given SER.

  Class labels are designed by a code letter: go to bed (B), put on jacket (D), exit the room (E), get up from bed (G), sit down (I), drink water (K), enter room (L), eat meal (M), take off jacket (N), mop floor (O), make a phone call (P), background activities (R), stand up (T).

Figure 6 :

 6 Figure 6: SER averages for each action class.

Figure 7 :

 7 Figure 7: Evolution of accuracy versus minimal SER.

Table 1 :

 1 Results on CUHA and TUM datasets.

		CUHA Dataset		TUM Dataset
		Precision Recall F 0.5 correct-classification rate
	MEMM (Sung et al., 2011)	69	57.3	64.2
	Hough forest based visual fea-				69,5
	tures (Yao et al., 2010)
	Hough forest based skeleton fea-				81,5
	tures (Yao et al., 2011)
	DTW baseline	83.7	74.5	77.2	76.3
	Short Term Position	80.2	86.3	81.4	76.6
	Short Term Motion	72.2	75.6	72.9	84.4
	Short Term Acceleration	67.9	74.2	69.1	68.35
	Short Term Markov	79.4	84.5	80.4	81.7
	Bag-of-Gestures	90.1	84.5	88.9	84.5
	Short Term Combination	70.5	71.4	70.7	90.8