Grégory Rogez
email: gregory.rogez@inria.fr

Deva Ramanan

Understanding Everyday Hands in Action from RGB-D Images

We analyze functional manipulations of handheld objects, formalizing the problem as one of fine-grained grasp classification. To do so, we make use of a recently developed fine-grained taxonomy of human-object grasps. We introduce a large dataset of 12000 RGB-D images covering 71 everyday grasps in natural interactions. Our dataset is different from past work (typically addressed from a robotics perspective) in terms of its scale, diversity, and combination of RGB and depth data. From a computer-vision perspective, our dataset allows for exploration of contact and force prediction (crucial concepts in functional grasp analysis) from perceptual cues. We present extensive experimental results with state-of-the-art baselines, illustrating the role of segmentation, object context, and 3D-understanding in functional grasp analysis. We demonstrate a near 2X improvement over prior work and a naive deep baseline, while pointing out important directions for improvement.

Introduction

Humans can interact with objects in complex ways, including grasping, pushing, or bending them. In this work, we address the perceptual problem of parsing such interactions, with a focus on handheld, manipulatable objects. Much previous work on hand analysis tends to focus on kinematic pose estimation [START_REF] Harrison | Nonverbal communication. Human Communication As a Field of Study: Selected Contemporary Views[END_REF][START_REF] Erol | Vision-based hand pose estimation: A review[END_REF]. Interestingly, the same kinematic pose can be used for dramatically different functional manipulations (Fig. 1), where differences are manifested in terms of distinct contact points and force vectors. Thus, contact points and forces play a crucial role when parsing such interactions from a functional perspective.

Problem setup: Importantly, we wish to analyze human-object interactions in situ. To do so, we make use of wearable depth cameras to ensure that recordings are mobile (allowing one to capture diverse scenes [START_REF] Pirsiavash | Detecting activities of daily living in first-person camera views[END_REF][START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF]) and passive (avoiding the need for specialized pressure sensors/gloves [START_REF] Bouzit | The rutgers master ii-new design force-feedback glove[END_REF][START_REF] Kry | Interaction capture and synthesis[END_REF]). We make no explicit assumption about the environment, such as known geometry [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]. However, we do make explicit use of depth cues, motivated by the fact that humans make use of depth for near-field analysis [START_REF] Fielder | Does stereopsis matter in humans?[END_REF]. Our problem formulation is thus: given a first-person RGB-D image of a hand-object interaction, predict the 3D kinematic hand pose, contact points, and force vectors.

Motivation: We see several motivating scenarios and applications. Our long-term goal is to produce a truly functional description of a scene that is useful for an autonomous robot. When faced with a novel object, it will be useful to know how if it can be pushed or grasped, and what forces and contacts are necessary to do so [START_REF] Saxena | Robotic grasping of novel objects[END_REF]. A practical application of our work is imitation learning or learning by demonstration for robotics [START_REF] Argall | A survey of robot learning from demonstration[END_REF][START_REF] Goodrich | Human-robot interaction: a survey[END_REF], where a robot can be taught a task by observing humans performing it. Finally, our problem formulation has direct implications for assistive technology. Clinicians watch and evaluate patients performing everyday hand-object interactions for diagnosis and evaluation [START_REF] Allin | Assessment of Post-Stroke Functioning using Machine Vision[END_REF]. A patient-wearable camera that enabled automated parsing of object manipulations would allow for long-term monitoring.

Why is this hard? Estimating forces from visual signals typically requires knowledge of object mass and velocity, which is difficult to reliably infer from a single image or even a video sequence. Isometric forces are even more difficult to estimate because no motion may be observed. Finally, even traditional tasks such as kinematic hand pose estimation are now difficult because manipulated objects tend to generate significant occlusions. Indeed, much previous work on kinematic hand analysis considers isolated hands in free-space [START_REF] Supancic | Depth-based hand pose estimation: data, methods, and challenges[END_REF], which is a considerably easier problem.

Approach: We address the continuous problem of pose+contact+force prediction as a discrete fine-grained classification task, making use of a recent 73-class taxonomy of fine-grained hand-object interactions developed from the robotics community [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF]. Our approach is inspired by prototype-based approaches for continuous shape estimation that treat the problem as a discrete categorical prediction tasks, such as shapemes [START_REF] Ren | Figure/ground assignment in natural images[END_REF] or poselets [START_REF] Bourdev | Poselets: Body part detectors trained using 3d human pose annotations[END_REF]. However, rather than learning prototypes, we make use of expert domain knowledge to quantize the space of manipulations, which allows us to treat the problem as one of (fine-grained) classification. A vital property of our classification engine is that it is data-driven rather than model-based. We put forth considerable effort toward assembling a large collection of diverse images that span the taxonomy of classes. We experiment with both parametric and exemplar-based classification architectures trained on our collection.

Our contributions: Our primary contribution is (1) a new "in-the-wild", large-scale dataset of fine-grained grasps, annotated with contact points and forces. Importantly, the data is RGB-D and collected from a wearable perspective. [START_REF] Allin | Assessment of Post-Stroke Functioning using Machine Vision[END_REF] We develop a pipeline for fine-grained grasp classification exploiting depth and RGB data, training on combinations of both real and synthetic training data and making use of state-of-the-art deep features. Overall, our results indicate that grasp classification is challenging, with accuracy approaching 20% for a 71-way classification problem. [START_REF] Argall | A survey of robot learning from demonstration[END_REF] We describe a simple post-processing exemplar framework that predicts contacts and forces associated with hand manipulations, providing an initial proof-of-concept system that addresses this rather novel visual prediction task.

Related Work

Hand pose with RGB(D): Hand pose estimation is a well-studied task, using both RGB and RGB-D sensors as input. Much work formulates the task as articulated tracking over time [START_REF] Kurata | A functionally-distributed hand tracking method for wearable visual interfaces and its applications[END_REF][START_REF] Kölsch | Hand tracking with flocks of features[END_REF][START_REF] Kölsch | An appearance-based prior for hand tracking[END_REF][START_REF] Athitsos | Estimating 3d hand pose from a cluttered image[END_REF][START_REF] Oikonomidis | Tracking the Articulated Motion of Two Strongly Interacting Hands[END_REF][START_REF] Stenger | Modelbased hand tracking using a hierarchical bayesian filter[END_REF][START_REF] Wang | Real-time hand-tracking with a color glove[END_REF]], but we focus on single-image hand pose estimation during object manipulations. Relatively few papers deal with object manipulations, with the important exceptions of [START_REF] Romero | Hands in action: real-time 3D reconstruction of hands in interaction with objects[END_REF][START_REF] Romero | Nonparametric hand pose estimation with object context[END_REF][START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF][START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF]. Most similar to us is [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF], who estimate contact forces during hand-object interactions, but do so in a "in-the-lab" scenario where objects of known geometry are used. We focus on single-frame "in-the-wild" footage where the observer is instrumented, but the environment (and its constituent ob-jects) are not.

Egocentric hand analysis: Spurred by the availability of cheap wearable sensors, there has been a considerable amount of recent work on object manipulation and grasp analysis from egocentric viewpoints [START_REF] Damen | You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video[END_REF][START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF][START_REF] Huang | How do we use our hands? discovering a diverse set of common grasps[END_REF][START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF][START_REF] Fathi | Learning to recognize objects in egocentric activities[END_REF]. The detection and pose estimation of human hands from wearable cameras was explored in [START_REF] Rogez | 3d hand pose detection in egocentric RGB-D images[END_REF]. [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF] propose a fully automatic vision-based approach for grasp analysis from a wearable RGB camera, while [START_REF] Huang | How do we use our hands? discovering a diverse set of common grasps[END_REF] explores unsupervised clustering techniques for automatically discovering common modes of human hand use. Our work is very much inspired by such lines of thought, but we take a data-driven perspective, focusing on large-scale dataset collection guided by a functional taxonomy.

Grasp taxonomies: Numerous taxonomies of grasps have been proposed, predominantly from the robotics community. Early work by Cutkosky [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF] introduced 16 grasps, which were later extended to 33 by Felix et al [START_REF] Feix | A comprehensive grasp taxonomy[END_REF], following a definition of a grasp as a "static hand postures with which an object can be held with one hand". Though this excluded two-handed, dynamic, and gravity-dependent grasps, this taxonomy has been widely used [START_REF] Romero | Spatiotemporal modeling of grasping actions[END_REF][START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF][START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF]. Our work is based on a recent fine-grained taxonomy proposed in [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF], that significantly broadens the scope of manipulations to include non-prehensile object interactions (that are technically not grasps, such as pushing or pressing) as well as other gravity-dependent interactions (such as lifting). The final taxonomy includes 73 grasps that are annotated with various qualities (including hand shape, force type, direction of movement and effort).

Datasets. Because grasp understanding is usually addressed from a robotics perspective, the resulting methods and datasets developed for the problem tend to be tailored for that domain. For example, robotics platforms often require an unavoidable real-time constraint, limiting the choice of algorithms, which also (perhaps implicitly) limited the difficulty of the data in terms of diversity (few subjects, few objects, few scenes). We overview the existing grasp datasets in Table 1 and tailor our new dataset to "fill the gap" in terms of overall scale, diversity, and annotation detail. [START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF] is imbalanced in that common grasps appear much more often than rare grasps (10 grasps suffice to explain 80% of the data). [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF] uses the same set of objects for the 4 subjects.

Figure 2. GUN-71: Grasp Understanding dataset. We have captured (from a chest-mounted RGB-D camera) and annotated our own dataset of fine-grained grasps, following the recent taxonomy of [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF]. In the top row, the "writing tripod" grasp class exhibits low variability in object and pose/view across 6 different subjects and environments. In the second row, "flat hand cupping" exhibits high variability in objects and low variability in pose due to being gravity-dependent. In the third row, "trigger press" exhibits high variability in objects and pose/view. Finally, in bottom, we show 6 views of the same grasp captured for a particular object and a particular subject in our dataset.

GUN-71: Grasp UNderstanding Dataset

We begin by describing our dataset, visualized in Fig. 2. We start with the 73-class taxonomy of [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF], but omit grasps 50 and 61 because of their overly-specialized nature (holding a ping-pong racket and playing saxophone, respectively), resulting in 71 classes.

Data capture

To capture truly in-the-wild data, we might follow the approach of [START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF] and monitor unprompted subjects behaving naturally throughout the course of a day. However, this results in a highly imbalanced distribution of observed object manipulations. [START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF] shows that 10 grasps suffice to explain 80% of the object interactions of everyday users. Balanced class distributions arguably allow for more straightforward analysis, which is useful when addressing a relatively unexplored problem. Collecting a balanced distribution in such a unprompted manner would be prohibitively expensive, both in terms of raw data collection and manual annotation. Instead, we prompt users using the scheme below.

Capture sessions: We ask subjects to perform the 71 grasps on personal objects (typical for the specific grasp), mimicking real object manipulation scenarios in their home environment. Capture sessions were fairly intensive and laborious as a result. We mount Intel's Senz3D, a wearable time-of-flight sensor [START_REF] Jang | 3d finger CAPE: clicking action and position estimation under selfocclusions in egocentric viewpoint[END_REF][START_REF] Damen | Egocentric real-time workspace monitoring using an rgb-d camera[END_REF][START_REF] Mann | Blind navigation with a wearable range camera and vibrotactile helmet[END_REF], on the subjects's chest using a GoPro harness (as in [START_REF] Rogez | 3d hand pose detection in egocentric RGB-D images[END_REF]). We tried to vary the types of objects as much as possible and considered between 3 and 4 different objects per subject for each of the 71 grasps. For each hand-object configuration, we took between 5 and 6 views of the manipulation scene. These views correspond to several steps of a particular action (opening a lid, pouring water) as well as different 3D locations and orientation of the hand holding the object (with respect to the camera).

Diversity: This process led to the capture of roughly 12,000 RGB-D images labeled with one of the 71 grasps. We captured 28 objects per grasp, resulting in 28 ⇥ 71 = 1988 different hand-object configurations with 5-6 views for each. We consider 8 different subjects (4 males and 4 females) in 5 different houses, ensuring that "house mates" avoid using the same objects to allow leave-one-out experiments (we can leave out one subject for testing and ensure that the objects will be novel as well). Six of our eight subjects were right handed. To ensure consistency, we asked the two left-handed subjects to perform grasps with their right hand. We posited that body shape characteristics might effect accuracy/generalizability, particularly in terms of hand size, shape, and movement. To facilitate such analysis, we also measured arm and finger lengths for each subject.

Figure 3. Contact point and forces. We show the 3D hand model for 18 grasps of the considered taxonomy. We also show the contact points (in green) and forces (in red) corresponding to each grasp. The blue points help visualize the shape of the typical object associated with each of these 18 grasps. We can observe that power grasps have wider contact areas on finger and palm regions, while precision grasps exhibit more localized contact points on finger tips and finger pads.

Synthetic (training) data generation

3D hand-object models: In addition to GUN-71, we construct a synthetic training dataset that will be used during our grasp-recognition pipeline. To construct this synthetic dataset, we make use of synthetic 3D hand models. We obtain a set of 3D models by extending the publiclyavailable Poser models from [START_REF] Romero | Hands in action: real-time 3D reconstruction of hands in interaction with objects[END_REF] to cover the selected grasps from [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF]'s taxonomy (by manually articulating the models to match the visual description of grasp).

Contact and force annotations: We compute contact points and applied forces on our 3D models with the following heuristic procedure. First, we look for physical points of contact between the hand and object mesh. We do this by intersecting the triangulated hand and object meshes with the efficient method of [START_REF] Moller | A fast triangle-triangle intersection test[END_REF]. We produce a final set of contact regions by connected-component clustering the set of 3D vertices lying within an intersection boundary. To estimate a force vector, we assume that contact points are locally stable and will not slide along the surface of the object (implying the force vector is normal to the surface of the object). We estimate this normal direction by simply reporting the average normal of vertices within each contact region. Note this only produces an estimate of the force direction, and not magnitude. Nevertheless, we find our heuristic procedure to produce surprisingly plausible estimates of contact points and force directions for each 3D model ( Fig. 3).

Synthetic training data: We use our 3D models to generate an auxiliary dataset of synthetic depth data, annotated with 3D poses, grasp class label, contacts, and force direction vectors. We additionally annotate each rendered depth map with a segmentation mask denoting background, hand, and object pixels. We render over 200, 000 training instances (3, 000 per grasp). We will release our models, rendering images, as well as GUN-71 (our dataset of realworld RGB-D images) to spun further research in the area.

Recognition pipeline

We now describe a fairly straightforward recognition system for recognizing grasps given real-world RGB-D images. Our pipeline consists of two stages; hand segmentation and fine-grained grasp classification.

Segmentation

The first stage of our pipeline is segmenting the hand from background clutter, both in the RGB and depth data. Many state-of-the-art approaches [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF][START_REF] Romero | Nonparametric hand pose estimation with object context[END_REF][START_REF] Romero | Hands in action: real-time 3D reconstruction of hands in interaction with objects[END_REF] employ userspecific skin models to localize and segment out the hand. We want a system that does not require such a user-specific learning stage and could be applied to any new user and environment, and so instead make use of depth cues to segment out the hand.

Depth-based hand detection: We train a P -way classifier designed to report one of P = 1500 quantized hand poses, using the approach of [START_REF] Rogez | First-person pose recognition using egocentric workspaces[END_REF]. This classifier is trained on the synthetic training data, which is off-line clustered into P pose classes. Note that the set of pose classes P We show the different steps of our segmentation stage: the depth map (a) is processed using a K-way pose classifier [START_REF] Rogez | First-person pose recognition using egocentric workspaces[END_REF], which reports a quantized pose detection k and associated foreground prior b ik (b) and mean depth µ ik (c) (used to compute a posterior following Eq. 1). To incorporate bottom-up RGB cues, we first extract superpixels (e) and then label superpixels instead of pixels to produce a segmentation mask (f). This produces a segmented RGB image in (g) , which can then be cropped (h) and/or unsegmented (i). We concatenate (deep) features extracted from (d), (g), (h), and (i) to span a variety of resolutions and local/global contexts. is significantly larger than the set of fine-grained grasps K = 71. We use the segmentation mask associated with this coarse quantized pose detection to segment out the hand (and object) from the test image, described further below.

Pixel model: We would like to use hand detections to generate binary segmentation masks. To do so, we use a simple probabilistic model where x i denotes the depth value of pixel i and y i 2 {0, 1} is its binary foreground/background label. We write the posterior probability of label y i given observation x i , all conditioned on pose class k as:

p(y i |x i , k) / p(y i |k)p(x i |y i , k) (1)
which can easily derived from Bayes rule . The first term on the right-hand-side is the "prior" probability of pixel i being fg/bg, and the second term is a "likelihood" of observing a depth value given a pose class k and label:

p(y i = 1|k) = b ik Bernoulli (2) p(x i |y i = 1, k) = N (x i ; µ ik , 2 ik ) N ormal (3)
p(x i |y i = 0, k) / constant Uniform (4) We use a pixel-specific Bernoulli distribution for the prior, and an univariate Normal and Uniform (uninformative) distribution for the likelihood. Intuitively, foreground depths tend to be constrained by the pose, while the background will not be. Given training data of depth images x with foreground masks y and pose class labels k, it is straightforward to estimate model parameters {b ik , µ ik , ik } with maximum likelihood estimation (frequency counts, sample means, and sample variances). We visualize the pixel-wise Bernoulli prior b ik and mean depth µ ik for a particular class k in Fig. 4-b and Fig. 4-c.

RGB-cues: Thus far, our segmentation model does not make use of RGB-based grouping cues such as color changes across object boundaries. To do so, we first compute RGB-based superpixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] on a test image and reason about the binary labels of superpixels rather than pixels:

label j = I ⇣ 1 |S j | X i2Sj p(y i |x i , k) > .5 ⌘ ( 5
)
where S j denotes the set of pixels from superpixel j. We show a sample segmentation in Fig. 4. Our probabilistic approach tends to produce more reliable segmentations than existing approaches based on connected-component heuristics [START_REF]Intel. Perceptual computing sdk[END_REF].

Fine-grained classification

We use the previous segmentation stage to produce features that will be fed into a K = 71-way classifier. We use state-of-the-art deep networks -specifically, Deep19 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] to extract a 3096 dimensional feature. We extract off-theshelf deep features extracted for (1) the entire RGB image, (2) a cropped window around the detected hand, and (3) a segmented RGB image (Fig. 4 (d,g,h,i)). We resize each window to a canonical size (of 224 x 224 pixels) before processing. The intuition behind this choice is to mix high and low resolution features, as well as global (contextual) and local features. The final concatenated descriptors are fed into a linear multi-class SVM for processing.

Exemplar matching: The above stages return an estimate for the employed grasp and a fairly accurate quantized pose class, but it is still quantized nonetheless. One can refine this quantization by returning the closest synthetic training example belonging to the recognized grasp and the corresponding pose cluster. We do this by returning the training example n from quantized class k with the closest foreground depth:

NN(x) = min n2Class k X i y n i (x n i µ ik ) 2 (6)
We match only foreground depths in the n th synthetic training image x n , as specified by its binary label y n . Because each synthetic exemplar is annotated with hand-object contact points and forces from its parent 3D hand model, we can predict forces and contact points by simply transferring them from the selected grasp model to the exemplar location in the 3D space.

Experiments

For all the experiments of this section, we use a leaveone-out approach where we train our 1-vs-all SVM classifiers on 7 subjects and test on the last 8 th subject. We repeat that operation with the 8 subjects and average the results. When analyzing our results, we refer to grasps by their id#. In the supplementary material, we include a visualization of all grasps in our taxonomy.

Baselines: We first run some "standard" baselines: HOG-RGB, HOG-Depth, and an off-the-shelf deep RGB feature [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. We obtained the following average classification rate: HOG-RGB (3.30%), HOG-Depth (6.55%), concatenated HOG-RGB and HOG-Depth (6.55%) and Deep-RGB (11.31%). Consistent with recent evidence, deep features considerably outperform their hand-designed counterparts, though overall performance is still rather low (Tab. 2).

Segmented/cropped data: Next, we evaluate the role of context and clutter.

Using segmented RGB images marginally decreases accuracy of deep features from 11.31% to 11.10%, but recognition rates appear are more homogeneous. Looking at the individual grasp classification rates, segmentation helps a little for most grasps but hurts the accuracy of "easy" grasps where context or object shape are important (but removed in the segmentation). This includes non-prehensile "pressing" grasps (interacting with a keyboard) and grasps associated with unique objects (chopsticks). Deep features extracted from a cropped segmentation and cropped detection increase accuracy to 12.55% and 13.67%, respectively, suggesting that some amount of local context around the hand and object helps.

Competing methods: [START_REF] Romero | Nonparametric hand pose estimation with object context[END_REF][START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF] make use of HOG templates defined on segmented RGB images obtained with skin detection. Because skin detectors did not work well on our (in-the-wild) dataset, we re-implemented [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF] using HOG templates defined on our depth-based segmentations and obtained 7.69% accuracy. To evaluate recent non-parametric methods [START_REF] Romero | Nonparametric hand pose estimation with object context[END_REF], we experimented with a naive nearest neighbor (NN) search using the different features extracted for the above experiments and obtained 6.10%, 6.97%, 6.31% grasp recognition accuracy using Deep-RGB, cropped-RGB and cropped+segmented-RGB. For clarity, these replace the K-way SVM classifier with a NN search. The significant drop in performance suggests that the learning is important, implying that our dataset is still not big enough to cover all possible variation in pose, objects and scenes.

Cue-combination: To take advantage of detection and segmentation without hurting classes where context is important, we trained our SVM grasp classifier on the concatenation of all the deep features. Our final overall classification rate of 17.97% is a considerable improvement over a naive deep model 11.31% as well as (our reimplementation of) prior work 7.69%. The corresponding recognition rates per grasp and confusion matrices corresponding to this classifier are given in Fig. 5. 4. Grasp classification for different sized taxonomies. We present the results obtained for K = 71 [START_REF] Liu | A taxonomy of everyday grasps in action[END_REF], K = 33 [START_REF] Feix | A comprehensive grasp taxonomy[END_REF] and K = 17 [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF], smaller taxonomies being obtained by selecting the corresponding subsets of grasps.

Easy cases: High-performing grasp classes (Fig. 5) tend to be characterized by limited variability in terms of viewpoint (i.e. position and orientation of the hand w.r.t camera) and/or object: eg. opening a lid (#10), writing (#20), holding chopsticks (#21), measuring with a tape (#33), grabbing a large sphere such as a basketball (#45), using screwdriver (#47), trigger press (#49), using a keyboard (#60), thumb press (#62), holding a wine cup (#72). Other highperforming classes tend to exhibit limited occlusions of the hand: hooking a small object(#15) and palm press (#55).

Common confusions: Common confusions in Fig. 6 suggest that finger kinematics are a strong cue captured by deep features. Many confusions correspond to genuinely similar grasps that differ by small details that might be easily occluded by the hand or the manipulated object: "Large diameter" (#1) and "Ring" (#31) are both used to grasp cylindrical objects, except that "Ring" only uses thumb and index finger. When the last three fingers are fully occluded by the object, it is visually impossible to differentiate them (see Fig. 6-c). "Adduction-Grip" (#23) and "Middle-over-Index"(#51) both involve grasping an object using the index and middle finger. Abduction-Grip holds the object between the two fingers, while Middle-over-Index holds the object using the pad of the middle finger and nail of the index finger (see Fig. 6-f). Best view: To examine the effect of viewpoint, we select the top-scoring view for each grasp class, increasing accuracy from 17.97% to 22.67% (Tab. 3). Comparing the two sets of recognition rates, best-view generally increases the performance of easy grasps significantly more than difficult ones -e.g., the average recognition rate of the top 20 grasps grow from 36.20% to 47.53%, while the top 10 grasps grows from 44.97% to 59.04%. This suggests that some views may be considerably more ambiguous than others.

Comparison to state-of-the-art. We now compare our results to those systems evaluated on previous grasp datasets. Particularly relevant is [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF], which presents visual grasp recognition results in similar settings, i.e. egocentric perspective and daily activities. In their case, they consider a reduced 17-grasp taxonomy from Cutkosky [START_REF] Cutkosky | On grasp choice, grasp models, and the design of hands for manufacturing tasks[END_REF], obtaining 31% with HOG features overall and 42% on a specific "machinist sequence" from [START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF]. Though these results appear more accurate than ours, its important to note that their dataset contains less variability in the background and scenes, and, crucially, their system appears to require training a skin detector on a subset of the test set. Additionally, it is not clear if they (or indeed, other past work) allow for the same subject/scene to be included across the train and testset. If we allow for this, recognition rate dramatically increases to 85%. This highly suggestive of overfitting, and can be seen a compelling motivation for the distinctly large number of subjects and scenes that we capture in our dataset.

Evaluations on limited taxonomies: If we limit our taxonomy to the 17 grasps from [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF], i.e. by evaluating only the subset of 17 classes, we obtain 20.53% and 23.44% (best view). See Tab. 4. These numbers are comparable to those reported in [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF]. Best-view may be a fair comparison because [START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF] used data that was manually labelled, where annotators were explicitly instructed to only annotate those frames that were visually unambiguous. In our case, subjects were asked to naturally perform object manipulations, and the data was collected "as-is". Finally, if we limit our taxonomy to the 33 grasps from Feix et al. [START_REF] Feix | A comprehensive grasp taxonomy[END_REF], we obtained 20.50% and 21.90% (best view). The marginal improvement when evaluating grasps from smaller taxonomies suggests that the new classes are not much harder to recognize. Rather, we believe that overall performance is somewhat low because our dataset is genuinely challenging due to diverse subjects, scenes, and objects.

Force and contact point prediction: Finally, we present preliminary results for force and contact prediction. We do so by showing the best-matching synthetic 3D exemplar from the detected pose class, along with its contact and force annotations. Fig. 7 shows frames for which the entire pipeline detection + grasp recognition + exemplar matching led to an acceptable prediction. Unfortunately, we are not able to provide a numerical evaluation as obtaining ground-truth annotation of contact and forces is challenging. One attractive option is to use active force sensors, Figure 7. Force and contact points prediction. We show frames for which the entire pipeline detection + grasp recognition + exemplar matching led to an acceptable prediction of forces and contact points. For each selected frame, we show from top to bottom: the RGB image, the depth image with contact points and forces (respectively represented by green points and red arrows, the top scoring 3D exemplar with associated forces and contact points, and finally the RGB image with overlaid forces and contact points. either embedded into pressure-sensitive gloves worn by the user or through objects equipped with force sensors at predefined grasp points (as done for a simplified cuboid object in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]). While certainly attractive, active sensing somewhat violates the integrity of a truly in-the-wild, everyday dataset.

Conclusions

We have introduced the challenging problem of understanding hands in action, including force and contact point prediction, during scenes of in-the-wild, everyday object manipulations. We have proposed an initial solution that reformulates this high-dimensional, continuous prediction task as a discrete fine-grained (functional grasp) classification task. To spur further research, we have captured a new large scale dataset of fine-grained grasps that we will release together with 3D models and rendering engine. Importantly, we have captured this dataset from an egocentric perspective, using RGB-D sensors to record multiple scenes and subjects. We have also proposed a pipeline which exploits depth and RGB data, producing state-of-the-art grasp recognition results. Our first analysis show that depth information is crucial for detection and segmentation, while the richer RGB feature allows for a better grasp recognition. Overall, our results indicate that grasp classification is challenging, with accuracy approaching 20% for a 71-way classification problem.

We have used a single 3D model per grasp. In future work, it would be interesting to (1) model within-grasp variability, capturing the dependence of hand kinematics on object shape and size and (2) consider subject-specific 3D hand shape models [START_REF] Khamis | Learning an efficient model of hand shape variation from depth images[END_REF], which could lead to more accurate set of synthetic exemplars (and associated forces and contacts).

Figure 1 .

 1 Figure 1. Same kinematic pose, but different functions: We show 3 images of near-identical kinematic hand pose, but very different functional manipulations, including a wide-object grasp (a), a precision grasp (b), and a finger extension (c). Contact regions (green) and force vectors (red), visualized below each image, appear to define such manipulations. This work (1) introduces a large-scale dataset for predicting pose+contacts+forces from images and (2) proposes an initial method based on fine-grained grasp classification.

Figure 4 .

 4 Figure 4. Segmentation.We show the different steps of our segmentation stage: the depth map (a) is processed using a K-way pose classifier[START_REF] Rogez | First-person pose recognition using egocentric workspaces[END_REF], which reports a quantized pose detection k and associated foreground prior b ik (b) and mean depth µ ik (c) (used to compute a posterior following Eq. 1). To incorporate bottom-up RGB cues, we first extract superpixels (e) and then label superpixels instead of pixels to produce a segmentation mask (f). This produces a segmented RGB image in (g) , which can then be cropped (h) and/or unsegmented (i). We concatenate (deep) features extracted from (d), (g), (h), and (i) to span a variety of resolutions and local/global contexts.

Figure 5 .

 5 Figure 5. RGB Deep feature + SVM. We show the individual classification rates for the 71 grasps in our dataset (a) and the corresponding confusion matrix in (b).

Figure 6 .

 6 Figure 6. Common confusions. The confusions occur when some fingers are occluded (a and c) or when the poses are very similar but the functionality (associated forces and contact points) is different (b, d, e and f).

Table 1 .

 1 Object manipulation datasets.[START_REF] Bullock | The yale human grasping dataset: Grasp, object, and task data in household and machine shop environments[END_REF] captured 27.7 hours but labelled only 9100 frames with grasp annotations. While our dataset is balanced and contains the same amount of data for each grasp,

	Dataset	View	Cam.	Sub. Scn	Frms	Label Tax.
	YALE [7]	Ego	RGB	4	4	9100	Gr.	33
	UTG [8]	Ego	RGB	4	1	?	Gr.	17
	GTEA [13]	Ego	RGB	4	4	00	Act.	7
	UCI-EGO [36]	Ego	RGB-D	2	4	400	Pose	?
	Ours	Ego	RGB-D	8	> 5 12, 000	Gr.	71

Table 2 .

 2 Grasp classification results.

	Features	Acc.	top 20 top 10	min	max
	HOG-RGB	3.30	7.20	9.59	0.00	28.54
	HOG-Depth	6.55	12.96	15.74	0.66	26.18
	HOG-RGBD	6.54	13.76	19.24	0.00	45.62
	Deep-RGB [41]	11.31	25.92	35.28	0.69	61.39
	Deep-RGB(segm.)	11.10	21.56	26.51	0.69	29.46
	HOG-RGB (cropped)	5.84	11.22	14.03	0.00	27.85
	Deep-RGB (cropped)	13.67	27.32	36.95	1.22	55.35
	HOG-RGB (crop.+segm.) [8]	7.69	15.23	18.65	0.69	30.77
	HOG-Depth (crop.+segm.)	10.68	22.04	27.99	0.52	42.40
	Deep-RGB (crop.+segm.)	12.55	22.89	27.85	0.69	37.49
	Deep-RGB (All)	17.97 36.20 44.97 2.71 68.48
				We present the result ob-
	tained when training a K-way linear SVM (K=71) with different
	types of features: HOG-RGB, HOG-Depth and Deep-RGB fea-
	tures, on the whole workspace, i.e. entire image, on a cropped
	detection window or on cropped and segmented image.
	View	Acc.	top 20 top 10	min	max
	All (All)	17.97	36.20	44.97 2.71 68.48
	Best scoring view 22.67 47.53 59.04	0	79.37

Table 3 .

 3 View selection. We present grasp recognition results obtained when training a K-way linear SVM on a concatenation of Deep features. We present the results obtained when computing the average classification rate over 1) the entire dataset and 2) over the top scoring view of each hand-object configuration.

		71 Gr. [28] 33 Gr. [14] 17 Gr. [9]
	All views	17.97	20.50	20.53
	Best scoring view	22.67	21.90	23.44
	Table

Acknowledgement. G. Rogez was supported by the European Commission under FP7 Marie Curie IOF grant "Ego-vision4Health" (PIOF-GA-2012-328288). J. Supancic and D. Ramanan were supported by NSF Grant 0954083, ONR-MURI Grant N00014-10-1-0933, and the Intel Science and Technology Center -Visual Computing. We thank our hand models Allon H., Elisabeth R., Marga C., Nico L., Odile H., Santi M. and Sego L. for participating in the data collection.


William Puech
email: william.puech@lirmm.fr

Loïc Dubois
email: loic.dubois@lirmm.fr

Jacques Blanc Talon
email: jacques.blanc-talon@dga.defense.gouv.fr

J Blanc-Talon

Smart Selective Encryption of H.264/AVC Videos using Confidentiality Metrics

Keywords: Selective encryption, H.264/AVC, Psychovisual metrics, Flickering, Visual confidentiality

  L'archive ouverte pluridisciplinaire

Introduction

With the rapid evolution in digital media, growth of processing power and availability of network bandwidths, digital videos are commonplace and their numbers are rising exponentially. Consequently, archived and transmitted data must be protected because they can be easily copied and modified. Data security and network security are two common solutions to solve these issues. The first one protects a network of vulnerable data while the second ensures the safety of all data which can be shared on unsecured networks. Data protection to ensure the security is generally preferred to the network security thanks to a better optimization of processing time and data-size. Furthermore, video data require compression in order to reduce the transmission time, and they need to be encrypted to ensuring their confidentiality. In video processing, full data encryption is rarely used because the processing time is twice that of the compression process. That is why Selective Encryption (SE) algorithms are usually recommended. SE algorithms aim to specify part of a video data bitstream to which the encryption algorithms are applied. This scheme guaranties visual confidentiality and protection without a data increase and while saving computation time. In order to compress video data, redundancy in the images is reduced by the prediction error of the compression algorithms, and SE algorithms use this information vector to spread its encryption through the video frames.

Moreover, SE algorithms can lean on psychovisual metrics. Psychovisual metrics are mathematical tools which measure the perceptual quality of a processed image relative to its original. The goal of this combination is to optimize the encryption scheme: psychovisual metrics guarantee the quality of each encrypted frame during the encryption process. This paper presents an analysis of video SE combined with similarity measures by taking into account the temporal aspect. In H.264 codec, a SE based on SE-CAVLC [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF] is spread in the inter frames while only encrypting the intra frames. Moreover, we use similarity measures like Strutural Similarity (SSIM) and Peak Signal-to-Noise Ratio (PSNR) to analyze the perceptual effect of this SE, and create a psychovisual measure, i.e. TSSIM which analyzes the flicker phenomenon between two neighbor frames of the encrypted video. Furthermore, we present a new Reduced Selective Encryption (RSE) approach which controls the amount of encrypted coefficients with respect to a good visual protection of each frame, GoP by GoP, in terms of SSIM, PSNR and TSSIM. Section 2 presents the H.264/AVC codec, the main previous work on SE of video and an overview of the actual psychovisual metrics. In Section 3, we present our approach and analysis in detail. The two main objectives are to highlight the propagation of an encryption effect through the predicted frames and the perceptual effect of this phenomenon. Furthermore, we analyze the proposed RSE method, which reduces the amount of encrypted coefficients in the entropic coder with respect to the perceptual confidentiality of the video. In Section 4, experimental results are given and discussed. In Section 5, concluding remarks and prospects for about the proposed scheme are discussed.

State of Art

In Section 2.1 we briefly present the H.264/AVC encoder. Next, a state of art on the SE of H.264/AVC encoder is described in Section 2.2. In Section 2.3, the psychovisual metrics that we used for our RSE method are presented.

H.264/AVC

H.264/AVC [START_REF]Draft ITU-T Recommendation and Final Draft International Standard of Joint VIdeo Specification (ITU-T Rec. H[END_REF], also known as MPEG-4 Part 10, is the video coding standard of ITU-T and ISO/IEC. In H.264/AVC, each frame is divided in Macro-Blocks (MBs) of 16x16 pixels. These macro-blocks are encoded separately; the encoding method is an Entire Transform followed by quantization of the MB, a prediction between MBs in intra (I frame) or inter (P and B frames), and an entropy coding using either run length coding (CAVLC) or arithmetic coding (CABAC), as presented in Fig. 1. In intra frame, the current MB is predicted spatially from neighboring MBs which were previously encoded and reconstructed. In inter frame, the current MB is predicted spatially and temporally from previous frames. The purpose of the reconstruction in the encoder is to ensure that both the encoder and the decoder use identical reference frames to create the predictions.

Selective Encryption

In the literature, several methods for video SE have been proposed [START_REF] Stütz | A Survey of H.264 AVC/SVC Encryption[END_REF]. SE, also known as partial encryption, is a encryption strategy which aims at saving computation time or enabling new system functionalities. In SE, a small part of the compressed bitstream is encrypted while still providing adequate data security [START_REF] Lookabaugh | Selective Encryption for Consumer Applications[END_REF] with respect to total encryption which would encrypt the whole bitstream. Moreover, SE fulfills the main tasks of video encryption, namely visual confidentiality and data protection. These tasks are performed by applying a SE in certain segments of the bitstream with respect to total encryption which encrypts the whole bitstream. Another challenge in SE is that both encrypted and non-encrypted informations should be appropriately identified and displayed [START_REF] Chen | Partial Encryption of Compressed Images and Videos[END_REF] in order that the SE bitstream will remain compliant with the H.264/AVC video standard.

In the field of video, different SE techniques have been developed which include permutation based on the Data Encryption Standard (DES) and the Advanced Encryption Standard (AES) [START_REF] Uhl | Image and Video Encryption -From digital Rights Management to Secured Personal Communication[END_REF]. Based on the location of the encryption stage in the video codec, these SE techniques can be divided into five broad categories.

These are based on the spatial position, the video codec structure, the matrix of transformed coefficients, the entropy encoding or the bitstream. Encryption in the entropy encoding module is often efficient and has been adopted by several authors. The use of the Huffman entropy coder as encryption cipher was studied in [START_REF] Wu | Design of Integrated Multimedia Compression and Encryption Systems[END_REF]. Despite providing a compliant video bitstream, the scheme is hampered by a bitrate increase which makes it less appropriate and limits real-time applications. Moreover, a SE of MPEG-4 video standard was studied in [START_REF] Wen | A Format-Compliant Configurable Encryption Framework for Acess Control of Video[END_REF], wherein DES was used to encrypt fixed length and variable length codes. In this approach, the encrypted bitstream is fully compliant with the MPEG-4 bitstream format but the bitstream size is increased. This particularity is a current observation in video SE. Furthermore, data security in intra mode is improved in [START_REF] Jiang | An Improved Selective Encryption for H.264 Video based on Intra Prediction Mode Scrambling[END_REF], where each frame receives a specific and synchronized encryption key. Moreover, each type of MB is encrypted differently with chaotic sequences in order to improve protection against plain-text attacks.

Perceptual encryption was also presented in [START_REF] Yeung | Perceptual Video Encryption using multiple 8x8 transforms in H.264 and MPEG-4[END_REF], where encryption is done with an alternative transform of the DCT coefficients with a singular key. Chaotic-maps have also been used to randomize the macro-bloc positions in order to improve protection against plain-text attacks [START_REF] Choi | Privacy-Preserving H.264 Video Encryption Scheme[END_REF]. The AES algorithm has also been used in SE-CAVLC [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF] by encrypting only a part of the quantized coefficients in various VLC tables. SE-CAVLC [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF] is performed by using the AES algorithm in Cipher Feedback (CFB) mode on a subset of codewords/bin-strings. The data information is selectively encrypted for each MB, and header information is never encrypted because it is used for prediction of the next MBs. In the entropy coder, the SE is performed in the multiple VLC tables used in CAVLC. In CAVLC, five syntax elements are used to code levels and runs: coeff token, signs of trailing ones, remaining non-zero levels, total number of zeros and runs of zeros, as shown in Fig. 2. Only signs of trailing ones and remaining non-zeros levels are encrypted in order to keep the bitstream compliant. The encrypted spaces are VLC codes spaces which means that the same code lengths as a standard compression may be kept. In [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF], the entire parts of the experimentation were carried out on QCIF video sequences.

Similarity Measures

This section provides a brief description of some objective quality measures: PSNR and SSIM-based measures. PSNR [START_REF] Avcibas | Statistical Evaluation of Image Quality Measures[END_REF] is widely used to measure the difference between two images based on pixel differences. For a N • M pixel image with pixel luminance values ranging from zero (black) to Lmax (white) depending of the image dynamic, PSNR is defined as:

P SN R = 10 • log 10 ( L 2 max M SE ), ( 1
)
where RMSE is the root mean square error defined as:

M SE = N i=1 M j=1 (Io(i, j) -Ir(i, j)) 2 N • M , (2)
where Io and Ir are respectively the original image and the recorded one, and N and M represent the image resolution.

SSIM and specific methods

SSIM [START_REF] Wang | Image Quality Assessment: From Error Visibility to Structural Similarity[END_REF] is an improved version of the universal image quality index. It is based on a top-down assumption that the Human Visual System (HSV) is highly adapted for extracting structural information from the scene, and therefore a measure of structural similarity should be a good approximation of the perceived image quality [START_REF] Seshadrinathan | Study of Subjective and Objective Quality Assessment of Video[END_REF]. SSIM is generally useful in high quality image cases, and for detecting local degradation in an image due to the inter-dependence of the closest pixels. SSIM is also a future candidate for rate-distortion optimizations in the design of image compression algorithms. Its mathematical computation can also be easily adapted for other types of signal. SSIM uses cross-variance coupled with the average and the variance, while PSNR uses only the mean square error (MSE). Mathematically, SSIM is defined as:

SSIM (x, y) = (2µ Io µ Ir + c 1 )(2cov IoIr + c 2 ) (µ 2 Io + µ 2 Ir + c 1 )(σ 2 Io + σ 2 Ir + c 2 ) , (3)
with µ the mean, σ the standard deviation, cov the covariance, and c 1 and c 2 are two constants to stabilize the division. This formula is principally applied on luma, where the correlation with the HVS is highest. For video quality, SSIM is usually applied on each frame and the conclusions are based on the mean and variance of SSIM of the whole video.

Multiscale SSIM (MS-SSIM) [START_REF] Wang | Multi-scale Structural Similarity for Image Quality Assessment[END_REF] is an extension of SSIM. It incorporates variations in the viewing condition in the calculation, which provides more adaptability for still images. The MS-SSIM parameters can be calibrated as a function of the relative importance of the different image scales.

A complex wavelet domain image similarity measure (CWSSIM) [START_REF] Wang | Translation insensitive image similarity in complex wavelet domain[END_REF] has been developed to be insensitive to luminance changes, contrast changes and spatial translation. This metric uses magnitude and/or phase changes of local wavelet transforms. This SSIM-based method is robust for small geometric distortions. However, it cannot underline large displacements and non-geometric deformations.

SSIM and the Scale-Invariant Feature Transform (SIFT) have been combined (SSIM SIFT) [START_REF] Decombas | A New Object Based Quality Metric Based On SIFT and SSIM[END_REF] in order to evaluate systems which do not preserve the positions and/or shapes of objects.

In [START_REF] Zujovic | Strutural Similarity Metrics for Texture Analysis and Retrieval[END_REF], the authors present a specific SSIM-based metric called Structural Texture SIMilarity (STSIM). This metric is used to improve measurement of the quality of textures in images using both intra-and inter-subband correlations and with incorporation of the color composition.

Note that few previous studies have presented methods that take temporal aspect into account. In [START_REF] Chikkerur | Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison[END_REF] temporal aspect of the video is analyzed to determine more precisely the quality. Different quality measures are used with standard deviation and mean, however flickering is not integrated in this method.

Proposed Method

An overview of our proposed method is presented in Section 3.1. In Section 3.2, we explain our encryption scheme where the prediction error of H.264/AVC is used as encryption vector. Section 3.3 introduces the flickering problem and our proposed psychovisual metric which analyzes this phenomenon. Moreover, we propose an adjustment of the encrypted coefficients depending on the psychovisual metrics in Section 3.4. In conclusion, in Section 3.5, we present different strategies to decode the encrypted video.

Overview

The goal of SE algorithms is to decrease the quantity of encrypted data in order to reduce the encryption process time, particularly during decoding. In H.264/AVC, the encryption of non-zero AC coefficients is generally sufficient to visually protect the video. Our proposed method aims to achieve a smart reduction in the encrypted AC coefficients with respect to the psychovisual measure of the encrypted video. We use a SE-CAVLC algorithm while encrypting just part of the non-zero coefficients, and we check whether the visual confidentiality remains efficient. The strategy of our proposed scheme (presented in Fig. 3), is applied on each Group of Pictures (GoP) of a video sequence. The first GoP is selectively encrypted with an initial number of encrypted coefficients C 1 , which guarantees a minimum degree of visual confidentiality. C 1 is then the number of encrypted coefficients by MB of the first frame (the Intra frame) of the GoP. After encryption, which is done during the entropy encoding, the full GoP is decoded and each frame is analyzed by a set of psychovisual metrics to check if the following GoP should be more encrypted or not with respect to the used metrics. This analysis allows us to set the value of C 2 , which is the number of encrypted coefficients by MB for the second GoP. So for a GoP i , we analyze the encryption of the GoP i-1 in order to set the value of C i .

Encryption through prediction errors

In this section, we present how the prediction errors of the H.264/AVC encoder are used to reduce the encryption ratio, which is the ratio between the number of encrypted bits and the video datasize (in bits): Encryption ratio = Size of the encrypted bits V ideo datasize . An overview of the compression and encryption scheme is presented in Fig. 4. In the H.264/AVC codec, the prediction error is used to reduce the bitstream size of video sequences. This prediction error is the difference between the current MB and a previous neighbor MB. A scan of each previously neighboring encoded MB is achieved in order to find the MB yielding the smallest prediction error. For the intra frames, the prediction is done on the current frame. Moreover, this prediction is used in the temporal domain in order to encode inter frames. During the decoding step, because of the selective encryption, a MB which has been decoded from an encrypted MB should be heavily distorted. We use this specificity to spread the encryption through each inter frame of a video sequence. The intra frames are selectively encrypted while the inter frames are not directly encrypted.

Psychovisual Metric

In this section, we present the flickering phenomenon, and we propose a new metric in order to measure this phenomenon in the case of encrypted videos.

Flickering phenomenon

The flickering phenomenon is a problem that disturbs the viewing of a video during the time. Research in this domain was previously carried out to reduce video flickering due to the compression algorithm [START_REF] Kuszpet | Post-processing for flicker reduction of H.264/AVC[END_REF][START_REF] Kanumuri | Temporal Flicker Reduction and Denoising in Video using Sparse Directional Transforms[END_REF][START_REF] Song | Noise Power Estimation For Effective De-noising in a Video Encoder[END_REF][START_REF] Chebbo | Objective Evaluation of Compressed Video's Temporal Flickering[END_REF]. In [START_REF] Kuszpet | Post-processing for flicker reduction of H.264/AVC[END_REF], a post-processing scheme is presented to reduce the discontinuity. This method is motion compensated-based. Sometimes the flickering is considered and processed like noise, as presented in [START_REF] Kanumuri | Temporal Flicker Reduction and Denoising in Video using Sparse Directional Transforms[END_REF][START_REF] Song | Noise Power Estimation For Effective De-noising in a Video Encoder[END_REF]. In [START_REF] Chebbo | Objective Evaluation of Compressed Video's Temporal Flickering[END_REF] a block-based method for objective evaluation is presented, where each block in a given frame is classified with respect to its co-located block. This flickering phenomenon also appears in encrypted video sequences and increases the visual confidentiality. Indeed, the greater is the flickering, more annoying is the video sequence. This temporal visual effect is crucial to preserve the visual confidentiality of the contents. For example, Fig. 5 presents four successive frames of an encrypted video sequence, and Fig. 6 illustrates the grey level variation of the luminance of one pixel of this video sequence with respect to the frame number. Note that there are high variations (more than 100 grey levels) between each frame. This phenomenon is highly annoying for a watcher, especially when the video is read at more than 25 frames per second.

Temporal Structural SIMilarity

In order to measure and quantify the flickering phenomenon, we have developed a psychovisual metric based on the SSIM applied on successive video frames. We have chose the SSIM-based metric because SSIM is a good metric in terms of psychovisual sensitivity with respect to the Human Visual System (HVS), as presented in [START_REF] Seshadrinathan | Study of Subjective and Objective Quality Assessment of Video[END_REF]. Our proposed metric, called Temporal SSIM (T-SSIM), allows us to measure variations between the difference of two original compressed frames and two encrypted frames:

T SSIM (Io,Ie) (i) = SSIM (|Io(i) -Io(i -1)|, |Ie(i) -Ie(i -1)|), (4)
where Io() is an original compressed frame and Ie() an encrypted one. In fact, T-SSIM is the SSIM of the absolute differences of two successive original frames and the same successive frames of the encrypted video sequence. With T-SSIM, we can analyze flickering between two successive frames of an encrypted video sequence. The range of values for T-SSIM is nearly the same that for SSIM, i.e. above 0.6 the flickering is not really marked, and below 0.4 we consider that it is sufficient for the visual confidentiality of the video. The last range between 0.4 and 0.6 is a transition range which needs to be analyzed case by case, so we consider that this last range is irrelevant for the visual confidentiality. Fig. 7 presents the quality measures of two successive images of two video sequences. Both of them are encrypted with a full encryption scheme from which we can extract the statistical results of PSNR, SSIM and TSSIM for perfect confidentiality: close to 7-8 dB for the PSNR and close 0 for SSIM and TSSIM.

Encrypted coefficient adjustment

In our SE scheme, the three color channels, luminance and two chrominances of each frame are affected by the selective encryption. For each GoP, we propose two solutions to apply the SE. In the first solution, only the first frame (intra frame) of the GoP i frame is encrypted. This solution is sufficient to ensure the confidentiality of the entire GoP i in the case of low resolution videos. In the second solution, we propose to encrypt the whole GoP i . Next, non-zero AC levels of lower frequencies are encrypted in priority according to the selected number of non-encrypted coefficients, as presented in Fig. 8. In the H.264/AVC compression scheme, high frequencies AC levels are encoded first due to the inverse zigzag scan in the entropic encoder. The combination of these two points results in the scheme presented in Fig. 3. We integrated psychovisual metrics between GoP i and GoP i+1 , these metrics measure the quality on the luminance component. These metrics control the amount of encrypted coefficients depending on the metric results of the previous GoP i . The metrics used for our experimentations are SSIM and T-SSIM, which are combined and one maximum threshold and one minimum threshold for each of them are set. This control is an algorithm which works like a selection trigger as presented in Algorithm 1. These psychovisual metrics control the visual confidentiality of each GoP i frame and if one of the frame result metrics Algorithm 1 Encrypted coefficient adjustment. Computation of C i , the number of encrypted coefficients for the GoP i , min and max are the upper and lower thresholds for the psychovisual metrics. SSIM i is the maximum SSIM for each frame of the GoP i . T SSIM i is the T-SSIM of GoP i or T-SSIM of the last frame of GoP i relative to the first frame of GoP i , depending on the encryption scheme.

if

(SSIM i > SSIMmax OR T SSIM i > T SSIMmax) then C i+1 ← C i + 1 else if (T SSIM i < T SSIM min AN D SSIM i < SSIM min ) then C i+1 ← C i -1 else C i+1 ← C i end if end if
is above one of the maximum thresholds, then the next GoP will be encrypted with more encrypted coefficients (C i+1 ← C i + 1), or if it is under the two minimum thresholds, we decrease the number of encrypted coefficients (C i+1 ← C i -1). With this control scheme, the encryption depends on the psychovisual measure and the confidentiality changes in accordance with the video sequence and its different contents.

Decoding step

During the decoding step, the process has to know the amount of encrypted coefficients per MB (value of C i ) in order to correctly decode the video frame. More precisely, the decoder has to know only if the C i value has to be incremented or decremented compared to the previous C i-1 value. Indeed, the initial frame is the only frame which requires the initial number of encrypted coefficients (C 0 ). In order to solve this problem, two methods can be applied: firstly, the information can be embedded in the header data of each MB. Secondly, the number can be hidden in the DC coefficient of the first MB of each frame by using a watermarking method. This last method is more advisable. The DC coefficient can be watermarked because the DC coefficients are encoded independently of the AC coefficients. Moreover, they are not encrypted. This data-hiding has to be performed during the loop filter in order to be considered for the prediction error as presented in [START_REF] Shahid | Considering the reconstruction loop for data hiding of intra-and inter-frames of H.264/AVC[END_REF]. If not, drifts would occur in the decoded frames. Another decoding method, takes into account the coefficient distributions in order to know if the macro-blocks are encrypted or not, as presented in [START_REF] Islam | Noise removing in encrypted color image by statistical analysis[END_REF]. In fact, this last method involves calculating the values during the decoding.

Experimental Results

For our experimental results 1 , we have used eight benchmark video sequences, four with CIF resolution (352×288 pixels): city, foreman, hall and mobile; two with a 640×352 pixel resolution: big buck bunny and venise-fly; two others in HD: movie with a 1920×800 pixel resolution and ardrone-fly with a 1280×720 pixel resolution. Fig. 9 presents an image of each video. These video sequences show different combinations of motions, colors, contrasts and objects. The results are presented with the most representative samples. In terms of encryption, we consider that we preserve the confidentiality if the PSNR is less than 13 dB, the SSIM less than 0.6 and the TSSIM less than 0.6. These thresholds have been determined with a subjective experiments. Human users have watched several encrypted videos with different levels of encryption. Also, we have compared these results with respect to the results of quality measures to find the confidentiality thresholds. All of the videos were compressed with a QP of 24, which represents a good trade off between quality and compression with a PSNR of around 35 dB for a compressed video sequence. The GoP size, which optimizes the encryption ratio and confidentiality, also has to be analyzed. A first study was proposed in [START_REF] Dubois | Reduced Selective Encryption of Intra and Inter Frames of H.264/AVC using Psychovisual Metrics[END_REF] and an interesting GoP size should be less than 16 frames for video sequences with 30 frames per second. If the GoP size is above than 16, then flickering due to the encryption algorithm is no longer visually annoying to ensure visual confidentiality. The four CIF video sequences and the two 640×352 pixel video sequences are encoded with a GoP of 4 images, the two HD videos are encoded with a GoP of 2 images, and with a GoP of 4 images. The results are separated into: low resolution (6 videos) and high resolution (2 videos). The experimental results on CIF and HD resolutions are presented because they are being used increasingly on new video devices, like smartphones, touchpads and computers.

Selective encryption of only the I frame of a GoP

In this section, we present experimental results on the encryption of only the I frame for a GoP. This section is divided into two parts: Section 4.1.1 for low resolution cases and Section 4.1.2 for high resolutions cases. Fig. 9 Benchmarks of videos with their corresponding names.

Results for low resolution video sequences

In the major parts of the results for low resolution videos, we conclude that the encryption scheme tends to guide the confidentiality metrics between the thresholds of confidentiality, as presented in Tab. 1. However, the amount of encrypted coefficients varies depending on the video with an encryption ratio which varies between 12.37% and 18.56% for our set of video sequences. This result highlights that our scheme depends on the video content. For instance, textures are often more affected by the encryption than uniform regions of the videos due to the quantity of the AC coefficients of the entire transform, particularly for mobile video sequences. But the T-SSIM is just measured during the switching of GoPs, and, in these cases, the T-SSIM is always under the bottom threshold, as presented in Fig. 10 for the foreman video sequence. This particular example is a good outline for the other videos. The encryption of only the first frame is not efficient, because between the P frames of a GoP, the T-SSIM is too high, and this method works only for short GoP cases as we will discuss in Section 4.1.2.

Results for high resolution video sequences

With a subjective analysis of the confidentiality of HD video sequences which are encoded with a 4-frame GoP, we immediately notice that the encryption of the only I frame is not sufficient. In the fourth frame of a GoP, and sometimes in the third frame, some ROI are clearly visible due to the low prediction and the reconstruction of the image through the GoP. A typical example is presented in Fig. 11: in the ROI containing a car; we can clearly see that this ROI is not well protected in the last two frames of the 4-frame GoP whereas the protection remains efficient for a 2-frame GoP. However, in the 2-frame GoP case, the results are still good, as presented in Fig. 12 and Fig. 13, which show that the encryption level depends on the content. In Fig. 12, for the ar-drone fly video sequence, the different camera variations affect the video content and also the encryption quality. This is why we can notice fast variations in the number of encrypted coefficients. In Fig. 13, for the movie video sequence, switching of scenes during the movie markedly affects the encryption. We can notice in this video that although the encryption is at its maximum value, the SSIM remains too high due to the content of the end at the video in this particular scene.

In Fig. 14, T-SSIM and SSIM are correctly controlled, and the number of unencrypted coefficients is at a stable level at the end of the video sequence. In Fig. 15, T-SSIM and SSIM are widely chaotic, especially T-SSIM, which is close to 1 at the end of the GoPs. In comparison with Fig 14 and Fig. 15, with reduced selective encryption of only the I frame of a 4-frame GoP, the respective results of Fig. 12 and Fig. 13 are better in terms of SSIM and TSSIM. This underlines the necessity of keeping a small GoP for HD video sequences. In high resolution cases, the MB size is small with respect to the size of the ROI of the video, also, and MBs do not have many AC coefficients after the transformation and quantization. This is why the visual protection for HD video sequences can be less efficient for the same number of encrypted coefficients with respect to other video sequences.

Selective encryption of all frames of a GoP

The experimental results of the encryption of all frames of a GoP are presented in section 4.2, which is divided into two parts: Section 4.2.1 for low resolution cases and Section 4.2.2 for high resolution cases.

Results for low resolution video sequences

Tab. 2 presents the results for the 6 low resolution videos of the benchmark. Note that the encryption tends to be between the two desired thresholds with different encryption ratios, which vary between 13.12% and 19.89% for our set of video sequences. The results in terms of SSIM and TSSIM are better for the mean and the standard deviation which underline the fact that the videos are more annoying to watch and consequently the confidentiality is increased. So this level depends on the video content as we predicted and with respect to the previous results presented in Tab. 1 we can notice that: even if the encryption ratio is similar, the encryption is more widespread through the GoP, which increases the visual confidentiality. In terms of standard deviation, the high variations in the T-SSIM results underline that even if the inter frames are encrypted it is advised to keep a small size for the GoP. Moreover, the encryption is a pseudo-random process and we cannot precisely predict the visual confidentiality of a next GoP when the amount of encrypted coefficients is set. However, we can guide the encryption in a good way in order to have the desired confidentiality, as presented in Fig. 16. Fig. 17 presents a comparison in terms of SSIM with the previous method SE-CAVLC [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF]. The encryption ratio is decreased by 1.09% (17.34% of the encryption ratio for the full SE of the foreman video sequence) and we maintain visual confidentiality between the two thresholds. The mean SSIM is increased by 0.02 for our method (0.36 as mean SSIM for a full SE of the foreman video sequence). Also, when the encryption is sufficient, we can lighten it when the previous method keeps a full SE.

Results for high resolution video sequences

Fig. 18 and Fig. 19 present reduced SE applied to HD videos. Note that the encryption is sufficient enough for the ar-drone video in terms of SSIM and T-SSIM. We can clearly see that with this Smart SE of a 4-frame GoP, the T-SSIM is less chaotic than the previous method, where only the intra frame was encrypted. The encryption is also efficient in terms of SSIM in the movie video but the flickering of the encrypted frames is not perfectly affected and this confirms the necessity of keeping small GoPs even if the flickering is stronger in a full SE case. This is probably caused by the lack of information in each MB of the video, if there are no-coefficients to scramble, the encryption cannot be properly done. These results underlined that in HD cases the effectiveness of the encryption is closely correlated with the amount of texture regions in the video. If the video presents some scenes where the ROI are expanded, like zooms or closeups, the amount of AC coefficients is not wide enough for the encryption. With the proposed method, SE is tailored to the switching of scenes. In this complete example, we can notice a control of the amount of encrypted coefficients with successively a decision due to the T-SSIM, the SSIM and both of them.

Global analysis and discussions

A first subjective analysis has given several confidentiality thresholds based on quality and flickering measures. These thresholds are used to control the proposed SSE-CAVLC to protect a video sequence with respect to SSIM and T-SSIM results. The proposed method presents several good results in terms of encryption ratio. With respect to [START_REF] Shahid | Fast Protection of H.264/AVC by Selective Encryption of CAVLC and CABAC for I & P frames[END_REF], the global encryption ratio of our method is 5% lesser in mean. However, the visual confidentiality of the encrypted videos is in mean near 11 dB which is adequate. To conclude Section 4, Fig. 20 presents some visual results of the encrypted video sequences.

Conclusion

The reduced selective encryption scheme we developed allows to control the encryption of video data depending on the desired confidentiality level. The use of psychovisual metrics is a good way to ensure the visual confidentiality, especially the proposed T-SSIM, which takes into account the temporal visual protection generated by the encryption, i.e. flickering. This T-SSIM quality metric allows us to have better control of the temporal visual confidentiality over the time. In most of the videos, the visual protection is adequate while minimizing the amount of encryption with respect to the previous SE scheme. We have a mean encryption ratio of 16% with a mean SSIM of 0.4 for our experimentation. However, the control of the visual protection through encryption is efficient but not integral, because the encryption method is pseudo-random, also we cannot predict the exact deterioration of the visual content.

Further studies are required to enhance these results. Firstly, we need to develop other psychovisual metrics which are closely correlated with the HVS in terms of low quality and encrypted images. Currently, psychovisual metrics are developed in order to be efficient in high quality cases. Secondly, we need to improve the T-SSIM while taking into account the HVS integration times because the T-SSIM measures two successive frames regardless of the frame frequency. T-SSIM needs also to be extended in order to measure the flickering on several frames for a more precise local measure. Moreover, the encryption can be improved by including some ROIs from objective measures. Finally, this encryption scheme is efficient for most video resolutions, but it deserves to be improved for HD videos.

Fig. 1

 1 Fig. 1 Overview of the H.264/AVC encoder.

Fig. 2

 2 Fig. 2 Syntax elements used in the CAVLC entropy encoder of H.264/AVC. The encrypted coefficients in SE-CAVLC [1] are shown in grey.

2. 3 . 1

 31 Peak-Signal-to-Noise-Ratio (PSNR)

Fig. 3

 3 Fig. 3 Overview of the proposed Reduced Selective Encryption scheme.

Fig. 4

 4 Fig. 4 Overview of the encryption and decryption method.

Fig. 5

 5 Fig. 5 Four successive frames of the encrypted mobile video sequence.

Fig. 6

 6 Fig. 6 Evolution of the luminance of one pixel of a selective encrypted mobile video sequence with respect to the frame number.

PSNR = 8 ,

 8 19 dB, SSIM = 0,001. PSNR = 8,04 dB, SSIM = 0,013. TSSIM = 0,012. PSNR = 8,924 dB, SSIM = 0,006. PSNR = 9,06 dB, SSIM = 0,016. TSSIM = 0,012.

Fig. 7

 7 Fig. 7 Full encryption of two successive frames of ar-drone (on top right) and movie (on bottom right) with the corresponding original frames (on left). The PSNR and the SSIM between the encrypted frame and the original one is indicated below each case. Moreover, TSSIM between the successive frames is indicated below the second frames for each video sequence.

Fig. 8

 8 Fig. 8 Example of 4x4 MB, with the inverse zigzag scan during entropic compression. The DC level is encoded independently, and AC non-zero levels are colored in red.

Fig. 10

 10 Fig. 10 Reduced selective encryption of only the I frame of a 4-frame GoP of the foreman video (300 frames ← 75 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 11 A

 11 Fig. 11 A ROI of the ar-drone video sequence: a) A reduced selective encryption of only the I frame of a 2-frame GoP, b) A reduced selective a encryption of only the I frame of a 4-frame GoP.

Fig. 12

 12 Fig. 12 Reduced selective encryption of only the I frame of a 2-frame GoP of the ar-drone video (1000 frames ← 500 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number: value -1 is the order for decreasing the encryption since both SSIM and T-SSIM are under their corresponding bottom thresholds. Value 0 is no change in the encryption. Value 1 is the increasing in the encryption due to a too high SSIM, value 2 is due to a too high T-SSIM and value 3 both of them. d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 13

 13 Fig. 13 Reduced selective encryption of only the I frame of a 2-frame GoP of the movie video (1000 frames ← 500 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 14

 14 Fig. 14 Reduced selective encryption of only the I frame of a 4-frame GoP of the ar-drone video (1000 frames ← 250 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 15

 15 Fig. 15 Reduced selective encryption of only the I of a 4-frame GoP of the movie video(1000 frames ← 250 GoPs): a) The number of unencrypted coefficients with respect to the GoP number. b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 16

 16 Fig. 16 Reduced selective encryption of a 4-frame GoP of the foreman video (300 frames ← 75 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d)The T-SSIM of the encrypted video with respect to the frame number.

Fig. 17

 17 Fig. 17 SSIM of the encrypted foreman video sequence with respect to the frame number. In blue with full SE, and in red with Smart SE.

Fig. 18

 18 Fig. 18 Reduced selective encryption of a 4-frame GoP of the ar-drone video (1000 frames ← 250 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 19

 19 Fig. 19 Reduced selective encryption of a 4-frame GoP of the movie video (1000 frames ← 250 GoPs): a) The number of unencrypted coefficients with respect to the GoP number, b) The SSIM of the encrypted video with respect to the frame number, c) The decision with respect to the GoP number, d) The T-SSIM of the encrypted video with respect to the frame number.

Fig. 20

 20 Fig. 20 Visual results of two successive encrypted frames of the different video sequences with their respecting names.

Table 1

 1 Results of reduced selective encryption of the six low definition videos where just the I frame of a 4-frame GoP is encrypted. The results are presented in terms of mean and standard deviation (Std) for the SSIM and the T-SSIM, in mean for the amount of unencrypted coefficients (AoUC) per macro-block and in percent for the encryption ratio for a Smart SE (ER-SSE) and a full SE (ER-SE).

		SIMM	T-SSIM	AoUC ER-SSE ER-SE [1]
		Mean Std	Mean Std	Mean	(%)	(%)
	Cit	0.46	0.17 0.40	0.24 9.05	16.78	19.31
	For	0.48	0.16 0.53	0.28 9.53	12.94	15.83
	Hal	0.51	0.05 0.72	0.38 12.1	12.37	17.53
	Mob	0.33	0.08 0.39	0.20 42.3	18.56	25.51
	BBB 0.53	0.17 0.71	0.33 28.6	13.92	20.15
	Ven	0.57	0.08 0.57	0.27 0.81	13.07	13.76

Table 2

 2 Results of the reduced selective encryption of the 6 low definition videos, where all of the 4-frames GoPs are encrypted. The results are presented in terms of mean and standard deviation (Std) for SSIM and T-SSIM, in mean for the amount of unencrypted coefficients (AoUC) per macro-block and in percent for the encryption ratio for a Smart SE (ER-SSE) and a full SE (ER-SE).

		SIMM	T-SSIM	AoUC ER-SSE ER-SE [1]
		Mean Std	Mean Std	Mean	(%)	(%)
	Cit	0.39	0.09 0.35	0.19 10.22	16.53	19.52
	For	0.38	0.12 0.46	0.26 1.44	16.25	17.34
	Hal	0.40	0.06 0.72	0.39 0.48	17.23	18.01
	Mob	0.32	0.08 0.37	0.19 41.04	18.87	26.51
	BBB 0.32	0.14 0.67	0.35 0.21	19.89	20.29
	Ven	0.54	0.08 0.55	0.27 0.14	13.22	13.93