Kumar Raja

Ivan Laptev

Patrick Pérez

Lionel Oisel

JOINT POSE ESTIMATION AND ACTION RECOGNITION IN IMAGE GRAPHS

Keywords: Action Recognition in still images, Pose estimation, Graph optimization

Human analysis in images and video is a hard problem due to the large variation in human pose, clothing, camera view-points, lighting and other factors. While the explicit modeling of this variability is difficult, the huge amount of available person images motivates for the implicit, datadriven approach to human analysis. In this work we aim to explore this approach using the large amount of images spanning a subspace of human appearance. We model this subspace by connecting images into a graph and propagating information through such a graph using a discriminativelytrained graphical model. We particularly address the problems of human pose estimation and action recognition and demonstrate how image graphs help solving these problems jointly. We report results on still images with human actions from the KTH dataset.

INTRODUCTION

We address the problem of human action recognition and pose estimation in still images. While human action recognition has been mostly studied in video, actions provide valuable description for many static images, hence, automatically identifying actions in such images could greatly facilitate their interpretation and indexing.

Human action recognition is known to be a hard problem due to the large variability in human pose, clothing, viewpoints, lighting and other factors. Identifying actions in still images is particularly challenging due to the absence of motion information helping action recognition in video. Several works have addressed human analysis in still images by identifying body pose [START_REF] Ramanan | Learning to parse images of articulated bodies[END_REF][START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Ferrari | Pose search: retrieving people using their pose[END_REF]. In particular, methods addressing human pose estimation and action recognition jointly have been recently proposed in [START_REF] Yang | Recognizing human actions from still images with latent poses[END_REF][START_REF] Yao | Modeling mutual context of object and human pose in human-object interaction activities[END_REF] motivated by the interdependency between the pose and the action. Such methods, formulated in terms of graphical models, are typically trained on manually annotated examples of person images and are then applied to individual images during testing.

The number of available annotated training images is usually limited due to the high costs associated with the manual annotation. At the same time, huge collections of images with no or noisy labels are now available online approximating the dense sampling of the visual world. Such collections have been successfully explored by recent work on object and scene recognition [START_REF] Torralba | 80 million tiny images: a large dataset for non-parametric object and scene recognition[END_REF][START_REF] Hays | Im2gps: estimating geographic information from a single image[END_REF] and in graphics [START_REF] Hays | Scene completion using millions of photographs[END_REF].

In this paper we aim to push the above ideas further and to explore dense image sampling for human analysis. We assume a large number of images is available spanning the subspace of particular human actions. We assume only some of these images are annotated and use the remaining images to propagate information between each other. The underlying assumption behind our method is that images with small distance in the image space will often have similar semantics such as human pose and actions. We formalize this intuition in a graphical model by connecting similar images of people in a graph as illustrated in Fig. 1. We in particular, address the problems of human pose estimation and action recognition and demonstrate how the proposed image graphs enable to improve solutions for both of these tasks when solved jointly.

Related work. Action recognition in still images was addressed by Ikizler et. al [START_REF] Nazli Ikizler | Recognizing actions from still images[END_REF] who used histogram of oriented rectangles as features and SVM classification. In [START_REF] Ikizler-Cinbis | Learning actions from the web[END_REF] action images were collected from the web using text queries and an action model was built iteratively. Actions in consumer photographs were collected and recognized in [START_REF] Delaitre | Recognizing human actions in still images: A study of bag-of-features and part-based representations[END_REF] using Bag-of-Words and LSVM classifiers.

Several other methods attempted action recognition by explicitly modeling the structure of the human body and its relation to manipulated objects. Graphical models have been used in [START_REF] Ramanan | Learning to parse images of articulated bodies[END_REF][START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Ferrari | Pose search: retrieving people using their pose[END_REF] to model relations among body parts. More recently, [START_REF] Yang | Recognizing human actions from still images with latent poses[END_REF][START_REF] Yao | Modeling mutual context of object and human pose in human-object interaction activities[END_REF] extended this work towards the joint modeling of human poses and actions. We build on top of this work and extend it by leveraging the large number of unlabeled images. In this regard our work is related to the methods of object and scene recognition using large collections of unlabeled images [START_REF] Torralba | 80 million tiny images: a large dataset for non-parametric object and scene recognition[END_REF][START_REF] Hays | Im2gps: estimating geographic information from a single image[END_REF] and extends it to human analysis.

Overview. First, we describe our joint graphical model for human pose and actions in a single image. The graph energy is defined in terms of image-dependent and imageindependent terms as described in Section 2. Next, Section 3 presents our contribution by extending the single-image graphical model to multiple images. This extension allows us to exploit unlabeled images while solving for the poses and actions in all images simultaneously. Experiments validating our approach are reported and discussed in Section 4. Finally, Section 5 concludes the paper.

JOINT MODEL FOR A SINGLE IMAGE

Motivated by the idea of pictorial structures [START_REF] Felzenszwalb | Pictorial structures for object recognition[END_REF] and following previous work [START_REF] Ramanan | Learning to parse images of articulated bodies[END_REF][START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Ferrari | Pose search: retrieving people using their pose[END_REF][START_REF] Yang | Recognizing human actions from still images with latent poses[END_REF][START_REF] Yao | Modeling mutual context of object and human pose in human-object interaction activities[END_REF], we model people using graphs. Our graphical model of a person (see Fig. 2(a)) contains six variable nodes encoding the positions of five body parts and the action label. We consider body parts p ∈ P = {H, RH, LH, RF, LF } corresponding to head, right-hand, left-hand, right-foot and left-foot, respectively, as well as K action classes A. The links between the nodes encode actiondependent constraints on the relative position of body parts and their appearance. Fig. 1 illustrates action labels and positions of the five body parts for some of our samples. Note that in some samples (depending on viewpoints) some parts may not be visible. So we include "occlusion" as one of the possible states of the part nodes. Using the part positions x p , p ∈ P, the pose vector is defined as P = [x p ] p∈P .

We define the energy of our pose-action graph in terms of image-dependent and image-independent potentials using pose P and action label A as variables:

E(A, P ) = φ(A; I) + p∈P φ p (x p , A; I) + ψ h (x RH , x LH , A) + ψ f (x RF , x LF , A)(1)
where image-dependent potentials φ(.; I) encode appearance of parts and actions in the image I, and image-independent potentials ψ(., ., A) encode relations between the body parts for an action. We estimate the pose and action by maximizing the energy E, over action labels and part locations:

(A * , P * ) = arg max (A,P ) E(A, P ) (2)
The arguments which maximize the above expression are found by the max-sum algorithm [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. The advantage of the proposed joint action-pose model will be shown experimentally in Section 4. Discretization of Pose Space. Pose estimation involves the maximization of ( 2) over all possible poses. The concatenation of five part positions as the pose vector makes the pose space huge and the maximization intractable. In order to overcome this, we discretize and narrow down the pose space by identifying the most probable locations of the part. To this end, we take several frames from different action videos in which the parts positions are annotated relative to the person bounding box. For each part, we form a set by combining the list of locations and random perturbations of them. This set constitutes the set of part node states and it drastically reduces the cardinality of the pose space. Fig. 3(a) illustrates the locations corresponding to the discrete states of the "head" and "righthand" part nodes. The number of states in the head and righthand nodes is 364 and 1921, respectively.

Image-dependent potentials. There are six imagedependent terms in [START_REF] Ramanan | Learning to parse images of articulated bodies[END_REF]. To model action potential φ(A; I), we learn a binary static-image action classifier for each of the K action classes using histogram of gradients (HoG)-features and LSVM detector [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF]. For a given image we obtain K action scores, we scale them linearly to the interval [0 1] and use them as values of φ(A; I).

Similarly, to define part potentials φ p (x p , A; I), p ∈ P, we train action-dependent body part detectors. In sample action images with five annotated body parts, we obtain HoGbased descriptors as features to train RBF-SVM parts detector using [START_REF] Joachims | Making large-Scale SVM Learning Practical[END_REF]. For each part node state (part location), the RBF-SVM detector score is evaluated and the scores scaled between 0 to 1 to get the potentials values φ(P, A; I). We fix the potential values corresponding to occlusion states to a constant corresponding to the frequency of a part being occluded among the training samples for a given action.

Image-independent potentials.

Image-independent terms ψ h (x RH , x LH , A) and ψ f (x RF , x LF , A) define relations between positions of hands and feet in the image. We model them in terms of the discrete states of the part nodes.

IMAGE GRAPH

Here we propose to extend the graphical model for a singleperson image in the previous section to multiple images. The rationale behind this extension is to take advantage of the large number of unlabeled person images and to propagate pose and action information among similar images. Such propagation should enable improved interpretation of samples when pose and action cannot be well-inferred from the training samples alone.

Given an image graph we introduce links between body parts of neighboring images. An example of a two-image pose-action graph is illustrated in Fig. 2(b). We connect all labeled (training) images and unlabeled (test) images into a graph and estimate the pose and the actions for unlabeled images simultaneously by maximizing the graph energy. We define the energy of the image graph as a function of pose and action in unlabeled images as

E G ({A} i , {P } i ) = i E i (A i , P i ) + i∼j p∈P d p (x p i , x p j ), (3)
where

P i = [x p i ]
p∈P , E i is the graph energy for a single image i defined in (1), and d p (x p i , x p j ) are neighbor potentials between images i and j penalizing the placement of matching parts p far apart. We define d p (x p i , x p j ) to be exp(-||x p i -x p j || 2 /2σ 2 ), where σ is the spread parameter. Potentials d p (x p i , x p j ) provide a strong prior on the position of parts in test images with direct connections to the training images. d p (x p i , x p j ) also encourages similar test images to have similar placement of parts. Note that action nodes of neighbor images are indirectly connected through the bodypart nodes. This enables propagation of action information through the graph.

Graph Construction. To construct an image graph, we connect every image (labeled and unlabeled) to its four nearest neighbors in terms of a distance measure. As the pose and the action labels are fixed for the training images, there is no flow of information among them. We therefore first connect every labeled image to its similar unlabeled images, then every unlabeled image is linked to its neighbors among labeled and unlabeled images. Our graph construction requires the notion of image similarity which can be defined in different ways. To validate our approach, in this work we find image similarity between a pair of images using the "true pose distance" which is defined as

D(i, j) = k∈Pv ||P k (i) -P k (j)|| 2 + nC o (4)
where P k (i) and P k (j) are the annotated positions of corresponding body parts in images i and j respectively, P v is the set of visible parts in both images and C o is the fixed occlusion cost penalizing a part with mismatched occlusion label.

In the future work we plan to substitute D(i, j) by a measure based on image information only. Note that the part positions in the unlabeled images are predicted from graph optimization.

EXPERIMENTS AND RESULTS

We validate our framework by estimating human pose and recognizing actions in still frames extracted from KTH dataset. The dataset contains images of multiple people performing six classes of actions: boxing, handclapping, handwaving, jogging, running and walking. Our training and test sets are separated by person identities and contain 461 and 328 cropped person images respectively. To train and test human pose estimation, we have manually annotated bounding boxes of head, hands and feet in all our images. A few samples and corresponding annotations from our dataset are illustrated in Fig. 1.

To evaluate action recognition, we measure accuracy by the ratio of correctly classified images to the total number of test images. To evaluate pose estimation, we report Average Precision (AP) values computed for the five body parts (head, hands and feet). We assume a body part is correctly localized if the overlap between its predicted and its true bounding boxes is greater than 0.5. 1 The detection score of a part corresponds to the sum of the terms in the graph energy in which the part location is a variable, i.e. it is the sum of the factors surrounding the part node in the pose-action graph. We do not distinguish between the left/right hands and left/right feet in the evaluation. Our framework allows for prediction of occluded parts. We consider a false detection if a part location is predicted for an occluded part, and vice-versa.

We evaluate human pose estimation and action recognition for different settings of the graphical model. To show the advantage of the joint approach to pose estimation and action recognition, we first consider an independent solution. For this purpose, in experiment E1 the action in an image is simply A * = arg max A φ(A; I). The location of a part p, p ∈ P is predicted as x p * = arg max x p max A φ(x p , A; I).

In the second experiment E2 we extend E1 and evaluate the advantage of action-dependent pose estimation by considering potentials φ p (x p , A; I) while still not modeling the relative position of parts, i.e. setting ψ(., ., A) = 0 in (1). In the experiment E3 we consider the solution provided by maximizing the full energy E in (1) of the single image graph. Finally, in experiment E4 we evaluate our extension of the single-image graph model to the multiple image graph. We report the solution obtained by maximizing the energy E G in (3).

The results for all four experiments E1-E4 are reported in Tables 1 and2 for action recognition and pose estimation respectively. By comparing results of E1 and E2, we observe the advantage of modeling the appearance of actions and parts jointly in E2. Additional modeling of relative positions of body parts in E3 demonstrates an improvement both for the action recognition and pose estimation compared to E1 and E2. Finally, the image graph proposed in this paper results in a clear improvement of action recognition and pose estimation in E4 compared to the single-image graph optimization in E3.

Experiment

Action Recognition(in %) E1. Action

CONCLUSIONS AND FUTURE WORK

We have extended a joint model of human action and pose to image graphs in a semi-supervised framework for human analysis. In this extension, inference on pose and action is performed in unlabeled images of an image graph using the image connections. A crucial aspect in this framework is the construction of the image graph in which neighboring images must have similar pose. The immediate future work to improve upon the presented ideas includes: 1) use of an automatic mechanism to determine the image distance used to construct image graphs, 2) building of image graphs with specific topologies to leverage the connections better, and 3) exploring a more realistic dataset of human actions such as consumer photographs.

Fig. 1 .

 1 Fig. 1. Joint pose estimation and action recognition in the image graph. Training images (red frames) are manually annotated with the position of body parts and action labels. Part positions and action labels in test images (yellow frames) are resolved by optimizing the global graph energy.

Fig. 2 .

 2 Fig. 2. Graphical models. (a) Pose-action graph for a single person. (b) Pose-action graph for two people in similar pose.

Fig. 3 .

 3 Fig. 3. (a): Discrete states (red) for positions of the head and the right-hand. (b): ψ h (x RH , x LH , A) showing possible left-hand locations (yellow) for handwaving action and a fixed right-hand position (red)

Table 1 .

 1 Accuracy of action recognition.

	classifier	78.35
	E2. ψ(., ., A) = 0		81.09
	E3. 1-image graph	82.62
	E4. N-image Graph	86.58
	Experiment	Head	Hands	Feet	mAP
	E1. Part Detectors	0.9645 0.3854 0.5357 0.6286
	E2. ψ(., ., A) = 0	0.9604 0.4105 0.6790 0.6833
	E3. 1-image Graph 0.9608 0.5206 0.9256 0.8023
	E4. N-image Graph 0.9892 0.8293 0.9745 0.9310

Table 2 .

 2 Average precision for part localization.

We measure the overlap between two bounding boxes a and b as the intersection over union: |a ∩ b|/|a ∪ b|.


Muhammad Muneeb Ullah

Ivan Laptev

ACTLETS: A NOVEL LOCAL REPRESENTATION FOR HUMAN ACTION RECOGNITION IN VIDEO

Keywords: Action recognition, local motion descriptors, supervised learning, actlets

This paper addresses the problem of human action recognition in realistic videos. We follow the recently successful local approaches and represent videos by means of local motion descriptors. To overcome the huge variability of human actions in motion and appearance, we propose a supervised approach to learn local motion descriptors -actlets -from a large pool of annotated video data. The main motivation behind our method is to construct action-characteristic representations of body joints undergoing specific motion patterns while learning invariance with respect to changes in camera views, lighting, human clothing, and other factors. We avoid the prohibitive cost of manual supervision and show how to learn actlets automatically from synthetic videos of avatars driven by the motion-capture data. We evaluate our method and show its significant improvement as well as its complementarity to existing techniques on the challenging UCF-sports and YouTube-actions datasets.

INTRODUCTION

Recognition of human actions in video is a highly challenging problem due to the large variations in person appearance, camera view points, backgrounds and other factors. An explicit way to address this problem is to reason about locations, poses, motion and interactions of people. While such a structural approach is appealing, it is difficult to implement in practice due to the modest performance of current methods for person detection and pose estimation in realistic video.

One alternative consists of representing human actions using statistics of local video descriptors. Despite being direct and simple, Bag-of-Features (BoF) type methods have recently been shown successful when applied to action recognition in realistic and challenging video data [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Yeffet | Local trinary patterns for human action recognition[END_REF][START_REF] Wang | Action recognition by dense trajectories[END_REF]. While the basic BoF model represents actions by disordered collections of local video descriptors, several ideas have been proposed aiming to improve BoF by modeling the spatial and temporal structure of activities [START_REF] Ullah | Improving bag-offeatures action recognition with non-local cues[END_REF][START_REF] Niebles | Modeling temporal structure of decomposable motion segments for activity classification[END_REF]. Complementary to these methods, the goal of this paper is to design improved local video descriptors providing better building blocks for statistical action representations.

Local video representations, e.g., [START_REF] Laptev | On space-time interest points[END_REF] are typically designed using unsupervised k-means quantization of local video descriptors into visual vocabularies [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] based on visual similarity. The rationale behind visual vocabularies is to group similar local events, such as upwards movements of hands, and to score occurrences of such events in subsequent recognition. While visual descriptors, e.g., HOG/HOF [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Wang | Action recognition by dense trajectories[END_REF] provide some invariance to variations of events in motion and appearance, unsupervised clustering may not be able to group similar events given the frequent variations of the video data due to changes in view points, lighting, background, clothing of people, style of actions and other factors.

Bourdev et al. [START_REF] Bourdev | Poselets: Body part detectors trained using 3d human pose annotations[END_REF] have recently proposed a supervised approach to learn appearance of body parts in static images. Body part detectors called poselets are trained to be invariant to irrelevant appearance variations using manual annotation of body parts in training images. Inspired by this representation, we in this paper propose a supervised approach to learn actlets, i.e., detectors of body parts undergoing specific patterns of motion. Learning actlets requires a substantial amount of annotated training data. To collect such data, we here propose to avoid the heavy burden of manual video annotation and generate annotated data automatically by synthesizing videos of avatars driven by the motion-capture data (cf. Fig. 1). We next develop a method that successfully deploys actlets for action recognition. We evaluate our method and demonstrate its significant improvement as well as complementarity to existing techniques on the UCF-sports and YouTube-actions datasets.

The rest of the paper is organized as follows: Sec. 2 presents details of our synthetic dataset used to train actlets in Sec. 3. Sec. 4 describes application of actlets to action recognition. Sec. 5 presents evaluation of our method. Sec. 6 concludes this paper with a discussion.

SYNTHETIC DATASET OF HUMAN MOTIONS

To train a representative set of actlets, we need a relatively large amount of training data. The training data should cover a diverse range of human movements and should contain annotated positions of body joints over time. Also, a significant amount of variation in terms of appearance (e.g. clothing and background), view-point, illumination, and camera motion, is required to span the expected variability of the test videos. While manual annotation of body joints and their motion in video is highly time-consuming and therefore impractical, we resort to animation techniques and use motion capture data to build a synthetic dataset. The main advantage in this approach, is the availability of the ground-truth positions of body-joints in each synthesized video provided by the 2D projections of 3D body-joint positions of the motion-capture data. We use the CMU motion capture database1 , containing a large number of human motion sequences; from simple locomotions and physical activities to more complex movements involving human interactions.

We perform motion re-targeting of CMU motion capture sequences on 3D humanoid characters in Autodesk Motion-Builder 2011, and render videos from a set of fixed locations. We use ten 3D characters including males and females of different physiques, wearing different clothes. We render videos Fig. 3. Illustration of motion clusters for four body joints. All trajectories within a cluster are shown in separate plots by blue curves. An example patch for each cluster is also shown. from a set of three different camera view points (front, left and right with respect to the character) while using five different static backgrounds. Additionally, we simulate the panning of the camera which follows the motion of the character in each video. We render one video for each motion capture sequence in the CMU database while randomly choosing a character, background, and a view point; and get 2549 synthetic video sequences in total. Fig. 2 illustrates a few example frames from our synthetic dataset together with the automatically annotated trajectories of body joints. We will make our dataset publically available upon acceptance of this paper.

TRAINING ACTLETS

In this paper we consider the motion of nine body-joints (head, left/right elbow, left/right wrist, left/right knee and left/right ankle), as these are expected to provide rich action description. To group body joints with similar motion patterns, we perform clustering of 2D trajectories associated with each of the nine body joints. We then extract video patches for each trajectory and use them to train one actlet classifier for each trajectory cluster. The details of the method are described below.

Trajectory representation

For each of the nine body-joints in a synthetic video, the associated 2D trajectory with spatial coordinates (x t , y t ) over time t ∈ 1...T is subdivided into overlapping subtrajectories, each having a length of L = 15 frames. The shape of a sub-trajectory encodes the local motion pattern associated with the body-joint. Following [START_REF] Sukthankar | Feature seeding for action recognition[END_REF], we represent the shape of a sub-trajectory with a velocity-based vector. Given a sub-trajectory of length L, we describe its shape by a sequence S = (∆P t , . . . , ∆P t+L-1 ) of displacement vectors ∆P t = (P t+1 -P t ) = (x t+1 -x t , y t+1 -y t ). The resulting vector S is normalized by the height of the character in the rendered video.

Clustering and training of actlets

We perform k-means clustering (we set k = 75) on all subtrajectories associated with each of the nine body joints in all 2549 synthetic videos. We avoid occluded body joints and remove trajectories of right/left joints from the videos synthesized for the left/right views of the person respectively. We perform both view-specific and view independent clustering, where trajectories from the three different views are clustered either separately or jointly. To select distinct clusters, we sort clusters for each body joint according to the decreasing sum of distances to other clusters and keep the top n = 50 clusters from them. Fig. 3 illustrates examples of our clusters.

To train an actlet for a given body joint and motion pattern, we extract video patches in the neghbourhood of trajectories from one trajectory cluster. These video patches serve as positive training samples for an actlet. For the negative training samples, we randomly extract 10,000 synthetic video patches, corresponding to trajectories from the remaining 49 clusters of the same body joint. We represent extracted video patches by histograms of optical flow (HOF) descriptors [START_REF] Laptev | Learning realistic human actions from movies[END_REF]. 2 We then train a linear SVM classifier for the HOF descriptors. This way, we obtain a total of 1000 linear SVM classifiers 3 , corresponding to the view-specific and view-independent actlets.

CLASSIFICATION

Actlets provide a means to detect specific motion patterns of body joints in video disregarding irrelevant variations of the data in terms of backgrounds, clothing, view points and other factors. Our next goal is to deploy such descriptors for action recognition in real video. Given a video, we extract denselysampled video patches and represent them by the HOF descriptors. For each HOF descriptor we obtain a set of actlet scores according to all trained actlet classifiers. We then use a 24-level spatio-temporal grid and concatenate maximum response value of each actlet within each grid cell into a vector representing the whole video. We refer to this video representation the Actlet channel. A somewhat similar approach called object banks has been previously proposed to represent still images [START_REF] Li | Objects as attributes for scene classification[END_REF]. For action classification based on the Actlet channel, we use a non-linear SVM [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] with RBF kernel.

We use Bag-of-Features (BoF) video representation as a baseline. BoF is typically based on k-means clustering, which is used to quantize local spatio-temporal descriptors into visual vocabularies, based on visual similarity. Here, we follow [START_REF] Wang | Evaluation of local spatio-temporal features for action recognition[END_REF], and build the Bag-of-Features video representation using the Harris3D detector [START_REF] Laptev | On space-time interest points[END_REF] in combination with the HOG/HOF descriptors [START_REF] Laptev | Learning realistic human actions from movies[END_REF]. We refer to this video representation as BoF channel (see [START_REF] Ullah | Improving bag-offeatures action recognition with non-local cues[END_REF] for more details).

For classification, we use a non-linear SVM [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] with a χ 2 kernel [START_REF] Laptev | Learning realistic human actions from movies[END_REF]. We use one-against-rest approach for multi-class classification for the both channels.

We integrate the Actlet channel with the BoF channel using multi-channel kernel [START_REF] Zhang | Local features and kernel for classification of texture and object categories: A comprehensive study[END_REF]:

K (x i , x j ) = exp - c 1 Ω c D x c i , x c j , (1)
where D(x c i , x c j ) is the distance computed using video channel x c between videos i and j, and Ω c is the normalization factor computed as an average channel distance [START_REF] Zhang | Local features and kernel for classification of texture and object categories: A comprehensive study[END_REF]. In our case, D(x c i , x c j ) is the Euclidean distance for the Actlet channel, whereas, χ 2 distance for the BoF channel.

EXPERIMENTS

We evaluate the performance of actlets on the task of action classification in two challenging datasets: UCF-Sports and YouTube Actions. The two datasets mainly contain sports action classes.

The UCF-Sports dataset [START_REF] Rodriguez | Action mach a spatio-temporal maximum average correlation height filter for action recognition[END_REF] contains 10 different types of human actions: swinging (on the pommel horse and on the floor), diving, kicking (a ball), weight-lifting, horseriding, running, skateboarding, swinging (at the high bar), golf swinging and walking (cf. Fig. 4, top). The dataset consists of 150 video samples which show a large intra-class variability. To increase the amount of data samples, we extend the dataset by adding a horizontally flipped version of each sequence to the dataset (similar to [START_REF] Wang | Evaluation of local spatio-temporal features for action recognition[END_REF]). We train a multi-class classifier and report the average accuracy over all classes. We use a leave-one-out setup and test on each original sequence while training on all other sequences together with their flipped versions (the flipped version of the tested sequence is removed from the training set).

The YouTube Actions dataset [START_REF] Liu | Recognizing realistic actions from videos 'in the wild[END_REF] contains 11 action categories: basketball shooting, biking/cycling, diving, golf swinging, horse back riding, soccer juggling, swinging, tennis swinging, trampoline jumping, volleyball spiking and walking with a dog (cf. Fig. 4, bottom). This dataset is challenging due to the large variations in camera motion, object appearance and pose, object scale, viewpoint, cluttered background and illumination conditions. The dataset contains a total of 1168 sequences. We follow the original setup [START_REF] Liu | Recognizing realistic actions from videos 'in the wild[END_REF] and perform leave-one-out cross validation for a pre-defined set of 25 folds. As for the UCF-Sports dataset, we report the average accuracy over all classes as the performance measure.

Results

Results on the UCF-Sports and YouTube datasets are presented in Table 1 and Table 2 respectively. We can notice that the performance of the Actlet channel is slightly lower than that of the baseline BoF channel on the UCF-Sports dataset, whereas, the Actlet channel performs better than the BoF channel on the YouTube dataset. Moreover, the combination of the Actlet channel with the BoF channel gives an improvement of approximately 6% over the BoF baseline in the case of UCF-Sports dataset and about 9% for the YouTube dataset. The better performance of the combined channels indicates their complementarity. Actlets focus on the characteristic local movements of people, whereas, BoF has a potential of capturing additional contextual information from the background. For the UCF-Sports dataset, we can observe that our proposed Actlet channel helped to significantly improve 5 out of 10 action classes (cf. Table 1), notably, the Golf swing and Skateboard action classes. On the YouTube dataset, the Actlet channel improved all the 11 action classes (cf. Table 2), specifically, the Volleyball spike class.

CONCLUSIONS

We have proposed a novel approach to represent local patterns of human motion in video, i.e., actlets. To get the required relatively large amount of annotated training data, we have avoided the expensive manual annotation and proposed to use synthetically generated videos of avatars driven by the motion capture data. We have shown how to train actlets for body parts undergoing particular motion, while making actlets insensitive to irrelevant variations in the video data. We have then proposed a new video representation based on actlets, and demonstrated significant improvement in human action recognition on the two challenging datasets, i.e., the UCFsports and YouTube-actions datasets.

Fig. 1 .

 1 Fig. 1. Illustration of actlets, trained on synthetic data (left) and localized on the real videos (right). The automatically annotated trajectories of body-joints are shown on the left.

Fig. 2 .

 2 Fig. 2. Sample frames from our synthetic dataset illustrating variability of generated videos in terms of view points, backgrounds, character clothing and motion. Color curves illustrate automatically annotated trajectories of body joints.

Fig. 4 .

 4 Fig. 4. Sample frames from video sequences of UCF-Sports (top), and YouTube Actions (bottom) datasets.

Table 1 .

 1 Accuracy for the UCF-Sports dataset.

		BoF [%] Actlet [%] Comb. [%]
	Dive	100.00	100.00	100.00
	Golf swing	61.25	51.25	90.00
	Kick ball	90.00	100.00	100.00
	Weight lift	100.00	83.33	100.00
	Horse ride	75.00	50.00	58.33
	Run	76.19	52.38	69.05
	Swing:pommel	85.00	100.00	95.00
	Skateboard	0.00	33.33	25.00
	Walk	90.91	81.82	95.45
	Swing:high bar	91.67	91.67	91.67
	Average accuracy	77.00	74.38	82.45
		BoF [%] Actlet [%] Comb. [%]
	Bike	72.29	68.98	83.87
	Dive	90.00	88.00	92.00
	Golf	77.00	77.00	87.00
	Soccer juggle	46.00	55.00	59.00
	Trampoline jump	66.00	63.00	73.00
	Horse ride	67.00	70.00	73.00
	Basketball shoot	22.33	42.00	38.67
	Volleyball spike	67.00	81.00	81.00
	Swing	73.00	79.00	76.00
	Tennis swing	51.00	44.00	55.00
	Walk	34.83	39.90	50.36
	Average accuracy	60.59	64.35	69.90

Table 2 .

 2 Accuracy for the YouTube Actions dataset.

Available at: http://mocap.cs.cmu.edu

We use motion descriptors only as we expect motion, in contrast to appearance, to transfer well between synthetic and real videos.

[START_REF] Wang | Action recognition by dense trajectories[END_REF] Front: 9 joints×50 clusters + left/right: 2×5 joints×50 clusters + viewindependent: 9 joints×50 clusters. We train actlets for clusters with the minimum of 50 trajectories.


Hossein Azizpour
email: azizpour@kth.se

Ivan Laptev
email: ivan.laptev@inria.fr

Object Detection Using Strongly-Supervised Deformable Part Models

 achieve state-of-the-art performance for object detection, but rely on heuristic initialization during training due to the optimization of non-convex cost function. This paper investigates limitations of such an initialization and extends earlier methods using additional supervision. We explore strong supervision in terms of annotated object parts and use it to (i) improve model initialization, (ii) optimize model structure, and (iii) handle partial occlusions. Our method is able to deal with sub-optimal and incomplete annotations of object parts and is shown to benefit from semi-supervised learning setups where part-level annotation is provided for a fraction of positive examples only. Experimental results are reported for the detection of six animal classes in PASCAL VOC 2007 and 2010 datasets. We demonstrate significant improvements in detection performance compared to the LSVM [1] and the Poselet [3] object detectors.

Introduction

Visual object recognition has achieved substantial progress in recent years, scaling up to thousands of object categories [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] and demonstrating industry-level performance in many applications such as face detection and face recognition. While large-scale visual categorization presents new challenges to the field, many single object classes remain to be very difficult to recognize. For example, the detection of birds, dogs, chairs and tables still achieves only modest performance in the recent PASCAL VOC evaluations [START_REF] Everingham | The PASCAL Visual Object Classes Challenge[END_REF]. This indicates the need of better models and learning methods able to handle objects with large variations in appearance due to intra-class variability, non-rigid deformations, wide range of views and other factors.

Pictorial structures [START_REF] Fischler | The representation and matching of pictorial structures[END_REF] provide a powerful framework for representing objects by non-rigid constellations of parts. Such models have been recently applied to a number of computer vision problems including object recognition [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Felzenszwalb | Pictorial structures for object recognition[END_REF][START_REF] Zhu | Latent hierarchical structural learning for object detection[END_REF], human pose estimation [START_REF] Ramanan | Learning to parse images of articulated bodies[END_REF][START_REF] Ferrari | Pose search: Retrieving people using their pose[END_REF][START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF], action recognition [START_REF] Yang | Recognizing human actions from still images with latent poses[END_REF] and facial feature detection [START_REF] Cristinacce | Feature detection and tracking with constrained local models[END_REF]. In particular, discriminatively-trained deformable part-based models (DPMs) [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Zhu | Latent hierarchical structural learning for object detection[END_REF] have shown excellent performance for object detection tasks. Learning such models, however, involves the optimization of a non-convex cost function over a set of latent variables standing for image locations of object parts and mixture

LSVM model

Strongly-supervised deformable part model Fig. 1: Dog detection (yellow rectangles) using deformable part-based models. Top: dog detections using the proposed strongly-supervised DPM method. Bottom: dog detections using regular DPMs [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF]. Note how the part structure of our model is adapted to the different dog images. In contrast, the original model attempts to explain all three images using the same deformable model. component assignment. The success of training DPMs, hence, depends on the good initialization of model parts in general [START_REF] Naderi Parizi | Reconfigurable models for scene recognition[END_REF].

While the latent SVM approach in [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] applies heuristics to automatically initialize model parts, several recent methods explore strong part-level supervision. For example, poselet-based methods [START_REF] Bourdev | Poselets: Body part detectors trained using 3D human pose annotations[END_REF][START_REF] Ott | Shared parts for deformable part-based models[END_REF][START_REF] Wang | Learning hierarchical poselets for human parsing[END_REF] learn appearance of human body parts from manually annotated limb locations. The joint learning of appearance and deformation parameters in DPMs using part-level supervision has been used for body pose estimation in [START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF] and object part localization in [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF].

The goal of this work is to develop and evaluate a strongly-supervised DPM framework for object detection. Our extensions of existing methods are motivated by the following. First, we expect the use of additional part-level supervision to enhance the performance of current DPM detectors [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Zhu | Latent hierarchical structural learning for object detection[END_REF]. Second, additional supervision should enable us to construct better, class-specific object models compared to the generic star models (see Figure 1). Finally, compared to the task of part localization [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF], object detection does not require precise part recovery and should benefit from a different learning objective compared to [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF].

Summarizing our contributions, we use strong supervision to improve the initialization of the LSVM object detector [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] (mixture component associations and part locations) and propose a class-specific optimization of the model structure. We formulate a learning objective which can deal with sub-optimal and incomplete object parts annotations, and which explicitly handles partial object occlusions. We provide extensive evaluation and favorably compare our method to previous object detectors [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Bourdev | Poselets: Body part detectors trained using 3D human pose annotations[END_REF] on the task of detecting six animal classes in PASCAL VOC 2007 and 2010 datasets.

The rest of the paper is organized as follows. Section 2 discusses related work. Next, we describe DPM and its proposed extensions in Section 3. Experimental evaluation of the method is presented in Section 4. Section 5 concludes the paper.

Related work

Object detection is a dynamic research area. Originating from work on person detection [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], the HOG descriptors with linear SVM classifiers have been a standard building block for object localization. While HOG represents objects by a rigid template, LSVM detector [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] extends [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] to a discriminatively-trained model of deformable HOG parts. Our work builds on DPMs in [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and extends it by introducing part-level supervision and optimization of the model structure.

Bag-of-Features (BOF) models have been highly successful for object retrieval [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] and image classification [START_REF] Zhang | Local features and kernels for classification of texture and object categories: a comprehensive study[END_REF]. Application of BOF to object localization, however, has been more problematic due to efficiency reasons. [START_REF] Chen | Active mask hierarchies for object detection[END_REF] combines BOF with HOG features in the latent SVM DPM framework. Region-level cues and image segmentation have also been explored to improve detection for textured objects such as cats and dogs in [START_REF] Parkhi | The truth about cats and dogs[END_REF]. Our work is complementary to these methods in that we use a single feature type only (HOG) while aiming to enrich the structure of the model and to explore part-level supervision.

Strong part-level supervision has been explored in a number of recent works. The work on Poselets [START_REF] Bourdev | Detecting people using mutually consistent poselet activations[END_REF] uses human limb annotation to learn independent HOG-SVM body-part detectors for person localization. In contrast to this work, we jointly learn part appearance and model deformation parameters and demonstrate improvements over [START_REF] Bourdev | Detecting people using mutually consistent poselet activations[END_REF] in Section 4. Part-level supervision has been used for human pose estimation [START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF][START_REF] Wang | Learning hierarchical poselets for human parsing[END_REF][START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] and object part localization [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF]. While [START_REF] Wang | Learning hierarchical poselets for human parsing[END_REF][START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF] learn part appearance independently, our approach is more related to [START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF][START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] using strong supervision to train DPMs. The task of part localization addressed in [START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF][START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF], however, is different from the task of object detection in this paper. In particular, the learning objectives in [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] enforce annotation-consistent part localization, which is not directly relevant for object detection and can hurt detection performance if some annotated parts are not discriminative or if their annotation is imprecise. On the contrary, our method optimizes the detection objective and learns discriminative locations and relative weights of parts. Finally, we use mixture of components (part trees) as opposed to the part mixtures in Yang and Ramanan [START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF]. The use of component mixtures enables us to adapt the structure of the model to different views of the object, as also explored in [START_REF] Zhu | Face detection, pose estimation, and landmark localization in the wild[END_REF].

In contrast with previous discriminatively-trained DPMs, we (i) explore classspecific optimization of the model structure, (ii) explicitly model part occlusions, and (iii) handle imprecise part-level annotation. We also explore and show benefits of semi-supervised learning where part annotation is provided for a fraction of positive training examples only.

Discriminative part-based model

We follow the framework of deformable part models [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Wang | Learning hierarchical poselets for human parsing[END_REF][START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF][START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF] and describe an object by a non-rigid constellation of parts appearance and location. Contrary to the above models we introduce a binary part visibility term in order to explicitly model occlusion. Each part in our model is defined by the location of a bounding box p i = (p x1 i , p y1 i , p x2 i , p y2 i ) in the image and the binary visibility state v i . One mixture component of the model has a tree structure with nodes U and edges E corresponding to object parts and relations among parts respectively. The score of a model β in the image I given model parts locations P = (p 0 , ..., p n ), and visibility states V = (v 1 , ..., v n ), v i ∈ {0, 1} is defined by the graph energy

S(I, P, V ) = max c∈{1..C} S(I, P, V, βc) (1)
S(I, P, V, βi) = i∈U SA(I, pi, vi, βi)

+ (i,j)∈E SD(pi, pj, βi)) (2)
where the unary term S A provides appearance score using image features φ(I, p i ) (we use HOG features [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF])

S A (I, p i , v i , β i ) = v i (F i .φ(I, p i ) + b i ) + (1 -v i )(F o i .φ(I, p i ) + b o i )
and the binary term S D defines a quadratic deformation cost Object parts are frequently occluded due to the presence of other objects and self-occlusions. Since occlusions often do not happen at random, the locations of occluded parts may have consistent appearance. We model occlusions by learning separate appearance parameters F o for occluded parts. The bias terms b i and b o i control the balance between occluded and non-occluded appearance terms in S A .

S D (p i , p j , β i ) = d i .ψ(p i -p j ) with ψ(p i -p j ) = {dx; dy; dx 2 ; dy 2 } where dx = p x1 i -(p x1 j + µ x ij ) and dy = p y1 i -(p y1 j + µ y ij ).

Learning

Given a set of labelled training samples D = (< x 1 , y 1 >, ..., < x N , y N >), where x = {I, P x , V x } is the part-annotated sample (V x = 0 for negative samples) and y ∈ {-1, 1} is class labels, we aim to learn linear parameters of the model β in a discriminative fashion. Towards this goal, similar to [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF][START_REF] Ott | Shared parts for deformable part-based models[END_REF], we minimize the objective function

L D (β) = 1 2 ||β|| 2 + C N i=1 max(0, 1 -y i f β (x i )). (3)
We make use of part-level supervision and constrain model parts to be approximately co-located with the manual part annotation (where available) on positive training images. This is achieved by maximizing the scoring function (1) over a subset of part locations and visibility states Z p , Z v consistent with the ground truth part annotations P x , V x :

f β (x) = max P ∈Zp(x),V ∈Zv(x) S(I x , P, V ) (4) zp(x) = {p ∈ P|O(p, px) > tovp} if px available P otherwise zv(x) = 1 vx = 1 {0, 1} vx = 0
where P is the set of all possible locations for a part bounding box. The overlap between two bounding boxes O(p i , p j ) is defined as the intersection over union of their areas, t ovp = 0.3 defines a fixed overlap threshold. Note that when annotation is not available for a part, v x = 0, we enforce no constraint. This can help us cope with missing part annotations due to annotators' errors. Note that, as opposed to [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF], we have used a loss function which only penalizes a low score for the whole object, while [START_REF] Branson | Strong supervision from weak annotation: Interactive training of deformable part models[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] penalizes any instances with different part locations to the annotation. The latter can be helpful for the task of pose estimation while our choice of loss functions allows for down-weighting a part which is not descriptive. This property appears important given the potentially noisy annotations and possibly non-discriminative appearance of some annotated parts.

Before optimizing for the objective function (3), we cluster our positive samples based on their pose, assign a structure to each mixture component and initialize part filters (see details in Sections 3.2 and 3.3). The objective in ( 3) is known to be non-convex due to the latent variables at positive samples [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF]. We solve the optimization problem adopting a coordinate-descent approach in which the latent variables for positive samples are fixed at the first step so that the formulation becomes convex in β. Then, in the second step, we apply stochastic gradient descent (SGD) to estimate model parameters. The steps of the algorithm can be seen in Algorithm 1. In practice we have used a modified version the Latent SVM implementation in [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and trained models by harvesting hard negative samples from negative training images.

Pose clustering

Mixture models (components) enable modelling of intra-class variation in the appearance of different samples. These variations are due to strong view-point changes, class subcategories and non-rigid deformations. Assignment of positive samples to components in LSVM formulation is a non-convex optimization and thus sensitive to initialization. [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] proposes simple heuristic and initializes assignments by grouping positive samples based on the aspect ratio of their bounding boxes. This strategy, however, is suboptimal as illustrated in Figure 1 where example images of dogs differ substantially in terms of appearance but do have bounding boxes with similar aspect ratio. -Model Structure: for each cluster c construct a relations graph Ec using an uncertainty measure u (section 3.3). endfor -Filter Initialization: for each part p in each cluster (c, Ec) initialize HOG-SVM part Fp and occlusion F o p filters. endfor -Coordinate Descent: do Positive Latent Fixation: Fix all positive latent variables using equation ( 4) Harvest hard negatives using the approach in [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] SGD: Minimize the objective (3) using stochastic gradient descent. while (hinge loss value in (3) does not change.)

In this work we intend to use part annotations to define better assignment of training samples to model components based on the pose. This allows us to align similar parts better within each component which is an important factor when using linear classifiers. Using annotated parts we parametrize the pose θ x of sample x by the following vector, θx = (p ′ 1 , ..., p ′ n , s1, ..., sn, a0, ..., an, v1, ..., vn).

Here

p ′ i = {p x1 i -p x1 0 ; p y1 i -p y1 0 }
is the relative position of ith part w.r.t. the object bounding box (indexed 0), s i is the relative scale of ith part width to object width, a is the aspect ratio of parts (including object) bounding boxes, and finally v i is the binary visibility annotation of part i. We take all positive pose vectors and cluster them using a modified k-means clustering algorithm. Since samples may have occluded parts (without any annotation for bounding box) in each maximization step of the algorithm we replace the missing blocks of vector θ by corresponding blocks of their associated cluster center. Under minor assumptions it will preserve the log-likelihood improvement guarantee of EM algorithm and thus its convergence to local minima [START_REF] Koller | Probabilistic Graphical Models: Principles and Techniques[END_REF]. To control the effect of each parameter during clustering, we define a weight for each block of θ by a prefixed parameter (W ). In addition, since in the mixture model, we are interested in using horizontally mirrored samples for training the same filters (using mirrored descriptors), we need them to fall into the same cluster. For this purpose we modify the distance between each sample x to a cluster c with center µ c to be d

(x, c) = min(||θ x -µ c ||, ||θ ′ x -µ c ||) where θ ′ x = (p ′′ 1 , ..., p ′′ n , s 1 , ..., s n , a 0 , ..., a n , v 1 , ..., v n ).
Note that the only dimensions modified in θ ′ is the relative position p ′′ i = {p x2 0p x1 i ; p y1 0p y1 i } of the horizontally mirrored sample. This is closely related to the work of Johnson et al. [START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF]. Contrary to [START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF], (a) we handle the missing data in a single modified k-means while they decouple the clustering of the fully annotated samples from the partially annotated ones; (b) we assign the same pose in mirrored images to the single cluster giving us more Average images generated from our clustering results are illustrated in the first column of Figure 2. The relatively sharp average images indicate consistency among training samples within a component, which can give better discriminative power from more aligned part positions. Note that using part visibility as part of the pose vector helps us to assign systematically similar occluded samples (e.g. cat upper body) to a separate cluster.

Model structure

The structure of the part based models (dependency graphs) is usually set manually [START_REF] Felzenszwalb | Pictorial structures for object recognition[END_REF][START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Ott | Shared parts for deformable part-based models[END_REF][START_REF] Chen | Active mask hierarchies for object detection[END_REF][START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF][START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF], either to a star model [START_REF] Felzenszwalb | Pictorial structures for object recognition[END_REF][START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Ott | Shared parts for deformable part-based models[END_REF], hierarchy of coarse to fine relations [START_REF] Chen | Active mask hierarchies for object detection[END_REF], or, in the case of human pose estimation, to a graph capturing the skeletal configuration of body parts [START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF][START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF]. However, in that case one should define a tree for each class-component which might not be intuitive or optimal. Here we propose to design an optimization procedure for constructing a dependency graph using object part annotations. As described in [START_REF] Felzenszwalb | Pictorial structures for object recognition[END_REF] the statistically optimal part relations in a generative pictorial structure model can be achieved by optimizing the connections over the following prior probability of samples (assuming each sample is generated independently),

E * = arg max E (u i ,u j )∈E c(ui, uj) = arg min E (u i ,u j )∈E -log c(ui, uj) (6)
c(ui, uj) = Np k=1 prob(p k u i , p k u j |d) (7)
Where -log c(u i , u j ) measures the uncertainty of a link over samples given the deformation model d, (i.e. how well the part locations are aligned along a graph edge). However, since we optimize our deformation model discriminatively we can not have d beforehand. Since the connectivities in our model are used to capture the spatial configuration, we approximate the above uncertainty value along each possible edge by variance of relative position of the two end parts.

Then we construct a fully connected graph H = (U H , E H ) with n + 1 nodes corresponding to object and parts bounding boxes, each edge e is defined as a 3-tuple (i, j, w). For each pair of nodes (i, j) we calculate the diagonal covariance matrix C of the translation (dx, dy) between the two corresponding bounding box centers. Inspired by the Harris corner detector [START_REF] Harris | A combined corner and edge detector[END_REF] we let the weight of each edge to be w e = trace 2 (C e )k det(C e ). This helps us to avoid edges with very small variation in one dimension and a large variation in the other as opposed to det(C e ). Finally, we want to find a tree T = (U ′ , E ′ ) which minimizes e∈E ′ w e . This is analogous to the problem of Minimum Spanning Tree(MST) in graph theory which is solvable O(2n 2 log (n)) using Kruskal algorithm. The graph structure is constructed using the above approach for each mixture component coming from pose clustering independently. Refer to Figure 2 for visualization of MST results.

Experimental evaluation

This section describes our experimental setup and presents a comparative performance evaluation of the proposed method. We report results for the detection of six animal classes in PASCAL VOC 2007 and 2010 datasets [START_REF] Everingham | The PASCAL Visual Object Classes Challenge[END_REF]. The considered classes (bird, cat, cow, dog, horse, sheep) are highly non-rigid and remain challenging for the current detection methods.

For each training image of an object we have annotated up to nine object parts roughly covering the surface of an object 1 . Each visible object part has been annotated with a bounding box as illustrated in Figure 3. Our method is not restricted to the types of parts used in this paper and can be easily extended Comparison with other methods. We compare results of our approach to the LSVM detector [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and Poselets detector [START_REF] Bourdev | Detecting people using mutually consistent poselet activations[END_REF]. On the VOC 2007 dataset we follow [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and apply post processing in terms of bounding box prediction to make our results comparable with results reported on the author's web-page [START_REF] Girshick | LSVM Release 4 Notes[END_REF] (see Table 1). For the VOC 2010 dataset (see Table 2) we retrain the detector [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and obtain Poselets [START_REF] Bourdev | Detecting people using mutually consistent poselet activations[END_REF] detection results from the VOC challenge web-page. Overall our method improves LSVM baseline by 6.5% and 5.1% on VOC 2007 and VOC 2010 datasets respectively measured in terms of mean Average Precision (mAP) for six animal classes. We also obtain increase of performance by 2.7% mAP compared to the Poselets method [START_REF] Bourdev | Detecting people using mutually consistent poselet activations[END_REF] which, similar to ours, uses additional part-level object annotation. The APM method [START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] uses part-level supervision and reports object detection results for cat and dog classes. Their evaluation setup on PASCAL VOC 2007 data, however, is not standard making comparison with [START_REF] Sun | Articulated part-based model for joint object detection and pose estimation[END_REF] difficult. We note, however, that APM does not improve LSVM results for the cat while our improvement over LSVM is significant (the cat AP increases by 7%, see Table 1).

Detailed evaluation of proposed extensions. To better understand contributions of the proposed extensions, we report results for the four variants of our method and the LSVM baseline [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF]. Results in Table 3 indicate a consistent improvement provided by the contributions of our approach. Figure 4 Semi-supervised learning. To reduce efforts for object part annotation, we investigate a semi-supervised learning setup where the part-level annotation is provided for a fraction of positive training samples only. As our method is able to handle samples with missing part annotation, we train detectors for s% positive training samples with part annotation and (100s)% samples without part annotation. We compare detection results to the method trained on fully annotated samples. Results at the bottom of Table 3 and the corresponding plots in Figure 5 show the benefit of this semi-supervised learning setup provided by the use of samples with no part annotation. Moreover, even a fraction of part-annotated samples can result in significant improvements by our method.

Number of mixture components. We investigate the sensitivity of LSVM, LSVM+Clus and SParts methods (see definition in Table 5 captions) to the number of mixture components. Evaluation of mAP for these three methods Fig. 5: Results of our model trained using different fractions of fully annotated samples. Red curves correspond to the training on s% samples with full part annotation. Green curves correspond to a semi-supervised training setup using X% fully-annotated samples as well as (100-s)% samples without part annotation. For each of the experiments (points in plot) 3 different random subsets are drawn. Mean average precision is computed over each three subsets. and different numbers of components is illustrated in Figure 6. As can be seen, the proposed methods benefit from the increasing number of components while the performance of LSVM degrades. This demonstrates the room for modelling more intra-class variation when using more elaborated clustering criteria than the aspect ratio.

Latent positions. Although object parts are provided in our training by the annotation, we treat part locations as a constrained latent variables, i.e., we allow the location of parts to be optimized in the neighborhood of their annotated position. This is expected to reduce the influence of the imprecise part annotation and the possibly low discriminative power of manually annotated object parts. To investigate the effect of latent positions, we conduct an experiment where we fix the positions of object parts to their annotated values. The comparison of detection results by our methods with and without latent part positions is summarized in Table 4 indicating the importance of the constrained latent variables in our method.

Part Localization. Our method localizes object parts as a bi-product of object detection.

No+Nv

, where N o and N v are the number of samples with the part in the occluded or visible states respectively. It can be seen that (execpt for the part "head") our method is able to detect the occlusion state fairly well. We have observed low numbers of samples with heads being occluded. This should explain low OD values for the head for some objects.

Implementation details. We use HOG features and an optimization procedure adopted from the on-line implementation of [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] (vocrelease 4.0 [START_REF] Girshick | LSVM Release 4 Notes[END_REF]). Training our detectors takes about 20 hours on a recent 8-core Linux machine for one class on average. Detection takes about 4 seconds for a 300x500 image. We cross-validated a few choices for the parameter W (pose clustering weight vector) and t ovp (minimum part overlap constraint) for the class "cat" on VOC 2007 and fixed them for all classes with the same value. We set W to uniform [START_REF] Girshick | LSVM Release 4 Notes[END_REF]. We train a filter only for the parts that are visible in at least 30% of each cluster positive samples. Five mixture components are used for the final evaluations.

Conclusion

In this work we have explored the use of part-level supervision for the training of deformable part-based models. In particular, we have shown how to use part annotations (i) to derive better DPM initialization, (ii) to re-define the tree structure of the model and (iii) to model occlusions. Evaluation of these contributions has shown consistent improvement for the tasks of detecting animal classes in PASCAL VOC datasets. Our improvements in the semi-supervised setting indicate that even a fraction of part-level annotated samples can significantly improve the overall quality of the detector.

  Notably, the score function (1) linearly depends on the model parameters β c = {F 0 ; ...; F n ; F o 0 ; ...; F o n ; d 1 ; ...; d n ; B}. To represent multiple appearances of an object, our full model combines a mixture of C trees described by parameters β = {β 1 ; ...; β C }.

Algorithm 1

 1 Learning frameworkRequire: D = (< x1, y1 >, ..., < xN , yN >) the training samples and annotations, C number of clusters -Pose Clustering: Cluster the positive samples to C clusters based on their poses. (section 3.2)

Fig. 2 :

 2 Fig. 2: Illustration of model initialization. Column one: average images of cows from three model components. Column two: star model. Column three: MSTbased tree model. Uncertainty in part localization is illustrated by ellipses and is lower for the MST-based tree model compared to the star model. Each row corresponds to one mixture component.

Fig. 3 :

 3 Fig. 3: Part annotation for training image examples of six animal classes.

Fig. 4 :

 4 Fig. 4: Detection results for three animal classes. The detections of the part bounding boxes are overlaid with the tree structure of the model connecting the parts. Colors of part bounding boxes are consistent for different samples of the same class. Dashed bounding boxes indicates the activation of occlusion filters.

Fig. 6 :

 6 Fig. 6: Detection results for different number of mixture components obtained by LSVM, LSVM+Clus, and Sparts methods. Mean AP results are shown for 6 animal classes of PASCAL VOC 2007.

  Here we evaluate object part localization on the VOC 2007 test set. We use the following three measures: (a)Occlusion Detection (OD) corresponding to the percentage of correctly recognized occlusion state 3 of the part, (b)Percentage of Correctly localized Parts (PCP 4 )[29] for non-occluded parts as well as (c)Occlusion sensitive Percentage of Correctly detected Parts (OPCP) for all parts defined as OD * No+P CP * Nv

1

  For each of the six animal classes we have annotated the following parts. bird: beak, closed wings, head, legs, open wings, tail; dog: front legs, tail, head, frontal face, profile face, ears, back; cat: front legs, tail, head, frontal face, profile face, ears, back; cow: back legs, front legs, head, frontal face, profile face, horns, ears, torso; horse: back legs, ears, front legs, frontal face, head, profile face, tail, torso; sheep: back legs, ears, front legs, frontal face, head, horns, profile face, tail, torso.

Table 1 :

 1 illustrates qualitative results of animal detection on a few example images from VOC 2007 Per-class results (Average Precision) for animal detection in VOC 2007.

		bird cat	cow dog horse sheep mAP
	LSVM w/o context [1] 10.0 19.3 25.2 11.1 56.8	17.8	23.4
	Ours	12.7 26.3 34.6 19.1 62.9 23.6 29.9
		bird cat	cow dog horse sheep mAP
	LSVM [1]	9.2 22.8 21.2 10.4 40.8	27.0	21.9
	Poselets [22] 8.5 22.2 20.6 18.5 48.2 28.0 24.3
	Ours	11.3 27.2 25.8 23.7 46.1 28.0 27.0

Table 2 :

 2 Per-class results (Average Precision) for animal detection in VOC 2010.

				bird cat	cow dog horse sheep mAP
	Supervised	LSVM [1] LSVM+Clus SParts Sparts+MST Sparts+MST+Occ	10.1 18.4 24.7 11.3 57.8 9.1 26.1 29.8 12.8 56.0 9.9 26.5 30.7 19.3 60.4 11.8 23.5 32.6 20.7 61.0 24.9 29.1 18.6 23.5 24.5 26.4 23.5 28.3 12.5 24.1 34.4 19.5 62.2 24.5 29.5
	Semi	Sup	Sparts+MST+Occ 75% 12.8 21.6 32.7 16.9 60.2 Sparts+MST+Occ 50% 12.1 22.5 31.1 14.3 60.0 Sparts+MST+Occ 25% 10.9 20.8 27.1 12.2 57.9	25.6 25.7 21.4	28.3 27.6 25.1

Table 3 :

 3 Step-by-Step results for PASCAL VOC07 w/o bounding box prediction. LSVM: Release 4.0 implementation of[START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] avaiable at authors webpage[START_REF] Girshick | LSVM Release 4 Notes[END_REF] LSVM+Clus: LSVM initialized with our pose clustering (see Section 3.2); Sparts: Supervised parts (instead of automatic LSVM parts) trained on top of pose clustering with a star model connectivity; Sparts+MST: Sparts with general acyclic connectivities using Minimum Spanning Trees; Sparts+MST+Occ: Sparts+MST with added occlusion filters and bias. Last rows show the results with partially annotated training data. test set. Although we do not require part localization, correct part placement indicates the expected interpretation of the image by our method.

Table 4 :

 4 Detection performance on PASCAL VOC 2007 animals using either laten or fixed part positions.

Table 5 :

 5 /65.7/65.7 30.3/37.3/35.8 65.2/39.3/46.7 91.3/57.7/66.0 48.3/32.0/39.6 sheep 26.1/29.4/28.9 65.5/17.6/34.3 78.6/10.9/33.7 76.5/57.1/65.1 80.8/2.4/61.4 Part Detection performance evaluated on PASCAL VOC 2007, reported numbers are Occlusion Detection/PCP/OPCP respectively weights and t ovp to 30% overlap. We set LSVM penalty cost C to 0.002 as in original LSVM implementation

		head	front legs	fore legs	torso/back	tail
	bird 21.4/25.4/25.3	-	30.6/12.1/18.2	-	55.4/6.1/19.9
	cat	33.3/60.0/59.7 70.5/8.9/34.9	-	43.9/17.2/29.1 84.2/1.7/53.4
	cow 27.3/36.3/35.8 65.0/25.9/34.6 57.5/37.1/42.5 84.2/58.2/66.5	-
	dog	0.0/40.5/40.2 64.4/23.1/37.1	-	36.0/6.7/20.1 86.6/0.9/61.9
	horse 66.7

Annotations of parts are publicly available and can be found at http://www.csc. kth.se/cvap/DPM/part_sup.html

We report a part occluded when either of the followings happen: a) the corresponding occlusion filter is activated b) a component is activated which does not include that part c) the part is detected with more than 70% of its area outside image

We follow the approach of[START_REF] Yang | Articulated pose estimation with flexible mixtures-ofparts[END_REF] when computing PCP in that we take all the detections in an image and for each ground truth object annotation we consider the detection with the highest score that has more

than 50% overlap with annotation.

Acknowledgement.This work was supported by the VINST project funded by Swedish Foundation for Strategic Research (SSF), by the EIT ICT labs, by the Quaero program funded by OSEO and ERC grant VideoWorld. Computing resources were provided by the Swedish National Infrastructure for Computing (SNIC 001-12-16) via PDC.