Gaurav Sharma
email: gaurav.sharma@mpi-inf.mpg.de

Patrick Perez

Latent Max-margin Metric Learning for Comparing Video Face Tubes

Comparing "face tubes" is a key component of modern systems for face biometrics based video analysis and annotation. We present a novel algorithm to learn a distance metric between such spatio-temporal face tubes in videos. The main novelty in the algorithm is based on incorporation of latent variables in a max-margin metric learning framework. The latent formulation allows us to model, and learn metrics to compare faces under different challenging variations in pose, expressions and lighting We propose a novel dataset named TV Series Face Tubes (TSFT) for evaluating the task. The dataset is collected from 12 different episodes of 8 popular TV series and has 94 subjects with 569 manually annotated face tracks in total. We show quantitatively how incorporating latent variables in max-margin metric learning leads to improvement of current state-ofthe-art metric learning methods for the two cases when the testing is done with subjects that were seen during training and when the test subjects were not seen at all during training. We also give results on a challenging benchmark dataset: YouTube faces, and place our algorithm in context w.r.t. existing methods.

Introduction

Automatic analysis of faces in digital images and videos is a very important biometrics problem and has attracted much attention in the computer vision community [START_REF] Ahonen | Face description with local binary patterns: Application to face recognition[END_REF]8,[START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF][START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Goh | The CMU Face In Action (FIA) database[END_REF][START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Hussain | Face recognition using local quantized patterns[END_REF][START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF][START_REF] Lee | Online learning of probabilistic appearance manifolds for video-based recognition and tracking[END_REF][START_REF] Kim | Face tracking and recognition with visual constraints in real-world videos[END_REF][START_REF] Lee | Visual tracking and recognition using probabilistic appearance manifolds[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]32,33,[START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF][START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF][START_REF] Wolf | Effective unconstrained face recognition by combining multiple descriptors and learned background statistics[END_REF]. It has many important applications in recognizing, searching, retrieving and indexing images, including: (i) Surveillance and video archives -find a person in large amounts of videos; (ii) Security -allow access to a person, or not, to a resource; (iii) Consumer databases -find a certain person in private or online image databases like Flickr or Facebook.

Recently, face verification, i.e. determining if two faces are of the same person or not, has emerged as an important research problem, e.g. [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]32,33]. The task

Comparison modulo Illumination

-Take the average distance between all possible pairs of faces with one single learnt metric?

-Take the minimum distance between the pairs compared with respective adapted (automatically learnt) metrics?

Same metric for all (existing) vs. adapted metrics for mined variations (proposed) Figure 1. In the context of the important face biometrics based problem of face tube comparison, the traditional methods learn one metric/projection for all types of face appearance variations e.g. expression, pose, illumination, and complex real-world combinations thereof (top row). We propose to learn automatically, with a latent variable based formulation, different projections for comparing different combinations of mined variations. As a related question, we investigate how taking an average of distances, using a single metric (traditional methods), between all possible pairs of faces from the two face tubes, compares with taking the minimum distance between faces compared using proposed, variations adapted, metrics.

is set in a supervised learning framework where an annotated set, containing pairs of face images (i) of the same person (taken at different times and conditions) and (ii) of different persons, is provided and the system has to predict if a new test pair (of unseen faces/person(s)) is of the same person or not. Current approaches address this under a metric learning framework [3, [START_REF] Davis | Information-theoretic metric learning[END_REF][START_REF] Frome | Learning globally-consistent local distance functions for shape-based image retrieval and classification[END_REF]39,[START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF] (see [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF] for a good survey), where a parametrized distance metric function is learnt for comparing face images [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]33] (with the benchmark Labeled Faces in the Wild (LFW) [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF] being a catalyst). Along with such verification task, the learnt distance function can also be used for other applications like identity based linking and clustering of faces.

Similar task of verification but with spatio-temporal tracks of faces in videos, a.k.a. face tubes, has also seen some interest, e.g. [8, [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF][START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF][START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF][START_REF] Wolf | The svm-minus similarity score for video face recognition[END_REF]. The task with realistic videos is much more challenging as, while in still image datasets (e.g. LFW [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF]) the faces are near frontal with good illumination and similar/constrained expressions, the faces in videos have unconstrained pose and expression variations and are taken in diverse and difficult illumination conditions (see Fig. 2). Cinbis et al. [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF] recently applied the distance metric learned from static face pairs to face tube matching showing that a cast-specific metric performs better than a metric learned from external data. While they addressed the case of unconstrained consumer videos, (i) they used frontal face detectors to construct the evaluation dataset and hence were limited to near frontal poses and (ii) they used static face metric for learning and hence ignored the information that comes from the natural association of faces in face tubes (see §3 for detailed discussion). A recent evaluation [8] of off-the-shelf face matchers shows that the faces with challenging poses, expressions and illuminations are the failure cases of such systems (Fig. 5 in [8]) .

In the present paper, we are interested in the difficult and important task of distance metric learning for face tubes under challenging realistic variations in pose, expression, illumination conditions, etc. Note that this is different from face recognition; we do not aim at learning person specific characteristics but at learning what makes faces similar or different. We propose a metric learning framework that incorporates and leverages factorization over such variations (and, more realistically, their complex combinations) in the metric as latent variables. This can also be viewed as learning separate metrics for specific combinations of variations. This is in contrast to the existing works, which learn same metric for any and all combinations of variations. Fig. 1 illustrates the point. We show the proposed latent metric can be learnt using an efficient stochastic gradient descent based algorithm. We provide experimental results on the challenging benchmark: YouTube Faces [START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF], to put our method in the context of existing works on the task of face verification. In addition, we propose a new challenging dataset -TV Series Face Tubes (TSFT) -to evaluate the difficult task of face tube matching in videos with high facial variability such as TV series. We use this dataset to evaluate our method for comparing new face tubes of unseen persons along with the standard cast specific setting, i.e. comparing face tubes of people already seen at training. Our results on TSFT dataset show that while face verification in static near-frontal faces can be done quite successfully (e.g. on LFW 1 ), the task of face tubes matching in videos with high facial variability is quite challenging, with room for much progress. We show with quantitative results that the method gives better performance, on TSFT dataset, than strong baselines and the state-of-the-art methods. We find expression variations to be particularly misleading for the system; this problem is amplified in the case of hard datasets with high variability, like TSFT, as the probability of find-1 http://vis-www.cs.umass.edu/lfw/results.html ing another person with a similar expression is very high. As collecting a dataset for sufficiently capturing identity dependent facial expression variations for verification is a very challenging task, we thus identify expression normalization as an intriguing problem for face verification.

We now set the context for the work in the next section. We then give the details of the approach ( §3) and the proposed dataset ( §4), followed by experimental evaluation ( §5) and discussions ( §6).

Related Works and Background

Face verification with still images is a very popular topic of research in computer vision, with the very popular Labeled Faces in the Wild (LFW) benchmark [START_REF] Huang | Labeled faces in the wild: A database for studying face recognition in unconstrained environments[END_REF] specially catalysing such research. Many recent papers address the problem with novel approaches, e.g. Berg and Belhumeur [START_REF] Berg | POOF: Part-based onevs.-one features for fine-grained categorization, face verification, and attribute estimation[END_REF] propose a discriminative part based approach, Li et al. [START_REF] Li | Probabilistic elastic matching for pose variant face verification[END_REF] propose a probabilistic elastic model, Simonyan et al. [33] use Fisher vectors [START_REF] Sanchez | Image classification with the fisher vector: Theory and practice[END_REF] with metric learning, Cao et al. [START_REF] Cao | Similarity metric learning for face recognition[END_REF] propose a novel regularization for similarity metric learning, Cui et al. [START_REF] Cui | Fusing robust face region descriptors via multiple metric learning for face recognition in the wild[END_REF] propose to fuse many descriptors using multiple metric learning, Sun et al. [START_REF] Sun | Hybrid deep learning for face verification[END_REF] use deep learning, Barkan et al. [4] propose a method which uses fast high dimensional vector multiplication and Weng et al. [START_REF] Weng | Robust feature set matching for partial face recognition[END_REF] use robust feature set matching for partial face recognition. Many of the most competitive approaches on LFW combine many features, e.g. [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF]28,[START_REF] Wolf | Similarity scores based on background samples[END_REF] and/or use external data, e.g. [START_REF] Berg | Tom-vs-pete classifiers and identity-preserving alignment for face verification[END_REF][START_REF] Kumar | Attribute and simile classifiers for face verification[END_REF] Metric learning has recently gained much interest, as in [3, [START_REF] Davis | Information-theoretic metric learning[END_REF][START_REF] Frome | Learning globally-consistent local distance functions for shape-based image retrieval and classification[END_REF]39,[START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF] (we refer the reader to Bellet et al. [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF] for an excellent survey) and it has been applied for comparing faces [START_REF] Cao | Similarity metric learning for face recognition[END_REF][START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]33] with simple extensions to comparing face tubes [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF] as well. Given two face descriptors v i and v j the task is formulated as learning a Mahalanobis like metric of the form D2

M (v i , v j ) = (v i -v j ) M (v i -v j )
, parametrized by the symmetric positive semi-definite (PSD) matrix M . Various objectives have been proposed to learn M (see [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF] for a survey). Since maintaining M as PSD is usually computationally expensive, M is often factorized as M = L L. Then the problem can be seen as that of finding a linear subspace, spanned by the rows of L, into which the features are embedded and compared as

D2 M (v i , v j ) = (v i -v j ) L L(v i -v j ) = Lv i -Lv j 2 2 .
(1) Such formulation of metric learning has been shown to be effective in comparing faces under lighting and expression variations [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]. Although the method is for comparing pairs of face images, it has been used in [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF] for comparing face tubes in videos by taking the minimum or average distances between all quadratic number of face pairs for the two tubes. While this was shown to give good results, it might not be optimal, as we discuss next. In a more recent work [START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF], it was shown that using strong features the face tube can be represented as a single descriptor vector (vs. one vector per face) achieving competitive results, when combined with existing metric learning methods.

Our work is also related to local metric learning methods notably [START_REF] Bohné | Large margin local metric learning[END_REF] (we encourage the reader to see the references in [START_REF] Bohné | Large margin local metric learning[END_REF] for more context on local metric learning). They propose to learn metrics by weighting a set of 'basis' metrics based on the location of the test pair vectors. They cluster the data projected into a low dimensional space, and are hence largely generative (although the low dimensional projection is learnt discriminatively). In the proposed method, the metrics and the implicit 'clustering' are automatically learnt in the same optimization.

Approach

Motivation. The traditional approach of using face pair metric learning with mininum or average face distances over the face tubes might not be optimal as: (i) A single projection matrix L is used for faces irrespective of their, i.e. pose, expression, lighting, etc., which means that faces with different variations are embedded into the same space for comparison; (ii) The learning is not done on face tubes with similar minimum or average distances, but rather on sampled positive and negative pairs of faces, which discards the information present by virtue of different faces being grouped into different tubes; (iii) While some degree of invariance against pose is achieved by aligning the faces [START_REF] Wolf | Effective unconstrained face recognition by combining multiple descriptors and learned background statistics[END_REF] or computing descriptors around face landmark detections [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF], invariance to other types of variations, especially expression, is not immediate. Even with pose, while current strategies work for near frontal faces (many standard face databases including LFW are constructed by using frontal face detectors), they might, however (a) add errors in facial landmark detections for harder non-frontal faces, which might then propagate to the later parts of the pipeline; (b) still not be optimal to compare aligned frontal images with aligned profile images directly and with the same projection matrix.

Such arguments motivate us to model variations which may be due to (combinations of) pose, illumination, expression, etc. as latent variables in metric learning framework which we detail in the following.

Proposed Method

Given spatio-temporal face tubes, which may be obtained by using face detection and/or tracking technologies, we are interested in learning a distance function for comparing them. Denoting s = [s 1:Ns ] ∈ R d×Ns a face tube (with N s faces, each represented with d dimensional feature), we propose to compare two face tubes with following distance: Compute ( , p, q) in (2) over this pair of tubes 7:

D 2 (s, t) = min ( ,p,q) L (s p -t q ) 2 2 , (2)
L ← L -r∇ L L st 8: b ← b -r∇ b L st 9: end for 10: Output: b, {L } k =1 where, ( , p, q) ∈ 1, k × 1, N s × 1, N t are latent vari- ables. Matrix L ∈ R d ×d , d
d, defines the linear embedding associated to a specific combination of facial variations and p, q specify the frames in the corresponding face tubes with features s p and t q ∈ R d . Each L can be thought of as a projection in a subspace where a certain combination of face-variations may be compared e.g. smiling-surprised, frontal-profile.

Given set T = {(s, t, y st )} of annotated tube pairs, with y st = 1 for face tubes of same person and y st = -1 of different persons, we learn the projection matrices L 's by minimizing the hinge loss

L = T max[0, 1 -y st (b -D 2 (s, t))] Lst , (3)
w.r.t. b and {L } k =1 . We perform the optimization with a stochastic gradient descent algorithm (Alg. 1) using the subgradients w.r.t. L ,

∇L Lst = 0, if yst(b -D 2 (s, t)) > 1 2ystL (sp -tq)(sp -tq) , otherwise, (4)
where the latent variables (p, q, ) are obtained for the current pair (s, t) using Eq. 2, and that w.r.t. b,

∇ b Lst = 0, if yst(b -D 2 (s, t)) > 1 -yst otherwise. ( 5
)
We note here, that we do not specify the kind of variations and related clustering (for different 's) we want to have-there are no explicit variation level annotations such as 'these people are smiling' or 'are in profile pose'. The factorizations over the variations is thus learnt automatically within a discriminative learning framework.

The lack of explicit regularization in the optimization objective (loss in Eq. 3) is compensated by (i) fixing the dimension of the projected space (d ), thereby limiting the rank of the learned metric [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF][START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF] and (ii) a combination of low learning rate and fixed number of iterations (inspired by the experiments with SGD for classification [START_REF] Akata | Good practice in large-scale learning for image classification[END_REF]).

While doing the stochastic updates, to generate much larger number of training points and to get smoother estimates of the projection matrices, at each stochastic step we not only randomly sample an annotated tube pair but also sample (up to) a fixed number (m) of images from each of the tubes. This allows us to construct many more virtual training tube pairs (especially positive pairs which are usually relatively few), which is important to help the algorithm update often all of the L matrices and provide a smoother estimation.

TV Series Face Tubes (TSFT) Dataset

Context. We are specifically interested in face tube matching algorithms in videos with very high variations in expression, pose, lighting, etc. and their combinations. Some face video datasets for evaluating face tracking and recognition exist, e.g. [START_REF] Goh | The CMU Face In Action (FIA) database[END_REF][START_REF] Kim | Face tracking and recognition with visual constraints in real-world videos[END_REF][START_REF] Lee | Visual tracking and recognition using probabilistic appearance manifolds[END_REF][START_REF] Lee | Online learning of probabilistic appearance manifolds for video-based recognition and tracking[END_REF][START_REF] Sivic | Who are you?" -learning person specific classifiers from video[END_REF][START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF], however they are either recorded with coorperative subjects in limited background/lighting variations or are based on near-frontal faces only. Also, many of them are generated automatically (with face detector) with a post processing step for eliminating duplicates and false positives and hence are dependent on and are limited by the statistics of the face detector used. While Sivic et al. [START_REF] Sivic | Who are you?" -learning person specific classifiers from video[END_REF] worked with TV series video including profile faces as well, the task they addressed was of learning character specific classifiers. The data they made publicly available is from two episodes of a single TV series with a relatively small number of subjects.

In a recent evaluation of commercial off-the-shelf face matchers applied to videos [8], it was found that faces with extreme face pose and illumination conditions were not enrolled by available systems (Fig. 5 in [8]) -we are interested in such high variability scenarios. To the best of our knowledge, a suitable publicly available large dataset to evaluate face tube matching algorithms in the presence of challenging high variations in expressions, pose, illuminations and their complex combinations, did not exist at the time of submission of this paper.

Proposed Dataset. We propose a novel dataset of manually annotated face tubes in popular TV series videos-TV Series Face Tubes (TSFT) dataset 2 . The dataset captures the many challenging variations present specifically in the case of videos. Tab. 1 gives the statistics of the dataset, Fig. 2 shows faces of the subjects from the dataset. Face tracks of 94 subjects who vary in age, build, race and sex were manually annotated in 12 episodes of 8 different series. Every tenth frame of the tubes was manually marked with a bounding box covering the face of the person and the intermediate bounding boxes were linearly interpolated. The boxes were expanded to make them square and padded with black pixels when they went out of the image. In total there are more than 32,000 faces in the dataset. Fig. 3 shows the distribution of track lengths and face sizes for the dataset. The average track duration is a bit over 2.2 seconds (55 frames) and the average face size is 121×121 pixels. Each face tube has labels assigned manually for the associated character in the series and the actor playing the character. The series and the scenes within, where the tracks are marked, are of highly diverse nature, they occur indoors (e.g. home, office, bar) and outdoors (e.g. playing field, street, market) with very challenging facial expressions, head motions, hence pose changes, and lighting conditions. There are a total of 2005 positive pairs and order of 100k negative pairs. Hence the dataset is large and very challenging for studying the problem of comparing face tubes in videos.

Proposed evaluation. We propose two evaluation settings with provided training and testing splits of the dataset.

(i) Known Persons. This is the classic cast-specific metric [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF] evaluation where there is at least one training face tube for each person present in the test set. To generate the train and test sets for this scenario, we randomly selected at least 50% face tubes of each subject in the dataset as training examples while keeping the rest as testing examples. This gave us all 94 subjects in the train set, with 780 positive and 75k negative tube pairs, and 85 subjects in the test set, with 189 positive and 19k negative tube pairs. (ii) Unknown Persons. In this setting the training and testing are done on face tubes of different subjects, i.e. the subjects in the test set were never seen during training. To generate the train and test sets for this scenario, we randomly selected 80% of the subjects for training and kept the rest for testing. This gave us 75 subjects with 1590 positive and about 1 million negative tube pairs for training and 19 subjects with 414 positive and 6k negative tube pairs for testing.

To evaluate the performance of the methods we report the average precision (AP) for the different methods on the respective test sets. If a method involves a random component we suggest running the method 10 times and reporting the mean and standard deviation of the AP for the 10 runs.

Experimental Results

TV Series Face Tubes (TSFT) dataset

Image representation. Recent works on face verification [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF][START_REF] Hussain | Face recognition using local quantized patterns[END_REF][START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF] showed that local pattern features are powerful facial descriptors. We thus use local binary pattern (LBP) [START_REF] Ahonen | Face description with local binary patterns: Application to face recognition[END_REF] as our base features. We extract LBP in 3×3 circular pixel neighborhoods (with the diagonal pixels bilinearly interpolated) and use the uniform LBP patterns, i.e. patterns with at most two bitwise transitions from 0 to 1, or vice versa, when the bit pattern is seen as circular. We extract such LBP densely at every pixel at 3 scales with face image resized to 120×120, 80×80 and 60×60 pixels. We make spatial histograms of the LBP features with cell size of 10×10 pixels and concatenate all the ( 1 normalized) cell histograms for all the scales to give a final vector of (12 2 + 8 2 + 6 2 ) × 59 = 14, 396 dimensions for each face image.

Baseline method (PCA+CS). Recently Chen et al. [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF] and Hussain et al. [START_REF] Hussain | Face recognition using local quantized patterns[END_REF] showed that projecting the local pattern histograms on to their PCA basis and then using cosine similarity with the projected vectors is a very strong baseline, which competes with many recently proposed supervised methods [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF][START_REF] Hussain | Face recognition using local quantized patterns[END_REF]. As a baseline: (i) We do 1 normalization followed by elementwise square root normalization of the local pattern histograms. This corresponds to the non-linear Helinger kernel map [START_REF] Vedaldi | Efficient additive kernels via explicit feature maps[END_REF] and was recently shown to give very good performance for facial analysis tasks [32]. With this kernel map, the Euclidean distances between mapped vectors correspond to the Bhattacharyya distance between the probability distributions represented by the original distances. (ii) We project these high dimensional vectors onto their low rank PCA basis. Similar to Hussain et al. [START_REF] Hussain | Face recognition using local quantized patterns[END_REF], we found that with such PCA based compression of vec-tors the performance practically stays the same upto a certain dimension and then falls as we go below. We found, with preliminary experiments, that 1000 PCA dimensions (almost similar to [START_REF] Hussain | Face recognition using local quantized patterns[END_REF] who suggested 900) as a good operating point. (iii) Finally we follow recent works [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF][START_REF] Hussain | Face recognition using local quantized patterns[END_REF] and use cosine similarity (CS), i.e. dot product divided by norms of the vectors, as the comparison measure. In practice, we 2 normalize the PCA projected vectors and then compare them using Euclidean distance, achieving the same effect. We stress again that this is a very strong baseline [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF][START_REF] Hussain | Face recognition using local quantized patterns[END_REF]. We denote this baseline as PCA+CS in the following.

Compared method: Supervised Metric Learning (ML). Supervised metric learning is currently one of the state-ofthe-art methods for face verification tasks [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]33]. As discussed above in §2, the main idea is to learn a Mahalanobis like metric -which is itself decomposed into a product of a low rank matrix with its transpose, effectively making it a low dimensional projection learning problem. Different losses such as hinge [33] and logistic [START_REF] Guillaumin | Is that you? Metric learning approaches for face identification[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF] have been optimized giving good results.

We compare with such supervised metric learning method by optimizing the hinge loss. Essentially we optimize the proposed objective (Eq. 3) with = N s = N t = 1.

We use a stochastic gradient descent algorithm [33] with a fixed number of one million iterations generating face pairs by first randomly sampling an annotated face tube pair and then randomly sampling one face each from the two face tubes. Once we learn a metric, we compare face tubes using either minimum or average distance [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF], i.e. take the minimum (resp. average) distance over all possible face pairs from the two tubes. We denote this method as ML in the following.

For such supervised learning methods, and similarly for the proposed metric learning with latent variables, two strategies have been used in the literature. Either, use a first compression method, like PCA, and then learn again another low dimensional projection with the PCA reduced vectors [START_REF] Chen | Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification[END_REF]. Or, learn the low dimensional projection directly with the original high-dimensional vectors [33]. We found, with preliminary experiments, that reducing the dimension using PCA and then learning the metric leads to similar performance as the second method while being faster. Hence, we follow the second approach and map the features first by PCA and then learn the metric for both proposed and compared methods.

Initialization. Methods with latent variables are generally based on non-convex optimizations and thus proper initialization is important for them. The experiments that follow require initialization for the supervised metric learning as well as our proposed latent metric learning. We follow recent work [33] for the compared supervised met-ric learning (ML), and initialize the projection matrix with low rank PCA basis, corresponding to the largest eigenvalues, whitened by dividing each of the PCA vectors with the square root of the corresponding eigenvalue.

We tried different strategies for initializing the projection matrices for the proposed latent metric learning. First, we did an unsupervised clustering of faces using k-means and used the clusters to initialize the projection matrices as in the case with ML above. The intuition is that the clustering will bring the faces that are similar in feature space, but (possibly) not of the same person, closer and then the projection matrices will specialize in better separating the confusing cases in each cluster. However, we found that this initialization doesn't work well in practice. Instead, we found that randomly selecting a small number (1500) of training vectors and initializing the projection matrix with the low rank whitened PCA matrix (as above for the supervised ML) of these vectors gave good results. We thus follow this initialization strategy.

Performance and comparison with existing methods. Tab. 2 gives the performances of the proposed method along with compared methods. On the Known persons experiments (while training and test face tubes are from different scenes/episodes all the test subjects were seen on training) the baseline of PCA projection to 1000 dimensions and then comparison with cosine similarity (CS) improves the 1.0 chance performance to 11.8 while compressing the features by 14×. Adding supervision and learning the metric by optimizing the hinge loss (ML), for differnt values of the projection dimension, improves this to up to 17.7 reducing the size of the vector from 14396 to 64. We see that increasing the projection dimension for ML leads to overfitting and the performance drops above d = 128. Finally, upon using the proposed latent max-margin metric learning we improve the performance to 18.4 while avoiding overfitting. For similar amount of compression (cf. d = 128 for ML), the proposed method achieves 18.3 (with = 2 and d = 64). Thus the proposed addition of latent variables in the standard metric learning formulation improves the performance for the task of Known persons verification.

It is interesting to note that the results using the two types of tube distances, i.e. minimum and average (take the minimum/average distance between all possible face pairs from the two tubes) are different from recent previous works [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF]. While Cinbis et al. [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF] had reported that average distance works better than the minimum distance on their dataset, we find that the opposite holds on the proposed dataset. We conjecture that this might be due to some overfitting of their method on the limited negative pairs as (i) the dataset was generated from one series only so the subjects were fewer and (ii) they used automatic negative pair generation by using co-occurrence arguments, i.e. face tubes which appear together are of different person, which leads to negative pairs with reduced diversity in appearances. Hence in their case they reported that some test negatives also have small distances which suggests possible overfitting on the negative set. However, in the present case the negatives are relatively much more diverse than the postives as, while the positive pairs come from the same series (albeit from different diverse scenes and from different episodes also) the negatives can come from completely different filming conditions of different series. Hence, in the present case the minimum distance works better, i.e. the system chooses to predict based on the best matching pair of faces for two test tubes.

In the more challenging and realistic situation of 'Unknown person' testing (no test subject was seen during training) the proposed method again shows improvements. The PCA+CS method improves the chance performance of 6.3 to 32.8 while the ML method improves it further to 36.4 (d = 128). The proposed latent metric learning improves the performance to up to 38.8 (for d = 64 and = 6).

The performances for the two evaluation scenarios should be seen relative to the chance performances. While, for the proposed method, the chance performance improves by about 18× (from 1.0 to 18.4) in the Known person evalu-ation, the same improvement is only about 6× (6.3 to 38.8) in the Unknown person setting underlying the much more challenging nature of the later.

Experiments with different number of negatives. Not surprisingly, the absolute magnitude of the AP depends on the random chance performance or the relative number of positives and negatives in the test set. When we vary the number of negatives (by random subsampling, for the Known persons evaluation) we see that the AP decreases with increasing number of negative examples (Tab. 3). With a chance performance around the same as the Unknown persons evaluation, the AP rises to 42.1 (cf. 38.8 for Unknown person), while when we take the maximum number of negatives available (around 20k) the AP is more than 2× lower. The probability of finding another person with almost same kind of expression and other variations increases with the possible number of negatives (see qualitative results below). We stress here that in a real-world system the number of negatives will far exceed the number of positives (as the number of images of the same person will be much lower that the total number of images in the database) and hence the expected performances will be much lower. Hence, we conclude that the task of face verification, especially in the presence of high facial variations, is very challenging with a large room for improvement.

Qualitative results. Fig. 4 shows some of the top false positives for the proposed method. As seen in the figure, in video databases it is much more likely to have two different persons with very similar expressions or in very similar poses and illumination conditions which makes it much more challenging. Upon visual inspection, we conclude that usually a combination of many factors, including expression, pose and illumination, contribute to the confusion. However, the high probability of the presence of very similar expressions seems to be a recurrent reason. We also attempted to visualize the implicit clustering obtained but we didn't get easily semantically interpretable results. As the initialization is done randomly, the mined variations seemed to be non-trivial combinations of expression, pose, illumination, etc.

YouTube Faces (YTF) dataset

We now give results on a standard challenging benchmark of video face verification: YouTube Faces (YTF) dataset [START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF]. YTF dataset contains 3425 unconstrained videos of 1595 celebrities, downloaded automatically from YouTube. The benchmark provides detected and aligned faces and has a standard evaluation protocol. It is divided into 10 splits, each split containing, randomly selected, 250 positive and 250 negative pairs of face tracks. The performance is reported as the average over 10 disjoint runs, where in each run 1 fold is used for testing and the rest 9

Method

AUC EER Random chance 50.0 50.0 ML (avg dist) [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]33] 85.9 22.5 ML (min dist) [START_REF] Cinbis | Unsupervised metric learning for face identification in TV video[END_REF][START_REF] Mignon | PCCA: A new approach for distance learning from sparse pairwise constraints[END_REF]33] 83. [START_REF] Barkan | Fast high dimensional vector multiplication face recognition[END_REF] [START_REF] Wolf | The svm-minus similarity score for video face recognition[END_REF] 86.9 21.2 FV (base method) [START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF] -16.2 DeepFace-single [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] 96.3 8.6 folds are used for training. We report the Area under the Receiver Operating Characteristic curve (AUC) and the Receiver Operating Characteristic Equal Error Rate (EER).

We use the three types of Local Binary Pattern (LBP) features provided by the authors i.e. LBP [START_REF] Ojala | A comparative study of texture measures with classification based on feature distributions[END_REF], Center-Symmetric LBP (CSLBP) [START_REF] Heikkila | Description of interest regions with center-symmetric local binary patterns[END_REF] and Four-Patch LBP [START_REF] Wolf | Descriptor based methods in the wild[END_REF]. We follow similar setting as for the TSFT dataset and first project the concatenated features to 1000 dimensions with PCA and then learn the lower dimensional embeddings.

Tab. 4 gives the result of the baseline ML, proposed method and some existing state-of-the-art methods. The results obtained with the proposed method are better than the dataset creators' [START_REF] Wolf | Face recognition in unconstrained videos with matched background similarity[END_REF] results (86.0 vs 82.6 AUC) and are competitive w.r.t. one of their recent works (86.9 AUC) [START_REF] Wolf | The svm-minus similarity score for video face recognition[END_REF] based on a more complicated learning setup. This validates the implementation of our model w.r.t. existing art, with similar features. State-of-the-art on this benchmark is primarily approached by engineering strong features [START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF] or by using large amounts of external data [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] and is hence not directly comparable to our results.

For the baseline method, the average distance performs better than minimum distance for the baseline ML (85.9 vs 83.4 AUC). The proposed method performs better than the baseline with minimum distance (86.0 vs 83.4 AUC), however, it performs similar to the baseline ML with average distance (+0.1 AUC and -0.1 EER). The main challenges of this dataset are filming conditions, video quality and motion blur. Pose robustness seems to be largely corrected by using alignment. First, the faces are outputs of a face detector and hence are constrained enough for reasonable landmark detections and second, empirically we observed a big gain (26 vs 22.5 EER) when using the features computed on aligned versions of the face images vs. those on unaligned versions (both provided with the dataset). The main challenges thus remaining are related largely to the quality of videos, where averaging the distances seems to smooth out the noise (e.g. in [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] as well). This also seems to be supported, perhaps surprisingly, by recent results where features were pooled/averaged for all the frames of the video together into one descriptor [START_REF] Parkhi | A compact and discriminative face track descriptor[END_REF]. Opportunistically choosing the minimum best distance between two faces of the two face tubes seems to be dominated by noise leading to the relatively lower performance of the baseline ML with minimum distance. This loss is largely recovered by using proposed latent ML which, however, is not able to surpass the averaged distance.

Discussion and conclusion

The task of video face track verification is an important and challenging face based biometrics problem. In the experiments reported, the performances were found to be far from saturated on the proposed challenging dataset. We note here that the performance depends on the random chance performance; in reality the number of negatives far exceeds the number of positives and the evaluation should mimic such challenging scenario.

We believe that a principle challenge for the task is that due to expression. In a large pool of negative candidates, it is highly probable to find another person with similar expression. Since annotating a large number of expression diverse faces will be quite a challenge, expression normalization would be a critical problem. The solutions could be inspired by the recent work using '3D frontalization' for face verification [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] and also the methods used for facial re-enactment and performance transfer e.g. [START_REF] Garrido | Automatic face reenactment[END_REF].

To conclude, we presented a metric learning algorithm incorporating latent variables. We showed results on the popular benchmark YouTube faces where the model performs competitively w.r.t. current art using similar features. We also proposed a challenging dataset-TV Series Face Tubes (TSFT)-with manual annotations for 569 face tubes of 94 different subjects appearing in 8 popular TV series. We showed that the method improves upon the current stateof-the-art metric learning algorithms on TSFT. The dataset will be publicly available upon publication of the article.

Algorithm 1 . 4 : 5 :

 145 SGD based learning algorithm. 1: Input: Annotated training pairs T = {(s, t, y st )}, rate (r) and number of stochastic updates (niters) 2: Initialize: b, {L } k =1 3: for i = 1, . . . ,niters do Randomly sample a training pair (s, t) from T Randomly sub-sample each tube to length (up to) m 6:

Figure 2 .Figure 3 .

 23 Figure 2. Example faces of the 94 subjects from the proposed TV Series Face Tubes (TSFT) dataset. The dataset has diverse set of subjects (age, gender, race, sex) who appear in different lighting conditions (home, office, bar, field, inside car, at day, at night) and have varied pose, expressions and motions.

Figure 4 .

 4 Figure 4. Typical false positive face pairs, selected from test face tube pairs, with the proposed latent metric learning algorithm.

Table 1 .

 1 Statistics for the TV Series Face Tubes (TSFT) dataset.

	Sl.	Series	Sub Tubes	S/E* + pairs -pairs
	1.	BBT*	10	71	1/1, 1/6	315	2871
	2.	Buffy	11	69	5/2	195	2610
	3.	Castle	12	97	3/1	439	5192
	4.	Dexter	12	78 1/1,2/24	267	3348
	5.	Homeland	11	74	1/1	269	3044
	6.	Mad Men	12	70 1/2,2/13	200	2685
	7.	Numb3rs	11	54	1/1	128	1613
	8.	Weeds	15	76	1/1,1/6	192	3118
	-	Total	94	569	-	2005	24681

(*BBT = The Big Bang Theory, *S/E = Season/Episode).

Table 2 .

 2 Results of the various experiments on the TV Series Face Tubes (TSFT) dataset. See §5 for a detailed discussion.

Table 3 .

 3 ML 53.2 40.8 28.9 25.0 18.9 17.6 LatML 55.2 42.1 31.6 25.9 22.2 18.3 Performance of methods, chance, baseline and proposed ( = 2), with different number of negative test pairs.

	#negs→	1k 2.5k	5k	10k	15k	20k
	Chance 15.9	7.0	3.6	1.9	1.2	1.0

Table 4 .

 4 Performance of the proposed method vs. baselines (top) and some existing method (bottom) on YTF dataset (see §5.2).

We will make the dataset publicly available upon publication.


est destinée au dépôt

1. FISCHER V. , HADDAD P., BERNARD F., (2013) :

« An open-source multi-FPGA modular system for fair benchmarking of true random number generators », 23rd international conference on field programmable logic and applications (FPL2013), pp. PS3_8, Porto, Portugal I. II. III.

1. 2.

3.

5.

4.

6.

•

  Evariste III: A new multi-FPGA system for fair benchmarking of hardware dependent cryptographic primitives Nathalie Bochard, Cédric Marchand, Oto Peťura, Lilian Bossuet, Viktor Fischer To cite this version: Nathalie Bochard, Cédric Marchand, Oto Peťura, Lilian Bossuet, Viktor Fischer. Evariste III: A new multi-FPGA system for fair benchmarking of hardware dependent cryptographic primitives. Workshop on Cryptographic Hardware and Embedded Systems, CHES 2015, Sep 2015, st-malo, France. 2015. ujm-01219840 Evariste III: A new multi-FPGA system for fair benchmarking of hardware dependent cryptographic primitives PUFs TRNGs SCAs First historical application, fair TRNG comparison thanks to: • Unified hardware platform for different FPGA and ASIC technologies • Linear power supply • High quality low pass filters • Evaluation of up to 6 modules in parallel with 6 motherboards placed in a box • JTAG chain for reconfiguration in situ • Zero insertion force connectors to facilitate exchange of modules • Access to power supply lines in different places of the design • SMA connectors for easier access and performance • Embedded ARM processor in new modules to study software primitives What's new? • 3 new FPGA modules, one with an embedded ARM processor • Motherboard with ZIF (zero insertion force) connectors • Serial connection of up to 6 modules via JTAG • Box with 6 motherboards interconnected and chained • 30 modules of each new FPGA family for PUF evaluation available • SMA connectors in all modules for side channel analysis added Inherited from Evariste II 1 Both scripting and fast acquisition data programs • Open source system • Cyclone III (3 versions) 4. Altera Aria II 5. Microsemi Fusion (2 versions) 6. ASIC controlled with Fusion FPGA • New motherboard -ZIF connectors -JTAG chain I/O Nathalie Bochard, Cédric Marchand, Oto Petura, Lilian Bossuet, Viktor Fischer Laboratoire Hubert Curien, UMR 5516 CNRS -Université Jean Monnet, Saint-Etienne, France http://labh-curien.univ-st-etienne.fr/wiki-evariste/ European Union's Horizon 2020 research and innovation program: HECTOR -Hardware Enabled Crypto and Randomness -grant agreement No 644052. ANR 2013 call: SALWARE -Salutary hardware design to fight against integrated circuit counterfeiting and theft -ANR-13-JS03-0003. NATO SfP (Science for Peace) 2013: SFPP 984520 "Secure implementation of post-quantum cryptography". Sources of funding: