Daniel Gill
email: gill@mta.ac.il

Yaniv Ninio
email: yaniv.ninio@amdocs.com

Learning Facial Expressions: From Alignment to Recognition

One of the main challenges in 'real-life' object recognition applications is keeping some invariance properties such as: translation, scaling, and rotation. However, trying to maintain such invariants can impair recognition capabilities, especially when the family of objects of interest has a large shape variability. We present a general family of shape metrics that generalizes Procrustes metric and within this framework learns the desired shape metric parameters from labeled training samples. The learnt distance retains invariance properties on one hand and emphasizes the discriminative shape features on the other hand. We show how these metrics can be incorporated in multi-class classification kernel SVMs. We demonstrate the merits of this approach on multi-class facial expressions recognition using the AR dataset. The results address some questions and cautions regarding the interpretation of classification results when using still images datasets collected in a controlled lab environment and their relevance for 'real-life' applications.

Introduction

In recent years both academia and industry have shown growing interest in the development of computer vision systems that can locate human faces and recognize their expressions. One of the challenges of computer vision is to make computers that interact with humans in a natural way, as humans interact with each other. Since spoken language, as one of the most natural ways of humans to interact, is ambiguous, human interaction is also based on body gestures and facial expressions, which transmit implicit information and help to interpret the explicit information.

The context and the information provided implicitly are extremely important for computers to get a full understanding of what is actually transmitted in a conversation. There are many practical uses of recognizing facial expressions, such as:

• Detecting boredom, fatigue or stress while driving a vehicle and providing appropriate alerts will help to prevent many accidents from happening. Facial expression recognition systems, among others, may contribute to the development of various accident-prevention safety on-road systems. • Automated detectors of fatigue, depression and anxiety could form another step towards personal wellness technologies. Automating such assessment becomes increasingly important in an aging population to prevent medical practitioners from becoming overburdened.

• Monitoring and interpreting facial signals can also provide important information to lawyers, police, security and intelligence agents regarding deception and true attitudes. Automated facial reaction monitoring could form a valuable toll in law enforcement as now only informal interpretations are typically used. Systems have the capability of alerting the appropriate authorities to cases of recognition of unfriendly or aggressive faces. • Recently, facial expression recognition becomes relevant in interactive video games as well. In order to enhance the interaction and attractiveness of the games -recognition of facial expressions, such as happiness, anger, fear, surprise and others provides an interpretation of the player's mood and uses it to adjust the game correspondingly.

Many methods have been applied to expression recognition such as sequence based methods and frame based methods. Sequence based methods use temporal information extracted from a sequence of frames and frame based methods do not use temporal information and rely on a single frame. The facial features are usually divided into two categories: geometric feature-based methods and appearance-based methods. Geometric facial features are usually expressed as locations of fiducial points (around the mouth, eyes, brows, and nose). Appearance-based methods usually apply Gabor wavelets to either specific regions in the face or to the whole face. For more details see [START_REF]Handbook of Face Recognition[END_REF]. As stated above, machine analysis of facial expressions attracted the interest of many researchers. However, although humans detect and analyze faces and facial expressions in a scene with little or no effort, development of an automated system that accomplishes this task is rather difficult. In addition, one of the main problems of automatic recognition of facial expression in real-life is the inhomogeneity of scale, orientation or translation of the visual stimuli. In this paper we describe a general framework that deals with shape classification in general and apply it to facial expressions recognition in particular.

The paper is organized as follows: in the following section we describe a generalized form of the Procrustes distance for landmark-based shapes. In section 3 we describe a metric learning approach for shapes. Section 4 describes Procrustean inner product kernels to be used by kernel machines. Experimental results are described in sections 5. Finally, in section 6 we draw conclusions.

Shape Metric

A natural choice of landmarks is of salient points which can be identified by computer and humans. In the general case, there is a considerable loss of information by extracting only certain points, and the transformed shape cannot be restored in details from the landmarks. Yet, many essential characteristics may remain in such representation. A set of k ordered points in 2D plane can be represented as a 2k-dimensional vector. Comparing two shapes can be based on corresponding points which are termed homologies. The term of distance (or similarity) between two shapes can be easily defined when using 2k-dimensional vectors by taking only their coordinates as attributes. It is obvious that the order of the points matters. Another convenient representation is called planar-by-complex and uses complex values to represent each 2-dimensional point, so the whole shape is represented as an k ×1 complex vector. The configuration matrix is a k × m matrix of real Cartesian coordinates of k landmarks in an m-dimensional Euclidian space. In a planar-by-complex representation the configuration is a k dimensional column vector of complex entries. From now on we will assume that all the shapes we deal with are two-dimensional and are given in the planar-by-complex representation. This framework can be extended to 3D objects in a straight forward manner, though extracting landmarks from 3D objects is more complicated.

Procrustes Shape Metric

A desired distance measure between two planar point based shapes should be insensitive to translation, scaling and rotation. Consider a configuration x = (x 1 , x 2 , . . . ,

x k ) ∈ C k , a centered configuration x satisfies x * 1 k = 0, which is accomplished by: x → x -1 k 1 T k x1 k
, where x * denotes the complex conjugate of x.

The full Procrustes distance between two configurations x and y is given by:

d F (x, y) = inf β,ϑ,a,b y y - x x βe iϑ -a -bi = 1 - y * xx * y x * xy * y 1 2
.

(

) 1
The full Procrustes mean shape µ of a set of configurations {w n i=1 } is the one that minimizes the sum of square full Procrustes distances to each configuration in the set, i.e.

µ = arg inf µ n i=1 d 2 F (w i , µ). (2)
It can be shown that the full Procrustes mean shape, µ, is the eigenvector corresponding to the largest eigenvalue of the following matrix:

S = n i=1 w i w * i w * i w i . (3)
(see [START_REF] Kent | The complex bingham distribution and shape analysis[END_REF]). The eigenvector is unique (up to rotations -all rotations of µ are also solutions, but these all correspond to the same shape) provided there is a single largest eigenvalue of S. In many morphometric studies several configurations are handled and pairwise fitted to a single common consensus in an iterative procedure [START_REF] Rholf | Extensions of the procrustes method for the optimal superimposition of landmarks[END_REF]. This process is called generalized Procrustes analysis. Scatter analysis, using generalized Procrustes analysis handles the superimposed configurations in an Euclidean manner and provides good linear approximation of the shape space manifold in cases where the configurations variability is small. The Procrustes superimposition tends to obtain less extreme magnitudes of point shifts.

The fact that in least-squares superimposition points are treated uniformly irrespective of their variance results in poor estimation, and reaches its extreme when all of the shape variation occurs at a single landmark, which is known as the Pinocchio effect [START_REF] Chapman | Conventional procrustes approaches[END_REF]. Moreover, such variations affect the configuration's center of mass and thus affect translation as well.

General Quadratic Procrustes Metric

A general quadratic distance metric, can be represented by a symmetric positive semidefinite k×k matrix Q (we use the Q = A * A decomposition and estimate A). Centering a configuration x according to the metric induced by Q means that x * Q1 k = 0, and this is done by:

x → x -1 k 1 T k Qx1 k .
For the rest of this section, we assume that all configurations are centered according to the metric induced by Q.

The general quadratic full Procrustes distance, according to matrix Q = A * A, between two configurations x and y is given by:

d 2 Q (x, y) = inf β,ϑ,a,b A y y Q -A x x Q βe iϑ -a -bi (4) = 1 - y * Qxx * Qy x * Qxy * Qy 1 2
, where x 2 Q = x * Qx is the square of the generalized norm. The general quadratic Procrustes mean shape µ Q , with a matrix Q = A * A, of a set of configurations {w i } n i=1 is the one that minimizes the sum of square generalized distances to each configuration in the set, i.e.

µ Q = arg inf µ n i=1 d 2 Q (w i , µ). (5)
Proposition 1 The general quadratic Procrustes mean shape is the eigenvector corresponding to the largest eigenvalue of the following matrix:

S Q = n i=1 Aw i w * i A * w * i A * Aw i , (6)
(the proof is similar to the Euclidean case).

Metric Learning

Many pattern recognition algorithms use a distance or similarity measures over the input space. The right metric should fit the task at hand, and understanding the input features and their importance for the task may lead to an appropriate metric. In many cases there is no such prior understanding, but estimating the metric from the data might result in a better performance than that achieved by off the shelf metrics such as the Euclidean [START_REF] Xing | Distance metric learning, with application to clustering with side-information[END_REF][START_REF] Goldberger | Neighbourhood components analysis[END_REF][START_REF] Globerson | Metric learning by collapsing classes[END_REF]. Fisher Linear Discriminant (FLD) is a classical method for linear projection of the data in a way that maximizes the ratio of the between-class scatter and the withinclass scatter of the transformed data (see [START_REF] Duda | Pattern Classification. 2nd Ed[END_REF]). Given a labeled data set consisting of 2D input configurations x 1 , x 2 , . . . , x n where x i ∈ C k and corresponding class labels c 1 , c 2 , . . . , c n , we define between-class scatter and within-class scatter both induced by the metric Q. In a similar way to FLD the desired metric Q is the one that maximizes the ratio of the generalized between-class and within-class scatters.

We denote the general quadratic Procrustes mean shape of the members of class j by µ Q j , and the full general quadratic Procrustes mean shape of all configurations by

µ Q . Denote ∆ Q k,l = x l x l Q β Q k,l e i > ϑ Q kl -µ Q k ( 7
)
and

∆ Q k = µ Q k β Q k e i > ϑ Q k -µ Q (8)
where β Q k,l , β Q k are the scaling solutions of eq. 4 for the l-th configuration towards the mean of class k, and scaling of the k-th mean configuration towards the global mean respectively. The angles ϑ Q kl , ϑ Q k are those which satisfy eq. 8 for rotation the l-th configuration towards the mean of class k, and rotation of the k-th mean configuration towards the global mean correspondently (the translations equal to zero if all configurations are previously centered).

The within class scatter according to a matrix Q is:

s Q W = m j=1 n i=1 r ij d 2 Q w i , µ Q = m j=1 n i=1 r ij ∆ Q j,i * Q∆ Q j,i (9)
where

r kl = 1 x l ∈ Class k 0 Otherwise ( 10
)
and m is the number of classes.

The between class scatter according to a matrix Q is:

s Q B = m k=1 n k ∆ Q k * Q∆ Q k , (11)
where n k is the number of samples belong to class k. The desired metric Q opt is the one that maximizes the ratio of the between-class scatter and within-class scatter:

Q opt = arg max Q s Q B s Q W . (12)
Contrary to the standard FLD, the suggested objective function f may have many local maxima. Thus, maximizing the objective function should be carried out carefully, and only a local maximum is guaranteed.

An example of the differences between the classic Procrustean metric and a learnt one is shown in Figure 1. This delineation uncovers discriminative landmarks in facial configurations means. The general quadratic Procrustes mean shape of females' faces is fitted using the learnt metric (general quadratic Procrustes fit) onto the general quadratic Procrustes mean shape of males' faces. It is evident that the learnt metric reveals differences between the two classes. The males' mandibles tend to be larger, and their forehead hairlines tend to be higher than those of females. These differences are not revealed when using the standard Procrustes metric.

Procrustes Distance Based Classifiers

One of the goals of distance learning is the enhancement of the performance of classifiers. In recent years, many studies have dealt with the design and analysis of kernel machines [START_REF] Cristianini | An Introduction to Support Vector Machines and Other Kernel-based Learning Methods[END_REF]. Kernel machines use inner-products functions where the decision function is not a linear function of the data. Replacing the predefined kernels with ones that are designed for the task at hand and are derived from the data itself, is likely to improve the performance of the classifier considerably, especially when training examples are scarce [START_REF] Aha | Feature weighting for lazy learning algorithms[END_REF]. In this section we introduce new kernels based on the general quadratic full procrustes distance where the learnt metric can be plugged in to produce new kernels with improved capabilities of shape classification.

General Quadratic Procrustes Kernels

Certain condition has to be fulfilled for a function to be a dot product in some high dimensional space (see Mercer's theorem [START_REF] Cristianini | An Introduction to Support Vector Machines and Other Kernel-based Learning Methods[END_REF]). Following the polynomial and radial basis function (RBF) kernels, we propose the following kernels.

Proposition 2 The following function is an inner product kernel for any positive integer p:

k(x, y) = y * Qxx * Qy x * Qxy * Qy p ( 13
)
For proof outline see appendix A.

Proposition 3

The following function is an inner product kernel for any positive semidefinite matrix Q and any positive γ:

k(x, y) = exp -γ 1 - y * Qxx * Qy x * Qxy * Qy ( 14
)
For proof see appendix B.

It should be emphasized that using Procrustean (classic or learnt) metric is not equivalent to aligning the configurations to some consensus configuration first and then using Euclidean distance, since such operation does not preserve all pairwise Procrustes distances among all the configurations at once.

Kernel design for Multi-Class Classifiers

The general framework of the general quadratic Procrustes metric was designed for many purposes. For classification tasks it has been demonstrated so far only by classifying binary classes using the pre-designed learnt kernels with SVM. The SVM classifier is basically designed for binary classification with a clear geometrical meaning of discriminating one class from another by a hyperplane that maximizes the margin. A crucial point is the choice of combining multi-class learnt metric into a kernel designed for dichotomies. Several approaches were suggested for the multiclass case where the implementation is done by combining several binary SVMs. Many studies have shown the superiority of the one vs. one based versions of multi-class SVMs over one vs. all versions. The one vs. one approach constructs one binary classifier for every pair of distinct classes. For each pair a distinct metric can be learnt and the classification is done according to the majority (or weighted majority) of votes of all M (M -1)/2 classifiers ( [START_REF] Duan | Which is the best multiclass SVM method? an empirical study[END_REF], [START_REF] Hsu | A comparison of methods for multiclass support vector machines[END_REF]) .

Experiments and Results

The performance of the Procrustes kernels was evaluated using the AR Face Database [START_REF] Martinez | The AR face database[END_REF]. This database contains over 4,000 images corresponding to 126 individuals (70 men and 56 women). Images include frontal view faces with different facial expressions, illumination conditions, and occlusions (sun glasses and scarf). For our current study, we selected 904 images of four facial expressions: neutral, anger, smile and scream (226 images for each expression). Figure 2 shows examples of the four expressions. A relatively small number (36) of crucial landmarks were chosen as the size of datasets is small (the landmarks were chosen according the ease of accurate marking). The chosen landmarks are: 6 points for each eyebrow, 4 points for each eye, 16 points for the mouth (the landmarks are shown in fig. 3). The extraction of the landmarks from the facial images was done by the Bayesian Tangent Shape Model (BTSM [START_REF] Zhou | Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference[END_REF]) followed by a manual correction. Six metrics (matrices) for six binary classifiers, denoted by A kl , were learnt for k = l where k, l ∈ {smile, scream, anger, neutral} and A kl = A lk . The matrices learning was limited to diagonal matrices.

Experiment A

The performance of three classifiers was evaluated in this experiment:

• SVM with standard RBF kernel with square Euclidean distance in the exponent. The configurations (complex vectors) were centered and normalized but remained in their original inclination (it should be noticed that the original dataset was prepared in a way where all faces have similar size and similar inclination). • SVM with full Procrustes distance based RBF kernel (Q = I).

• SVM with learnt Procrustes distance based RBF kernel (learnt Q).

The error rates were evaluated by 4-fold cross validation. The results are given in confusion matrices (Tables 123). It can be observed that the Procrustes kernel (Q = I) does not have a significant advantage over the Euclidean based RBF kernel. Three possible explanations for this are: (i) The photographs were taken in advance in artificial conditions where the location and size of the heads are similar. This fact does not provide significant advantage to the Procrustes invariance properties. (ii) It is possible that in the anger expression case people tend to tilt their heads a little bit more frequently than in the neutral case. This cue is lost in the Procrustean approach, but remains in the Euclidean RBF kernel. However, in a more realistic scenario the head's inclination is not a reliable feature. (iii) The anger and the neutral expressions are very similar (even human subjects have identification difficulties when looking at the full detailed images). The classic Procrustes alignment tends to align the configurations in a way that minimizes the landmark distances and make the configurations to look more similar. The Procrustes learnt kernel is preferable over the two other kernels. Though it loses information conveyed by rotation (which is usually not helpful in real-life applications) it still has strong abilities to discriminate landmark based configurations.

Experiment B

The purpose of this experiment is to evaluate the contribution of the rotation feature (ϑ or ϑ Q as in eq. 1 and 4 correspondently). Two kernels were evaluated:

• SVM with a kernel constructed of a product of full Procrustes distance based RBF kernel (Q = I) and a kernel that is based on the rotation angle between configurations:

k 1 (x, y) = exp -γ 1 d 2 F (x, y) • exp -γ 2 (ϑ xy ) 2 .
• SVM with a kernel constructed of a product of a learnt Procrustes distance based RBF kernel (learnt Q) and a kernel that is based on the learnt metric rotation angle between configurations:

k 2 (x, y) = exp -γ 1 d 2 Q (x, y) • exp -γ 2 ϑ Q xy 2 .
γ 1 and γ 2 are kernel hyperparameters. The error rates were evaluated by 4-fold cross validation. The results are given in confusion matrices (Tables 4 and5). Indeed, the relative rotation angle between two configurations convey some cues regarding the class membership. The modified kernels accomplish significant additional improvement.

Conclusions

In this paper we have introduced a general framework of shape distances. Aligning configurations according to the learnt metric enables a visualization that uncovers the discriminative landmarks. Improvement in classification performance was demonstrated by using multi-class kernel SVM with a data driven kernel design. The main contribution of the learnt metric is the meaningful alignment -it is of particular importance in cases where the training sets are small. Classifiers with Euclidean distance based kernels must hold much more samples of each facial expression as they are not taking into consideration some variations in head position in space. Procrustes distance (learnt or classical) kernel based classifiers are insensitive to translation, scaling and rotation, which is a key factor in recognizing facial expressions both in still images and video sequences in 'real-life' environment. The comparison between Procrustes metric (Q = I) and Euclidean metric discovers a discriminative feature -the head's anglewhich is helpful when using input images taken in lab environment, but might be useless in 'real-life' conditions where there is a large variability in the face position within the frame. This property should be taken into consideration when dealing with 'real-life' systems designed by training sets produced in lab conditions.

Fig. 1 .

 1 Fig.1. Superimpositions of mean facial configurations: females (solid line) and males (dashed line) according to the full Procrustes metric (left) and the learnt Procrustes metric (right). The mean shapes are of 78 females and 32 males.

Fig. 2 .Fig. 3 .

 23 Fig. 2. The four facial expressions of the AR dataset: (a) smile, (b) scream, (c) anger, and (d) neutral

Table 1 .

 1 SVM classification results using Radial Basis Function Kernel with Euclidean distance. The mean error rate: 13.9 ± 2.4%.

	Actual\Predicted Smile Scream Anger Neutral
	Smile	99.6% 0%	0% 0.4%
	Scream	0.4% 99.6% 0%	0%
	Anger	0%	0% 68.3% 31.7%
	Neutral	1.8% 0% 21.4% 76.8%

Table 2 .

 2 SVM classification results using Procrustes Kernel (Q = I). The mean error rate: 12.8 ± 1.3%.

	Actual\Predicted Smile Scream Anger Neutral
	Smile	99.6% 0%	0% 0.4%
	Scream	0.4% 99.6% 0%	0%
	Anger	0%	0% 75.2% 24.8%
	Neutral	1.3% 0% 24.3% 74.4%

Table 3 .

 3 SVM classification results using learnt Procrustes Kernel. The mean error rate: 10.4 ± 1.3%.

	Actual\Predicted Smile Scream Anger Neutral
	Smile	99.6% 0%	0% 0.4%
	Scream	0% 100% 0%	0%
	Anger	0%	0% 77.4% 22.6%
	Neutral	1.7% 0% 16.9% 81.4%

Table 4 .

 4 SVM classification results using Procrustes Kernel (Q = I) with rotation feature. The mean error rate: 11.1 ± 1.3%.

	Actual\Predicted Smile Scream Anger Neutral
	Smile	99.6% 0%	0% 0.4%
	Scream	0.4% 99.6% 0%	0%
	Anger	0%	0% 77.9% 22.1%
	Neutral	1.3% 0% 19.9% 78.7%

Table 5 .

 5 SVM classification results using learnt Procrustes Kernel with rotation feature. The mean error rate: 9.0 ± 1.2%.

	Actual\Predicted Smile Scream Anger Neutral
	Smile	99.6% 0%	0% 0.4%
	Scream	0% 100% 0%	0%
	Anger	0%	0% 80.3% 19.7%
	Neutral	1.3% 0% 14.6% 84.1%

Appendix A.

Proof outline: First we show that the following function is an inner product kernel for any positive integer p:

We have to show that this kernel satisfies Mercer's theorem. This is done by proving that:

for any function g with finite l 2 norm. Each term of the multinomial expansion has a non-negative value:

(r 1 , r 2 , . . . , l 1 , l 2 , . . . , )!

and hence the integral is non-negative. Showing that:

Satisfies Mercer's theorem is done in a similar way by using eigen-decomposition the non-negativity of Q's eigenvalues.

Appendix B.

Proof:

The first factor on the right side is positive and the second factor can be arbitrarily close approximated by polynomial of the exponent with positive coefficients, thus using proposition 3 we have a sum of semi-definite functions, which is also a semi-definite function.


Augusto Destrero
email: augusto.destrero@imavis.com

Alberto Lovato
email: alberto.lovato@imavis.com

Francesca Odone
email: odone@disi.unige.it

A prototype system for unconstrained face verification based on statistical learning

This paper discusses the video processing modules of a prototype face verification system. The main modules (face detection, registration, and verification) are all based on a feature selection plus classification pipeline that implements recently proposed statistical learning algorithms. All these modules are running on the prototype since January 2008, performing face verification in real time. Here we report quantitative results obtained both off-line, on a previously recorded dataset, and on-line, on the verification system.

Introduction

After many years of theoretical and algorithmic investigations, machine learning findings are nowadays mature for exploitation and technology transfer. One application domain where such a transition is already taking place is image and video processing. In particular, the literature is rich with many interesting approaches related to face detection and face recognition that may lead to effective solutions for the biometrics market.

In this paper we propose a prototype verification system based on a simple hardware infrastructure and on software modules that combine state-of-the-art video processing and statistical learning algorithms. The adopted solution allows us to achieve real time face verification that may be applied to monitor entrances in unconstrained environments. Here we mainly focus on the prototype video-processing functionalities, in particular on face detection, registration and verification in videos. All these phases are addressed according to a common pipeline: during training we first look for a data-driven image representation by means of a regularized feature selection algorithm recently proposed [START_REF] Daubechies | An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[END_REF][START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF]. Then, we use this representation to train an appropriate classifier -here we simply adopt an SVM classifier [START_REF] Vapnik | Statistical Learning Theory[END_REF]. At run time we evaluate the video stream on a frameby-frame basis: for what concerns face detection we look for faces in each frame, as for face verification we analyse each frame of a video and then compute a cumulative feedback (genuine user or impostor) based on the evidence gathered from all frames.

The choices adopted are based on the following system requirements:

1. do not put constraints on users posture and behaviours 2. allow for fast computation (overall processing in real time) 3. set the basis for a simple enrollment phase 4. produce compact models for future engineering (e.g., storage of visual identifiers on small memory buffers)

The limited amount of constraints allowed calls for the use of models that are robust to appearance variations. At the same time, robust model-based approaches (e.g., 3D models) are not suitable for real time processing. Learning from examples guarantees a trade-off between the two requirements. We address both detection and verification as binary classification problems, relying on appropriate datasets acquired in the calibration and enrollment phase respectively. The adopted learning tools allow us to obtain satisfactory performances with relatively small datasets, with a clear advantage for the design of the enrollment phase. Requirement 3 is also addressed by exploiting video information, since videos are easier to capture than independent sets of images, and by designing an appropriate operative environment: users walk down a corridor in front of the camera and therefore datasets of reasonable size are easily collected. As for requirement 4 we rely on a data-driven feature selection mechanism to find the most appropriate description for each enrolled individual. Starting off from an over-complete description, based on features successfully used for face recognition (LBP features), we select meaningful features for each enrolled individual via data-driven feature selection. Let us briefly sketch pertinent state-of-the-art in order to set the adopted algorithms in the appropriate frame. In recent years many face-related problems have been addressed by learning from examples. In particular this paradigm dominated the face detection scene since the early 90s and contributed to obtaining promising solutions that deal with view-point changes, rotations, scale and illumination variations -see [START_REF] Yang | Detecting faces in images: a survey[END_REF]. Among such a vast literature, componentbased approaches highlighted the fact that local areas are often more descriptive and more appropriate for dealing with occlusions and deformations [START_REF] Mohan | Example-based object detection in images by components[END_REF][START_REF] Ullman | Visual features of intermediate complexity and their use in classification[END_REF]. Very popular approaches are derived from Adaboost (see, for instance, [START_REF] Viola | Robust real-time face detection[END_REF][START_REF] Zhang | Real-time face detection using boosting in hierarchical feature spaces[END_REF]). As for face recognition [START_REF] Zhao | Face recognition: A literature survey[END_REF] our approach relates to feature-based methods, specifically on template-based descriptions. Possibly the first local approach to face recognition is due to Pentland and his co-workers [START_REF] Pentland | View-based and modular eigenspaces for face recognition[END_REF]. Among local descriptors for face recognition, Local Binary Patterns (LBP) [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF] are becoming very popular for their ability to capture descriptive features of a given texture and in particular of faces (see, for instance, [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF][START_REF] Hadid | Learning personal specific facial dynamics for face recognition from videos[END_REF][START_REF] Tan | Enhanced local texture feature sets for face recognition under difficult lighting conditions[END_REF]).

The paper is organized as follows. Section 2 is an overview of the prototype system architecture, Section 3 reports the various video analysis phases, from preprocessing to verification. Sec. 4 reports an analysis of the performances of the system. The results reported are mainly releted to face verification, in particular on the effectiveness of the feature selection strategy for this specific problem, and on the way information coming from various frames is condensed in a unique feedback. Sec. 5 is left to a final discussion and to an account of future works.

The face verification prototype is composed by (1) a PC-based verification unit, (2) a RFID card reader, and (3) a monitoring and feedback unit -see Figure 1, left. The verification unit is based on the ImAnalysis library3 . Together with video acquisition and processing functionalities it contains the users database and the events database, and may be accessed remotely in particular to perform administration tasks. The RFID card reader is a MIFARE (ISO/IEC 14443) proximity reader with an activation range of [START_REF] Mohan | Example-based object detection in images by components[END_REF][START_REF] Ullman | Visual features of intermediate complexity and their use in classification[END_REF][START_REF] Viola | Robust real-time face detection[END_REF]cm; it is configured to read RFID tags that store the ID of the enrolled users. The monitoring and feedback unit is composed of a color analog video camera and a monitor used to attract the users attention and to send them feedbacks on the verification status (Fig. 1, right). The prototype is installed in a corridor with the card reader installed on one side and the monitoring unit on the other side, by the side of the monitored entrance. The user has to walk in front of the camera for a few seconds. In the case of a genuine user, as soon as the system has gathered enough positive evidence, it returns a positive feedback via the monitor.

We now analyse the system operative phases, first considering the case of a user known to the system (i.e., previously enrolled). Verification starts when the user brings the RFID tag in his possession near the RFID reader. Upon a positive feedback from the reader, obtained in a few seconds, the user may walk down the corridor at a normal speed. Considering the corridor extension (about 4 meters) it takes about 10 seconds to reach the other end of the corridor. In the meanwhile, the card reader associates to the current user an identity to be verified and sends the verification system, via a TCP/IP connection, the RFID-id. The verification system, that hosts the identities database, retrieves the gallery associated to the RFID-id, then starts building the current appearance model of the user walking down the corridor. The verification phase is based on comparing the gallery with current model, as described in Section 3.

To enroll a new user, the system associates to him the enrollment state, where user personal data have been inserted into the database, but no gallery is available. If the user is in the enrollment state, when he slides the RFID tag, the system acquires data to the purpose of building a dataset for training and validating the module.

Notice that the gallery of a user is based on all the data collected when he is in the enrollment mode, and could consist of information obtained from various videos. Instead, when the user is in verification mode a new test model (probe) is built each time he requires his identity to be verified. In this case the probe is derived from current video only. refer in particular to such functionalities. Figure 2 shows the video processing workflow. The pre-processing phase, based on motion detection and skin detection, was reported in [START_REF] Destrero | A system for face detection and tracking in unconstrained environments[END_REF]; we do not detail it any further.

Data-driven feature selection

In this section we briefly report the feature selection method upon which we base all our image representations. The method was originally proposed in [START_REF] Daubechies | An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[END_REF] and was adopted in [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF] for face detection. In [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF] a comparison with other dimensionality reduction methods is reported, in particular with Adaboost feature selection. Here we simply recall that the main advantages of regularized feature selection are, on one side, a well ground theory to rely on, and on the other size, an easy to implement algorithm that leads to satisfactory results with a relatively small number of examples. We consider a binary classification setting, therefore we start from a training set of positive and negative examples described with a dictionary of features appropriate for the problem. If this dictionary is descriptive of the content carried by the data we may assume a linear dependence between input and output:

Af = g (1)
where A is the matrix of processed image data, and each {A ij } is obtained from i = 1 . . . n images each of which is represented by j = 1, . . . , p features. In a binary classification setting we may set g i ∈ {-1, 1}; f is the unknown vector that weighs the importance of the features for each image of the training set. In this setting, feature selection means looking for a sparse solution f = (f 1 , . . . , f p ) : features corresponding to non-zero f i are relevant to model the diversity of the two classes.

In general one could solve Problem (1) by matrix inversion, but, as we will see later in this section, the systems we consider are largely under-determined and may be severely ill-conditioned because of the features redundancy. One possible way to address these problems is to resort to some regularization strategy. In particular, since we aim at selecting a subset of features, a sparsity-enforcing penalty is ideal for our case. We consider the L 1 norm that leads to a feasible problem of the form

f L = arg min f {|g -Af | 2 2 + 2τ |f | 1 (2)
( |f | 1 is the L 1 -norm), usually referred to as lasso problem [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. We solve it by an iterative method called thresholded Landweber that has been shown to converge to the minimizer of ( 2) under appropriate conditions on the matrix A

[1]. The iterative step is (f 0 L = 0 ): f t+1 L = S τ [f t L + A (g -Af t L )] t = 0, 1, . . . ; (3)
where S τ is a soft-thresholder:

(S τ h) n = h n -τ sign(h n ) if|h n | ≥ τ 0 otherwise. (4)
As we will see later, in our case, problem (3) involves the manipulation of matrices that may be very big. We therefore consider a re-sampling procedure based on solving S smaller sub-problems obtained each time extracting, with replacement, a subset of m features from the original set, m p. The S intermediate solutions are combined and used as input of a second regularized feature selection. Details and motivations of this 2-stages feature selection pipeline, as well as a discussion on parameter tuning, can be found in [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF].

The solution obtained at the end of the two stages is sparse and consistent, but usually it is not small enough for our processing and storage requirements. We then apply a third selection stage on the features that survived the previous two stages, that further reduces the amount of redundancy in the set of selected features. We first restrict the set starting from a random feature and adding only features that are distant or appear not correlated according to the Spearman's correlation test. The complete feature selection module will then consist of 3 stages. We call the set of features obtained from the 3-stages method S out .

Once features have been selected they are used to represent the data and then to address the classification problem of interest. In all the phases of our prototype we adopt a linear SVM as a classification algorithm, as it gives us the best compromise between performance and computational speed. In the next section we discuss classification issues specific of real-time object detection.

Coarse-to-fine classification

In general object detection must face the problem of a strong unbalance between positives and negatives in a given image. It is well known how, to perform real-time object detection, one has to consider a number of methods and heuristics to allow for a fast elimination of the millions of negative data. This issue has been debated in [START_REF] Viola | Robust real-time face detection[END_REF] where a coarse-to-fine procedure -a cascade of classifiers -is adopted. Here we take inspiration from that approach to build a cascade of SVM classifiers. Each classifier of the cascade is built extracting from S out small subsets of at least 3 distant features that are able to reach a fixed target performance on a validation set. We start by 3 mutually distant features and add further features until a target performance is obtained, testing a validation set with a linear SVM.

Target performance is chosen so to build weak SVM classifier that will not be likely to miss positive occurrences: we set the minimum hit rate to 99.5% and the maximun false positive rate to 50%. These modest targets allow us to achieve good performances with the global classifier, since global performance of the cascade will be computed according to the following [START_REF] Viola | Robust real-time face detection[END_REF]:

H = K i=1 h i and F = K i=1 f i
where h i is the hit rate and f i is the false positive rate for each weak classifier i. In our case, assuming a cascading of 10 weak classifiers, we will get H = 0.995 10 ∼ 0.9 and F = 0.5 10 ∼ 3 × 10 -5 .

Face detection and registration

Face detection refers to the problem of localizing human faces in digital images. We are mainly interested in detecting frontal faces, and to this purpose, we start from a dataset of 19×19 frontal face and non-face images acquired by our system. The dataset we consider to build the face detection module contains 4000 training examples, 2000 validation, and 3500 test examples, evenly distributed between positives and negatives [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF]. Alternatively one may use a benchmark dataset, such as the CBCL-MIT -see [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF] for results with this dataset.

For face detection we consider a dictionary of rectangle features [START_REF] Viola | Robust real-time face detection[END_REF], proved effective to describe the intrinsic structure of faces and their symmetries. We then proceed as follows:

-The linear system Af = g is built from the available training set. Each row of matrix A corresponds to a training image, each column to a specific rectangle feature of a given type, size, aspect ratio, position. As for vector g, we associate +1 to each positive (face) image, -1 to negative (non face) images.

-Considering 19 × 19 images we end up with about 64000 rectangle features;

thus the system we consider is hugely under-determined and, because of images auto-correlation properties very much ill-conditioned. This justifies the use of the regularized method described earlier in the section. Also, because of the size of A the re-sampling strategy discusses is recommended. Parameter tuning τ is performed on the validation set, with a cross-validation on the second selection stage [START_REF] Destrero | A sparsity-enforcing method for learning face features[END_REF]. The full 3-stages system leaves us with a compact set of 42 features to be used as face descriptions, shown in Fig. 3 (left). To evaluate the appropriateness of the selected features we test their generalization ability w.r.t. the test set. A classifier is trained, tuned, and tested over a dataset represented by the selected features. The ROC curves obtained from this analysis are shown in Figure 3 (right): here we report a comparison between the results obtained with the features selected by the first 2 stages and the compact set of features obtained adding the 3rd stage. A negligible performance loss is countered by an increase in description compactness.

In order to keep up with real time processing the face detection module implements the coarse-to-fine procedure described above.

We conclude this section on face detection by noticing that since our system does not put any constraints on the individual posture, some images will be inappropriate for recognition. Having at disposal a whole video for each session we discard poor quality images by means of a eye detector. Since eyes are the simplest feature to detect in a human face we discard faces where eyes are not clearly visible. The eye detection module replicates exactly the face detection module. Only the dataset has changed (extracted in this case from the FERET dataset), according to a data-driven philosophy. The eye-detection module is also used to automatically register the detected faces. Registration is very rough and simply consists of cropping an appropriate area of the face relative to the detected eyes positions (see Fig. 4, left). Faces below 40×40 pixels are discarded, and the remaining ones are scaled down to 40 × 40. (Fig. 4, right) shows samples extracted from various videos, and illustrates how the registration quality is quite reasonable.

Face verification

Let us assume that for each individual enrolled in the system we dispose of a dataset of faces gathered and registered automatically by the face detection module. The dataset is divided, as usual, in training, validation and test. The feature dictionary we consider is based on the so-called local binary patterns (LBP) [START_REF] Ahonen | Face description with local binary patterns: application to face recognition[END_REF]. We now describe how we represent each individual I:

-We start off from a set of 40×40 positive images I p , p = 1, . . . , P of individual I and a set of negative images I n n = 1, . . . , N randomly sampled from other individuals. -For each image I i , each pixel neighbourhood is represented as a LBP. The neighbourhood size we consider is obtain choosing 8 sampling points on a circle of radius 2 around the pixel. Then, for each image, we compute L LBP histograms on rectangular regions of at least 3 × 3 pixels, on all locations and aspect ratios; thus the description of image I i is a list of histograms H 1 i , . . . , H L i . -Notice that, here again, a feature selection procedure is important, since we obtained L = 23409 LBP histograms. We resort to the data-driven feature selection. Instead of exploiting directly positive and negative examples the linear system is built so to express the intra-personal and extra-personal variation of each histogram. With a similar approach to [START_REF] Hadid | Learning personal specific facial dynamics for face recognition from videos[END_REF] we compute each row of the matrix A as follows: for each pair of images I A and I B we compare corresponding LBP histograms (after normalization) using the χ 2 distance. That is, for each pair of input images we obtain a feature vector x i whose elements are χ 2 distances:

x i = (χ 2 (I 1 A , I 1 B ), . . . , χ 2 (I L A , I L B )
). We associate a label g i ∈ {+1, -1}, where g i = 1 if both I A and I B are positive images,

g i = -1 if either I A or I B is negative.
For a given set of examples, the number of feature vectors we compute is very high -for a set of positive images of size P we have P 2 positive feature vectors and many more negative feature vectors. In the current implementation, in order to obtain balanced systems of reasonable size we simply randomly sample at most 2000 positive and 2000 negative feature vectors and build matrix A. The vector g is composed of the corresponding labels g i and the vector f of unknowns will weight the importance of each LBP histogram for intra-person and extra-person comparison. Once we have built a system for a given individual described, we select features following the 3-stages protocol.

Notice that for each individual we obtain a different set of distinctive features, the ones that best represent his/her peculiarity. Figure 5 (left) shows the top 5 features extracted for some individuals (n. 6, 8, 13, 14) of our training set (dataset V AL01, see Sect. 4) . Although the detected features do not necessarily reflect elements that a human observer would find meaningful, they are localized in the most distinctive areas of the face for each individual. For person 6, for instance, the beard is spotted. The most distinctive area of person 8 seems to be the top part, the eyes region. Person 14 is mainly characterized by vertical features. The top 2 features of person 13 are localized where she wears a fringe.

In conclusion, we briefly comment on the appropriateness of the 3-stages feature selection strategy for the problem of face authentication. We report similar experiments to the ones shown for the face detection case for a randomly chosen individual I A (from dataset V AL01, Sect. 4). After feature selection we train and tune a classifier that should discriminate between images of I A and images of impostors. Note that, unlike the experiments reported in Sect. 4, here we consider images instead than videos. Figure 5 (right) shows a comparison of the results obtained for the 3 stages. Similarly to face detection we notice that the three results are comparable, with a slight performance loss when adding the third stage, which is although responsible of a consistent decrease in the number of features and therefore allows for a very compact representation of each person (the models computed from dataset V AL01 range from 10 to 30 LBP features per individual).

Video-based face verification

At the end of the enrollment procedure of a given individual I, once s features have been selected as a "good" description of I, we choose g training images of I, {G I 1 , . . . , G I g } as a gallery. This will lead us, in the verification stage, to evaluate g test data per each new probe P t : < G I i , P t > i = 1, . . . , g, t = 1, . . . , T curr . We classify each of these test data, with a linear SVM classifier associated to identity I, to evaluate how likely is P t to depict individual I. We then associate a final confidence value to P t , in the range [0, 1], related to how many images of the gallery are similar to P t .

When processing a video each frame is evaluated according to the above procedure. Different strategies may be adopted to associate to the video a unique positive or a negative feedback. So far we adopted two families of methods, compared in the experiments section: the first one, which we call multiple peak, returns a positive feedback (i.e., a positive verification) as soon as N frames not necessarily adjacent of the video are associated a confidence value greater than a threshold T 1. The second one, based on computing a running average of confidence values, returns a positive feedback if the average of the latest M confidence values is greater than a threshold T 2. Sect. 4 will report comparative experiments, that will highlight how the strategy choice does not seem to be crucial. The system is currently implementing a feedback based on a multiple peaks, with N=3.

Experiments on face verification

The datasets

We start describing in some details how we gathered a reference dataset V AL01 that we use for training, tuning, and testing the system.

The dataset V AL01 was built from several weeks of acquisition: at the end of two weeks we manually labeled the stored videos (a video is stored if faces are detected in it) and build models for all the individuals that have a rich dataset (≥ 20 frames). From the third week on data were gathered and labeled for testing: individuals that did not previously appear were stored to be used as negative examples (impostors) for all models. In total, [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] For what concerns real time experiments, the same 15 users received a RFID card each, they can use at their discretion. We do not have prior knowledge on the amount of impostors and not even on how collaborative the users are. In order to analyse the results obtained data (video, system output) are stored for 60 seconds after the RFID reader is activated. One video is captured for each person walking in the scene within that time frame. Data storage followed by manual labeling is important for a quantitative evaluation. A simple interface allows us to manually label the previously verified identities and thus to estimate the percentage of true positives (correctly verified users) and true negatives (correctly identified impostors). As a side effect we are also gathering a wider dataset to be available for future experiments. This dataset V AL02, currently under construction, may be naturally used as a test set extension to V AL01. The experiments reported here refer to about 20 days recording. We labeled the data in two different ways: (i) all walks evaluated by the system are labeled in genuine users or impostors depending on the person tracked, (ii) only clean walks, containing 1 person only, are labeled. In the first case we gather 192 genuine users and 37 impostors; in the second case 144 genuine users and 22 impostors.

The data stored in V AL02 are quite challenging for various reasons. First, there is a 6 months temporal gap between the acquisition of V AL01 that we use to model the individuals, and the run time experiments that generate V AL02. This translates in various scene changes (in particular in a strong illumination change from winter to summer sun exposure) and the obvious appearance variation of the individuals enrolled in the system. Fig. 7 shows some illumination and appearance changes produced by time. Second, we do not put any restriction in people's behaviour, and we do not forbid multiple passage (that in a real access control system would inhibit entrance). Fig. 6 depicts some common behaviours, including people walking in groups, people not looking at the camera, occlusions caused by people crossing the individual to verify.

Results

A first (offline) evaluation on V AL01 test set allows us to discuss the appropriateness of the face representation based on automatic LBP selection with respect to (1) a traditional global description based on PCA (2) the manually selected LBP features adopted in [START_REF] Hadid | Learning personal specific facial dynamics for face recognition from videos[END_REF]. Table 1 reports the results obtained with the three methods in terms of acceptance rate and rejection rate. In these experiments we have assumed an equal probability of having a genuine user or an impostor, therefore for each individual we extract randomly from other individuals datasets as many negative test data as the number of available positives. The As for run time experiments (forming current V AL02) Fig. 8 shows very promising verification performances, while comparing various voting strategies to associate a single feedback to a video sequence. There is no real gain in adopting one strategy or another. In the multiple peaks case N = 4 produces slightly better results if one wants to keep false acceptance rates low. In the case of running average a temporal window of size M=3 is a good choice. Fig. 9, instead, shows that there is not a great advantage in keeping clean walks only. In absolute values the multiple peaks strategy, N=4, leads to 4 false acceptance (FA) and 40 false rejection (FR) over the full dataset and 2 (FA) and 27 (FR) over the clean dataset. Similarly, adopting running average over temporal windows with M = 3, we obtain 4 FA and 43 FR on the full dataset, and 2 FA and 25 FR on the clean one. The ROC of the multiple peaks case with N = 3 shows current on-line performance of the prototype.

Discussion and future work

This paper describes a prototype face verification system based on videos. The system allows a user to declare his identity via a RFID card and verifies the identity in real-time by means of fast video-processing capabilities. The verification algorithms implement various recent findings from the statistical learning theory, in particular in obtaining a data-driven image representation. The very same feature selection + classification pipeline is implemented in various modules of the system, in particular to detect faces, to register them via eye-detection, and to verify identities. The video-processing workflow has been evaluated offline, on a previously gathered dataset on videos, and is currently being evaluated on-line on the prototype system performing real-time. We report very promising quantitative results obtained by a manual labeling of videos acquired on a 20 days time frame. The system is currently installed in a very much unconstrained environment, where illumination may change drastically due to changes of natural light, and scene clutter may be significant, since we do not require that the scene is empty during verification. We highlight the fact that no real system would work under such challenging circumstances. We are currently implementing an update procedure that will allow us to keep up with appearance variations in time. Also we are evaluating different ways to exploit data redundancy derived from the temporal component, in particular adopting Extended Volume LBP [START_REF] Hadid | Learning personal specific facial dynamics for face recognition from videos[END_REF].

Fig. 1 .

 1 Fig. 1. Left: a sketch of the system architecture and of the operative environment. Right: a picture of the monitoring and feedback unit.

Fig. 2 .

 2 Fig. 2. The video processing workflow implemented on the verification unit.

Fig. 3 .

 3 Fig. 3. Left: the 42 rectangle face features selected by the 3-stages feature selection method. Right: generalization performance of the features selected with a 2-stages and a 3-stages method.

Fig. 4 .

 4 Fig. 4. Face registration: on the left, the areas considered to form the new registered face. On the right, examples of automatically registered faces.

Fig. 5 .

 5 Fig. 5. Left: Top 5 features for some individuals. Right: Generalization performance of the features obtained from the three stages, for an individual arbitrarily chosen.

  individuals were included in the training phase, while a total of 64 individuals were gathered for testing. V AL01 contains a set of training and tuning video frames per each of the 15 individuals, in the range of [30 -200] depending on the individual. It then contains about 1700 test frames of the 64 test individuals.

Fig. 6 .

 6 Fig.6. Sample frames from videos in V AL02. Being the system entirely unconstrained the data include non collaborative people and clutter due to the presence of multiple people.

Fig. 7 .

 7 Fig. 7. The same person in different day times and different days.

Fig. 8 .

 8 Fig. 8. Strategies to associate a single feedback to a video. Left: multiple peak (acceptance if N confidence values are above T 1 -see text). Right: running average (acceptance if the average of M consecutive values is above T 2 -see text). ROC curves are build varying T 1, T 2 between 0.1 and 1.

Fig. 9 .

 9 Fig. 9. Comparison between two different data labeling (the complete test set V AL02 and a set including only videos with one person at a time). Left: multiple peak, N = 4. Right: running average, M = 3.

Table 1 .

 1 Performance of different representation methods on the dataset VAL01.

	Method	Acceptance Rate (AR) Rejection Rate (RR)
	PCA (global) [16]	0.68	0.70
	manual LBP select. [12]	0.87	0.94
	data-driven LBP select. [our method]	0.93	0.97

table shows how a local approach outperforms a global one. In particular PCA is not appropriate in our case, since images are only approximately registered. Manual selection of the most discriminative LBP features leads to good results, although inferior to our automatic feature selection, confirming that LBP are a good way to capture face peculiarities. Besides the very good results achieved, an important effect of our automatic feature selection procedure, is that an ad hoc description for each enrolled individual is achieved, allowing to capture different appearance features.

Video processingThe verification unit described in Sec. 2 consists of various software and system modules including the video processing functionalities. In this section we 3 http://www.imavis.com


Himaanshu Gupta
email: hgupta@objectvideo.com

Xiaochun Cao
email: xcao@objectvideo.com

Niels Haering
email: nhaering@objectvideo.com

Map-based Active Leader-Follower Surveillance System

We propose a generic framework for an active leader-follower surveillance system. The system can ingest inputs from a variety of multimodal leader sensors: Radars, Automatic Identification Systems (AIS), or cameras. Rule and learning based pattern recognition techniques are performed on the leader sensors to infer unusual behaviors and events. In order to calibrate the follower PTZ cameras to the leader sensors, we break up the calibration process into two independent steps: mapping leader cameras' image coordinates to latitude/longitude values, and mapping the latitude/longitude values to the follower's pan-tilt-zoom settings. This calibration method provides straightforward scalability to complex camera networks since newly added cameras need to be calibrated only with respect to the site map (latitude/longitude), and the Radar or AIS leader sensors also fit in seamlessly. Upon detection of an unusual event by one of the leader sensors, follower PTZ cameras move to the location where the event was triggered, detect the target of interest in the field of view, and then track the target actively by automatically adjusting the PTZ settings. We also propose a system to learn various PTZ camera characteristics which facilitate stable and accurate control of the camera during active tracking. Finally, we evaluate the performance of each of the components individually, as well as that of the entire system.

Introduction

A leader-follower system is an effective and commonly used solution for wide area surveillance. In such a system, a fixed wide FOV sensor acts as the leader, and directs follower PTZ cameras to zoom in on the targets of interest. One of the drawbacks of a typical active leader-follower system is that the follower camera can only follow the target as long as it remains in the leader's FOV. Also, due to inaccuracies in the leader-follower calibration, the follower camera might be unable to zoom-in to tightly on the target. To automate the follower cameras, we add active tracking capability. There are two challenges faced by the follower camera: 1) update the pan/tilt angles to follow it over long durations, and 2) automatically refine the zoom levels for the optimal tracking and visualization performance, i.e. as big as possible without losing targets. During this following process, the intrinsic PTZ camera properties such as actuation/query delays and pan/tilt/zoom velocities significantly affect the performance. However, deployed video monitoring systems are likely to interact with a wide range of COTS follower PTZ cameras in various states of working order, which renders manual calibration or the use of manufacturer specifications impractical. We thus propose a new method for automatically acquiring the PTZ parameters of a PTZ camera by sending sequences of commands to the camera and analyzing the incoming video. Several methods [START_REF] Davis | Calibrating pan-tilt cameras in widearea surveillance networks[END_REF][START_REF] Zhou | A masterslave system to acquire biometric imagery of humans at distance[END_REF][START_REF] Jain | Using stationary-dynamic camera assemblies for wide-area video surveillance and selective attention[END_REF][START_REF] Sankaranarayanan | An efficient active camera model for video surveillance[END_REF] are proposed to calibrate pan-tilt cameras. Davis and Chen [START_REF] Davis | Calibrating pan-tilt cameras in widearea surveillance networks[END_REF] utilize a moving LED as the virtual landmark. But they require multiple calibrated stationary cameras to infer the 3D locations via stereo triangulation. Zhou et al. [START_REF] Zhou | A masterslave system to acquire biometric imagery of humans at distance[END_REF] builds a look up table for each pixel coordinate in the leader camera and the pan-tilt angles in the follower camera. This method has the disadvantage that the calibration time will increase quadratically with respect to the number of leader and follower cameras. Jain et al. [START_REF] Jain | Using stationary-dynamic camera assemblies for wide-area video surveillance and selective attention[END_REF] focuses on the close views where the displacement between rotation axis and the optical center matters. Sankaranarayanan and Davis [START_REF] Sankaranarayanan | An efficient active camera model for video surveillance[END_REF] focus on top view cameras and introduce the elliptical locus for the inferring of pan-tilt orientations. Our united leader/follower calibration method has two independent steps: mapping leader camera image coordinates to the latitude and longitude, and mapping the latitude and longitude to follower pan-tilt-zoom values. Our approach is related to [START_REF] Zhou | A masterslave system to acquire biometric imagery of humans at distance[END_REF]. However, the calibration for each new leader and follower is independent and the system can be expanded to n leader and follower cameras in O(n) calibration time, comparing to O(n 2 ) using [START_REF] Zhou | A masterslave system to acquire biometric imagery of humans at distance[END_REF]. In addition, other leader sensors such as Radar fit in seamlessly since they provide latitude/longitude data directly. Another major component in an active leader follower camera system is online pattern recognition in the leader cameras. The leader camera needs to perform real-time detecting, tracking, classifying and monitoring of suspicious targets, and automatic analysis of their behaviors. We improved the traditional background subtraction method [START_REF] Toyama | Wallflower: Principles and practice of background maintenance[END_REF] to detect targets in complex environment. Temporal data association for the detected targets is performed using both motion and appearance properties of targets to solve occlusions and handle detection errors and sensor noise. The rule and learning based pattern recognition algorithms are applied to the target physical information after geo-registration. In summary, our leader and follower system is illustrated in Figure 1. These cameras are all geo-registered into a common world coordinate system as described in Section 3. In Section 2, rule and learning based pattern recognition techniques are applied to the physical properties of geo-registered targets and generate alerts. The alerts come with latitude and longitude that can be converted to pan-tilt-zoom values of a follower camera after the leader/follower calibration in Section 3. The follower PTZ cameras' characteristics, including pan/tilt/zoom profiles and actuation/query delays, are thoroughly analyzed in Section 4. After automatically zooming into the target of interest, an active tracking algorithm is used in Section 5 to follow the target. We present experimental results in Section 7.

Leader Sensor Processing

In order to trigger the follower to track the targets of interest, the leader camera needs to be able to detect those targets. For detection in a stationary camera, we improved the subtraction method [START_REF] Toyama | Wallflower: Principles and practice of background maintenance[END_REF] using a multi-modal spatiotemporal representation of the background to detect targets in dynamic scenes, e.g. water. Temporal data association for the detected targets is performed using both motion and appearance properties of targets to handle occlusions, detection errors and sensor noises. The tracked targets are then classified into predefined classes including humans, groups of humans, vehicles, watercraft (and its subclasses: freighters, cruise ships, recreational boats, etc.) using Gentle AdaBoost [START_REF] Friedman | Additive logistic regression: a statistical view of boosting[END_REF]. The classifier uses multiple cues that include C2 Features [START_REF] Serre | Robust object recognition with cortex-like mechanisms[END_REF], metric size, metric velocity, aspect ratio, and any other available contextual information. Pattern recognition is performed in both the image and world coordinate systems. While the methods for both coordinate systems are essentially similar, in this work we focus on the world coordinate system. Given detected targets with classification types from the stationary cameras, their physical information is computed based on geo-registration results as described in Section 3.1. This physical information is defined in a unified coordinate system, i.e. the world. Then a data fusion service links disconnected target trajectories inside a sensor and observations across sensors in a probability framework by modeling a target's geodesic location, speed and heading. Global consistency is enforced by temporally smoothing target locations and removing spurious targets. We use tripwire and area of interest (AOI) rules specified by the users to generate alerts. We also learn historical target properties for the detection of unusual events. Target properties include target width, height, size and speed. In our system, we learn the boat speed and density information based on the Automatic Identify System (AIS) readings. For each map pixel, we fit a Gaussian model for the normal speed. Therefore, given the estimated target physical speed on one location, we decide whether they are consistent with the historical observation. Radar and AIS data can be seamlessly integrated into this framework and treated as leader sensors. For example, the Radar readings have geo-location and speed information. Geo-location data can be used to trigger map rule based alerts. Together with geo-location data, speed information can be used to compare against historical target speed maps.

Leader-Follower Calibration

We propose a camera calibration scheme which involves registering the leader camera to a world coordinate system, where each point is mapped to unique pan-tilt-zoom values of each follower camera. We break this calibration scheme into two steps: geo-registration of leader camera, and calibration of PTZ camera to geo-locations. This two step process allows a flexible approach as far as integrating various kinds of leader sensors (Radar/AIS/camera) in the framework as the second step remains the same.

Georegistration of Static Cameras

We use a planar homography to map the camera field of view to a satellite image/map that has pixel level geo-registration, e.g. Google Earth [8]. This relationship is estimated using geo-positioning information broadcasted by the mobile transponders as detailed in [START_REF] Haering | Automatic imaging sensor calibration using objects that broadcast positional information[END_REF]. The transponders can be persons or cars on land, and boats on water. The footprints of the transponders in camera views are automatically detected and tracked. Note that in an uncontrolled environment, neither do all detected targets in the camera view transmit geo-positional data nor do all targets transmitting this data have to be visible. Therefore, one-toone correspondences between target footprints and geo-positional data are not available. In our experiments, accidental correspondences (noise) significantly out-number the true correspondences (typically noisy data is more than 80% of the total data), where a RANSAC framework [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] is not guaranteed to find an optimal homography. To handle this challenge, aggressive data filtering is first utilized to reject outliers based on motion properties, lengths and confidence of trajectories, global consistency of motion, as well as geometrical constraints.

Map to PTZ Camera Calibration

During the calibration process, users manually slew the follower camera to a landmark at P W = [Latitude, Longitude] T , and record the pan and tilt angles as θ and ψ. The two readings θ and ψ define a 3D ray casting from the center of the follower camera's pan-tilt head. This ray intersects a virtual plane Π, which is parallel to the camera's mount plane at unit distance, at location

P v = [x v , y v ] T as: x v = tan(θ) sin(ψ), (1)
y v = tan(θ) cos(ψ). (2)
Given more than four non-degenerate landmarks and corresponding pan and tilt readings, we find the planar homography H [10] that projects the ground plane points P W to points P v on the virtual plane Π, which provides θ and ψ after triangular operations. This completes the calibration for the pan and tilt values. The optimality of zoom is subjective and varies from applications. While users would like to zoom to the faces of a detected person as much as possible, a fast moving person might go out of field of view due to the detecting, panning and responding latencies. To approximate the user's desire for a particular zoom function, then, the calibration procedure fits a least-squares line through the zoom values given by the user based on the calculated distance from the target to the camera location on the world plane

PTZ Camera Autocalibration

As in any control system, visual feedback (as opposed to open-loop control) enables an active PTZ controller to overcome calibration and modeling errors when tracking moving targets. Calibration of the PTZ head is an important step in implementing active control, in particular for tuning the response and stability of the controller. Deployed systems are likely to interact with a wide range of COTS PTZ cameras in various state of working order, which renders manual calibration or the use of manufacturer specifications impractical. As a result, we have developed calibration techniques to automatically acquire the parameters of the PTZ camera by sending sequences of PTZ commands and analyzing the resulting video.

Focal Length and Zoom Profile

Many COTS cameras implement an arbitrary non-linear scale between zoom commands and actual magnification. Knowledge of this zoom profile is required for the zoom controller to maintain a visual target at an optimal size on the image plane. Furthermore, knowledge of the focal length at each zoom level allows pixel offsets to be converted into pan and tilt angles for active camera control. We model the zoom profile as a high dimensional polygon relating zoom to magnification. As the first step in this calibration process, the camera is set to minimum zoom and commanded to follow a short scan path over a pan range of +/-0.5 radians. Captured frames are processed using the usual mosaic building algorithms. After the first complete scan is detected, the mosaic optimization algorithms [START_REF]A framework for wide-area visual surveillance using a scanning camera[END_REF] are applied to estimate the focal length. The estimated focal length is recorded as the base value to which the zoom profile is applied to obtain the focal length at a given magnification. To estimate the zoom profile, a series of zoom commands over the full working range are sent to the camera, and the frame before and after execution of each command is logged. The magnification between each pair of frames is then calculated using a brute force search over scales. Multiplying the scales over all frame pairs yields the complete zoom profile. Finally, a high dimensional polygon (of order 10 in the current implementation) is fitted to the measurements to model the zoom profile.

Pan and Tilt Velocity Profiles

The pan and tilt velocity profiles characterize the relationship between the commanded pan/tilt velocity and the actual velocity, along with the maximum physical velocity for each axis. The profile is represented by fitting a high order polynomial to measured velocities uniformly spaced over the input velocity range.

For each measurement, the camera is driven at the fixed input speed starting from the minimum pan/tilt range for a fixed interval T , and the PTZ position is queried before and after each motion. The measurement is then repeated from the minimum pan/tilt angle for an interval of 2T . The angular difference between the two motions divided by T gives the measured velocity for the given input. Measuring the velocity this way ensures that errors due to actuation delays and acceleration of the axes are cancelled out when taking the difference. The maximum pan/tilt velocity is the highest input setting that results in a unique velocity measurement.

Zoom Position and Duration Profiles

While the pan and tilt velocities are typically independent of the camera position, the zoom velocity varies with zoom setting. Thus, we can not represent the zoom profile as a simple velocity input/output relationship. Instead, we construct a zoom position profile which gives the zoom setting attained after zooming in from the minimum zoom for a given duration at a given speed setting. We also represent the inverse of this curve, the zoom duration profile to simplify computation during velocity control of the camera. In theory, the zoom profiles should be measured for each zoom speed setting of the camera; in practice, we always use the minimum zoom speed setting and therefore only measure one profile pair. The profiles are represented as high order polynomials fitted to a set of measurements, where each measurement (T, z) is obtained by zooming the camera from the minimum zoom for duration T and querying the final zoom z.

Minimum Query Interval and Query Delay

A query command returns the current location of the pan, tilt and zoom axes from the PTZ controller. The minimum query interval is the minimum time between sending consecutive query commands, and the query delay is the duration between sending a query command and receiving a response from the controller. In the best case, the minimum query interval is equal to the query delay. The minimum query interval is measured by issuing pairs of query commands temporally separated by increasing multiples of 100 ms and taking the smallest interval for which the second query returns a valid response. The query delay is estimated as the difference in system time between sending a query and receiving a response, averaged over several commands.

Actuation Delay

The actuation delay is the time between issuing a velocity command and observing the resulting camera motion, and results from signal processing delays and mechanical characteristics such as acceleration. To determine when the camera starts moving, each captured frame is classified as moving or stationary based on frame differencing and thresholding with the previous frame. The actuation delay is then measured as the difference in system time between sending a motion command and observing the first moving frame, averaged over several motion commands. The actuation delay is measured separately for the pan/tilt and zoom axes, since these use different physical mechanisms.

Active Tracking

Once the PTZ camera is pointed in the vicinity of the target of interest using the leader-follower calibration described above, the tracking algorithm assumes the task of segmenting the target and estimating the state (position, orientation, scale and velocity) in subsequent frames. Our tracking algorithm exploits the fact that watercraft are generally sparsely distributed. This enables the effective use of adaptively selected discriminative features [START_REF] Collins | On-line selection of discriminative tracking features[END_REF][13] and tracking of targets independently without explicit consideration of multiple target tracking. New targets are initialized from the detection using the same algorithm described in Section 2 that do not overlap with existing targets. Our tracking algorithm uses adaptively selected discriminative color subspace features based on the approach in [START_REF] Collins | On-line selection of discriminative tracking features[END_REF]. A discriminative subspace provides good separation between foreground and background pixels; the tracking algorithm evaluates the most discriminative subspaces in each frame and uses those to track the target in the subsequent frame. During tracking, the current frame is projected into each color subspace.

A foreground histogram φ f g is accumulated from pixels in the target ellipse and a background histogram φ bg is accumulated from pixels in a surrounding elliptical ring. A log likelihood ratio is then computed for each histogram bin as:

L(i) = max (φ f g (i), δ) max (φ bg (i), δ) (3)
where δ = 1/ max (N f g , N bg ), and N f g , N bg are the number of foreground and background pixels respectively. Finally, a likelihood image is constructed by projecting each pixel into the N f most discriminative color sub-spaces from the previous frame and averaging the likelihood ratio for the corresponding histogram bins. Target pixels in the resulting image are expected to have a positive likelihood while background pixels have a negative likelihood. The discriminability of each color subspace is measured after updating the target in each frame, and the best N f subspaces are used to track the target in the subsequent frame. Since evaluating multiple subspaces is computationally expensive, two measures are implemented to maintain real-time tracking performance. Firstly, the likelihood map is only computed in a local region around the predicted target location. Furthermore, the image is dynamically sub-sampled to maintain an approximately constant number of pixels on target over all scales. A mean-shift procedure [START_REF] Collins | Mean-shift blob tracking through scale space[END_REF] is used to update the location and scale of the target given the likelihood image. Before applying mean-shift, the target location is predicted from the previous frame using the estimated velocity and inter-frame sample time. Targets can undergo a significant change in scale while moving from far field to near field in the camera field of view. To maintain track, we use two inter-leaved mean-shift searches over location and scale respectively, similar to [START_REF] Collins | Mean-shift blob tracking through scale space[END_REF]. Finally, a Kalman Filter is applied to the new target location from the mean-shift procedure to estimate the location and velocity of the target ellipse. Most of the standard PTZ control algorithms issue absolute pan, tilt, and zoom commands to the camera. While this approach is sufficient for tracking and maintaining the target in the camera FOV, it leads to jerky camera motion.

In order to improve the viewing experience for the user by having the camera move smoothly and continuously, we implemented a velocity-based active PTZ controller that issues velocity commands rather than position commands. Furthermore, increments in velocity are typically less perceptible than increments in position. Figure 2 provides an illustration of the active PTZ control loop. While no targets are visible, the controller remains in a wait state. When a new active tracking event occurs, the controller queries the current location of the PTZ and enters the main control loop. The first step in the control loop is to predict the change in camera position (due to previously issued velocity commands) between the time at which the frame was processed and the time when the camera would execute the command. The change in camera location is predicted by adding the integral of all velocity commands issued between the time when the last video frame was processed and the time of command execution (current time + actuation delay). The next step is to determine the desired pan, tilt, zoom correction of the camera in order to center the target at the desired resolution.

The system computes the camera zoom level at the time of frame processing by aggregating all the zoom commands issued since the last camera position query up to the time at which the last frame was processed to the last queried zoom level. Once the zoom level is known, using the zoom profile(Section 4.1), the focal length of the camera at that particular frame can be determined. The pan and tilt correction can then be computed as follows:

△pan = atan( frame center x -target center x focal length ) (4) △tilt = atan( frame center y -target center y focal length ) (5)
The PTZ set point is the desired positional correction of the camera, and is computed by extrapolating the last known location of the tracked target. The target velocity is estimated using the previous two target centering differentials and the aggregate camera motion between the previous two target sightings. This velocity is integrated over the time difference between the time of last frame processing and the time when we anticipate the target to be centered in the camera FOV (current time + actuation delay + target centering duration). The difference between the predicted PTZ position (p) and predicted setpoint (s) serves as the control loop error signal from which a proportional velocity command (v ) is computed. The controller gain is determined indirectly by a parameter called the target centering duration (d ), which is the duration required for the camera to move to the current setpoint. The controller gain, in combination with the control loop period τ , has a significant impact on the stability of the controller.

Since the period of the control loop is not known ahead of time, we estimate it online and compute the centering duration as a factor (f ) of the control loop period. Thus, the new velocity command is computed as:

v = s -p d = s -p f τ (6)
The pan and tilt velocity commands issued to the camera are computed using the pan/tilt velocity profiles computed in 4.2. The duration of the zoom command is determined by the required zoom correction and the zoom duration profile (4.3). After issuing a new velocity command, the camera is queried for its current position. If the query delay is significant, the control loop can be configured to query less frequently than every cycle in order to reduce the control loop period and improve controller stability. The control loop period for the current loop is estimated using standard system timers, and is combined using a fading memory average with previous estimates.

Experimental Results

In this section, we present the experimental results of the various technologies that we have proposed in the paper so far.

PTZ Autocalibration

Once the PTZ autocalibration process is complete, the system has knowledge about the various characteristics of the camera that are essential for accurate control during active tracking. We used a COTS Pelco Spectra III camera for our experiments. Figures 3(a)&(b) depict the mapping between input pan and tilt velocity in camera units and the output velocity in radians/sec. Figures 3(c)&(d) depict the zoom position and duration profiles respectively. Figure 3(e) depicts the mapping of camera zoom levels to image magnification. For the Pelco Spectra III under test, the minimum query interval was 500 ms and the query delay was about 400 ms. The estimated pan/tilt and zoom actuation delays were 70ms and 130ms respectively.

PTZ to Map calibration

To test the accuracy of our proposed calibration algorithm, we performed a qualitative and quantitative analysis to gauge the performance. In a real world setting, the camera at our test site was calibrated to the map, and upon clicking at a point on the map, the camera was commanded to go to the corresponding pan, tilt and zoom values (computed using the calibration). A human observer then verified if the camera was pointing at the correct location. We used synthetic data to measure the quantitative performance of the system. Figure 4(a) shows a pictorial representation of setup used for the synthetic data analysis. We used 8 training points (red) to compute the calibration matrix, and used 300 test points (green). The PTZ camera was moved in a circle around the points to measure the effect of the camera viewing angle on the calibration accuracy. We added a uniform noise of 1 pixel to take into account typical operator error during the calibration process.

Active Tracking

Figure 5 illustrates the results of some the intermediate stages of the active tracking algorithm. Figure 5(a) is the video frame from the PTZ camera after it has been pointed in the target's vicinity using the map-PTZ camera calibration. Figure 5(b) shows the foreground mask detected using the algorithm described in Section 2. Figure 5(c) shows the sorted feature discrimination score for the candidate color subspaces. Figure 5(d) shows the averaged likelihood map image computed using the two features with the greatest feature discrimination score. with the maximum discrimination score. The detected target ellipse computed after performing mean-shift iterations on the likelihood image is overlayed on the original video frame and shown in Figure 5(a). Figure 6(a) shows the snapshots of four PTZ camera at the test site while tracking the same target. Note that some cameras have less than desired resolution on the target since they are already fully zoomed in.

Event Detection

The event detection performance was also evaluated on the ground truth data. In this evaluation, we defined a virtual tripwire as the event trigger. Tripwires were randomly drawn and the ground truth for each tripwire crossing was manually labeled. Figure 6(b) shows a boat crossing one such tripwire along with the mapped target trajectory and target snapshot. The hit/miss/false alarm rates for seven test video clips were measured. Overall, the system was able to detect 55 out of 56 tripwire events with a total of 7 false alarms. The precision and recall for event detection is 88.71% and 98.21% respectively.

Fig. 1 .

 1 Fig. 1. Active leader-follower surveillance system.

Fig. 2 .

 2 Fig. 2. Active tracking controller

Fig. 3 .Fig. 4 .

 34 Fig. 3. PTZ Velocity Profiles

Figure 4 (

 4 b) shows the plot of the resulting average pan/tilt error versus the PTZ camera viewing angle. As can be seen from the plot, the camera positioning is accurate up to a degree for almost all viewing angles.

Fig. 5 .

 5 Fig. 5. Active tracking results

Fig. 6 .

 6 Fig. 6. (a) Active tracking PTZ snapshots (b) Estimated FOVs of four high resolution cameras overlaid on satellite imagery, along with map rules and target trajectories.