Antitza Dantcheva
email: antitza.dantcheva@inria.fr

Franc ¸ois Brémond
email: francois.bremond@inria.fr

F Brémond

Gender estimation based on smile-dynamics

Keywords: soft biometrics, gender estimation, facial dynamics. I

Gender estimation based on smile-dynamics

Antitza Dantcheva, François

estimation systems. This is why the majority of previous works have extracted and studied appearance-based features under the simplifying assumption of neutral face expressions with reasonably good results.

A. Gender and emotional expression

Deviating from such works, we here introduce the usage of a set of dynamic facial features for gender estimation. Specifically, we focus on extracting dynamic features from a common facial expression, namely the smile, and study how smile-dynamics encrypt gender evidence. The hypothesis is that male and female smile-dynamics differ in parameters such as intensity and duration. This hypothesis is supported in part by a number of cognitive-psychological studies, showing evidence for gender-dimorphism in the human expression [START_REF] Cashdan | Smiles, speech, and body posture: How women and men display sociometric status and power[END_REF], [START_REF] Simon | Gender and emotion in the united states: Do men and women differ in self-reports of feelings and expressive behavior? 1[END_REF], [START_REF] Hess | Facial appearance, gender, and emotion expression[END_REF], [START_REF] Adams | The intersection of genderrelated facial appearance and facial displays of emotion[END_REF], [START_REF] Liébart | Smile line and periodontium visibility[END_REF], [START_REF] Deutsch | What is in a smile?[END_REF]. A main observation of such studies has been that females express emotions more frequently than males, and in the context of smile, females tend to smile more often than men in a variety of social contexts [START_REF] Deutsch | What is in a smile?[END_REF]. Such observations follow the theorem of men exhibiting restrictive emotionality and thus being unwilling to self-disclose intimate feelings. It is interesting to note, that a gender-based difference in emotional expression is observed as early as in 3 months old, shaped by how caregivers interact to male and female infants [START_REF] Fogel | Mother-infant face-to-face interaction in japan and the united states: A laboratory comparison using 3-monthold infants[END_REF]; and also observed in toddlers, which appears to be further trained in social interactions [START_REF] Chaplin | Gender differences in emotion expression in children: a meta-analytic review[END_REF], [START_REF] Malatesta | The development of emotion expression during the first two years of life[END_REF], [START_REF] Malatesta | Learning display rules: The socialization of emotion expression in infancy[END_REF]. Moreover, females are more accurate expressers of emotion, when posing deliberately and when observed unobtrusively, which is consistent across cultures [START_REF] Brody | Gender and emotion in context[END_REF]. The same work assigns happiness and fear as female-gender-stereotypical expressions. On the other hand, faces showing anger are considered more masculine [START_REF] Hess | Facial appearance, gender, and emotion expression[END_REF], [START_REF] Hess | Who may frown and who should smile? dominance, affiliation, and the display of happiness and anger[END_REF], [START_REF] Hess | When two do the same, it might not mean the same: The perception of emotional expressions shown by men and women[END_REF], [START_REF] Hess | Why the same expression may not mean the same when shown on different faces or seen by different people[END_REF], [START_REF] Bayet | Angry facial expressions bias gender categorization in children and adults: behavioral and computational evidence[END_REF], [START_REF] Becker | The confounded nature of angry men and happy women[END_REF], [START_REF] Zebrowitz | Social psychological face perception: Why appearance matters[END_REF] in the context of human gender recognition.

B. Contributions

Motivated from the above, we propose the use of an automated framework for facial dynamics extraction based on signal displacement of facial distances between key facial landmarks. We analyze the properties of 27 such facial distances in smile-video-sequences with emphasis on spontaneous, as well as posed smiles. The proposed dynamic features are fully complementary to appearance based features, and when combined with appearance, can pose an increased difficulty for spoof-attacks. We have adopted the approach from Dibeklioglu et al. [START_REF] Dibeklioglu | A smile can reveal your age: Enabling facial dynamics in age estimation[END_REF], [START_REF] Dibeklioglu | Are you really smiling at me? spontaneous versus posed enjoyment smiles[END_REF], where it has been used for age estimation, as well as spontaneous vs. posed smile detection based on facial dynamics, see also [START_REF] Dibeklioglu | Recognition of genuine smiles[END_REF], [START_REF] Dibeklioglu | Combining facial dynamics with appearance for age estimation[END_REF].

The use of the framework is instrumental in answering following questions:

• Do facial dynamics provide information about gender in (a) spontaneous smile-and (b) posed smile video sequences? • Can facial smile dynamics improve the accuracy of appearance based gender estimation systems? • Which gender can pose smiles more genuinely? Related work of a holistic smile-based gender estimation algorithm can be found in Biliński et al. [START_REF] Bilinski | Can a smile reveal your gender[END_REF].

C. Structure of paper

This work is organized as follows: Section I-D revisits existing works on gender estimation. Section II proceeds to describe the proposed method, elaborating on individual steps (face detection, landmark location, selected features, statistics of dynamic features, feature selection, classification and used appearance features). Section III presents the employed dataset and the subsequent Section IV depicts and discusses related experimental results. Finally Section V concludes the paper.

D. Related work

Gender estimation Existing introductory overviews for algorithms related to gender estimation include the works of Ng et al. [START_REF] Ng | Vision-based human gender recognition: A survey[END_REF], Bekios-Calfa et al. [START_REF] Bekios-Calfa | Revisiting linear discriminant techniques in gender recognition[END_REF], Ramanathan et al. [START_REF] Ramanathan | Age progression in human faces: A survey[END_REF], Mäkinen and Raisamo [START_REF] Mäkinen | Evaluation of gender classification methods with automatically detected and aligned faces[END_REF] and Dantcheva et al. [START_REF] Dantcheva | What else does your biometric data reveal? a survey on soft biometrics[END_REF].

Based on these works we can conclude that gender estimation remains a challenging task, which is inherently associated with different biometric modalities including fingerprint, face, iris, voice, body shape, gait, signature, DNA, as well as clothing, hair, jewelery and even body temperature. The forensic literature [START_REF] Loth | Sex determination[END_REF] suggests that the skull, and specifically the chin and the jawbone, as well as the pelvis, are the most significant indicators of the gender of a person; in juveniles, these shapebased features have been recorded to provide classification accuracy of 91% -99%.

Humans are generally quite good at gender recognition from early in life (e.g., [START_REF] O'toole | An other-race effect for classifying faces by sex[END_REF], [START_REF] Quinn | Representation of the gender of human faces by infants: A preference for female[END_REF]), probably reflecting evolutive adaptation. As pointed out by Edelman et al. [START_REF] Edelman | Sex classification of face areas: how well can a linear neural network predict human performance?[END_REF], humans perform facial image based gender classification with an error rate of about 11%, which is commensurate to that of a neural network algorithm performing the same task.

Dynamics have been used in the context of body-based classification of gender. Related cues include body sway, waist-hip ratio, and shoulder-hip ratio (see [START_REF] Mather | Gender discrimination in biological motion displays based on dynamic cues[END_REF]); for example, females have a distinct waist-to-hip ratio and swing their hips more, whereas males have broader shoulders and swing their shoulders more.

Despite these recent successes, automated gender recognition from biometric data remains a challenge and is impacted by other soft biometrics, for example, age and ethnicity; gender dimorphism is accentuated only in adults, and varies across different ethnicities.

Automated Image-based Gender Estimation from Face In gender estimation from face, feature-based approaches extract and analyze a specific set of discriminative facial features (patches) in order to identify the gender of a person. This is a particularly challenging problem, as is implied from the fact that female and male average facial shapes are generally found to be very similar [START_REF] Langlois | Attractive faces are only average[END_REF].

Another challenge comes to the fore in unconstrained settings with different covariates, such as illumination, expressions and ethnicity. While in more constrained settings, face-based gender estimation has been reported to achieve classification rates of up to 99.3% (see Table I), this performance though significantly decreases in more realistic and unconstrained settings.

The majority of gender classification methods contain two steps preceding face detection, namely feature extraction and pattern classification.

Feature extraction: Notable efforts include the use of SIFT [START_REF] Toews | Detection, localization, and sex classification of faces from arbitrary viewpoints and under occlusion[END_REF], LBP [START_REF] Mäkinen | Evaluation of gender classification methods with automatically detected and aligned faces[END_REF], semi-supervised discriminant analysis (SDA) [START_REF] Bekios-Calfa | Robust gender recognition by exploiting facial attributes dependencies[END_REF] or combinations of different features [START_REF] Guo | Is gender recognition affected by age?[END_REF], [START_REF] Wang | Boosting dense sift descriptors and shape contexts of face images for gender recognition[END_REF].

Classification: A number of classification methods have been used for gender estimation, and a useful comparative guide of these classification methods can be found in Mäkinen and Raisamo [START_REF] Mäkinen | Evaluation of gender classification methods with automatically detected and aligned faces[END_REF]. One interesting conclusion of their work was that image size did not greatly influence the classification rates. This same work also revealed that manual alignment affected the classification rates positively, and that the best classification rates were achieved by SVM.

The area of gender estimation has also received some other contributions such as those that go beyond using static 2D visible spectrum face-images. Interesting related work include the work of Han et al. [START_REF] Han | Gender classification based on 3D face geometry features using SVM[END_REF], exploring 3D images, Gonzalez-Sosa et al. [START_REF] Gonzalez-Sosa | Image-based gender estimation from body and face across distances[END_REF], studying jointly body and face, and Chen and Ross [START_REF] Chen | Evaluation of gender classification methods on thermal and near-infrared face images[END_REF], [START_REF] Ross | Can gender be predicted from near-infrared face images[END_REF], using near-infrared (NIR) and thermal images for gender classification.

Expression Recognition Automated expression recognition has received increased attention in the past decade, since it is particularly useful in a variety of applications, such as human computer interaction, surveillance and crowd analytics. The majority of methods aim to classify 7 universal expressions namely neutral, happy, surprised, fearful, angry, sad, and disgusted [START_REF] Yu | Learning realistic facial expressions from web images[END_REF] based on the extracted features used. Classical approaches follow Ekman's facial action coding system (FACS) [START_REF] Ekman | Facial expression and emotion[END_REF], assigning each facial unit to represent movement of a specific facial muscle. In this context, intensity and number of facial units have been studied, as well as of action unit combinations, towards expression recognition. Interesting work can be found in related survey papers [START_REF] Zeng | A survey of affect recognition methods: Audio, visual, and spontaneous expressions[END_REF], [START_REF] Martinez | A model of the perception of facial expressions of emotion by humans: Research overview and perspectives[END_REF], [START_REF] Sandbach | Static and dynamic 3D facial expression recognition: A comprehensive survey[END_REF] and in a related recent expression-recognition-challenge-study [START_REF] Valstar | Fera 2015-second facial expression recognition and analysis challenge[END_REF]. Latest advances involve deep learning [START_REF] Zhao | Facial expression recognition via deep learning[END_REF], [START_REF] Juefei-Xu | Deepgender: Occlusion and low resolution robust facial gender classification via progressively trained convolutional neural networks with attention[END_REF].

Inspired by cognitive, psychological and neuroscientific findings, facial dynamics have been used previously towards improving face recognition [START_REF] Hadid | Combining appearance and motion for face and gender recognition from videos[END_REF], gender estimation [START_REF] Demirkus | Gender classification from unconstrained video sequences[END_REF], age estimation [START_REF] Dibeklioglu | A smile can reveal your age: Enabling facial dynamics in age estimation[END_REF], as well as kinship recognition reported in a review article by Hadid et al. [START_REF] Hadid | On the use of dynamic features in face biometrics: recent advances and challenges[END_REF].

II. DYNAMIC FEATURE EXTRACTION IN SMILE-VIDEO-SEQUENCES

Deviating from the above works on gender estimation, we propose to extract dynamic features in smile-video-sequences. The general scheme is shown in Fig. 1. Specifically we focus on signal displacement of facial landmarks, as we aim to study among others the pertinence of different facial landmarks, as well as the pertinence of different statistical properties of facial dynamics (e.g. intensity and duration) in the effort of gender estimation.

Towards extraction of such dynamic features, we assume a near frontal pose of the subject and an initial near-neutral expression of the subject (given in the used dataset).

A. Face Detection and Extraction of Facial Landmarks

Firstly we detect the face using the well established Viola and Jones algorithm [START_REF] Viola | Robust real-time face detection[END_REF]. We here note that the faces were robustly detected in all video sequences and frames. Within the detected face we identify facial feature points corresponding to points in the regions of the eye brows, eyes, nose and lips (see Fig. 5). Specifically we employ the facial landmark detection algorithm proposed in the work of Asthana et al. [START_REF] Asthana | Incremental face alignment in the wild[END_REF]. The algorithm is an incremental formulation for the discriminative deformable face alignment framework [START_REF] Xiong | Supervised descent method and its applications to face alignment[END_REF], using a discriminative 3D facial deformable shape model fitted to a 2D image by a cascade of linear regressors. The detector was trained on the 300W -dataset (a dataset introduced in the context of the 300 faces in-the-wild challenge [START_REF] Sagonas | 300 faces in-the-wild challenge: The first facial landmark localization challenge[END_REF]) and detects 49 facial landmarks (see Fig. 5). For the UvA Nemo-dataset the facial landmarks were detected robustly in all video sequences and frames. We use these points to initialize a sparse optical flow tracking algorithm, based on the Kanade-Lucas-Tomasi (KLT) algorithm [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] in the first frame of each video-sequence. For the here proposed framework we select a subset of facial-points in three different face regions: (a) eye brow region, (b) eye region, (c) mouth region (see Fig. 2) and proceed to extract dynamic features thereof.

B. Extraction of Dynamic Features

We extract dynamic features corresponding to the signaldisplacement in facial-distances depicted in Table II. We have selected 27 such facial-distances based on findings on facial movements during smile-expressions [START_REF] Richardson | Digitizing the moving face during dynamic displays of emotion[END_REF].

1) Temporal smile-segmentation: Generally, the human smile is caused by the contraction of the zygomatic major muscle, which raises the corners of the lips [START_REF] Ekman | Felt, false, and miserable smiles[END_REF], corresponding to "Action Unit Nr. 12" in Ekman's facial action coding system [START_REF] Ekman | Facial expression and emotion[END_REF]. Temporally segmented, the human smile contains three phases: (a) onset: contraction of the zygomatic major muscle and alteration from neutral to expressive state, (b) apex: peak period of the expressive state, and (c) offset: relaxation of the zygomatic major muscle and change from expressive to neutral state. We here note that there are dozens of smileclasses, differing in appearance and meaning.

The next step in our method is to temporally segment the signal-displacement functions as: (a) onset: duration of monotonous increase, (b) apex: phase between onset and offset, (c) offset: duration of monotonous decrease. Fig. 3 illustrates two examples of signal-displacement in the mouth-region (D 5 , mouth length), leading to a smile-curve with differently pronounced onset, apex and offset phases.

We smoothen each of the 27 signal displacement functions by the 4253H-twice smoothing algorithm [START_REF] Velleman | Definition and comparison of robust nonlinear data smoothing algorithms[END_REF] to flatten minor tracking-flaws.

2) Statistics of Dynamic Features: We proceed to extract statistics from each dynamic function with respect to the particular smile-phases, denoted by the superindices ( + ) for onset, ( a ) for apex, and ( -) for offset, which we summarize in Table III. We compute the speed as V (t) = dD dt and the acceleration as

A = d 2 D dt 2 = dV dt .
We denote the number of frames by η, frame rate of the video sequence by ω. Each of the defined 27 signal-displacement-functions are represented by a set of 24 features, resulting in a 648-dimensional feature vector.

C. Feature Selection

We use the Min-Redundancy Max-Relevance (mRMR) algorithm [START_REF] Peng | Feature selection based on mutual information criteria of max-dependency, max-relevance, and minredundancy[END_REF] for selecting the permanent dynamic proposed features. mRMR minimizes the redundancy, while selecting the most relevant information:

max fj ∈F -Sm-1   I(f j , c) - 1 m -1 fi∈Sm-1 I(f j , f i )   , (1)
with I being the mutual information function, c the target class, F the feature set, and S m-1 set of m -1 features. The mutual information I of a feature f j and the target class c is computed based on the related probability density functions p(f j ), p(c) and p(f j , c) as follows

I(f j ; c) = p(f j , c) log p(f j , c) p(f j )p(c) df j dc. (2)

D. Classification

A pattern classifier, trained on labeled data, is used to classify the feature vector into one of two classes: male or female.

We utilized linear Support Vector Machines (SVM) [START_REF] Chang | Libsvm: A library for support vector machines[END_REF], AdaBoost [START_REF] Bekios-Calfa | Revisiting linear discriminant techniques in gender recognition[END_REF] and Bagged Trees [START_REF] Breiman | Bagging predictors[END_REF] in this work. For SVM Total Amplitude

(D + ) (|D -|) Net Amplitude (D + ) -(|D -|) Amplitude Ratio (D + ) (D + )+ (D -) (D -) (D + )+ (D -) Maximal Speed max(V + ) max(V -) Mean Speed mean(V + ) mean(V -) Maximum Acceleration max(A + ) max(A -) Mean Acceleration mean(A + ) mean(A -)
Net Ampl., Duration Ratio

( (D + )-(|D -|))ω η(D)
the Gaussian RBF kernel is used. The optimum values for C and the kernel parameter γ are obtained by a grid-search of the parameter space based on the training set.

E. Extracted Appearance Features

OpenBR [START_REF] Klontz | Open source biometric recognition[END_REF] is a publicly available open source software for biometric recognition and evaluation. We utilize the gender estimation algorithm, based on the work of Klare et al. [START_REF] Klare | Face recognition performance: Role of demographic information[END_REF]. Specifically, a face image is represented by extracting histograms of local binary pattern (LBP) and scale-invariant feature transform (SIFT) features computed on a dense grid of patches. Subsequently, the histograms from each patch are projected onto a subspace generated using Principal Component Analysis (PCA) in order to obtain a feature vector. Support Vector Machine (SVM) is used for the final gender estimation. The OpenBR gender classification algorithm has been validated on a FERET5 subset, attaining accuracies of 96.91% and 82.98% for male and female classification, respectively and an overall true classification rate of 90.57% [START_REF] Chen | Impact of facial cosmetics on automatic gender and age estimation algorithms[END_REF], outperforming other algorithms (Neural Network, Support Vector Machine, etc.) on the same dataset [START_REF] Mäkinen | Evaluation of gender classification methods with automatically detected and aligned faces[END_REF].

how-old.net is a website (http://how-old.net/) launched by Microsoft for online age and gender recognition. Images can be uploaded and as an output age and gender labels are provided. The underlying algorithm and training dataset are not publicly disclosed.

Commercial Off-the-Shelf (COTS) is a commercial face detection and recognition software, which includes a gender classification routine. The underlying algorithm and the training dataset that were used are not publicly disclosed. The system does not provide a mechanism to re-train the algorithm based on an external dataset; instead it is a black box that outputs a label (i.e., male or female) along with a confidence value.

Since the video-sequences of the UvA-NEMO dataset start with the neutral expression of the portrayed subject, the first frame is utilized to extract appearance features.

F. Fusion of Dynamic and Appearance Features

We concatenate score-levels obtained from the appearance based-algorithms with features obtained from the feature selection step of the dynamics-framework. We utilize PCA to reduce the dimension and obtain a fused feature vector.

III. UVA-NEMO SMILE-DATASET

The UvA-NEMO Smile Dataset6 , introduced by Dibeklioglu et al. [START_REF] Dibeklioglu | Are you really smiling at me? spontaneous versus posed enjoyment smiles[END_REF], consists of multiple video sequences of 400 subjects (185 females, 215 male). The age of the subjects ranges from 8 to 76 years, see Fig. 4 for the age-distribution. For the most of the subjects there are two videos per subject displaying: (a) spontaneous smile and (b) posed smile. To elicit spontaneous smiles, each subject was displayed a short funny video segment. Each video starts and ends with neutral or a near-neutral expression of the subject (see Fig. 5). The pose of the subjects is frontal and the illumination condition is reasonably constant across subjects. The resolution of the videos is 1920 x 1080 pixels at a framerate of 50 frames per second. This dataset has been used for the analysis of smiles for different ages [START_REF] Dibeklioglu | Are you really smiling at me? spontaneous versus posed enjoyment smiles[END_REF] and for smile-based age analysis [START_REF] Dibeklioglu | A smile can reveal your age: Enabling facial dynamics in age estimation[END_REF].

We note that the ethnicity of subjects in the UvA-NEMO dataset is predominantly Caucasian, hence the current study does not reflect on covariates such as ethnicity, as well as social and cultural background.

A. Effect of Age

The UvA-NEMO dataset consists of images of subjects in the age-range of 8 to 76 years. The ability of dynamics to predict age, and thus the impact of age on a small set of facial dynamics has been previously assessed in the work of Dibeklioglu et al. [START_REF] Dibeklioglu | A smile can reveal your age: Enabling facial dynamics in age estimation[END_REF], where results suggest that facialdynamics change significantly with age. Consequently we present our results based on age-categories.

IV. RESULTS

In order to evaluate the performance of the proposed gender estimation algorithm, we employ a 15-fold cross-validation scheme. Here, the UvA-NEMO dataset is divided into 15 folds with approximately 24 subjects in each fold. 14 folds are used for training the dynamic gender-estimation algorithm, and the remaining fold is used for testing it. This is repeated 15 times and reported results are the average thereof. Note that the subjects in the training set are not present in the test set.

A. Dynamics versus Appearance

Table IV firstly depicts the discriminative power of the two complementary characteristics individually for spontaneous smiles. As mentioned above, we report age-based gender recognition accuracy. Since training is required for the dynamics based gender estimation (and hence larger amount of subjects per group), we merge age-groups to two main groups: < 20 years and > 19 years and provide the associated results in Table IV. We observe that the appearance based gender algorithms perform significantly better for the age category > 19 years and rather poorly in the age category < 20 years. This can be due to age-unbalanced training sets or merely due to poor feature performance for toddlers and adolescents, due to low sexual dimorphism. The related confusion matrices for the age category > 19 years are shown in Table V.

Dynamics based gender estimation: Interestingly, dynamic features (True Gender Classification Rate T GCR = 59.44%) outperform two of the three appearance based features (T GCR OpenBR = 52.45% and T GCR how-old.net = 51.05%) in the first age-category. While, appearance-based features are more reliable for the age category > 19 years with T GCR OpenBR = 78.04%, T GCR how-old.net = 93.46%, T GCR COT S = 92.52%; dynamics-based features obtain a noticeable accuracy of 67.81%. The latter suggests that facial smile-dynamics carry substantial cues related to gender of the subject. The confusion matrix is rather balanced in the dynamics-based gender estimation (Table V

(d)).
We note that fusion of appearance and smile-dynamicbased gender estimation either increases the performance of appearance based algorithms (e.g., for OpenBR in both age classes, for how-old.net in the younger age-class and for COTS in the older age-class) or does not impact it negatively. Related confusion matrices are shown in Table V.

In our related work [START_REF] Bilinski | Can a smile reveal your gender[END_REF], we have presented a holistic approach for smile-based gender estimation, that extracts spatiotemporal features based on dense trajectories, represented by a set of descriptors encoded by Fisher Vectors. The associated true gender classification rates account for 86.3% for adolescents, and 91.01% for adults.

B. Spontaneous versus posed smile

We also provide results on the posed-smile subset of the UvA-NEMO dataset presented in Table VI. Interestingly, the associated dynamics-based gender-estimation accuracy resembles strongly the spontaneous-smile-case. The difference in performance origins in the slightly larger posed-smile subsetsize, that contributes to larger trainings-sets in the case of dynamics-based gender classification, as well as in the fusion of appearance and dynamics-based features. Nevertheless, the results suggest that dynamics of posed smiles carry significant cues on gender, similarly to spontaneous smiles. The related confusion matrices are shown in Table VII.

This result is in agreement with psychological findings, that show that females are more accurate expressers of emotion, when posing deliberately and when observed unobtrusively [START_REF] Brody | Gender and emotion in context[END_REF], hinting that posing a smile carries gender-specific cues. C. Gender divergence in spontaneous and posed smiles

We seek to answer the question, whether males or females pose smiles more genuinely and whether there is a significant divergence. Towards this, we combine features in all possible sets and compute Euclidean distances between sets in the spontaneous and the associated sets in the posed-smile-case. Fig. 6 illustrates the related results for the most diverging case between males and females. Females have slightly lower distances, suggesting that females pose smiles more realistically; however, the disparity is not significant. This tendency conforms with previous psychological findings [START_REF] Brody | Gender and emotion in context[END_REF].

D. Discriminative Features

We here analyze the individual discriminability of the selected dynamic-features for the 27 distances. Towards this, we estimate gender based on each feature individually. Hence, we train and test an SVM-classifier with each feature individually for the two age groups, < 20 and > 19 years. We report for each age group the most discriminative features respectively (see Table VIII and Table IX). The most striking outcome is that the majority of discriminative features are in the mouth region. It is also interesting to note that while for the younger group D 10 (Center of mouth to right mouth corner) and D 7 (Center of mouth to left side of upper lip) and the onset-phase are predominant, for the older group D 5 (Length of mouth) and mainly the offset-phase is more profound. This hints that sexual dimorphism can be gleaned from the asymmetricalonset in adolescents. On a related note, a recent psychological study [START_REF] Carr | The two sides of spontaneity: Movement onset asymmetries in facial expressions influence social judgments[END_REF] has found that expressions shown on the left hemi-face (LHF) were rated as more intense, and furthermore that spontaneous expressions start earlier in the LHF. Hence expressions in both hemi-faces are not fully redundant. Description of most discriminative features In adolescents, females tended to show longer Duration Ratio -Offset and longer Duration -Onset on the right side of the mouth and higher Amplitude Ratio -Onset on the left side of the mouth, than males. In adults, females tended to show higher Mean Amplitude -Apex of mouth opening, higher Maximum Amplitude on the right side of the mouth, as well as faster Mean Speed -Offset on the left side of the mouth, than males. Figure 7 illustrates the boxplots for the five most discriminative features in the age category > 19 years for spontaneous smile.

We here note, that the selected features for the proposed al-gorithm in previous sections do not correspond to the presented features in this section, since a mutual information function prunes out correlated features in the selection process, which we do not consider here.

V. CONCLUSIONS

In this work we introduced smile-based dynamic facial feature extraction for gender estimation. The proposed dynamicsbased gender estimation algorithm predominantly improves the performance of three state-of-the-art appearance-based gender estimation algorithms. We observe that dynamics can outperform appearance-based features for subjects younger than 20 years old; while facial appearance features are more discriminative for older subjects. We show that appearance and dynamics-based features are complementary and the combination thereof beneficial. Our results further suggest that gender is mainly exhibited in dynamics in the mouth-region among the studied facial dynamic-features. Finally, we analyzed the gender-dimorphism of both, spontaneous and posed smiles and observe that both carry substantial cues for gender.

APPENDIX A

Fig. 1 .

 1 Fig. 1. Proposed framework for automatic gender estimation.

Fig. 2 .

 2 Fig. 2. Subset of landmarks extracted by the Asthana et al. algorithm [2] used in the proposed algorithm.

Fig. 3 .

 3 Fig. 3. Signal displacement in mouth-region (D 5 , mouth length) for (a) female and (b) male subject from the UvA-NEMO dataset [28]. Example (a) shows the three profound smile phases: onset as the monotonic increasing phase, apex the (relatively) flat peak phase and offset the monotonic decreasing phase, example (b) on the other hand has a less pronounced apex-phase.

  mean(D + ) mean(D a ) mean(D -)

Fig. 4 .Fig. 5 .

 45 Fig. 4. Age and gender distributions of the subjects in the UvA-Nemo database, part 'spontaneous smile' containing 357 subjects.

Fig. 6 .

 6 Fig.6. Distributions of Euclidean distances between posed and spontaneous feature vectors for male and female subjects in the UvA-NEMO dataset.

(a) D 11 Fig. 7 .

 117 Fig.7. Boxplots of most discriminative features in age category > 19 years. Females tended to show longer Mean Amplitude Apex of mouth opening, a higher Maximum Amplitude on the right side of the mouth, as well as a shorter Mean Speed Offset on the left side of the mouth, than males. Further the Mean Acceleration Offset of the mouth length is shorter for females than for males.

TABLE I .

 I OVERVIEW OF FACE-BASED GENDER CLASSIFICATION ALGORITHMS. ABBREVIATIONS USED: PRINCIPAL COMPONENT ANALYSIS (PCA), INDEPENDENT COMPONENT ANALYSIS (ICA), SUPPORT VECTOR MACHINES (SVM), GAUSSIAN PROCESS CLASSIFIERS (GPC), ACTIVE APPEARANCE MODEL (AAM), LOCAL BINARY PATTERN (LBP), ACTIVE SHAPE MODEL (ASM), DISCRETE COSINE TRANSFORM (DCT), SEMI-SUPERVISED DISCRIMINANT ANALYSIS (SDA).

	Work	Features	Classifier	Datasets used	Performance
				for evaluation	numbers
	Bekios-Calfa et al. (2007) [7]		SVM	UCN (nonpublic), 10,700 images	93.46% ± 1.65%
		PCA	LDA	FERET, 994 images	93.57% ± 1.39%
		LCA	LDA	PAL, 576 images	93.57% ± 1.39%
	Xia et al. (2008) [80]	LBP, Gabor	SVM	CAS-PEAL, 10,784 images	93.74%
	Mäkinen and Raisamo (2008) [55]	LBP	SVM	FERET, 411 images	86.54%
	Baluja and Rowley (2008) [3]	Raw pixels	Adaboost	FERET, 2,409 images	93%
	Gao and Ai (2009) [34]	ASM	Adaboost	Private, 1,300 images	92.89%
	Toews and Arbel (2009) [75]	SIFT	Bayesian	FERET, 994 images	83.7%
	Shan (2010) [72]	LBP	Adaboost	LFW, 7,443 images	94.44%
	Guo et al. (2009) [36]	LBP, HOG, BIF	SVM	YGA, 8,000 images	89.28%
	Wang et al. (2010) [79]	SIFT, context	Adaboost	FERET, 2,409 images	95.0%
	Nazhir et al. (2010) [60]	DCT	KNN	SUMS, 400 images	99.3%
	Ross and Chen (2011) [69]	LBP	SVM	CBSR NIR, 3,200 images	93.59%
	Cao et al. (2011) [12]	Metrology	SVM	MUCT, 276 images	86.83%

[START_REF] Hu | Fusion of region-based representations for gender identification[END_REF]

[START_REF] Hu | Fusion of region-based representations for gender identification[END_REF]

Filter banks SVM Flickr, 26,700 images 90.1% Bekios-Calfa et al. (2011) [8] SDA PCA Multi-PIE, 337 images 88.04% Shan (2012) [73] Boosted LBP SVM LFW, 7,443 94.81% Ramón-Balmaseda (2012) [66] LBP SVM MORPH, LFW, Images of Groups, 17,814 75.10% Jia and Cristianini (2015) [46] Multi-scale LBP C-Pegasos Private, 4 million images 96.86%

TABLE II .

 II EXTRACTED SIGNAL-DISPLACEMENT-FUNCTIONS CONTRIBUTING TO DYNAMIC FEATURES. ρ DENOTES THE DISTANCE BETWEEN FACIAL LANDMARKS, l i DENOTES THE i th LANDMARK POINT, AS ILLUSTRATED IN FIG. 2.

	Facial Distance	Description	Description by facial landmarks
	D1 D2	Width of right eye Width of left eye	ρ( ρ( l 14 +l 15 2 l 8 +l 9 2 ,	, l 11 +l 12 l 17 +l 18 2 2 )	)
	D3	Length of right eye		ρ(l13, l16)
	D4	Length of left eye		ρ(l7, l10)
	D5	Length of mouth		ρ(l19, l24)
	D6	Width of mouth		ρ(l22, l25)
	D7 D8 D9 D10 D11 D12	Center of mouth to left side of upper lip Center of mouth to right side of upper lip Center of mouth to left mouth corner Center of mouth to right mouth corner Center of mouth to upper lip Center of mouth to average distance of two mouth corners	ρ(	ρ( ρ( ρ( ρ( ρ( l 22 +l 25 l 22 +l 25 2 l 22 +l 25 2 l 22 +l 25 2 l 22 +l 25 2 l 22 +l 25 2 2 , l 19 +l 24 , l21) , l23) , l19) , l24) , l27) 2	)
	D13	Left side of right eyebrow to nose		ρ(l4, l26)
	D14	Right side of left eyebrow to nose		ρ(l3, l26)
	D15	Center of right eyebrow to right side of the right eyebrow		ρ(l5, l6)
	D16	Center of right eyebrow to left side of the right eyebrow		ρ(l5, l4)
	D17	Center of left eyebrow to right side of the left eyebrow		ρ(l2, l3)
	D18	Center of left eyebrow to left side of the left eyebrow		ρ(l2, l1)
	D19	Distance between eyebrows		ρ(l3, l4)
	D20	Left corner of left eye to left mouth corner		ρ(l7, l19)
	D21 D22	Right corner of left eye to center of mouth Left corner of right eye to center of mouth		ρ(l10, ρ(l13,	l 22 +l 25 2 l 22 +l 25 2	) )
	D23	Right corner of right eye to right mouth corner		ρ(l16, l24)
	D24 D25 D26 D27	Upper side of left eye to right corner of left eyebrow Upper side of right eye to left corner of right eyebrow Upper side of left eye to left corner of left eyebrow Upper side of right eye to right corner of right eyebrow		ρ( ρ( l 14 +l 15 l 8 +l 9 , l3) 2 , l4) 2 ρ( l 8 +l 9 , l1) 2 ρ( l 14 +l 15 2 , l6)

TABLE III .

 III EXTRACTED DYNAMIC FEATURE STATISTICS. η DENOTES THE NUMBER OF FRAMES, D DENOTES THE RESPECTIVE DYNAMIC FEATURE, V (t) = dD dt DENOTES THE SPEED, A = d 2 D dt 2 = dV dt DENOTES THE ACCELERATION, ω DENOTES THE FRAME RATE OF THE VIDEO SEQUENCE. THE SUPERSCRIPT + DENOTES THE ONSET, SUPERSCRIPT a DENOTES THE APEX, SUPERSCRIPT -DENOTES THE OFFSET.

TABLE VI .

 VI POSED SMILE. TRUE GENDER CLASSIFICATION RATES. AGE GIVEN IN YEARS.

	Combined Age-Groups	< 20	> 19
	Subj. amount	143	225
	Dynamics (SVM)	59.44%	66.22%
	OpenBR + Dynamics (Bagged Trees)	60.8%	80%
	how-old.net	51.05%	93.78%
	how-old.net + Dynamics (SVM)	60.8%	92.89%
	COTS	76.92%	92%
	COTS + Dynamics (Bagged Trees, PCA)	76.92%	92.89%

TABLE IV .

 IV SPONTANEOUS SMILE. TRUE GENDER CLASSIFICATION RATES. AGE GIVEN IN YEARS.

	Age	< 10	10 -19	20 -29	30 -39	40 -49	> 49
	Subj. amount	48	95	60	49	72	33
	OpenBR	58.33%	50.53%	81.67%	75.51%	75%	81.82%
	how-old.net	39.58%	56.84%	95%	87.76%	98.61%	87.88%
	COTS	77.08%	76.84%	93.33%	89.8%	94.44%	90.91%
	Merged Age-Groups	< 20		> 19
	Subj. amount	143			214
	Dynamics (PCA, SVM)	59.44%		67.81%
	OpenBR	52.45%		78.04%
	OpenBR + Dynamics (Bagged Trees)	60.1%		78.97%
	how-old.net	51.05%		93.46%
	how-old.net + Dynamics (Tree)	60.8%		93.46%
	COTS	76.92%		92.52%
	COTS + Dynamics (Tree)	76.92%		93%

TABLE V .

 V SPONTANEOUS SMILE IN AGE CATEGORY > 19: CONFUSION MATRIX FOR MALES AND FEMALES FOR (A) APPEARANCE FEATURES #1 (OPENBR) (DENOTED AS APP. 1), (B) APPEARANCE FEATURES #2 (how-old.net) (DENOTED AS APP. 2) , (C) APPEARANCE FEATURES #3 (COTS) (DENOTED AS APP. 3), (D) DYNAMIC FEATURES (DENOTED AS DYN.), (E) DYNAMIC AND APPEARANCE FEATURES #1 (DENOTED AS DYN. + APP. 1), (F) DYNAMIC AND APPEARANCE FEATURES #2 (DENOTED AS DYN. + APP. 2), (G) DYNAMIC AND APPEARANCE FEATURES #3 (DENOTED AS DYN. + APP. 3).

		(a) App. 1	(b) App. 2	(c) App. 3	(d) Dynamics	(e) Dyn. + App. 1	(f) Dyn. + App. 2	(g) Dyn. + App. 3
		Male	Female	Male	Female	Male	Female	Male	Female	Male	Female	Male	Female	Male	Female
	Male	61.8%	38.2%	94.5%	5.5%	99.1%	0.9%	70.0%	30.0%	78.18%	21.82%	94.5%	5.5%	99.1%	0.9%
	Female	4.8%	95.2%	7.7%	92.3%	14.4%	85.6%	34.62%	65.38%	20.19%	79.81%	7.7%	92.3%	12.5%	86.54%

TABLE VII .

 VII POSED SMILE IN AGE CATEGORY > 19:CONFUSION MATRIX FOR MALES AND FEMALES FOR (A) APPEARANCE FEATURES #1 (OPENBR) (DENOTED AS APP. 1), (B) APPEARANCE FEATURES #2 (how-old.net) (DENOTED AS APP. 2) , (C) APPEARANCE FEATURES #3 (COTS) (DENOTED AS APP. 3), (D) DYNAMIC FEATURES (DENOTED AS DYN.), (E) DYNAMIC AND APPEARANCE FEATURES #1 (DENOTED AS DYN. + APP. 1), (F) DYNAMIC AND APPEARANCE FEATURES #2 (DENOTED AS DYN. + APP. 2), (G) DYNAMIC AND APPEARANCE FEATURES #3 (DENOTED AS DYN. + APP. 3).

		(a) App. 1	(b) App. 2	(c) App. 3	(d) Dynamics	(e) Dyn. + App. 1	(f) Dyn. + App. 2	(g) Dyn. + App. 3
		Male	Female	Male	Female	Male	Female	Male	Female	Male	Female	Male	Female	Male	Female
	Male	62.34%	37.66%	94.81%	5.19%	98.7%	1.3%	68.1%	31.9%	75%	25%	93.97%	6.03%	99.14%	0.86%
	Female	1.52%	98.48%	7.58%	92.42%	15.15%	84.85%	35.78%	64.22%	14.68%	85.32%	8.26%	91.74%	13.76%	86.24%

TABLE VIII .

 VIII MOST DISCRIMINATE DYNAMIC FEATURES FOR AGE < 20. TGCR...TRUE GENDER CLASSIFICATION RATE.

	Distance	Feature	TGCR
	D10	Duration Ratio Offset	65.73%
	D7	Amplitude Ratio Onset	63.64%
	D10	Duration Onset	62.94%
	D10	Total Amplitude Onset	62.24%
	D9	Maximum Amplitude	62.24%
	D10	Mean Amplitude Onset	62.24%
	D7	Amplitude Ratio Offset	62.24%

TABLE IX .

 IX MOST DISCRIMINATE DYNAMIC FEATURES FOR AGE > 19. TGCR...TRUE GENDER CLASSIFICATION RATE.

	Distance	Feature	TGCR
	D11	Mean Amplitude Apex	62.15%
	D8	Maximum Amplitude	61.68%
	D9	Mean Speed Offset	61.54%
	D5	Mean Acceleration Offset	60.28%
	D5	Amplitude Ratio Offset	60.28%
	D5	Duration Offset	60.28%
	D5	Maximum Acceleration Offset	60.14%

http://www.nist.gov/itl/iad/ig/colorferet.cfm

http://www.uva-nemo.org

ACKNOWLEDGMENT This work was carried out during the tenure of an ERCIM "Alain Bensoussan" and Labex Fellowship Programmes. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement nr. 246016 and the French Government (National Research Agency, ANR) through the Investments for the Future Program reference #ANR-11-LABX-0031-01.